A Comprehensive Benchmark Suite for Topological Deep Learning

Assess how your model compares against state-of-the-art topological neural networks.

Overview • Get Started • Tutorials • Neural Networks • Liftings • Datasets • References

📌 Overview

TopoBench (TB) is a modular Python library designed to standardize benchmarking and accelerate research in Topological Deep Learning (TDL). In particular, TB allows to train and compare the performances of all sorts of Topological Neural Networks (TNNs) across the different topological domains, where by topological domain we refer to a graph, a simplicial complex, a cellular complex, or a hypergraph. For detailed information, please refer to the TopoBench: A Framework for Benchmarking Topological Deep Learning paper.

The main pipeline trains and evaluates a wide range of state-of-the-art TNNs and Graph Neural Networks (GNNs) (see ⚙️ Neural Networks) on numerous and varied datasets and benchmark tasks (see 📚 Datasets ). Additionally, the library offers the ability to transform, i.e. lift, each dataset from one topological domain to another (see 🚀 Liftings), enabling for the first time an exhaustive inter-domain comparison of TNNs.

🧩 Get Started

Create Environment

If you do not have conda on your machine, please follow their guide to install it.

First, clone the TopoBench repository and set up a conda environment tb with python 3.11.3.

git clone [email protected]:geometric-intelligence/topobench.git
cd TopoBench
conda create -n tb python=3.11.3

Next, check the CUDA version of your machine:

/usr/local/cuda/bin/nvcc --version

and ensure that it matches the CUDA version specified in the env_setup.sh file (CUDA=cu121 by default). If it does not match, update env_setup.sh accordingly by changing both the CUDA and TORCH environment variables to compatible values as specified on this website.

Next, set up the environment with the following command.

source env_setup.sh

This command installs the TopoBench library and its dependencies.

Run Training Pipeline

Next, train the neural networks by running the following command:

python -m topobench 

Thanks to hydra implementation, one can easily override the default experiment configuration through the command line. For instance, the model and dataset can be selected as:

python -m topobench model=cell/cwn dataset=graph/MUTAG

Remark: By default, our pipeline identifies the source and destination topological domains, and applies a default lifting between them if required.

The same CLI override mechanism also applies when modifying more finer configurations within a CONFIG GROUP. Please, refer to the official hydradocumentation for further details.

🚲 Experiments Reproducibility

To reproduce Table 1 from the TopoBench: A Framework for Benchmarking Topological Deep Learning paper, please run the following command:

bash scripts/reproduce.sh

Remark: We have additionally provided a public W&B (Weights & Biases) project with logs for the corresponding runs (updated on June 11, 2024).

⚓ Tutorials

Explore our tutorials for further details on how to add new datasets, transforms/liftings, and benchmark tasks.

⚙️ Neural Networks

We list the neural networks trained and evaluated by TopoBench, organized by the topological domain over which they operate: graph, simplicial complex, cellular complex or hypergraph. Many of these neural networks were originally implemented in TopoModelX.

Graphs

Model	Reference
GAT	Graph Attention Networks
GIN	How Powerful are Graph Neural Networks?
GCN	Semi-Supervised Classification with Graph Convolutional Networks
GraphMLP	Graph-MLP: Node Classification without Message Passing in Graph

Simplicial complexes

Model	Reference
SAN	Simplicial Attention Neural Networks
SCCN	Efficient Representation Learning for Higher-Order Data with Simplicial Complexes
SCCNN	Convolutional Learning on Simplicial Complexes
SCN	Simplicial Complex Neural Networks

Cellular complexes

Model	Reference
CAN	Cell Attention Network
CCCN	Inspired by A learning algorithm for computational connected cellular network, implementation adapted from Generalized Simplicial Attention Neural Networks
CXN	Cell Complex Neural Networks
CWN	Weisfeiler and Lehman Go Cellular: CW Networks

Hypergraphs

Model	Reference
AllDeepSet	You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks
AllSetTransformer	You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks
EDGNN	Equivariant Hypergraph Diffusion Neural Operators
UniGNN	UniGNN: a Unified Framework for Graph and Hypergraph Neural Networks
UniGNN2	UniGNN: a Unified Framework for Graph and Hypergraph Neural Networks

Combinatorial complexes

Model	Reference
GCCN	TopoTune: A Framework for Generalized Combinatorial Complex Neural Networks

Remark: TopoBench includes TopoTune, a comprehensive framework for easily designing new, general TDL models on any domain using any (graph) neural network as a backbone. Please check out the extended TopoTune wiki page for further details on how to leverage this framework to define and train customized topological neural network architectures.

🚀 Liftings & Transforms

We list the liftings used in TopoBench to transform datasets. Here, a lifting refers to a function that transforms a dataset defined on a topological domain (e.g., on a graph) into the same dataset but supported on a different topological domain (e.g., on a simplicial complex).

Structural Liftings

The structural lifting is responsible for the transformation of the underlying relationships or elements of the data. For instance, it might determine how nodes and edges in a graph are mapped into triangles and tetrahedra in a simplicial complex. This structural transformation can be further categorized into connectivity-based, where the mapping relies solely on the existing connections within the data, and feature-based, where the data's inherent properties or features guide the new structure.

We enumerate below the structural liftings currently implemented in TopoBench; please check out the provided description links for further details.

Remark:: Most of these liftings are adaptations of winner submissions of the ICML TDL Challenge 2024 (paper | repo); see the Structural Liftings wiki for a complete list of compatible liftings.

Graph to Simplicial Complex

Name	Type	Description
DnD Lifting	Feature-based	Wiki page
Random Latent Clique Lifting	Connectivity-based	Wiki page
Line Lifting	Connectivity-based	Wiki page
Neighbourhood Complex Lifting	Connectivity-based	Wiki page
Graph Induced Lifting	Connectivity-based	Wiki page
Eccentricity Lifting	Connectivity-based	Wiki page
Feature‐Based Rips Complex	Both connectivity and feature-based	Wiki page
Clique Lifting	Connectivity-based	Wiki page
K-hop Lifting	Connectivity-based	Wiki page

Graph to Cell Complex

Name	Type	Description
Discrete Configuration Complex	Connectivity-based	Wiki page
Cycle Lifting	Connectivity-based	Wiki page

Graph to Hypergraph

Name	Type	Description
Expander Hypergraph Lifting	Connectivity-based	Wiki page
Kernel Lifting	Both connectivity and feature-based	Wiki page
Mapper Lifting	Connectivity-based	Wiki page
Forman‐Ricci Curvature Coarse Geometry Lifting	Connectivity-based	Wiki page
KNN Lifting	Feature-based	Wiki page
K-hop Lifting	Connectivity-based	Wiki page

Pointcloud to Simplicial

Name	Type	Description
Delaunay Lifting	Feature-based	Wiki page
Random Flag Complex	Feature-based	Wiki page

Pointcloud to Hypergraph

Name	Type	Description
Mixture of Gaussians MST lifting	Feature-based	Wiki page
PointNet Lifting	Feature-based	Wiki page
Voronoi Lifting	Feature-based	Wiki page

Simplicial to Combinatorial

Name	Type	Description
Coface Lifting	Connectivity-based	Wiki page

Hypergraph to Combinatorial

Name	Type	Description
Universal Strict Lifting	Connectivity-based	Wiki page

Feature Liftings

Feature liftings address the transfer of data attributes or features during mapping, ensuring that the properties associated with the data elements are consistently preserved in the new representation.

Name	Description	Supported Domains
ProjectionSum	Projects r-cell features of a graph to r+1-cell structures utilizing incidence matrices (B_{r}).	All
ConcatenationLifting	Concatenate r-cell features to obtain r+1-cell features.	Simplicial

Data Transformations

Specially useful in pre-processing steps, these are the general data manipulations currently implemented in TopoBench:

Transform	Description
OneHotDegreeFeatures	Adds the node degree as one hot encodings to the node features.
NodeFeaturesToFloat	Converts the node features of the input graph to float.
NodeDegrees	Calculates the node degrees of the input graph.
NodeDegrees	Keeps only the selected fields of the input data.
KeepOnlyConnectedComponent	Keep only the largest connected components of the input graph.
InfereRadiusConnectivity	Generates the radius connectivity of the input point cloud.
InfereKNNConnectivity	Generates the k-nearest neighbor connectivity of the input point cloud.
IdentityTransform	An identity transform that does nothing to the input data.
EqualGausFeatures	Generates equal Gaussian features for all nodes.
CalculateSimplicialCurvature	Calculates the simplicial curvature of the input graph.

📚 Datasets

Graph

Dataset	Task	Description	Reference
Cora	Classification	Cocitation dataset.	Source
Citeseer	Classification	Cocitation dataset.	Source
Pubmed	Classification	Cocitation dataset.	Source
MUTAG	Classification	Graph-level classification.	Source
PROTEINS	Classification	Graph-level classification.	Source
NCI1	Classification	Graph-level classification.	Source
NCI109	Classification	Graph-level classification.	Source
IMDB-BIN	Classification	Graph-level classification.	Source
IMDB-MUL	Classification	Graph-level classification.	Source
REDDIT	Classification	Graph-level classification.	Source
Amazon	Classification	Heterophilic dataset.	Source
Minesweeper	Classification	Heterophilic dataset.	Source
Empire	Classification	Heterophilic dataset.	Source
Tolokers	Classification	Heterophilic dataset.	Source
US-county-demos	Regression	In turn each node attribute is used as the target label.	Source
ZINC	Regression	Graph-level regression.	Source

Simplicial

Dataset	Task	Description	Reference
Mantra	Classification, Multi-label Classification	Predict topological attributes of manifold triangulations	Source

Hypergraph

Dataset	Task	Description	Reference
Cora-Cocitation	Classification	Cocitation dataset.	Source
Citeseer-Cocitation	Classification	Cocitation dataset.	Source
PubMed-Cocitation	Classification	Cocitation dataset.	Source
Cora-Coauthorship	Classification	Cocitation dataset.	Source
DBLP-Coauthorship	Classification	Cocitation dataset.	Source

🔍 References

To learn more about TopoBench, we invite you to read the paper:

@article{telyatnikov2024topobench,
      title={TopoBench: A Framework for Benchmarking Topological Deep Learning}, 
      author={Lev Telyatnikov and Guillermo Bernardez and Marco Montagna and Pavlo Vasylenko and Ghada Zamzmi and Mustafa Hajij and Michael T Schaub and Nina Miolane and Simone Scardapane and Theodore Papamarkou},
      year={2024},
      eprint={2406.06642},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2406.06642}, 
}

If you find TopoBench useful, we would appreciate if you cite us!

🐭 Additional Details

Hierarchy of configuration files

├── configs                   <- Hydra configs
│   ├── callbacks                <- Callbacks configs
│   ├── dataset                  <- Dataset configs
│   │   ├── graph                    <- Graph dataset configs
│   │   ├── hypergraph               <- Hypergraph dataset configs
│   │   └── simplicial               <- Simplicial dataset configs
│   ├── debug                    <- Debugging configs
│   ├── evaluator                <- Evaluator configs
│   ├── experiment               <- Experiment configs
│   ├── extras                   <- Extra utilities configs
│   ├── hparams_search           <- Hyperparameter search configs
│   ├── hydra                    <- Hydra configs
│   ├── local                    <- Local configs
│   ├── logger                   <- Logger configs
│   ├── loss                     <- Loss function configs
│   ├── model                    <- Model configs
│   │   ├── cell                     <- Cell model configs
│   │   ├── graph                    <- Graph model configs
│   │   ├── hypergraph               <- Hypergraph model configs
│   │   └── simplicial               <- Simplicial model configs
│   ├── optimizer                <- Optimizer configs
│   ├── paths                    <- Project paths configs
│   ├── scheduler                <- Scheduler configs
│   ├── trainer                  <- Trainer configs
│   ├── transforms               <- Data transformation configs
│   │   ├── data_manipulations       <- Data manipulation transforms
│   │   ├── dataset_defaults         <- Default dataset transforms
│   │   ├── feature_liftings         <- Feature lifting transforms
│   │   └── liftings                 <- Lifting transforms
│   │       ├── graph2cell               <- Graph to cell lifting transforms
│   │       ├── graph2hypergraph         <- Graph to hypergraph lifting transforms
│   │       ├── graph2simplicial         <- Graph to simplicial lifting transforms
│   │       ├── graph2cell_default.yaml  <- Default graph to cell lifting config
│   │       ├── graph2hypergraph_default.yaml <- Default graph to hypergraph lifting config
│   │       ├── graph2simplicial_default.yaml <- Default graph to simplicial lifting config
│   │       ├── no_lifting.yaml           <- No lifting config
│   │       ├── custom_example.yaml       <- Custom example transform config
│   │       └── no_transform.yaml         <- No transform config
│   ├── wandb_sweep              <- Weights & Biases sweep configs
│   │
│   ├── __init__.py              <- Init file for configs module
│   └── run.yaml               <- Main config for training

More information regarding Topological Deep Learning

Topological Graph Signal Compression

Architectures of Topological Deep Learning: A Survey on Topological Neural Networks

TopoX: a suite of Python packages for machine learning on topological domains