arXiv:2609.08709v1 [cs.LG] 8 Sep 2026
Chimaera: A Mixture-of-Graph-Experts Architecture for Cross-Task and Cross-Dataset Graph Learning Jonathan Frank
David Richerby
Ansgar Scherp
University of Ulm Ulm, Germany [email protected]
University of Essex Colchester, UK [email protected]
University of Ulm Ulm, Germany [email protected]
Abstract—Designing foundation models for graphs is challenging due to the irregular structure of graphs and the different sizes and characteristics of embeddings. Chimaera integrates mixture-of-experts with graph foundation models (GFM). It integrates different GFM architectures, such as graph prompts and linear GNN models. Large language models are used to generate embeddings, and experts can be trained and combined following different strategies, GFMs, embeddings, etc. Furthermore, Chimaera extends existing linear GNNs to support link-level and graph-level tasks in addition to node-level tasks. Empirical analyses are performed on same-task and cross-task experiments with node, link, and graph classification tasks using six benchmark text-attributed graph datasets. The experiments demonstrate the effectiveness of Chimaera and its capabilities for transfer across tasks and datasets. Further insights include the need to use both large and small language models to generate embeddings for the experts, a strong cross-task transferability of simple but effective linear GNNs, and using few samples only to provide strong results. The source code: https://github.com/ascherp/ChIMERA-dev Index Terms—graph foundation models, mixture-of-experts
I. I NTRODUCTION The success of foundation models [1] in computer vision [2], [3] and natural language processing [4], [5], [6] inspired the development of foundation models specifically for graphs [7]. However, the significant successes in natural language processing have not yet been replicated in graphs, because of their non-i.i.d. nature. Learning from graphs is inherently more complex due to their irregular structure and the lack of a standardized vocabulary [8]. Challenges faced by graph foundation models (GFMs) [7] include handling graph features of different kinds and sizes. For example, a text-attributed graph may be encoded with sparse embeddings like TF-IDF or dense neural embeddings such as a BERT model [9]. Additionally, there are two key characteristics one expects from a foundation model: homogenization [10] and generalizability (also known as emergence) [11]. Homogenization refers to a model’s applicability to different tasks. Graph learning tasks are node classification, link prediction, and graph classification. Generalizability is the model’s ability to perform well on unseen datasets and tasks with an increasing amount of training data or parameters. Even though graph neural networks (GNNs) [12], [13], [14], [15] emerge as a specific family of graph learning methods that exhibit impressive performance in tasks such as node classification [13], [16], [17], [18] and link prediction [14],
Input
Output
Encoders Experts Gating
Classifier
Fig. 1: Mixture-of-Graph-Experts configuration with Chimaera. We use n+1 input encoders η (one encoder is from the original dataset). The input is combined via n GNN experts ξ in a gating function γ, followed by a classifier κ.
they are limited to the same feature and label spaces as they are trained on. In other words, a GNN trained on a specific dataset for a certain task does not generalize to other datasets and tasks. Hence, one has to try different models and configurations for each new dataset and task. A mixture-of-experts architecture can learn to combine different GNNs depending on their strengths and, therefore, can mitigate this problem. However, the dependency on the same feature and label spaces remains. A solution for the feature space dependency in text-attributed graphs is provided by large language models (LLMs), which transfer the text features of the node into the same embedding space. For the label space dependency, the solution can be found in prompting [19] and linear GNNs [20], [21]. To solve these problems and create a GFM based on GNNs, we introduce Chimaera,1 the first model that combines existing GFMs, such as prompting and linear GNNs, with a mixtureof-experts (MoE) architecture. In this context, we extend linear GNNs from node classification to include link-level and graph-level tasks. Our model is illustrated in Figure 1, where the experts ξ (i) represent a sequence of different GNNs operating on potentially different embeddings. We use different embeddings generated by neural encoders η (i) , e. g., BERT [9]. Consequently, Chimaera is fundamentally a GNNbased model but also functions as a GNN+LLM-based model for text-attributed graphs. The gating function γ combines the output embedding of the experts and resembles a MoE. The classifier κ is a GFM that uses the combined embeddings for 1“Chimaera” (Greek mythology) is a creature part lion, goat, and serpent. In a similar way, we combine familiar elements to create something new.
classification. Thus, our model aims to generalize by integrating graphs, and converts text-attributed graphs into Levi graphs. various experts and embeddings while ensuring homogenization GPT4Graph [27] uses a graph description language and the through existing GFM methods. Chimaera supports node-, link-, language capabilities of GPT4 [4] to process graph data. and graph-level classification tasks and can transfer knowledge GNN+LLM-based GFMs combine features of GNNs and LLMs across tasks and datasets. Thus, it is a GFM itself. to create synergies. For example, OFA [28] uses LLMs to We experiment with various LLMs as encoders, GNNs as improve embeddings for GNNs by incorporating graph features experts, gating functions, and GFMs as classifiers. We compare and text prompts. Another model, GOFA [29], interleaves co-training with pre-training and evaluate our method in both randomly initialized GNN layers into a frozen pre-trained LLM, same-task and cross-task test settings, following [22]. Co- allowing for a natural integration of semantic and structural training applies a trained model to similar datasets, while pre- modeling capabilities. These methods are limited to textual training involves using trained models on datasets not included features within the graph or require textual prompts. in the training phase. Same-task models transfer knowledge A Mixture-of-Experts (MoE) [30] combines a set of experts, within a specific task type, such as node classification; cross- e. g., neural networks, using a mixture function and returns a task models focus on transferring knowledge between different distribution over the experts. Fedus et al. [31] suggest using tasks, e. g., from node classification to graph classification. only one expert at a time and encourage a balanced workload Additionally, we use varying numbers of samples for the among experts by incorporating an auxiliary loss [32], [33], classification process using linear GNNs and prompting. [34]. To enhance the scaling power of MoE, GRIN incorporates Using six benchmark datasets on four tasks (node classi- sparse gradient estimation for expert routing and configures fication, link prediction, reasoning, and graph classification), model parallelism [35]. GMoE [36] is based on GNNs, and we observe that Chimaera performs well when trained on the enhances the network’s adaptability to diverse training graphs same datasets and tasks, achieving scores similar to the baseline with minimal computational overhead. DA-MoE [37] is a graph GNNs. In cross-task or pre-training scenarios, Chimaera demon- MoE that uses a GNN, in contrast to the linear projections used strates strong generalization capabilities, achieving performance by GMoE, to better capture complex patterns and dependencies close to that of models directly trained on the target dataset within the data. Previous work, e. g., [38], has primarily focused in more than half of the cases. This highlights the robustness on using MoE to enhance performance on specific graph-related of Chimaera, especially given that, unlike the baselines, our tasks, such as link prediction. In contrast, our research aims model was not explicitly trained for those specific tasks or to use MoE to transfer knowledge across multiple tasks and datasets. In summary, our contributions are: datasets. • We introduce Chimaera, a model that combines GFMs III. C HIMAERA M O E-A RCHITECTURE FOR GFM S with MoE. As a feature by design, our model itself is a Given a graph G = (V, E) with nodes V and edges E, we GFM that shows generalizability and homogenization. represent its features as (X, A), where X ∈ R|V |×c is the • An analysis of complexity and an examination of a gating feature matrix of the nodes and A ∈ R|V |×|V | is the adjacency function across different contexts. matrix. The set of classes is denoted by Y , which specifies • Experiments in 64 settings on 6 benchmarks with 4 tasks demonstrate Chimaera’s effectiveness in cross-task settings a task-dependent aspect of the graph. We consider four tasks. and transferring to new datasets with few training samples. In node classification, each node gets a label from Y . For We summarize related work below. Section III introduces link prediction, a pair of nodes is predicted to have a 1 if Chimaera. The experimental apparatus is described in Sec- the edge between these two nodes exists or 0 otherwise, i. e., tion IV. An overview of the achieved results is reported in Y = {0, 1}. In reasoning, the edges get a label from Y . In graph classification, we assign a label to each graph G. Section V and discussed in Section VI, before we conclude. As shown in Figure 1, the input is first encoded by the II. R ELATED W ORK encoders η (i) to create the embeddings X (i) ∈ R|V |×c before There are recent comprehensive surveys of Graph Foundation passing X (i) to the expert ξ (i) , e. g., a GNN. The experts Models (GFMs) [7], [23]. Below, we describe selected models themselves provide their embeddings H (i) ∈ R|V |×d to the relevant to this work. Generally, GFM architectures can be gating function γ. All experts have the same output dimension, divided into those based on GNNs, LLMs, or both [7], [23]. but their input dimension my differ depending on the encoder. PRODIGY [19] is a pre-trained GNN-based GFM that can di- A skip connection is provided from the input to the gating rectly perform downstream classification tasks on unseen graphs function γ by the encoder η (γ) . using in-context learning and graph prompts. GraphAny [21] is The gating function γ combines the embeddings a GNN-based model that combines different linear GNNs [24] H (1) , . . . , H (n) from the experts and the embedding X (γ) with an attention module. The linear GNNs need no training and created by the encoder η (γ) to create an embedding F ∈ are adapted for each graph using an analytical solution. Only the R|V |×d . Then a classifier κ, a GFM, returns a prediction attention module in GraphAny is trained. TrainlessGNN [20] Z ∈ R|V |×|Y | for F . Given the total of n + 1 embeddings X is a another linear GNN adapted for graph learning tasks. from the n + 1 encoders η, a gating function γ, and a clas(γ) (1) (n) The LLM-based GLM [25] uses the encoder-decoder language sifier κ, we define Chimaera as χ(X , X , . . . , X ) = (γ) 1 (1) n (n) model T5 [26], and modifies the adjacency matrix to support κ γ X , ξ (X ), . . . , ξ (X ) .
The choice of encoders, experts, gating functions, and classifiers is an optimization aspect of Chimaera that is not typically found in other GNNs or GFM architectures. We introduce the implementations of encoders, experts, gating functions, and GFMs. The GFMs we consider include linear GNNs [21], [20] and the prompt-based PRODIGY [19]. Subsequently, we explain how to train Chimaera effectively. a) Encoders: We use language models as encoders to create node embeddings for our text-attributed graphs. Our encoders are BERT [9], Sentence-BERT [39] (ST), E5 [40], Llama2 7B, and Llama2 13B [6]. We use LLMs as encoders, but any model capable of producing node embeddings could also be integrated into Chimaera. b) Experts: Experts in Chimaera can differ in architecture, size, and parameters. Their objective is to create embeddings H (i) by enhancing the input embeddings X (i) with additional information or refining them. We use the following GNNs as our expert models: GCN [12], GIN [13], and GAT [14]. We use these GNNs as they show state-of-the-art performance when optimized properly [41], [42]. c) Gating Functions: For simplicity, we refer here to the embedding X (γ) created by the encoder η (γ) for the gating function γ as input X. The gating function combines the experts’ output embeddings. One straightforward approach for a gating function to calculate an embedding matrix F of the nodes is to calculate the average of the embeddings H (1) , . . . , H (n) (recall that these embeddings have the same dimension). Alternatively, we can concatenate the embeddings and feed them through an MLP to generate the embedding F . We refer to these methods as mean and MLP, respectively. The mixture functions used in MoE approaches are particularly well suited to serve as gating functions. The original MoE formulation [30] combines a set of experts ξ (1) , . . . , ξ (n) using a mixture function Q (e. g., an MLP) that outputs a distribution, represented as a vector, over the experts given the input X. Chimaera uses a mixture function as a gating function with γ = P n (i) (i) i=1 Q(X)i ·ξ (X ), where Q(X)i denotes the ith item in the output vector of Q(X). Thus, Q determines the weighting of the experts appropriate to a given input. GMoE [36] employs a noisy top-k gating design [32] and is defined as Q(X) = Softmax(TopK(XWg +ϵ Softplus(XWn )), k)). Here, k is the number of selected experts, ϵ ∼ N (0, 1) is standard Gaussian noise, and Wg ∈ Rc×n and Wn ∈ Rc×n are learnable weights. DA-MoE [37] uses a structure-based gating network instead of linear projection to obtain the scores for each expert: Q(X) = Softmax(TopK(T (X)+ϵ Softplus(T (X)), k)) with T (X) = σ((1+α)X +AX), where α is a learnable parameter adjusting the contribution of the node’s own feature and σ is a two-layer fully connected neural network with a nonlinear activation function applied between the layers. d) Classifiers: As classifiers, we employ the linear GNN models LinearGNN from GraphAny [21] and TrainlessGNNs [20], and use the prompt-based PRODIGY [19]. To extend Linear GNNs to all task types, we convert the original graph G into a task graph GT = (VT , ET ). The nodes VT represent entities in the original graph G. Depending on
the task, VT represents the original nodes for the node-level tasks, edges for link-level tasks, and graphs for the graph-level tasks. We use the embeddings in F of the gating function to create the feature matrix P of the nodes VT depending on the task. For node-level tasks, P = F ∈ R|V |×d , and for a link-level task with m edges, we concatenate the respective node embeddings in F to create P ∈ Rm×2d . In graph-level tasks with m graphs, we use mean as a pooling function to create P ∈ Rm×d from F . We use only P for classification and do not update the features of the task graph GT , so we set ET = ∅. The labeled nodes in the task graph GT are represented by VL , along with their corresponding embeddings, denoted as PL . These labeled nodes have the same label as the entities in the original graph they represent. The prediction of a linear GNN is Z = P W . The learnable weight matrix W ∈ Re×|Y | has the dimensionality e = 2d for link-level tasks and e = d for the other tasks. We distinguish between two types of linear GNNs to establish the weight matrix W. The weight matrix of a LinearGNN from GraphAny [21] is determined by a closed-form solution W = PL+ YL ∈ Re×|Y | , where PL+ is the pseudo-inverse of the embeddings of the labeled nodes in VL , and YL ∈ R|VL |×|Y | are the label vectors of the nodes in VL . In contrast, TrainlessGNNs [20] use virtual class nodes for creating the weight matrix. The virtual class nodes are connected to the nodes in VL that belong to the same class via edges with weight 1. The virtual nodes are connected to nodes belonging to different classes by edges with a weight defined by the hyperparameter ω. This results in BL ∈ {0, 1}|VL |×|Y | . The matrix BL acts as the incidence matrix between virtual class nodes and VL . One round of message-passing is applied [20] to obtain the weight matrix W ∈ Re×|Y | : W T = (B − ω T |Y | 1) PL , where 1 is the all-ones matrix of size |VL | × |Y |. In the context of graph prompting, a prompt set S consists of k example graphs per class. The objective is to use the prompt set S to classify the entities in a query set Q. PRODIGY applies this approach for classification in the graph domain [19]. It uses a prompt graph as a unified representation for k-shot prompts across |Y | labels with o queries. A prompt graph is composed of data graphs and a task graph. Each entity from the prompt set S and the query set Q is represented by a data node in the task graph. The data graphs are used to compute the embeddings of these data nodes. In Chimaera, the embeddings P of the data nodes are created using the embeddings in F from the gating function. For graph tasks, the data node embeddings are obtained through mean pooling, while for node tasks, the respective node embeddings are used. In the case of link tasks, the embeddings of the nodes forming the edge and the max pooling over all node embeddings [43] are concatenated with each other. With an additional linear projection layer, the embedding sizes are converted back to d. Each possible class y is represented by a class node vy in the task graph. The embeddings Py of the class nodes can be initialized with a random Gaussian or with additional information about the class. However, we initialize all entries with zero to ensure consistent conditions for our experiments.
In total, a task graph contains |Y | · k + o data nodes and |Y | class nodes. For the prompts, one connects each data node to all the class nodes and vice versa. The edges are labeled with “True” if the prompt example belongs to the class of the label, and otherwise “False”. For the query set, one adds single-directional edges from all class nodes to each data point in the query set, which is labeled with “?”. An attention-based GNN [19] is then applied to the task graph to produce an updated representation PQ of the query nodes and PC of the class nodes. The classification is done by using the cosine similarity sim between PQ and PC , i. e., Z = sim(PQ , PC ) . e) Training Chimaera: Chimaera is trained stepwise along the components shown in Figure 1. For a task on a dataset divided into training, validation, and test sets, we construct and train Chimaera as follows. We first train the experts using a simple classifier on the dataset with the embeddings from the respective encoder and validate them on the validation set. Subsequently, the classifiers are removed, and the experts’ parameters are frozen. The validation results are used to select the experts and their encoders. We propose three strategies for using GCN, GAT, and GIN as experts. The first strategy, the multi-GNN strategy, selects the best GCN, GAT, and GIN with their encoders based on their validation performance. Second, the multi-encoder strategy chooses the best GNN with its encoder according to its validation results and combines it with two GNNs that are the same GNN but use different encoders. Third, the multi-layer strategy again selects the best GNN and its encoder and combines it with two GNNs that differ from it solely in the number of layers and have the same encoder. The chosen expert and encoder pairs are then combined using a gating function. The gating function has a simple classifier on top and is trained. Afterward, the classifier is discarded, and the parameters are frozen. Then, the gating function is used to generate input for a chosen GFM as a classifier. The GFM is trained, and then its parameters are frozen. The training of Chimaera reduces both the overall training time and complexity when compared to training all components simultaneously. This approach reduces the hyperparameter space required for grid search and enables the parallel training of the experts. Additionally, the training time for each individual component remains unaffected since the hyperparameters of the underlying components are kept frozen. f) Complexity Analysis: For the complexity analysis, we look into the components of Chimaera. The inference complexity of an ℓ-layer GNN expert is O(ℓ (|E|d + nd2 )), where |E|, n, and d are the number of edges, the number of nodes, and the hidden dimension, respectively. Chimaera can use various gating functions with differing complexities. For the gating using mean, no additional overhead is added because it has a linear complexity of O(d). The MLP gating, GMoE, and MoE use an MLP for gating with complexity O(d2 ). It was also shown that GMoE [36] brings negligible overhead on the inference cost compared with its GNN counterpart. For DA-MoE, an ℓ-layered GNN is used for gating, which has a complexity of O(ℓ (|E|d + nd2 )).
Given m experts with ℓ1 , . . . , ℓm GNN layers, Pmrespectively, the inference complexity of our gating is O( i=1 ℓi (|E|d + nd2 )). Overall, the cost of gating on the inference cost is negligible when the gates and experts can be run simultaneously. A linear GNN has pre-processing complexity of O(|E|), optimization complexity of O(|VL |), and inference of O(|V |). Therefore, Pm with a linear GNN, the complexity of Chimaera is O( i=1 ℓi (|E|d + nd2 )). PRODIGY uses a GNN on the task graph with |Y | k + o data nodes and |Y | class nodes. We approximate the complexity of creating the task graph and the inference on it with an ℓ0 -layer GNN as O(ℓ0 (|E|d + nd2 )) because, in general, the task graph is much smaller than the original Pmgraph. The resulting Chimaera inference complexity is O( i=0 ℓi (|E|d + nd2 )). IV. E XPERIMENTS a) Datasets: We use six datasets from TAGLAS [44], an atlas of text-attributed graph datasets and benchmarks. An overview of the dataset characteristics is provided in Table I. For node classification, we use the datasets Cora, PubMed [28], [45], and WikiCS [46], [28]. In link-level tasks, we distinguish between link prediction, which classifies whether an edge exists, and reasoning, which aims to label existing edges. We use the Cora and PubMed datasets [28], [45] for link prediction. For reasoning, we use the WN18RR dataset [28]. Finally, for graphlevel tasks, we include the BBBP and BACE datasets [47]. b) Procedure: We use several pre-trained models to obtain embeddings from text-attributed graphs, including BERT [9], Sentence-BERT [39], E5 [40], Llama2 7B, and Llama2 13B [6]. The embeddings generated by BERT and Sentence-BERT have dimension c = 768, while those created by E5 are c = 1, 024. The largest embeddings are created by Llama2 7B with c = 4, 096 and Llama2 13B with c = 5, 120. For the experts, we employ GIN [13], GAT [14], and GCN [12]. All experiments are conducted on an NVIDIA H100 GPU with 80GB. Main Experiments: We train various combinations of embeddings and experts using classification heads on the training data, in which the encoders are frozen. The combinations are aggregated using one of our gating functions. To select the experts for the next steps, we apply one of our three strategies. These strategies are the multi-GNN strategy, the multi-encoder strategy, and the multi-layer strategy (see Section III). After determining the experts, we freeze their parameters and train the gating function using the training data. Subsequently, we use one of our gating functions or the best GNN model as input for the GFM, i. e., the classifier κ (see Figure 1). As classifiers, we use LinearGNN [21], TrainlessGNN [20], and PRODIGY [19]. Only the prompt-based PRODIGY is trainable. Thus, it is the only model trained on the training data. See also the explanation of GFMs as classifiers in Section III. We use trained experts and the gating function as our baselines. GNNs serve as valid baselines for graph-related tasks [41]. A model that has been explicitly trained on a specific task and dataset naturally serves as a baseline for another model that has not been trained on that task or dataset, particularly when it needs to transfer knowledge between different tasks or datasets.
TABLE I: Characteristic of TAGLAS datasets [44]. Task type
Dataset
Avg. |V |
Avg. |E|
# graphs
Node-level
Cora PubMed WikiCS
2 708 19 717 11 701
21 112 44 338 216 123
Link-level
Cora PubMed WN18RR
2 708 19 717 40 943
Graph-level
BBBP BACE
24.06 34.09
Domain
Split (train/val/test)
# classes
1 1 1
Co-citation Co-citation Wikipedia page
140/500/2 068 60/500/19 157 580/1 769/5 847
7 3 10
21 112 44 338 93 003
1 1 1
Co-citation Co-citation Knowledge graph
17 944/1 056/2 112 150 700/8 866/17 730 86 835/3 034/3 134
2 2 11
51.91 73.72
2 039 1 513
1 631/204/204 1 210/151/152
2 2
Molecular Molecular
Experiments with Full Chimaera and Using PRODIGY as MoE employs a 2-layer MLP with an embedding size Classifier: First, we train Chimaera with a linear GNN and matching the experts. GMoE uses a 2-layer MLP with the parameters unfrozen. The full Chimaera is trained after our same embedding size and varies k between 2 and 3. DAregular training procedure. Second, we train Chimaera with MoE selects the best k experts for its gating function, with PRODIGY, similar to our training procedure except that we k ∈ {2, 3}. It uses the best GNN for its mixture function. The train PRODIGY for node-, link-, and graph-level tasks by MLP as a gating function takes three times the embedding combining the loss of these tasks in each training step. size of the experts as input and consists of two layers with a For the testing procedure, we follow [22], and vary the hidden size equal to the embedding size. We train the gating number of shots per class. We use either 3, 10, or 20 shots functions for 50 epochs, again using early stopping with a per class, with a maximum of 10 for WikiCS due to the patience of 5 epochs. These models are optimized with the limited number of examples in WikCS. We test five times with Adam optimizer [48], learning rate 0.01, and weight decay 0. different shots and random seeds, averaging the results over We train PRODIGY for Chimaera on one task of one these five runs. We explore the same-task and cross-task test dataset using the procedure from [19], using three shots per settings and differentiate between co-training and pre-training, example. Training is conducted with 2, 000 batches. We use as described in the introduction. For each combination of the standard values from the PRODIGY implementation: Adam settings, we evaluate the models’ performance on the specific optimizer [48], learning rate 0.001, and weight decay 0.001. test set, using accuracy for tasks with more than two classes For training the whole of Chimaera with a linear GNN, and the ROC curve (AUC) for binary tasks. we train it similarly to PRODIGY, but for 500 batches. For Experiments on Gating Functions: To investigate how MoE TrainlessGNNs, we set ω = 0. For the multi-task PRODIGY, behaves for tasks and data for which it has not been trained, in addition to the training dataset, we incorporate the Cora we track the scores of the gating function of DA-MoE with dataset to introduce node-level and link-level tasks, as well k = 3 for each expert for each tested sample. This analysis as the BACE dataset to add a graph-level task to the training is exclusive to DA-MoE because it uses the graph’s structure process. If a task is already included in the training dataset, in its gating function. As a result, we obtain a distribution of we do not use the additional dataset for that specific task. scores for each expert, which we can further categorize by class Each batch consists of samples for each task. All tasks are and training data. We employ three different strategies and executed separately during each training step, and their losses compare their outcomes. Our objective is to identify variations are combined. All models are optimized with the Adam based on class, expert, and the strategy implemented. optimizer [48], learning rate 0.001, and weight decay 0.001. c) Hyperparameter Optimization: For the hyperparameter V. R ESULTS optimization, we use grid search. We use GIN, GAT, and GCN Our experiments show that the effect of the number of as experts with input generated by either BERT, SentenceBERT, E5, Llama2 7B, or Llama2 13B. The encoders are shots on PRODIGY in Chimaera is minimal. Additionally, the not trained and are only used for inference. The experts are performance of linear GNNs tends to improve with a higher trained for 50 epochs on the training set, employing early number of shots. Unfreezing the parameters and learning a stopping with a patience of 5 epochs. We conduct a grid search multi-task PRODIGY does not enhance Chimaera’s overall over all combinations of embedding size ∈ {64, 128, 256}, performance. The fully trained Chimaera with linear GNNs number of layers ∈ {1, 2, 3}, dropout ∈ {0, 0.2}, and weight performs less well than training step-by-step and freezing. The decay ∈ {0, 0.01}. For TrainlessGNNs, we optimize ω ∈ results for Chimaera with PRODIGY and multi-training are {−1, 0, 0.001, 0.01, 0.1, 1}, which is an edge weight modeling mostly better for unseen datasets than for its variant with singlethe influence of classes of which the node under consideration task training. Nevertheless, Chimaera with linear GNNs still is not a member. For node and graph classification tasks, we outperforms it for unseen datasets. use a 1-layer MLP as the classification head, while for link Table II shows how the best Chimaera (in the co/pre-training tasks, we employ a 2-layer MLP. All models are optimized and same/cross-task settings) performs compared to the best using the Adam optimizer [48] with a learning rate of 0.01. experts and gating functions, with the latter always being trained
♣
♦
♦
♥
♠
♥
♥
♥
♦
♣
♦
♥
♥
♠
♥
♥
♦
♦
♣
♥
♥
♥
♥
♥
♥
♥
♥
♣
♥
♥
♥
♥
♠
♥
♥
♥
♣
♦
♥
♥
♥
♠
♥
♥
♦
♣
♥
♥
TABLE II: Best test accuracy/AUC of the experts, gating functions, and Chimaera. In training, pre-/co- indicate pre-/cotraining, S/X = same/cross-task. – means no suitable dataset.
Node
Cora
Train
Pubmed
co-S co-X pre-S pre-X
WikiCS
Link
WN18RR
co-S co-X pre-S pre-X
Cora
Pubmed
♥
♥
♥
♥
♥
♥
♣
♦
♥
♥
♥
♥
♥
♥
♦
♣
co-S co-X pre-S pre-X
Graph
BBBP
Node
BACE
BBBP
Pubmed
Cora
WN18RR
n O
WikiCS
Cora
n O in ed Te s
te d
Tr a
Pubmed
BACE
Link
co-S co-X pre-S pre-X
Accuracy AUC Expert Gating Chimaera Expert Gating Chimaera Cora (node) Cora (link) 0.8 0.82 0.81±0.00 0.99 0.99 0.60±0.01 n/a n/a 0.76±0.00 n/a n/a 0.68±0.00 n/a n/a 0.67±0.00 n/a n/a 0.69±0.00 n/a n/a 0.67±0.00 n/a n/a 0.71±0.00 PubMed (node) PubMed (link) 0.79 0.80 0.80±0.00 0.98 0.99 0.90±0.00 n/a n/a 0.74±0.00 n/a n/a 0.82±0.04 n/a n/a 0.61±0.00 n/a n/a 0.75±0.01 n/a n/a 0.70±0.00 n/a n/a 0.89±0.00 WikiCS (node) BBBP (graph) 0.82 0.82 0.81±0.00 0.65 0.64 0.65±0.00 n/a n/a – n/a n/a – n/a n/a 0.48±0.02 n/a n/a 0.64±0.00 n/a n/a 0.53±0.01 n/a n/a 0.61±0.00 WN18RR (link) BACE (graph) 0.73 0.79 0.40±0.00 0.77 0.76 0.74±0.00 n/a n/a – n/a n/a – n/a n/a – n/a n/a 0.63±0.04 n/a n/a 0.40±0.00 n/a n/a 0.71±0.01
Graph
(a) Chimaera with PRODIGY ♣
♦
♦
♥
♠
♥
♥
♥
♦
♣
♦
♥
♥
♠
♥
♥
♦
♦
♣
♥
♥
♥
♥
♥
♥
♥
♥
♣
♥
♥
♥
♥
♠
♥
♥
♥
♣
♦
♥
♥
♥
♠
♥
♥
♦
♣
♥
♥
♥
♥
♥
♥
♥
♥
♣
♦
♥
♥
♥
♥
♥
♥
♦
♣
Node
Cora
Pubmed
WikiCS
Link
WN18RR
Cora
Pubmed
Graph
BBBP
Node
Link
BACE
BBBP
Pubmed
Cora
WN18RR
WikiCS
Pubmed
Cora
n O
d te
Te s
Tr a
in
ed
O
n
BACE
Graph
(b) Chimaera with LinearGNN
Fig. 2: Chimaera’s performance averaged over five runs using accuracy for tasks with more than two classes and AUC for binary tasks. The circles in the tiles indicate the experimental setting. If a left semi-circle is white, it denotes co-training; if it is black, it indicates pre-training. The right semi-circle is white for the same-task setting, and black for cross-task.
on the dataset and task it is tested on (always co-training and same-task). Chimaera achieves performance comparable to the trained expert and gating functions for node classification in Cora and PubMed, link prediction in PubMed, and graph classification in BBBP and BACE independent of the test settings. The heatmaps in Figure 2 show the best performance for Chimaera with PRODIGY and with LinearGNN (from GraphAny). The heatmap for Chimaera with TrainlessGNN is similar to that of LinearGNN and is therefore omitted. For cross-task and co-training, the linear GNNs effectively transfer knowledge between node classification and link prediction tasks. When we apply pre-training on the same task, the linear GNNs perform well. In pre-training and cross-task scenarios, the linear GNNs excel in transferring knowledge from all tasks to link prediction and graph classification. No variant of Chimaera had one gating function and one strategy that were best in all tested scenarios. However, in more than 93% of our test cases, a gating function is the best input for a GFM. The number of shots can influence which gating function is the most suitable input for an expert. Table III displays how well each combination of gating function and strategy performs directly on the test sets compared to the best GNNs listed in Table IV. Overall, the gating function enhances test accuracy or AUC for all tasks except those at the graph-level. For both GMoE and DA-MoE, using k = 3 yields better results than k = 2. The best expert-encoder pairs are shown for each dataset and task in Table IV. For node classification, Llama2 13B and E5 have the best results. In reasoning, the optimal combination is Llama2 13B with GCN, while for link prediction, the best
TABLE III: Test accuracy and AUC of all MoE combinations. The distribution of scores can be employed to distinguish Underlined: MoE models better or equal to the best GNN. between classes, as illustrated in Figure 4. Bold: best MoE. DA-MoE and GMoE with k = 2 omitted as k = 3 is either better or similar. N = Node, L = Link, G = Graph. VI. D ISCUSSION Accuracy AUC Gating Cora PubMed WikiCS WN18RR Cora PubMed BBBP BACE a) Key Insights: In more than half of the cross-task or Function (N) (N) (N) (L) (L) (L) (G) (G) pre-training scenarios, Chimaera achieves performance close Multi-GNN strategy mean 0.788 0.787 0.816 0.751 0.987 0.942 MLP 0.782 0.786 0.810 0.791 0.980 0.908 MoE 0.773 0.768 0.762 0.750 0.987 0.987 GMoE 0.779 0.777 0.784 0.754 0.990 0.961 DA-MoE 0.787 0.793 0.806 0.736 0.986 0.939 Multi-encoder strategy mean 0.773 0.793 0.811 0.720 0.988 0.983 MLP 0.756 0.798 0.803 0.762 0.987 0.982 MoE 0.681 0.791 0.788 0.729 0.989 0.984 GMoE 0.756 0.785 0.798 0.728 0.989 0.984 DA-MoE 0.778 0.800 0.810 0.716 0.990 0.982 Multi-layer strategy mean 0.818 0.774 0.816 0.744 0.987 0.987 MLP 0.762 0.776 0.816 0.770 0.987 0.988 MoE 0.754 0.774 0.780 0.745 0.989 0.988 GMoE 0.810 0.772 0.813 0.749 0.987 0.988 DA-MoE 0.809 0.778 0.815 0.744 0.987 0.989
0.627 0.753 0.634 0.436 0.617 0.755 0.618 0.758 0.642 0.732 0.611 0.745 0.626 0.730 0.614 0.740 0.620 0.745 0.623 0.707 0.621 0.712 0.630 0.379 0.631 0.658 0.625 0.733 0.632 0.709
TABLE IV: Test accuracy and AUC for all encoders and models with the best validation accuracy and AUC. #L = #Layers Dataset Encoder Model Layer Size #L Acc AUC Cora (Node) Llama2 13B GIN 256 1 0.795 – PubMed (Node) E5 GCN 64 1 0.794 – WikiCS (Node) Llama2 13B GAT 64 1 0.815 – WN18RR (Link) Llama2 13B GCN 128 1 0.725 – Cora (Link) E5 GAT 128 1 – 0.989 PubMed (Link) E5 GAT 128 1 – 0.982 BBBP (Graph) Llama2 7B GCN 64 1 – 0.648 BACE (Graph) ST GCN 64 3 – 0.767
pairing is E5 with a GAT. The best performance for graph classification is achieved with a GCN in combination with the sentence encoder or Llama2 7B. The GCN shows strong performance in graph classification and reasoning tasks, while the GAT is effective for graph classification and link prediction. The scores of the gating function of DA-MoE vary depending on the dataset and the strategy employed. Figure 3 shows the average scores for each expert trained on the Cora (Node), WN18RR, and BACE datasets. Although the average scores differ across training data and strategies, the three examples provide similar insights into their behavior. Notably, the gating function trained on BACE demonstrates consistent behavior across all test sets, using each expert with a comparable frequency. Additionally, the gating function’s performance varies depending on the specific test set employed. Another observation is that the gating functions trained on Cora (Node) and Cora (Link) exhibit similar behaviors. Most classes show unique score distributions that differentiate them from others.
to that of models trained directly on the target dataset and task. Therefore, it shows strong generalization capabilities. It also shows that Chimaera can use already trained GNNs and MoE (gating functions) for pre-training and cross-task scenarios. For same-task and co-training, Chimaera works better with PRODIGY, outperforming linear GNNs as a classifier. However, when it comes to pre-training or cross-task scenarios, linear GNNs tend to be superior. Overall, linear GNNs derive more benefits from the trained experts and gating mechanisms than PRODIGY. A possible explanation is that PRODIGY requires training for different tasks, whereas Chimaera effectively transfers knowledge across a single task and dataset. However, even with multi-task training, the Chimaera variant with PRODIGY is not better than using linear GNNs as a classifier in Chimaera. Linear GNNs generally exhibit strong transfer capabilities across different tasks and datasets. Even when working with a limited number of examples, Chimaera enables the application of linear GNNs to various tasks while introducing new parameters that can be optimized. As a result, Chimaera enhances both the homogenization and generalizability of linear GNNs. Additionally, our training method benefits linear GNNs. The experiments demonstrate that training Chimaera as a whole reduces the transferability of linear GNNs. In other words, training Chimaera as a whole reduces the transferability of the experts and the gating functions. Thus, our training procedure, which consists of stepby-step training and freezing components, benefits Chimaera. A Chimaera combining the experts with one of our strategies usually outperforms a Chimaera that uses no gating function and only one expert. Moreover, combining different experts enhances performance across all tasks, except for graph-level tasks. The score of a gating function depends on the training set, test set, and the strategy employed. Additionally, we demonstrate that the gating functions exhibit varying behavior across most classes. Using a larger number of experts tends to yield better results than using fewer. This is demonstrated by GMoE and DA-MoE, as k = 3 performs better than k = 2 for our MoE and, most of the time, also for our GFMs. Overall, the multi-layer strategy is the most effective for the MoE, as it typically leads to performance improvements. We recommend DA-MoE, as it uses the graph for gating and can be further tailored to specific needs. The performance of experts with encoders using larger embeddings improves the results for datasets with a larger number of classes in a task. This trend is consistent across all tasks we examined. An explanation is that larger embeddings make it easier to distinguish more classes. Our findings suggest that the choice of encoder plays a crucial role in a GNN’s performance. The size of the embeddings and the specific encoder selected strongly influence a GNN’s
1.0
0.22 ±0.05 0.28 ±0.09 0.56 ±0.06 0.21 ±0.00 0.28 ±0.13 0.40 ±0.14 0.51 ±0.01 0.28 ±0.00 3
0.8 0.6 0.4 0.2 0.0
0.45 cora_node ±0.17 0.43 pubmed_node ±0.13 0.43 wikics ±0.12 0.31 wn18rr ±0.14 0.54 cora_link ±0.09 0.22 pubmed_link ±0.01 0.22 bbbp ±0.00 0.35 bace ±0.00 1
(a) Cora (Node)
0.23 ±0.07 0.27 ±0.10 0.24 ±0.05 0.28 ±0.13 0.24 ±0.08 0.23 ±0.05 0.25 ±0.00 0.34 ±0.01 2 Expert
(b) WN18RR
0.32 ±0.16 0.30 ±0.11 0.33 ±0.12 0.40 ±0.15 0.22 ±0.02 0.55 ±0.06 0.53 ±0.01 0.31 ±0.00 3
1.0 0.8 0.6 0.4 0.2 0.0
Trained On
0.34 ±0.17 0.37 ±0.13 0.22 ±0.05 0.58 ±0.00 0.28 ±0.12 0.25 ±0.06 0.27 ±0.01 0.34 ±0.01 2 Expert
Trained On
Trained On
0.44 cora_node ±0.17 0.34 pubmed_node ±0.12 0.22 wikics ±0.03 0.21 wn18rr ±0.00 0.44 cora_link ±0.16 0.35 pubmed_link ±0.13 0.22 bbbp ±0.00 0.38 bace ±0.01 1
0.58 cora_node ±0.00 0.48 pubmed_node ±0.01 0.23 wikics ±0.00 0.21 wn18rr ±0.00 0.28 cora_link ±0.02 0.22 pubmed_link ±0.00 0.22 bbbp ±0.00 0.36 bace ±0.00 1
0.21 ±0.00 0.25 ±0.01 0.49 ±0.00 0.58 ±0.00 0.22 ±0.00 0.26 ±0.01 0.26 ±0.01 0.33 ±0.00 2 Expert
0.21 ±0.00 0.27 ±0.01 0.28 ±0.00 0.21 ±0.00 0.50 ±0.02 0.52 ±0.01 0.52 ±0.01 0.31 ±0.00 3
1.0 0.8 0.6 0.4 0.2 0.0
(c) BACE
cora_node
1/21
pubmed_node
2/21
wikics
1/21
0.8
wn18rr 10/21
0.6
cora_link
0/21
0.4
pubmed_link
0/21
bbbp
1/21
bace
0/21
Trained On
1.0
0.2 0.0
cora_node
0/1
pubmed_node
0/1
wikics
0/1
0.8
wn18rr
0/1
0.6
cora_link
1/1
0.4
pubmed_link
0/1
bbbp
1/1
bace
1/1
All Expert
All Expert
(a) Cora (Node)
(b) Cora (Link)
cora_node
9/55
pubmed_node
8/55
1.0
wikics 10/55
0.8
wn18rr 11/55
0.6
cora_link 32/55 pubmed_link 31/55 bbbp 17/55 bace
9/55 All Expert
(c) WN18RR
0.4 0.2 0.0
Trained On
Trained On
Trained On
Fig. 3: The average scores for each expert on the test sets given the gating function of DA-MoE with k = 3. The dataset for the test set is indicated in the sub-caption, and the gating function uses a multi-layer strategy.
1.0
0.2 0.0
cora_node
0/1
pubmed_node
0/1
wikics
1/1
0.8
wn18rr
0/1
0.6
cora_link
0/1
0.4
pubmed_link
1/1
bbbp
0/1
bace
0/1
1.0
0.2 0.0
All Expert (d) BACE
Fig. 4: The subfigures demonstrate the number of class pairs that exhibit a similar distribution of scores generated by the gating function. This gating function is part of the DA-MoE model with k = 3 and uses a multi-layer strategy. The dataset for the test set is specified in the sub-caption.
effectiveness. Thus, selecting the right encoder is an important optimization consideration when applying GNNs to graphs. b) Limitations: We only train our models for one task type. This creates an issue with PRODIGY when it is not trained for a link-level task. In this case, the linear projection for the edge embeddings is not trained, and PRODIGY performs worse on link-level tasks. However, to address this, we conduct experiments by training PRODIGY for multiple tasks. Despite this effort, linear GNNs still outperform PRODIGY when dealing with unknown datasets. Another limitation is that we do not train the experts, gating functions, and GFMs simultaneously without any pre-training. However, the training with linear GNN and unfrozen parameters does not improve performance; in fact, it appears to decrease it. Additionally, testing all hyperparameter combinations of Chimaera significantly increases the time required for hyperparameter optimization when using grid search. We use a maximum of three experts. However, even with this limit, we demonstrate that having more than one expert is advantageous. We leave the exploration of the optimal number of experts open for future work. VII. C ONCLUSION AND F UTURE W ORK Chimaera is the first model to integrate gating functions and GFMs using multiple LLM embeddings. The combination of GNNs through gating functions enhances performance. Additionally, using a gating function combined with MoE achieves better results for Chimaera than using the best expert. Overall, Chimaera demonstrates strong generalization capabilities. It enables linear GNNs to be applied to all classification tasks. The application of linear GNNs facilitates transfer capabilities across different tasks and datasets, resulting in only a minimal decrease in performance compared to models specifically trained for those tasks. Future research may explore additional strategies for combining experts and optimizing the gating functions (MoE). This may also include using a broader range of encoders, including those not based on LLMs, such as Node2Vec. Moreover,
exploring different training procedures, including pre-training methods for GNNs and transductive GNNs, may lead to improved transfer performance. VIII. E THICAL C ONSIDERATIONS Our paper presents foundational research and is not linked to specific applications. While our work has broad implications, making GNN and similar models more applicable to broader training and testing settings, we believe that no specific societal consequences require immediate emphasis in this context. Acknowledgments: The authors acknowledge support by the state of Baden-Württemberg through bwHPC. This work was co-funded by the Deutsche Forschungsgemeinschaft (DFG) as part of the CodeInspector Project - 504226141. R EFERENCES [1] R. Bommasani, D. A. Hudson, E. Adeli, R. B. Altman, S. Arora, et al., “On the opportunities and risks of foundation models,” arXiv:2108.07258, 2021. [2] L. Yuan, D. Chen, Y. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, C. Liu, M. Liu, Z. Liu, Y. Lu, Y. Shi, L. Wang, J. Wang, B. Xiao, Z. Xiao, J. Yang, M. Zeng, L. Zhou, and P. Zhang, “Florence: A new foundation model for computer vision,” arXiv:2111.11432, 2021. [3] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A frontier large vision-language model with versatile abilities,” arXiv:2308.12966, 2023. [4] OpenAI, “GPT-4 technical report,” arXiv:2303.08774, 2023. [5] R. Anil, S. Borgeaud, Y. Wu, J. Alayrac, J. Yu, et al., “Gemini: A family of highly capable multimodal models,” arXiv:2312.11805, 2023. [6] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv:2307.09288, 2023. [7] J. Liu, C. Yang, Z. Lu, J. Chen, Y. Li, M. Zhang, T. Bai, Y. Fang, L. Sun, P. S. Yu, and C. Shi, “Towards graph foundation models: A survey and beyond,” arXiv:2310.11829, 2023. [8] H. Mao, Z. Chen, W. Tang, J. Zhao, Y. Ma, T. Zhao, N. Shah, M. Galkin, and J. Tang, “Position: Graph foundation models are already here,” in ICML, 2024. [9] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in NAACLHLT, pp. 4171–4186, Association for Computational Linguistics, 2019. [10] R. Bommasani, D. A. Hudson, E. Adeli, R. B. Altman, S. Arora, et al., “On the opportunities and risks of foundation models,” arXiv:2108.07258, 2021. [11] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, et al., “Emergent abilities of large language models,” Tr. Mach. Learn. Res., 2022. [12] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in ICLR, OpenReview.net, 2017. [13] K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?,” in ICLR, OpenReview.net, 2019. [14] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in ICLR, OpenReview.net, 2018. [15] W. L. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in NeurIPS, pp. 1024–1034, 2017. [16] P. Li, Y. Guo, Y. Luo, X. Wang, Z. Wang, and X. Liu, “Graph neural networks based memory inefficiency detection using selective sampling,” in SC, pp. 85:1–85:14, IEEE, 2022. [17] Q. Sun, Y. Liu, H. Yang, R. Zhang, M. Dun, M. Li, X. Liu, W. Xiao, Y. Li, Z. Luan, and D. Qian, “CoGNN: Efficient scheduling for concurrent GNN training on GPUs,” in SC, pp. 39:1–39:15, IEEE, 2022. [18] M. Zhang, Q. Hu, C. Wan, H. Wang, P. Sun, Y. Wen, and T. Zhang, “Sylvie: 3d-adaptive and universal system for large-scale graph neural network training,” in ICDE, pp. 3823–3836, IEEE, 2024. [19] Q. Huang, H. Ren, P. Chen, G. Krzmanc, D. Zeng, P. Liang, and J. Leskovec, “PRODIGY: Enabling in-context learning over graphs,” in NeurIPS, 2023. [20] K. Dong, Z. Guo, and N. V. Chawla, “You do not have to train graph neural networks at all on text-attributed graphs,” arXiv:2404.11019, 2024.
[21] J. Zhao, H. Mostafa, M. Galkin, M. Bronstein, Z. Zhu, and J. Tang, “GraphAny: A foundation model for node classification on any graph,” arXiv:2405.20445, 2024. [22] Z. Chen, H. Mao, J. Liu, Y. Song, B. Li, W. Jin, B. Fatemi, A. Tsitsulin, B. Perozzi, H. Liu, and J. Tang, “Text-space graph foundation models: Comprehensive benchmarks and new insights,” arXiv:2406.10727, 2024. [23] Z. Wang, Z. Liu, T. Ma, J. Li, Z. Zhang, X. Fu, Y. Li, Z. Yuan, W. Song, Y. Ma, Q. Zeng, X. Chen, J. Zhao, J. Li, M. Jiang, P. Lio, N. V. Chawla, C. Zhang, and Y. Ye, “Graph foundation models: A comprehensive survey,” arXiv:2505.15116, 2025. [24] F. Wu, A. H. S. Jr., T. Zhang, C. Fifty, T. Yu, and K. Q. Weinberger, “Simplifying graph convolutional networks,” in ICML, PMLR, 2019. [25] M. Plenz and A. Frank, “Graph language models,” arXiv:2401.07105, 2024. [26] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res., vol. 21, pp. 140:1–140:67, 2020. [27] J. Guo, L. Du, and H. Liu, “GPT4Graph: Can large language models understand graph structured data? An empirical evaluation and benchmarking,” arXiv:2305.15066, 2023. [28] H. Liu, J. Feng, L. Kong, N. Liang, D. Tao, Y. Chen, and M. Zhang, “One for all: Towards training one graph model for all classification tasks,” arXiv:2310.00149, 2023. [29] L. Kong, J. Feng, H. Liu, C. Huang, J. Huang, Y. Chen, and M. Zhang, “GOFA: A generative one-for-all model for joint graph language modeling,” arXiv:2407.09709, 2024. [30] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” Neural Comput., vol. 3, no. 1, 1991. [31] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” J. Mach. Learn. Res., vol. 23, pp. 120:1–120:39, 2022. [32] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” in ICLR, OpenReview.net, 2017. [33] N. Shazeer, Y. Cheng, N. Parmar, D. Tran, A. Vaswani, and others, “Mesh-tensorflow: Deep learning for supercomputers,” in NeurIPS, 2018. [34] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “GShard: Scaling giant models with conditional computation and automatic sharding,” in ICLR, OpenReview.net, 2021. [35] L. Liu, Y. J. Kim, S. Wang, C. Liang, Y. Shen, and others, “GRIN: GRadient-INformed MoE,” arXiv:2409.12136, 2024. [36] H. Wang, Z. Jiang, Y. You, Y. Han, G. Liu, J. Srinivasa, R. Kompella, and Z. Wang, “Graph mixture of experts: Learning on large-scale graphs with explicit diversity modeling,” in NeurIPS, 2023. [37] Z. Yao, C. Liu, X. Meng, Y. Zhan, J. Wu, S. Pan, and W. Hu, “DA-MoE: Addressing depth-sensitivity in graph-level analysis through mixture of experts,” arXiv:2411.03025, 2024. [38] L. Ma, H. Han, J. Li, H. Shomer, H. Liu, X. Gao, and J. Tang, “Mixture of link predictors on graphs,” in NeurIPS, 2024. [39] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in EMNLP-IJCNLP, ACL, 2019. [40] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei, “Text embeddings by weakly-supervised contrastive pretraining,” arXiv:2212.03533, 2022. [41] Y. Luo, L. Shi, and X. Wu, “Classic GNNs are strong baselines: Reassessing GNNs for node classification,” in NeurIPS, 2024. [42] O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann, “Pitfalls of graph neural network evaluation,” arXiv:1811.05868, 2018. [43] Q. Huang, H. Ren, and J. Leskovec, “Few-shot relational reasoning via connection subgraph pretraining,” in NeurIPS, 2022. [44] J. Feng, H. Liu, L. Kong, Y. Chen, and M. Zhang, “TAGLAS: An atlas of text-attributed graph datasets in the era of large graph and language models,” arXiv:2406.14683, 2024. [45] Z. Chen, H. Mao, H. Li, W. Jin, H. Wen, X. Wei, S. Wang, D. Yin, W. Fan, H. Liu, and J. Tang, “Exploring the potential of large language models (LLMs) in learning on graphs,” arXiv:2307.03393, 2023. [46] P. Mernyei and C. Cangea, “Wiki-CS: A Wikipedia-based benchmark for graph neural networks,” arXiv:2007.02901, 2020. [47] H. Zhao, S. Liu, M. Chang, H. Xu, J. Fu, Z. Deng, L. Kong, and Q. Liu, “Gimlet: A unified graph-text model for instruction-based molecule zeroshot learning,” NeurIPS, vol. 36, 2024. [48] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR, 2015.
[49] Z. Wang, Z. Zhang, N. V. Chawla, C. Zhang, and Y. Ye, “GFT: graph foundation model with transferable tree vocabulary,” in NeurIPS, 2024. [50] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in ICLR, OpenReview.net, 2019. [51] P. Velickovic, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm, “Deep graph infomax,” arXiv:1809.10341, 2018. [52] S. Thakoor, C. Tallec, M. G. Azar, M. Azabou, E. L. Dyer, R. Munos, P. Velickovic, and M. Valko, “Large-scale representation learning on graphs via bootstrapping,” in ICLR, OpenReview.net, 2022. [53] Z. Hou, X. Liu, Y. Cen, Y. Dong, H. Yang, C. Wang, and J. Tang, “Graphmae: Self-supervised masked graph autoencoders,” in KDD, pp. 594–604, ACM, 2022. [54] E. Chien, W. Chang, C. Hsieh, H. Yu, J. Zhang, O. Milenkovic, and I. S. Dhillon, “Node feature extraction by self-supervised multi-scale neighborhood prediction,” in ICLR, OpenReview.net, 2022. [55] K. Ding, J. Wang, J. Li, K. Shu, C. Liu, and H. Liu, “Graph prototypical networks for few-shot learning on attributed networks,” in CIKM, pp. 295– 304, ACM, 2020. [56] S. Wang, K. Ding, C. Zhang, C. Chen, and J. Li, “Task-adaptive few-shot node classification,” in KDD, pp. 1910–1919, ACM, 2022. [57] S. Wang, C. Chen, and J. Li, “Graph few-shot learning with task-specific structures,” in NeurIPS, 2022. [58] Z. Tan, S. Wang, K. Ding, J. Li, and H. Liu, “Transductive linear probing: A novel framework for few-shot node classification,” in Learning on Graphs Conf., vol. 198, p. 4, PMLR, 2022.
A PPENDIX Cora
HIV
FB15K237
WN188RR
Arxiv
WikiCS
Pubmed
Cora
Tr Te aine st ed d O n O n
Graph
Edge
Node
To compare Chimaera with other GFMs, we use the setup by GFT [49], which uses cross-domain and cross-task Pubmed graph datasets. It includes citation networks such as Cora, PubMed, arXiv, and the web link network WikiCS for node WikiCS classification. Additionally, two Knowledge Graphs, WN18RR and FB15K237, are used for reasoning tasks. For graph-level Arxiv tasks, the molecule network HIV is used. The characteristics of the datasets and the specific training parameters are detailed WN18RR in Table V. Node embeddings are generated using SentenceBERT [39], following the approach outlined by GFT. All experiments are conducted ten times with different random FB15K237 seeds. For the HIV dataset, we evaluate using test AUC, while for the other datasets, we focus on accuracy. HIV For the comparison, we train a one-, two- and three-layer GCNs, all with a hidden size of 768. Then, we combine these GCNs with a DA-MoE using a one-layer GCN with hidden Node Edge Graph size 768 as a gating function. The gating function is also trained in the same manner as the GCNs. During training, Fig. 5: Chimaera’s performance on the datasets of the comparwe use a weight decay of 0.001, a drop rate of 0.2, and the ison. parameters shown in Table V. These parameters are based on the default settings used by GFT but have been slightly adjusted to accelerate the training process. For optimization, settings, even though it sometimes performs worse than the baselines. Regarding the WN18RR knowledge graph, the results we use the AdamW optimizer [50]. For the classification tasks, we employ a linear GNN with can be found in Table XI. Here, the Chimaera model trained different numbers of shots as the classifier for Chimaera. The on FB15K237 either matches or exceeds the performance of experiment is conducted exclusively on Chimaera, while results OFA. In the context of graph classification, the Chimaera for the other models are referenced from GFT [49]. In the demonstrates similar results for the HIV dataset, regardless of first part, we evaluate all trained Chimaera instances using the whether it was trained on this specific dataset, as indicated in complete set of training examples for the linear GNNs. Figure 5 Table XII. In general, Chimaera achieves results that are similar to or illustrates the performance of Chimaera across all datasets. It better than those of OFA. Additionally, it can outperform GFT demonstrates that Chimaera effectively transfers knowledge on arXiv when working with a smaller number of classes. Our between most datasets. Table VI compares the performance results show that Chimaera, which was trained on FB15K237, of Chimaera and its components against other GFMs in the performs effectively on other datasets, demonstrating its ability same-task and co-training setting. Overall, Chimaera and its to transfer knowledge between tasks and domains. Additionally, components demonstrate competitive performance compared it can outperform models trained on multiple datasets. Overall, to the other GFMs, showing promise for further development. Chimaera performs well on datasets with a small number of The performance decrease when using an untrained linear classes. Furthermore, the high standard deviation in our results GNN instead of a trained classifier is typically less than five suggests that providing high-quality examples contributes percentage points. positively to its performance. As a result, future improvements In the second part of our comparison, we evaluate Chimaera to Chimaera may also focus on selecting good examples and under various few-shot settings across different datasets. These eliminating poor ones for the adaptation of the linear GNN. settings differ in the number of shots and the number of classes. The tested classes are randomly chosen. The results for the arXiv dataset are presented in Tables VII and VIII. In the 3-way classification, Chimaera trained on FB15K237 outperforms all other baselines and Chimaeras. Overall, the Chimaera model trained on FB15K237 demonstrates strong performance on the arXiv dataset, surpassing models such as Prodigy and OFA. The results for the Cora dataset, shown in Table IX, indicate that the Chimaera trained on FB15K237 is the best-performing variant, again surpassing OFA. However, it is outperformed by GFT, which was adapted using additional fine-tuning instances. For the FB15K237 dataset, Table X shows that the Chimaera trained on arXiv achieves reasonable performance across all
TABLE V: The dataset characteristics (upper block) and training parameters (lower block) used for the comparison. ∗ Number of vertices and edges for HIV is the mean per graph.
Graphs Vertices Edges Classes
Node Classification Reasoning Graph Classification Cora PubMed WikiCS arXiv WN18RR FB15K237 HIV 1 1 1 1 1 1 41 127 2 708 19 717 169 343 11 701 14 541 40 943 25.5∗ 10 556 44 338 1 166 243 216 123 310 116 93 003 27.5∗ 7 3 40 10 237 11 2
1 000 Epochs Batch size − 200 Early stop Learning rate 0.005
1 000 − 200 0.005
1 000 − 200 0.0001
2 000 − 500 0.005
3 000 1 024 200 0.005
3 000 1 024 200 0.0001
100 1 024 20 0.005
TABLE VI: Model performance in pre-training and fine-tuning setting with SD
DGI [51] BGRL [52] GraphMAE [53] GIANT [54] GFT [49]
Node Classification Cora PubMed WikiCS .721±.003 .731±.006 .753±.010 .712±.003 .753±.013 .765±.007 .731±.004 .743±.003 .776±.004 .751±.005 .723±.005 .766±.009 .786 ± .012 .772 ± .020 .794 ± .004
1-layer GCN [12] 2-layer GCN [12] 3-layer GCN [12] DA-MoE [37] Chimaera
.716±.023 .750±.025 .750±.026 .778±.015 .709±.043
Method
.741±.019 .745±.025 .748±.023 .769±.022 .739±.030
Reasoning Graph Classification arXiv WN18RR FB15K237 HIV .692±.002 .758±.006 .813±.002 .596±.012 .712±.002 .754±.003 .807±.003 .640±.011 .709±.003 .790±.005 .853±.002 .610±.006 .701±.003 .844±.003 .875±.005 .654±.014 .719±.001 .919 ± .003 .897 ± .002 .727 ± .014
.762±.005 .712±.002 .765±.009 .722±.003 .770±.010 .695±.069 .788±.006 .728 ± .002 .468±.041 .710±.003
.687±.005 .687±.005 .669±.019 .691±.003 .649±.015
.525±.013 .466±.067 .386±.049 .504±.023 .503±.026
.483±.020 .485±.029 .514±.036 .543±.033 .524±.023
TABLE VII: Few-shot learning performance on arXiv (Part 1). “# trains” indicates the number of fine-tuning instances per class for GFT. The best performance is bold, and the best Chimaera is underlined. 5-way 3-way 5-shot 3-shot 1-shot 5-shot 3-shot 1-shot GPN [55] .505±.031 .483±.038 .386±.016 .623±.049 .585±.030 .485±.056 .608±.075 .560±.089 .456±.107 .742±.099 .705±.115 .594±.136 TENT [56] GLITTER [57] .560±.044 .574±.049 .471±.027 .621±.109 .609±.121 .592±.055 .501±.088 .462±.079 .358±.086 .629±.117 .584±.113 .463±.108 TLP-BGRL [58] TLP-SURGL [58] .779 ± .065 .742 ± .076 .618± .101 .863±.075 .838±.089 .735±.127 PRODIGY [19] .611±.059 .586±.058 .482±.062 .736±.069 .714±.073 .616±.085 OFA [28] .599±.013 .587±.064 .528±.039 .722±.033 .718±.016 .605±.027 GFT (# train = 5) [49] .680±.019 .660±.025 .582±.042 .786±.040 .740±.032 .662±.041 GFT (# train = 10) [49] .724±.036 .717±.029 .624±.026 .788±.012 .762±.042 .699±.038 GFT (# train = 20) [49] .731±.028 .717±.024 .642±.021 .807±.023 .793±.023 .729±.034 GFT (# train = 30) [49] .747±.030 .733±.037 .651±.038 .796±.026 .768±.027 .732±.034 Method
Chimaera (Cora) Chimaera (PubMed) Chimaera (WikiCS) Chimaera (WN18RR) Chimaera (FB15K237) Chimaera (HIV)
.628±.159 .663±.191 .641±.150 .562±.143 .762± .127 .632±.189
.592±.156 .548±.178 .556±.204 .478±.194 .742± .175 .589±.188
.469±.197 .746±.181 .787±.118 .743±.118 .297±.183 .741±.192 .676±.182 .565±.173 .413±.192 .742±.146 .705±.180 .583±.229 .267±.134 .703±.134 .585±.179 .517±.214 .586± .163 .871± .108 .875± .095 .803± .147 .332±.199 .685±.224 .739±.160 .432±.224
TABLE VIII: Few-shot learning performance on arXiv (Part 2). “# trains” indicates the number of fine-tuning instances per class for GFT. The best performance is bold, and the best Chimaera is underlined. Method
40-way 20-way 10-way 5-shot 3-shot 1-shot 5-shot 3-shot 1-shot 5-shot 3-shot Prodigy [19] .255±.001 .237±.001 .214±.002 .343±.004 .313±.007 .292±.010 .508±.018 .474±.030 OFA [28] .240±.006 .221±.009 .213±.013 .363±.005 .326±.002 .294±.012 .496±.027 .481±.037 GFT (# train = 5) [49] .363±.010 .344±.010 .265±.011 .458±.011 .426±.012 .350±.010 .564±.035 .524±.011 GFT (# train = 10) [49] .418±.009 .391±.019 .308±.006 .497±.019 .469±.015 .410±.018 .602±.011 .576±.016 GFT (# train = 20) [49] .451±.012 .439±.014 .350±.015 .533±.014 .509±.017 .430±.018 .635±.016 .614±.030 GFT (# train = 30) [49] .467± .011 .446± .010 .359 ± .012 .540 ±.014 .519 ± .012 .438 ± .019 .644 ± .013 .628 ± .09 Chimaera (Cora) Chimaera (PubMed) Chimaera (WikiCS) Chimaera (WN18RR) Chimaera (FB15K237) Chimaera (HIV)
.275±.065 .297±.047 .283±.052 .280±.063 .285±.055 .274±.060 .243±.069 .245±.048 .288± .062 .308± .085 .271±.097 .284±.062
.147±.074 .137±.070 .123±.041 .096±.047 .195± .058 .148±.058
.389±.115 .447±.124 .411±.122 .341±.169 .463± .113 .421±.160
.371±.078 .392±.106 .357±.077 .307±.104 .442± .112 .389±.113
.158±.114 .524±.144 .163±.132 .543±.147 .170±.089 .493±.136 .089±.067 .453±.166 .254± .131 .601± .158 .184±.121 .529±.176
.468±.139 .446±.150 .393±.124 .394±.141 .582± .159 .496±.167
1-shot .411±.063 .395±.054 .444±.026 .481±.025 .532±.018 .547± .016 .212±.154 .257±.172 .239±.101 .168±.095 .413± .167 .264±.190
TABLE IX: Few-shot learning performance on Cora. “# trains” indicates the number of fine-tuning instances per class for GFT. The best performance is bold and the best Chimaera is underlined. 7-way 3-shot − − − −
Method GPN [55] TENT [56] TLP-BGRL [58] TLP-SURGL [58]
5-shot − − − −
1-shot − − − −
5-shot − − − −
5-way 3-shot − − − −
1-shot − − − −
OFA [28] .321±.018 .360±.021 .304±.024 .423±.024 .313±.026 .237±.017 GFT (# train = 1) [49] .436±.074 .433±.081 .414±.080 .523±.066 .515±.066 .498±.068 GFT (# train = 2) [49] .565±.035 .559±.035 .536±.045 .637±.034 .624±.044 .605±.046 GFT (# train = 5) [49] .674±.043 .673±.044 .661±.044 .741±.043 .744±.045 .727±.049 GFT (# train = 10) [49] .740 ± .039 .743 ± .037 .726 ± .038 .785 ± .030 .789 ± .026 .769 ± .022 Chimaera (PubMed) Chimaera (WikiCS) Chimaera (arXiv) Chimaera (WN18RR) Chimaera (FB15K237) Chimaera (HIV)
.531±.044 .437±.051 .580±.040 .409±.045 .620± .036 .558±.056
.446±.043 .335±.062 .540±.052 .300±.062 .575± .054 .463±.084
.290±.048 .568±.090 .270±.058 .478±.062 .422±.068 .633±.085 .226±.039 .426±.052 .458± .067 .686± .066 .313±.070 .595±.097
.504±.068 .402±.098 .612±.102 .348±.101 .643± .088 .506±.133
5-shot .638±.029 .590±.024 .813±.019 .925 ± .010
2-way 3-shot − − − −
1-shot .561±.021 .543±.021 .592±.025 .815±.021
.722±.038 .622±.012 .519±.044 .750±.041 .763±.036 .0729±.046 .823±.040 .817±.038 .780±.063 .870±.034 .860±.033 .860±.034 .879±.029 .885 ± .024 .884 ± .029
.351±.065 .707±.083 .335±.064 .712±.090 .480±.108 .831±.072 .288±.028 .587±.105 .527± .079 .867± .054 .358±.112 .770±.095
.661±.102 .669±.116 .793±.062 .638±.078 .822± .069 .695±.115
.559±.103 .657±.083 .691±.146 .518±.116 .745± .143 .596±.151
TABLE X: Few-shot learning performance on FB15K237. “# trains” indicates the number of fine-tuning instances per class for GFT. The best performance is bold and the best Chimaera is underlined. Method
40-way 10-way 5-way 5-shot 3-shot 1-shot 5-shot 3-shot 1-shot 5-shot 3-shot 1-shot Prodigy [19] .620±.006 .596±.002 .543±.007 .843±.078 .796±.083 .661±.099 .881±.007 .880±.005 .876±.008 OFA [28] .665±.003 .658±.005 .635±.009 .836±.062 .831±.015 .835±.041 .914±.006 .911±.007 .910±.010 GFT (# train = 10) [49] .611±.016 .615±.013 .608±.014 .788±.018 .791±.016 .792±.018 .863±.011 .860±.018 .877±.009 GFT (# train = 20) [49] .704±.017 .706±.021 .702±.014 .854±.021 .856±.013 .859±.015 .918±.011 .918±.062 .918±.015 GFT (# train = 30) [49] .750 ± .010 .746 ± .007 .750 ± .009 .891± .017 .885 ± .022 .881 ± .014 .919 ± .01 .923 ± .019 .924 ± .013 Chimaera (Cora) Chimaera (PubMed) Chimaera (WikiCS) Chimaera (arXiv) Chimaera (WN18RR) Chimaera (HIV)
.542±.148 .533±.135 .546±.121 .520±.111 .556± .107 .488±.155
.485±.157 .497±.135 .485±.121 .489±.102 .499± .123 .472±.121
.425±.158 .431±.159 .394±.148 .442± .131 .415±.136 .406±.122
.668±.128 .694±.143 .628±.151 .710± .137 .624±.141 .697±.132
.625±.160 .668±.145 .609±.150 .707± .155 .597±.153 .682±.147
.577±.174 .552±.220 .568±.190 .638± .186 .488±.148 .590±.190
.778±.064 .817± .097 .693±.193 .816±.112 .648±.135 .786±.075
.754±.120 .747±.146 .709±.168 .801± .118 .676±.129 .781±.113
.685±.161 .676±.173 .676±.169 .756± .152 .602±.173 .625±.148
TABLE XI: Few-shot learning performance on WN18RR. “# trains” indicates the number of fine-tuning instances per class for GFT. The best performance is bold and the best Chimaera is underlined. 10-way 5-way 3-way 5-shot 3-shot 1-shot 5-shot 3-shot 1-shot 5-shot 3-shot 1-shot OFA [28] .326±.016 .306±.010 .258±.011 .483±.032 .450±.024 .344±.015 .607±.038 .613±.026 .518±.027 GFT (# train = 1) [49] .355±.046 .355±.050 .353±.042 .488±.036 .485±.037 .481±.044 .626±.027 .607±.039 .584±.038 GFT (# train = 2) [49] .424±.031 .425±.029 .420±.030 .559±.026 .548±.023 .544±.022 .663±.019 .664±.017 .649±.033 GFT (# train = 5) [49] .448±.029 .449±.031 .448±.035 .580±.026 .577±.022 .574±.025 .689±.022 .697±.021 .688±.010 GFT (# train = 10) [49] .512± .046 .513 ± .048 .509 ± .042 .627 ± .049 .630 ± .059 .636 ± .063 .722±.037 .726 ± .049 .726 ± .048 Method
Chimaera (Cora) Chimaera (PubMed) Chimaera (WikiCS) Chimaera (arXiv) Chimaera (FB15K237) Chimaera (HIV)
.296±.048 .299±.046 .290±.045 .307±.038 .312± .036 .308±.048
.259±.026 .274±.062 .249±.030 .313±.044 .317± .038 .296±.070
.190±.050 .455±.117 .208±.046 .478±.131 .186±.046 .442±.133 .243±.078 .518±.144 .256± .064 .533± .147 .210±.097 .492±.140
.424±.107 .386±.066 .362±.101 .514±.146 .530± .146 .440±.128
.366±.116 .293±.101 .295±.069 .388±.127 .429± .132 .337±.175
.599±.155 .614±.151 .573±.125 .711±.154 .729± .161 .644±.172
.625±.126 .550±.120 .514±.117 .690±.158 .720± .151 .624±.151
.479±.172 .404±.124 .458±.151 .531±.197 .596± .189 .563±.165
TABLE XII: Few-shot learning performance on HIV. “# trains” indicates the number of fine-tuning instances per class for GFT. The best performance is bold and the best Chimaera is underlined. Method
2-Way 10-shot 5-shot 3-shot 1-shot OFA [28] .544±.049 .576±.037 .593 ± .030 .572±.018 GFT (# train = 10) [49] .532±.118 .542±.106 .576±.105 .583±.091 GFT (# train = 20) [49] .587 ± .075 .588 ± .069 .584±.073 .599 ± .071 GFT (# train = 30) [49] .581±.053 .586±.051 .583±.054 .591±.052 Chimaera (Cora) Chimaera (PubMed) Chimaera (WikiCS) Chimaera (arXiv) Chimaera (WN18RR) Chimaera (FB15K237)
.506±.024 .525± .037 .500±.030 .524±.022 .524±.031 .518±.022
.514±.027 .522±.037 .495±.026 .528± .022 .526±.032 .522±.025
.512±.024 .523±.036 .494±.026 .526± .021 .525±.031 .518±.022
.512±.023 .524±.037 .500±.029 .529± .023 .527±.033 .522±.025