One Model, Many Graphs: Learning over Attributed Graphs across Heterogeneous Modalities with Vision-Language Models Jiayi Yang1* , Yifang Chen1* , Yuanfu Sun1,2 , Jiajin Liu1,2 , Qiaoyu Tan1† 1 New York University Shanghai, 2 New York University {jy4656,yc6990,qiaoyu.tan}@nyu.edu
Abstract
Title: Attention Is All You Need Abstract: The dominant sequence...
1
TAG TAG
MAG
Image-centric Framework Text-centric Framework
TAG... TAG MAG
TAG
MAG
IAG
Unified framework
IAG
I/TAG MAG
MAG RevGAT GraphVLM
100
GraphSAGE OMG-VLM(Ours)
99.1
75
UniGraph2
92.5
67.85
50 25
0
IAG (RedditS)
TAG (arXiv)
MAG (Arts)
Figure 1: Heterogeneous modalities over attributed graphs. Prior methods target TAGs or MAGs, while OMG-VLM handles heterogeneous modality schemas. Bars compare OMG-VLM with representative baselines on IAG (RedditS), TAG (arXiv), and MAG (Arts).
2021; Veličković et al., 2018). In these settings, node attributes provide semantic signals beyond graph topology, enabling models to reason jointly over structure and content. However, attribute modalities vary substantially across domains: some graphs contain textual attributes, such as titles or abstracts in citation networks; others contain visual attributes, such as product or user images; and some combine multiple modalities (Wei et al., 2019; Wang et al., 2023; He and McAuley, 2016; Ning et al., 2025). This modality heterogeneity reflects real-world data collection processes, where available attributes depend on domain-specific constraints. Developing models that can operate across such heterogeneous attributed graphs is therefore an important yet underexplored challenge.
Introduction
Existing graph learning methods are largely built around fixed modality assumptions. GNNs typically operate on predefined feature vectors and require modality-specific encoders to convert raw attributes into numeric representations (Kipf and Welling, 2017; Veličković et al., 2018; Yan et al., 2023; He et al., 2024a; Zolnai-Lucas et al., 2024). This often leads to separate architectures or pre-
Attributed graph learning is a fundamental problem in data mining and machine learning, with applications in social networks, e-commerce platforms, citation graphs, and knowledge-centric systems (Kipf and Welling, 2017; Hamilton et al., 2018; Hu et al., * Equal contribution. †
IAG IAG
Accuracy (%)
arXiv:2607.19128v1 [cs.LG] 21 Jul 2026
Vision-language models (VLMs) provide a unified representation space for textual and visual information, yet their potential as generalpurpose backbones for graph-structured data remains largely unexplored. In practice, attributed graphs exhibit substantial modality heterogeneity: some graphs contain only textual node attributes, others only visual attributes, while still others provide both. Existing graph learning approaches are typically designed for fixed modality schemas, requiring separate models for different settings and limiting scalability and cross-graph generalization. To bridge this gap, we present OMGVLM (One Model, Many Graphs with VisionLanguage Models), a unified framework for learning over attributed graphs across heterogeneous modality schemas. OMG-VLM leverages a pretrained VLM as a shared backbone and introduces structure-aware graph adapters that integrate neighborhood information while remaining compatible with the VLM’s native embedding space. This design enables effective learning over text-attributed, imageattributed, and multi-attributed graphs within a single model. Extensive experiments across diverse domains show that OMG-VLM consistently outperforms state-of-the-art GNN- and LLM-based baselines on attributed graph learning tasks such as node classification and link prediction, while exhibiting strong generalization to unseen graphs and varying modality schemas. The source code is available at https://github.com/Jo-eyang/OMG-VLM.
LuPro Japanese LPAT- ...
Corresponding author.
1
processing pipelines for text-attributed, imageattributed, and multimodal graphs, limiting reuse and cross-graph generalization. Recent LLM-based graph methods provide more flexibility for textattributed graphs by incorporating neighborhood information through prompting or in-context learning (Lv et al., 2025; Tang et al., 2024; Chen et al., 2024a; He et al., 2024b). However, they remain text-centric and typically rely on external vision encoders for visual attributes (Fan et al., 2026; Yan et al., 2025; He et al., 2025), which limits their ability to support image-only or heterogeneous multimodal graphs within a unified model. To address this gap, we study attributed graph learning under modality heterogeneity through the lens of VLMs. Pretrained VLMs provide a shared representation space for textual and visual information (Bai et al., 2023; Liu et al., 2023), making them a natural foundation for unified learning across text-attributed, image-attributed, and multiattributed graphs. Instead of relying on separate modality-specific encoders, we investigate whether a single VLM-based model can serve as a shared backbone across heterogeneous attributed graphs. Such a model can reduce the need for separate architectures, support transfer across graphs with different attribute schemas, and improve scalability in real-world settings where modality availability varies across domains, as motivated in Figure 1. However, applying VLMs to heterogeneous attributed graphs remains challenging. First, graph neighborhoods introduce variable-sized and structure-dependent context that does not naturally fit the sequential input format of VLMs, making naïve neighborhood injection inefficient and prone to noise (Chen et al., 2023). Second, graph-derived information must be integrated in a way that is compatible with the VLM’s native textual and visual processing pipeline. External encoders or mismatched representation spaces may introduce additional gaps, making unified multimodal graph learning more difficult (Shu et al., 2025). We propose OMG-VLM (One Model, Many Graphs with Vision-Language Models), a unified framework for attributed graph learning across heterogeneous modality schemas. OMG-VLM uses a pretrained VLM as a shared backbone and introduces structure-aware graph adapters to inject neighborhood information while remaining native to the VLM’s embedding space. It further employs modality-specific but backbone-native mechanisms to incorporate textual and visual neighborhoods, to-
gether with a co-optimization strategy that enables effective adaptation to graph-structured inputs. Our contributions are summarized as follows: • We study unified attributed graph learning under heterogeneous modality schemas, where a single model operates across textattributed, image-attributed, and multimodalattributed graphs. This setting reflects realistic deployment scenarios in which different graphs expose different attribute modalities, while existing methods typically rely on modality-specific designs, only handling graphs with a specific modality schema. To the best of our knowledge, this is the first work to systematically study this problem from a VLM perspective. • We propose OMG-VLM, a unified VLM-based framework for attributed graph learning across heterogeneous modality schemas. OMG-VLM introduces structure-aware graph adapters that incorporate neighborhood information compatible with the VLM’s native representation space, enabling end-to-end learning across graphs with different modality configurations. • Through extensive experiments on node classification and link prediction across diverse domains and modality schemas, we show that OMG-VLM consistently outperforms state-of-the-art GNNand LLM-based baselines. It also demonstrates strong generalization capabilities across various domains and modality schemas, supporting the effectiveness of VLMs as unified backbones for attributed graph learning.
2
Related Works
Text-Attributed Graph (TAG) Learning. GNNs, including GCN (Kipf and Welling, 2017), GraphSAGE (Hamilton et al., 2018), and RevGAT (Li et al., 2022), are widely used for node- and linklevel prediction via neighborhood propagation. For TAGs, prior work commonly encodes node texts with pretrained language models and then applies graph propagation (He et al., 2024b,a; Chen et al., 2024b; Wang et al., 2025; Yan et al., 2023). Recent LLM-based methods, such as LLaGA (Chen et al., 2024a), GraphGPT (Tang et al., 2024), and GraphPrompter (Lv et al., 2025), instead serialize neighborhoods and graph structure into textual prompts for generative graph reasoning (Sun et al., 2025; Zhang et al., 2024). However, these meth2
E-Commerce Network
“Given the target product information …”
Citation Network
Social Network
Domains
Looks amazing!
FFN CenterConditioned Visual Fusion
IAG
Cross-Attention Multi-Head Cross-Attention Cross-Attention
Text Bridge
Target-Aware Queries
Per-Neighbor Compression
Target-Conditioned Neighbor Retrieval Multi-Head Cross-Attention Cross-Attention
RoPE Self-Attention
Cross-Attention Multi-Head Cross-Attention Cross-Attention
Neighbor Image Tokens
Target-Aware Textual Aggregation
Concatenate
Visual Context Bank
Self-Attention
Anchor Image Tokens
TAG
“Which category does the target …”
Structure-Aware Query Generation
Concatenate
So cool! Thanks
MAG
Aggregated Text Tokens
Enhanced Image Tokens
Graph-Aware Visual Representation Learning L layers
Where is this?
Modalities
“This product belongs to the category of …”
VLM Decoder
Concatenate
Concatenate
Learnable Query
Anchor Text
Learnable Query
Native Feature Extraction
Shared Embedding Initialization
Visual Encoder
VLM Tokenizer
Many Graphs
One Model
Neighbor Text Token Bank
Neighbor Text Tokens
Frozen Tuning
Figure 2: Overview of the OMG-VLM framework. OMG-VLM uses a pretrained VLM as a shared backbone for reasoning over attributed graphs with heterogeneous modality schemas. Textual and visual neighborhoods are incorporated via Target-Aware Textual Aggregation and Graph-Aware Visual Representation Learning, enabling a single model to handle TAGs, IAGs, and MAGs within a shared embedding space.
ods remain text-centric and are not designed for non-textual node attributes.
earization is constrained by input length and primarily targets fully multimodal attributed graphs.
Multimodal Attributed Graph (MAG) Learning. MAG learning extends TAGs by incorporating both textual and visual node attributes, requiring models to jointly capture multimodal features and graph structure (Yan et al., 2025). Early methods are largely GNN-based and often target domainspecific applications such as recommendation (Wei et al., 2019; Wang et al., 2023). Recent work leverages multimodal encoders such as CLIP (Radford et al., 2021) for multimodal attributed graph reasoning (Zhu et al., 2025; Liu et al., 2026; Ning et al., 2025; Yan et al., 2025; Sun et al., 2026). UniGraph2 (He et al., 2025) aligns textual and visual attributes into a shared embedding space, while MLaGA (Fan et al., 2026) and Graph4MM (Ning et al., 2025) integrate multimodal features through external encoders and fusion modules. These methods are effective for MAGs but mainly assume fully multimodal-attributed graphs, leaving heterogeneous modality schemas underexplored.
3
Problem Statement
Attributed Graph Learning with Heterogeneous Modality Schemas. We are given a collection of attributed graphs {Gm = (Vm , Em )}K m=1 , where different graphs may follow different modality schemas. We consider three common cases: (1) text-attributed graphs (TAGs), where nodes are associated with textual attributes; (2) imageattributed graphs (IAGs), where nodes are associated with visual attributes; and (3) multimodalattributed graphs (MAGs), where nodes are associated with both textual and visual attributes. For a node v ∈ Vm , let Tv and Iv denote its textual and visual attributes. Either modality may be absent depending on the graph schema. We denote the available modality set of v as Av ⊆ {Tv , Iv }. Similarly, let N (v) denote the neighborhood of v. We define NT (v) and NI (v) as the available respective textual and visual attributes associated with neighbors in N (v), depending on availability. Learning Objective. We study generative reasoning on attributed graphs with heterogeneous modality schemas. Given a target node v, its available attributes Av , its modality-specific neighborhood context NT (v) and NI (v), and a task instruction P, the goal is to learn a unified model that generates the target sequence Y under a shared conditional generation objective:
Vision-Language Models (VLMs). VLMs align visual and textual representations through multimodal pretraining and instruction tuning, enabling strong visual understanding and language reasoning (Li et al., 2023; Liu et al., 2023; Bai et al., 2023). However, they are not designed to process graph-structured context natively. GraphVLM (Liu et al., 2026) applies VLMs to graphs by serializing nodes, edges, or structural descriptions into prompts. While promising, this prompt-based lin-
L(Θ) = log P Y | Av , NT (v), NI (v), P; Θ ,
3
(1)
where Θ denotes the model parameters. In this paper, we focus on two representative graph reasoning tasks: node classification and link prediction. For node classification, the query is a target node v and Y denotes its label. For link prediction, the query is a node pair (vi , vj ) and Y denotes whether an edge exists between them.
4
Conditioned on Xv , the VLM generates the target output sequence autoregressively: P (Y | G, P) =
By converting graph-derived textual and visual context into native VLM token representations, OMG-VLM avoids explicit cross-modal realignment while preserving the pretrained multimodal interface of the backbone. The following sections describe how ΦT and ΦI aggregate textual and visual neighborhoods, respectively, before jointly optimizing them with the generative objective.
We introduce OMG-VLM, a generative framework for reasoning over attributed graphs with heterogeneous modality schemas. We first formulate graph reasoning as generative modeling (Sec. 4.1), then present two graph adapters for incorporating neighborhood context: Target-Aware Textual Aggregation (Sec. 4.2) and Graph-Aware Visual Representation Learning (Sec. 4.3). Finally, we describe a unified training objective for optimizing these modules within the shared VLM backbone (Sec. 4.4).
4.2
OMG-VLM builds graph reasoning on top of a pretrained VLM with parameters Θ. The key idea is to represent heterogeneous graph context as a sequence of VLM-compatible tokens, so that attributed graphs with different modality schemas can be handled by the same generative backbone. Rather than introducing separate modality encoders and then aligning them with the VLM, OMGVLM operates directly in the VLM’s native multimodal embedding space. Textual and visual node attributes, together with their modality-specific neighborhoods, are first transformed into compact token representations by graph adapters and then concatenated with the task instruction. Given a target node v, OMG-VLM constructs the input sequence Xv as
graph context
Target-Aware Textual Aggregation
Textual neighbors provide useful graph context, but their relevance to the target node varies. Existing methods typically encode neighborhood text with external encoders or concatenate it into the input (Liu et al., 2026; Chen et al., 2024a; Fan et al., 2026). For VLM backbones, however, external encoders require additional cross-modal alignment, while concatenation yields noisy and inefficient inputs. OMG-VLM instead incorporates textual graph context through an adapter built on top of the VLM’s own textual representations, reducing the need for explicit cross-space alignment. We introduce Target-Aware Textual Aggregation ΦT , which performs target-conditioned token-level retrieval over textual neighbors. Shared Embedding Initialization. The target text and textual neighbor attributes are embedded using the token embedding matrix of the pretrained VLM. Given the target text Tv , its token embeddings are:
Overview
Xv = ΦT (Tv , NT (v)) ∥ ΦI (Iv , NI (v)) ∥ EP , |{z} | {z }
PVLM yt | y<t , Xv ; Θ . (3)
t=1
Methodology
4.1
|Y | Y
T ETv = Embed Tokenizer(Tv ) ∈ RLv ×d , (4) where LTv is the target text length and d is the hidden dimension of the backbone VLM. For textual neighbors NT (v), we concatenate all token embeddings into a textual neighbor bank:
(2)
instruction
where EP = Embed(P). Here, ΦT and ΦI denote the textual and visual graph adapters, which map modality-specific node attributes and neighborhood context into compact token sequences compatible with the VLM backbone. If a modality is unavailable for a given graph, the corresponding adapter output is omitted. This design allows the same model to process TAGs, IAGs, and MAGs under a unified input format, as illustrated in Figure 2.
T Embed Tokenizer(Tu ) ∈ RLN ×d , (5)
BTv = u∈NT (v)
where LTN is the number of textual neighbor tokens. Target-Aware Query Generation. Since the VLM tokenizer lacks an explicit pooling token such as [CLS] (Bai et al., 2023; Devlin et al., 2019), we introduce MT learnable query tokens QT ∈ RMT ×d as adaptive semantic probes. We concatenate them 4
Per-Neighbor Compression. Each visual neighbor is compressed into MI tokens using learnable visual queries QI ∈ RMI ×d :
with the target text embeddings and apply a RoPEaware self-attention block (Su et al., 2023): HTv = AttnRoPE ETv ∥QT . (6)
CIu = MHCA QI , EIu , EIu ,
The final MT tokens are used as target-conditioned textual queries, which summarize multiple semantic aspects of the center node: b T = HT [−MT :, :] ∈ RMT ×d . Q v v
(7)
Center-Conditioned Visual Aggregation. OMGVLM integrates visual neighborhood information by refining the target visual tokens in place. At layer ℓ, the target visual tokens attend to the compressed visual context bank:
(9)
HI,(ℓ+1) = FFN MHCA HI,(ℓ) , BIv , BIv + HI,(ℓ) v v v I,(0)
with Hv = EIv . After L aggregation layers, the resulting graphaware visual representation is:
Graph-Aware Visual Representation Learning
ΦI (Iv , NI (v)) = XIv = HI,(L) . v
Visual neighborhoods pose a distinct challenge for graph reasoning. Each image is represented as patch tokens, so directly incorporating K visual neighbors yields KLI tokens, where LI is the patch sequence length per image. Aggregating these high-dimensional, spatially redundant tokens can dilute salient cues and incur substantial computational overhead during inference. To address this challenge, we introduce a GraphAware Visual Adapter ΦI with two stages: (i) per-neighbor visual compression and (ii) centerconditioned visual aggregation. Native Feature Extraction. We first extract visual features using the VLM’s frozen visual encoder. In modern VLMs, a Vision Transformer (ViT) serves as the inherent visual encoder that maps raw images into visual token sequences (Dosovitskiy et al., 2021; Bai et al., 2023). For the target node v and each visual neighbor u ∈ NI (v): I
u ∈ {v} ∪ NI (v),
(12)
u∈NI (v)
where ETsep is a delimiter embedding acting as a text bridge, e.g., ”Neighbor Text Information:”.
EIu = ViT(Iu ) ∈ RL ×d ,
CIu ∈ R|NI (v)|MI ×d .
BIv =
This relevance-based retrieval preserves finegrained neighborhood information, avoiding the uniform averaging in static pooling strategies. The final textual adapter output is:
4.3
(11)
Here, CIu ∈ RMI ×d and MI ≪ LI , reducing the visual context from |NI (v)|LI to |NI (v)|MI tokens, while enabling adaptive selection of informative visual regions. The compressed visual neighbor tokens are concatenated into a visual context bank:
Target-Conditioned Neighbor Retrieval. Given b T , we retrieve informative textual neighborhood Q v content through multi-head cross-attention: b T , BT , BT ∈ RMT ×d . (8) ZTv = MHCA Q v v v
ΦT (Tv , NT (v)) = XTv = ETv ∥ETsep ∥ZTv ,
u ∈ NI (v).
(13)
The refined tokens preserve the original visual token length, allowing direct substitution into the VLM input sequence without increasing downstream sequence length. 4.4
Co-Optimization on Attributed Graphs with Heterogeneous Modality Schemas
A key advantage of leveraging a pretrained VLM is that textual and visual inputs are mapped into a shared embedding space by the backbone tokenizer and visual encoder. This unified representation reduces the need for explicit cross-modal alignment, which is often required in LLM-based graph methods for TAGs or MAGs. OMG-VLM therefore supports joint optimization while preserving pretrained multimodal representations. Training Strategy. We jointly optimize the LoRA-adapted VLM parameters and the two graph adapters, while keeping the remaining backbone parameters frozen. This adapts the VLM to graphstructured inputs while preserving its native visual encoder and tokenizer. Training uses a heterogeneous collection of attributed graphs, including TAGs, IAGs, and MAGs,
(10)
where LI is the number of visual tokens and d is the hidden dimension of the backbone VLM. The target visual tokens EIv serve as the reference for visual neighborhood aggregation. 5
Baselines. We compare OMG-VLM with five categories of baselines: MLP (Rosenblatt, 1958); GNN-based methods, including GCN (Kipf and Welling, 2017), GraphSAGE (Hamilton et al., 2018), RevGAT (Li et al., 2022), and UniGraph2 (He et al., 2025); text-only LLM-based methods, including LLaGA (Chen et al., 2024a) and GraphPrompter (Lv et al., 2025); multimodal LLM-based methods, including MLaGA (Fan et al., 2026) and Graph4MM (Ning et al., 2025); and VLM-based prompting methods, such as GraphVLM (Liu et al., 2026). We follow standard training and evaluation protocols for all baselines (Chen et al., 2024a; Lv et al., 2025; Fan et al., 2026; Ning et al., 2025; Liu et al., 2026). Evaluation Metric. Following prior work (Chen et al., 2024a; Fan et al., 2026; Liu et al., 2026), we use accuracy as the primary metric for all graph reasoning tasks. We also report additional MacroF1 results in Appendix C.3.
across multiple domains and graph reasoning tasks. This exposes OMG-VLM to diverse modality schemas, encouraging structure-aware reasoning that generalizes beyond a single modality schema. Let D be training set, where each example is: z = (v, Tv , Iv , NT (v), NI (v), P, Y ).
(14)
The input sequence Xv is constructed by Eq. 2 according to the modalities available in each example. The model maximizes: |Y | X L(Θ) = Ez∼D log PVLM yt | y<t , Xv ; Θ . (15) t=1
All trainable components are updated jointly via backpropagation, allowing neighborhood retrieval, aggregation, and generation to co-adapt during optimization. This enables OMG-VLM to operate as a single model across attributed graphs with heterogeneous modality schemas.
5
5.2
Experiments
We evaluate OMG-VLM under in-domain and transfer settings to assess (i) in-domain effectiveness and (ii) cross-graph generalization across heterogeneous modality schemas. 1 OMG-VLM In-Domain Performance (RQ1). ⃝ achieves the best performance on all in-domain benchmarks across TAG, IAG, and MAG settings. As shown in Table 1, OMG-VLM ranks first on all four training graphs. On TAG arXivNC , it achieves the highest accuracy, 67.85 (+3.26 over the best baseline), indicating effective use of semantic neighborhood information. The same trend holds on IAG RedditS, showing effectiveness over visual neighborhoods. Gains are especially pronounced on MAGs: OMG-VLM reaches 92.50 on ArtsNC , outperforming the strongest LLM-based baseline LLaGA (88.74) by +3.76 points, and achieves 95.30 on MoviesLP , exceeding UniGraph2 (91.51) by +3.79 points. 2 OMG-VLM Transfer Performance (RQ2). ⃝ generalizes effectively to unseen graphs with strong performance across modality schemas. Without dataset-specific fine-tuning, OMG-VLM achieves the best results on most transfer benchmarks. It improves over the strongest baseline by +6.42 points on TAG PubMedNC and up to +20.15 points on MAG RedditMLP , with consistent gains on all other transfer datasets. These datasets span TAG, IAG, and MAG settings, yet OMG-VLM remains competitive across modalities. In contrast,
We conduct experiments to address the following four research questions (RQs): RQ1: How does OMG-VLM compare with graph reasoning baselines on in-domain tasks? RQ2: How well does OMG-VLM generalize to unseen graphs under transfer settings? RQ3: How much do the textual adapter, visual adapter, and co-optimization strategy contribute? RQ4: How sensitive is OMG-VLM to key hyperparameters for graph context modeling? 5.1
Overall Comparison (RQ1 and RQ2)
Experiment Setup
Datasets. We evaluate OMG-VLM on four primary datasets across e-commerce, citation, and social networks. For node classification, we use MAG Amazon-Arts (Majumder et al., 2020; He and McAuley, 2016) and TAG ogbn-arXiv (Hu et al., 2021); for link prediction, we use MAG AmazonMovies and IAG RedditS (Hamilton et al., 2018). To assess cross-graph generalization, we further evaluate all models on held-out graphs unseen during training: TAG PubMed (He et al., 2024a), TAG Cora (Orbifold, 2019), MAG RedditM (Hamilton et al., 2018), and MAGs Amazon-VideoGames and Amazon-CDs (Majumder et al., 2020; He and McAuley, 2016). We mark node classification and link prediction datasets with NC and LP , respectively. Dataset statistics are reported in Appendix A, and prompt templates are shown in Appendix F. 6
Category
Model
In-Domain
Transfer
Overall
ArtsNC arXivNC MoviesLP RedditSLP Avg. PubMedNC CDsNC RedditMLP VideoGamesLP CoraLP Avg. † ⋄ † †
Avg.
73.75
64.59
59.42
56.25
63.50
37.96
1.42
55.20
50.12
50.90
39.12
49.96
GNNs & GCN GNN-based RevGAT GraphSAGE UniGraph2
63.95 71.89 74.77 78.87
58.71 62.15 62.78 41.26
52.38 49.54 52.22 91.51
49.77 49.77 49.87 78.85
56.20 58.34 59.91 72.62
20.79 40.26 42.77 38.40
10.79 3.54 3.67 10.83
51.05 50.25 50.80 62.35
50.61 50.08 52.99 54.41
50.00 50.00 50.30 51.10
36.65 38.83 40.11 43.42
45.34 47.50 48.91 56.40
LLM-based LLaGA GraphPrompter
88.74 83.21
63.70 56.28
53.27 63.82
– –
68.57 67.77
1.65 56.92
4.14 30.71
51.60 44.65
57.98 70.30
0.70 50.00
23.21 50.52
40.22 56.99
LLM w/ Ext. Enc.
MLaGA Graph4MM
89.63 84.05
– 52.75
76.20 88.45
90.10 –
85.31 75.08
– 25.95
31.10 10.25
64.30 72.30
58.46 59.40
– 14.90
51.29 36.56
68.30 51.01
VLM-based GraphVLM OMG-VLM
82.55 92.50
63.99 67.85
50.74 95.30
55.70 99.10
63.25 88.69
72.13 78.55
29.47 32.56
57.35 92.45
48.73 77.99
50.20 51.80
51.58 66.67
56.76 76.46
MLP
MLP
Table 1: Main results under in-domain and transfer settings. In-domain results are evaluated on the four training graphs, while transfer results are evaluated on unseen graphs. Subscripts † and ⋄ denote TAG and IAG datasets, respectively; all remaining datasets are MAGs. Bold and underline indicate the best and second-best results. – indicates that a model cannot process the dataset due to architectural constraints. Additional baseline implementation details and differences from originally reported results are discussed in Appendix C.
5.3
Accuracy (%)
no baseline consistently ranks second to OMGVLM in more than three settings. Overall, OMG-VLM achieves strong in-domain performance and effective transfer to unseen graphs, demonstrating the effectiveness of unified multimodal graph representation learning under heterogeneous modality schemas.
100
6
80
4
60
2
40
0
20
Ablation Studies (RQ3)
LP LP LP NC iv ies ditS ditM arX Mov Red Red
NC CDs
Best Mean
Ours
Best Attn.
−2
Figure 3: Effectiveness of graph adapters. Bars compare OMG-VLM with the strongest mean-pooling and attention-pooling adapter replacements across textual, visual, or both adapters. The line shows the gap between OMG-VLM and the best pooling variant.
In this section, we ablate the proposed aggregation modules and training paradigm. Effectiveness of Graph Adapters for Neighbor Aggregation. We compare the full model with variants that replace (i) textual adapter, (ii) visual adapter, or (iii) both adapters with mean pooling or center-query-conditioned attention pooling, while keeping other components unchanged. Results are detailed in Appendix Table 6. 3 Modality-specific graph adapters are crucial ⃝ for neighborhood aggregation. Replacing both adapters consistently degrades performance. On MoviesLP , mean and attention pooling reduce accuracy by 4.55 and 21.49 points, respectively; on RedditSLP , both variants incur drops over 22 points. Replacing only the visual adapter also severely hurts IAGs, reducing RedditSLP by 22.85 and 24.15 points under mean and attention pooling. Replacing the textual adapter lowers performance by up to 19.30 points on PubMedNC and 6.22 points on CDsNC . These results show that simple pooling, even with center-query attention, cannot replace the proposed modality-specific adapters. Per-Neighbor Compression in Image Aggregation. To evaluate per-neighbor compression, we
compare OMG-VLM with and without compression under varying neighborhood sizes, measuring both accuracy and computational cost. 4 Per-neighbor compression improves effi⃝ ciency while preserving performance. As shown in Appendix Table 7, compressed and uncompressed variants achieve comparable accuracy across datasets and modality settings, with differences typically within 1–2 points (e.g., 91.17– 91.88 vs. 91.20–92.16 on ArtsNC ; Figure 4). Meanwhile, Figure 5 shows that compression reduces FLOPs by approximately 75% across neighborhood sizes, from 10.74B–107.37B to 2.68B– 26.84B. These results show that per-neighbor compression enables scalable visual neighborhood aggregation with little performance loss. End-to-End Co-Optimization vs. Staged Training. We compare end-to-end co-optimization, which jointly optimizes all trainable components, with a two-stage strategy that separates adapter pretraining from structure-aware tuning. Results 7
91 MoviesLP
10
40
100 10
70
# Visual Neighbors
40
70
70
# Visual Neighbors
85
75
50
70 0 10
40
70
100
# Visual Neighbors Acc. w/o comp. FLOPs w/o comp.
Acc. w/ comp. FLOPs w/ comp.
Figure 5: Effect of per-neighbor compression as the number of visual neighbors increases. Bars show average accuracy, while lines show computational cost.
are shown in Appendix Table 8. We exclude full-parameter fine-tuning due to its substantially higher memory cost, which is infeasible under our hardware setup and less practical than parameterefficient adaptation (Bai et al., 2023). 5 End-to-end co-optimization matches or out⃝ performs two-stage training on most datasets while simplifying optimization. Co-optimization performs better on 7 of 9 benchmarks, with gains of +1.33 on MoviesLP , +2.16 on RedditSLP , and +6.75 on RedditMLP (Figure 6). It also reduces training time by 17.54% (81.05 → 66.83 hours). These results suggest that a unified objective improves co-adaptation between neighborhood modeling and reasoning while avoiding staged training schedules. 5.4
2
NC Arts
LP LP LP LP ies mes ditS ditM Mov Red oGa Red Vide Two-stage Co-Optimization
kens per neighbor MI , which controls the visual information retained before aggregation. 7 Moderate compression preserves perfor⃝ mance, while larger token budgets provide limited gains. As shown in Table 9, accuracy remains stable across MI . For example, ArtsNC stays within 92.36–92.85, MoviesLP varies by less than 0.5 points, suggesting that compact visual summaries preserve key cues for graph reasoning. Aggressive compression (MI =8) can remove useful detail, while increasing to MI =64 does not consistently improve results. We therefore set MI =32 by default to keep visual context compact while maintaining strong performance. 8 OMGNumber of Aggregated Neighbors. ⃝ VLM is stable across neighborhood sizes. Table 9 shows that accuracy remains relatively stable across K. In many cases, K=10 already achieves the best or near-best performance, e.g., ArtsNC obtains 92.50 at K=10 vs. 91.17 at K=40, indicating that a small neighborhood captures most useful structural signals. Larger neighborhoods can occasionally improve transfer performance, e.g., CoraLP improves from 51.80 to 61.90 at K=70, but such gains are not consistent. We set K=10 by default for efficient neighborhood aggregation.
100 FLOPs (B)
Avg. Accuracy (%)
Figure 4: Effect of image token compression under increasing visual neighborhood sizes. 80
4
Figure 6: Training-strategy ablation. Bars show accuracy under two-stage training and co-optimization; the line shows the relative gain of co-optimization.
100
w Compression (MI =32)
w/o Compression
80
6
0
RedditSLP ∆max = 1.06
∆max = 1.45
90
Gain (%)
95.5 95 94.5 94
8
∆max = 1.34
∆max = 0.99
92
100
VideoGamesLP Accuracy (%)
Accuracy (%)
93
Accuracy (%)
ArtsNC
6
Sensitivity Analysis (RQ4)
Conclusion
We study attributed graph reasoning under heterogeneous modality schemas, covering text-, image-, and multi-attributed graphs. We propose OMGVLM, a unified framework that uses a pretrained VLM as a shared backbone and incorporates neighborhood context through structure-aware graph adapters compatible with the backbone embedding space. Experiments on node classification and link prediction across domains and modality schemas show that OMG-VLM consistently outperforms representative baselines and generalizes well to unseen graphs. Ablations further validate the proposed aggregation modules and training strategy.
6 Injected Text Neighborhood Token Length. ⃝ OMG-VLM is stable across different aggregated text token lengths. Table 9 of Appendix shows that accuracy varies slightly across MT , typically within 2 points. Increasing MT from 1 to 8 improves performance on several datasets, such as MoviesLP (94.70 → 95.30) and RedditMLP (79.10 → 92.45), while larger values do not consistently help. We set MT =8 by default, balancing performance with compact textual context. Per-Neighbor Image Token Length. We study sensitivity to the number of compressed visual to8
Overall, our results show that VLMs provide a strong foundation for unified attributed graph learning under heterogeneous modality schemas, and we hope this work encourages further exploration of VLM-based graph reasoning.
of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv preprint. ArXiv:2010.11929 [cs].
Limitations OMG-VLM studies unified reasoning over attributed graphs with heterogeneous textual or visual information. Although this provides a general framework for handling different modality schemas, the current work focuses on static graph reasoning tasks under a standard supervised learning setting. It does not yet consider more interactive paradigms, such as agent-based graph reasoning, where models may dynamically decide how to access, query, or use graph information. Exploring such interactive extensions remains an important direction for future work.
Dongzhe Fan, Yi Fang, Jiajin Liu, Djellel Difallah, and Qiaoyu Tan. 2026. Mlaga: Multimodal large language and graph assistant. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. William L. Hamilton, Rex Ying, and Jure Leskovec. 2018. Inductive Representation Learning on Large Graphs. arXiv preprint. ArXiv:1706.02216 [cs]. Ruining He and Julian McAuley. 2016. Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering. In Proceedings of the 25th International Conference on World Wide Web, pages 507–517, Montréal Québec Canada. International World Wide Web Conferences Steering Committee.
References Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv preprint. ArXiv:2308.12966 [cs].
Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi. 2024a. Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning. arXiv preprint. ArXiv:2305.19523 [cs].
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-VL Technical Report. arXiv preprint. Version Number: 2.
Yufei He, Yuan Sui, Xiaoxin He, and Bryan Hooi. 2024b. UniGraph: Learning a Unified Cross-Domain Foundation Model for Text-Attributed Graphs. Yufei He, Yuan Sui, Xiaoxin He, Yue Liu, Yifei Sun, and Bryan Hooi. 2025. UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs. arXiv preprint. ArXiv:2502.00806 [cs].
Jinsong Chen, Chang Liu, Kaiyuan Gao, Gaichao Li, and Kun He. 2023. Tokenized Graph Transformer with Neighborhood Augmentation for Node Classification in Large Graphs. arXiv preprint. ArXiv:2305.12677 [cs].
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2021. Open Graph Benchmark: Datasets for Machine Learning on Graphs. arXiv preprint. ArXiv:2005.00687 [cs].
Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. 2024a. LLaGA: Large Language and Graph Assistant. arXiv preprint. ArXiv:2402.08170 [cs].
Thomas N. Kipf and Max Welling. 2017. SemiSupervised Classification with Graph Convolutional Networks. arXiv preprint. ArXiv:1609.02907 [cs].
Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, and 1 others. 2024b. Exploring the potential of large language models (llms) in learning on graphs. ACM SIGKDD Explorations Newsletter, 25(2):42–61.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. 2022. Training Graph Neural Networks with 1000 Layers. arXiv preprint. ArXiv:2106.07476 [cs]. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pretraining with Frozen Image Encoders and Large Language Models. arXiv preprint. ArXiv:2301.12597 [cs].
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference
9
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llavanext: Improved reasoning, ocr, and world knowledge.
Yuanfu Sun, Zhengnan Ma, Yi Fang, Jing Ma, and Qiaoyu Tan. 2025. GraphICL: Unlocking Graph Learning Potential in LLMs through Structured Prompt Design. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2440–2459, Albuquerque, New Mexico. Association for Computational Linguistics.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. arXiv preprint. ArXiv:2304.08485 [cs].
Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. 2024. GraphGPT: Graph Instruction Tuning for Large Language Models. arXiv preprint. ArXiv:2310.13023 [cs].
Jiajin Liu, Dongzhe Fan, Chuanhao Ji, Daochen Zha, and Qiaoyu Tan. 2026. Graphvlm: Benchmarking vision language models for multimodal graph learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9491–9500.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. arXiv preprint. ArXiv:1710.10903 [stat].
Rui Lv, Zaixi Zhang, Kai Zhang, Qi Liu, Weibo Gao, Jiawei Liu, Jiaxia Yan, Linan Yue, and Fangzhou Yao. 2025. GraphPrompter: Multi-stage Adaptive Prompt Optimization for Graph In-Context Learning. arXiv preprint. ArXiv:2505.02027 [cs].
Qifan Wang, Yinwei Wei, Jianhua Yin, Jianlong Wu, Xuemeng Song, and Liqiang Nie. 2023. DualGNN: Dual Graph Neural Network for Multimedia Recommendation. Trans. Multi., 25:1074–1084.
Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. 2020. Interview: Large-scale Modeling of Media Dialog with Discourse Patterns and Knowledge Grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8129–8141, Online. Association for Computational Linguistics.
Shijie Wang, Jiani Huang, Zhikai Chen, Yu Song, Wenzhuo Tang, Haitao Mao, Wenqi Fan, Hui Liu, Xiaorui Liu, Dawei Yin, and 1 others. 2025. Graph machine learning in the era of large language models (llms). ACM Transactions on Intelligent Systems and Technology, 16(5):1–40.
Xuying Ning, Dongqi Fu, Tianxin Wei, Wujiang Xu, and Jingrui He. 2025. Graph4MM: Weaving Multimodal Learning with Structural Information. arXiv preprint. ArXiv:2510.16990 [cs].
Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019. MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video. In Proceedings of the 27th ACM International Conference on Multimedia, pages 1437–1445, Nice France. ACM.
Orbifold. 2019. The Cora dataset. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv preprint. ArXiv:2103.00020 [cs].
Hao Yan, Chaozhuo Li, Ruosong Long, Chao Yan, Jianan Zhao, Wenwen Zhuang, Jun Yin, Peiyan Zhang, Weihao Han, Hao Sun, Weiwei Deng, Qi Zhang, Lichao Sun, Xing Xie, and Senzhang Wang. 2023. A Comprehensive Study on Textattributed Graphs: Benchmarking and Rethinking. Advances in Neural Information Processing Systems, 36:17238–17264.
F. Rosenblatt. 1958. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6):386–408.
Hao Yan, Chaozhuo Li, Jun Yin, Zhigang Yu, Weihao Han, Mingzheng Li, Zhengxin Zeng, Hao Sun, and Senzhang Wang. 2025. When graph meets multimodal: benchmarking and meditating on multimodal attributed graph learning. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 5842–5853.
Dong Shu, Haiyan Zhao, Jingyu Hu, Weiru Liu, Ali Payani, Lu Cheng, and Mengnan Du. 2025. Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability. arXiv preprint. ArXiv:2501.01346 [cs].
Mengmei Zhang, Mingwei Sun, Peng Wang, Shen Fan, Yanhu Mo, Xiaoxiao Xu, Hong Liu, Cheng Yang, and Chuan Shi. 2024. GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Tasks.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv preprint. ArXiv:2104.09864 [cs].
Jing Zhu, Yuhang Zhou, Shengyi Qian, Zhongmou He, Tong Zhao, Neil Shah, and Danai Koutra. 2025. Mosaic of modalities: A comprehensive benchmark for multimodal graph learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14215–14224.
Yuanfu Sun, Kang Li, Pengkang Guo, Jiajin Liu, and Qiaoyu Tan. 2026. Mario: Multimodal graph reasoning with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19219–19228.
10
all available 1-hop neighbors of the center node. If the neighborhood budget is not filled, we then include 2-hop neighbors, and continue to higher-hop neighbors until reaching the maximum budget. For baselines, we follow the neighborhood construction or sampling strategy specified in the corresponding paper (Fan et al., 2026; Chen et al., 2024a; Ning et al., 2025; He et al., 2025; Lv et al., 2025).
Aaron Zolnai-Lucas, Jack Boylan, Chris Hokamp, and Parsa Ghaffari. 2024. Stage: Simplified textattributed graph embeddings using pre-trained llms. arXiv preprint arXiv:2407.12860.
A
Dataset Formulation
Table 2 summarizes the datasets used for training and evaluation, including their sizes, modality schemas, and task types. The benchmark spans TAGs, IAGs, and MAGs, covering both node classification and link prediction. We follow the official train/test splits for all datasets. For link prediction datasets, we sample positive edges from the corresponding split and pair them with an equal number of negative edges for training and evaluation. All datasets, pretrained models, and baseline implementations used in this work are publicly available research artifacts, and the associated textual data are in English. We use these artifacts only for research evaluation, follow their standard researchuse settings, and adopt protocols consistent with prior work on graph reasoning and multimodal attributed graph learning (Chen et al., 2024a; Fan et al., 2026; Liu et al., 2026). To enable balanced mixed training across tasks, domains, and modality schemas, we downsample selected training graphs to a comparable scale. In particular, ogbn-arXiv is downsampled from its full training split to avoid dominating the mixed training process due to its substantially larger size, from 101,606 to 15,545 nodes. Therefore, reproduced baseline results under our unified protocol may differ from originally reported results, which are often obtained under dataset-specific settings. These differences are discussed in Appendix C.
C
Baseline Details
All results are reported from single runs conducted under the same hardware environment, using a single NVIDIA H100 80GB GPU. C.1
Unavailable Baseline Results
Certain baseline results are unavailable because the corresponding models are designed for specific modality schemas and cannot be directly applied to all dataset types considered in our evaluation. Text-only LLM-based baselines, including LLaGA (Chen et al., 2024a) and GraphPrompter (Lv et al., 2025), are designed for TAGs. Their input construction relies on textual node attributes and serialized graph context, without a mechanism for processing visual node attributes. Therefore, they cannot be applied to RedditS, which is an IAG containing only visual inputs. Multimodal LLM-based baselines also have architecture-specific modality requirements. Graph4MM (Ning et al., 2025) is designed for MAGs and uses an MM-QFormer module to extract visual information through text-derived queries. This design requires textual attributes and, therefore, cannot be directly applied to IAGs without text. Conversely, MLaGA (Fan et al., 2026) reasons over graph structure using a visual branch and a multimodal fusion module, which assumes the availability of visual inputs. As a result, it is not directly applicable to TAGs, where only textual attributes are available. These unavailable results highlight the motivation for our heterogeneous modality setting. Unlike prior methods that are specialized for TAGs, IAGs, or MAGs, OMG-VLM is designed as a unified framework that can operate across graphs with different modality schemas.
B Training Setup and Neighbor Sampling OMG-VLM is trained for 3 epochs, where the visual encoder is frozen, while the VLM transformer backbone is adapted using LoRA. We optimize OMG-VLM with AdamW, using a learning rate of 1 × 10−5 , weight decay of 0.1, β2 = 0.95, a warmup ratio of 0.01, and a cosine learningrate scheduler. For LoRA adaptation, we use rank r = 64, scaling factor α = 16, and dropout 0.05. LoRA is applied to the attention projection modules and feed-forward layers of the VLM backbone. For neighborhood construction of OMG-VLM, we select neighbors from predefined graph edges during both training and evaluation. Following prior graph-based multimodal learning protocols (Ning et al., 2025), we first randomly include
C.2
Differences from Reported Baselines
Some reproduced baseline results differ from those reported in the original papers, particularly MLaGA on MoviesLP and LLaGA on ogbnarXivNC . These gaps mainly stem from differences 11
Setting
Dataset
#Train
#Test
Domain
Modality
Task
In-Domain
Amazon-Arts ogbn-arXiv Amazon-Movies RedditS
16,917 15,545 10,000 6,000
5,639 47,416 3,169 1,991
E-commerce Citation Network E-commerce Social Network
Multimodal Attributed Graph Text Attributed Graph Multimodal Attributed Graph Image Attributed Graph*
Node Classification Node Classification Link Prediction Link Prediction
Transfer
PubMed RedditM Amazon-VideoGames Amazon-CDs Cora
– – – – –
3,944 2,000 2,608 7,255 1,000
Citation Network Social Network E-commerce E-commerce Citation Network
Text Attributed Graph Multimodal Attributed Graph Multimodal Attributed Graph Multimodal Attributed Graph Text Attributed Graph
Node Classification Link Prediction Link Prediction Node Classification Link Prediction
Table 2: Datasets details for in-domain and transfer evaluation. *Textual information of RedditS dataset is omitted to simulate Image Attributed Graph. Dataset
LLaGA
GraphPrompter
UniGraph2
Graph4MM
GraphVLM
OMG-VLM
ArtsNC ogbn-arXivNC MoviesLP RedditSLP PubMedNC RedditMLP VideoGamesLP CDsNC CoraLP
88.51 74.71 52.13 – 1.52 44.50 46.36 4.88 1.70
17.04 70.83 18.18 – 11.59 8.95 0.50 11.83 1.40
79.60 32.84 91.51 65.24 21.70 61.25 49.08 5.97 51.80
85.64 67.14 75.39 – 1.01 73.90 63.61 0.44 52.60
66.19 42.52 49.10 90.81 63.09 47.00 49.15 33.65 62.60
92.13 76.53 95.08 99.40 75.99 87.50 77.84 34.29 69.10
Table 3: Additional reproduced baseline results, with ogbn-arXivNC no longer downsampled. – indicates that the model cannot process the dataset due to architectural constraints or unavailable results. Bold and underline indicate the best and second-best results.
between prior dataset-specific protocols and our unified heterogeneous-modality benchmark. Our setting trains and evaluates models across heterogeneous graphs, including TAGs, IAGs, and MAGs, whereas many baselines are designed for narrower modality settings, such as LLaGA for TAGs and MLaGA for MAGs (Chen et al., 2024a; Fan et al., 2026). Our protocol also jointly mixes citation, ecommerce, and social graphs, rather than training within a single domain or modality schema. MLaGA. For MLaGA, the gap on MoviesLP is likely related to link-prediction split construction. Since MLaGA does not release the exact train/test edge splits, our reproduction may use different sampled edges for training and evaluation. Moreover, MLaGA is originally evaluated in a narrower Amazon-domain MAG setting (Fan et al., 2026), while our protocol covers heterogeneous domains and modality schemas. LLaGA. For LLaGA, the gap on arXivNC mainly comes from the balanced mixed-training setup. To prevent large graphs from dominating joint training, we downsample ogbn-arXiv training nodes, as detailed in Appendix A, which differs from the original LLaGA setting, which uses the full ogbnarXiv split (Chen et al., 2024a). To isolate the effect of training-set size, we additionally evalu-
ate reproduced baselines with the full ogbn-arXiv training data. As shown in Table 3, LLaGA improves to 74.71 on ogbn-arXiv, while OMG-VLM further improves to 76.53, suggesting that the lower reproduced LLaGA result in Table 1 is largely attributable to the balanced split rather than implementation changes. Meanwhile, OMG-VLM remains the best-performing model across datasets, outperforming LLaGA by 1.82 points. Overall, these results indicate that differences from originally reported baselines mainly arise from adapting specialized methods to a unified benchmark over heterogeneous domains and modality schemas. Meanwhile, OMG-VLM shows competitive performance in both training settings. C.3
Additional Evaluation Metrics
We report Macro-F1 for representative strong baselines and OMG-VLM in Table 4. Macro-F1 provides a complementary view of performance by weighting classes equally, which is especially useful when class distributions are imbalanced. OMGVLM achieves the best Macro-F1 on most datasets, with particularly large gains on PubMedNC (73.34 vs. 59.64), and RedditMLP (92.14 vs. 75.32). These results are consistent with the accuracy trends in the main paper and further support the effectiveness 12
Dataset
GraphP. UniG2 G4MM G-VLM OMG
ArtsNC arXivNC MoviesLP RedditSLP PubMedNC RedditMLP CDsNC
84.81 23.68 15.34 – 43.80 37.00 22.89
80.78 22.82 91.46 78.85 38.40 62.35 5.15
87.87 37.39 85.95 – 33.13 75.32 12.05
80.06 52.94 33.15 54.90 59.64 46.53 26.34
93.21 54.07 95.19 99.10 73.34 92.14 27.07
E
Ablation and Sensitivity Results
E.1
Complete Main Ablation Results
Table 6 details the performance of OMG-VLM, text-pooled variant, image-pooled variant, and both-pooled variant of our model on all datasets. Table 7 details the performance of OMG-VLM with and without per-neighbor compression across different neighborhood sizes (10-100). Table 8 details the performance of OMG-VLM under co-optimization and two-stage training.
Table 4: Macro-F1 results for representative strong baselines and OMG-VLM. – indicates that the model cannot process the dataset due to architectural constraints. Bold and underline indicate the best and second-best results.
E.2 Model
GraphVLM OMG-VLM
Backbone
VLM VLM
Train Infer Avg. Prompt (ms/sp) (ms/sp) Tokens 1428.30 925.42
397.85 358.17
Table 9 details the performance of OMG-VLM with different textual token lengths, visual token lengths, and the number of aggregated neighbors.
1444.07 464.30
Table 5: Computation comparison. Runtime is reported in milliseconds per sample. Avg. Prompt denotes the average number of input prompt tokens of ArtsNC . Bold highlights the better result among VLM-based methods.
E.3
Backbone Ablations
OMG-VLM utilizes Qwen-VL-7B as VLM backbone (Bai et al., 2023). Our use of Qwen-VL-7B follows its standard research-use setting and is consistent with prior VLM-related graph reasoning work (Liu et al., 2026). Here, we examine whether the effecti veness of OMG-VLM depends on a specific VLM backbone. In addition to the Qwen-VLbased OMG-VLM used in the main experiments, we instantiate the same framework with two alternative backbones, LLaVA-v1.6-7B (Liu et al., 2024) and Qwen3-VL-4B (Bai et al., 2025). Table 10 reports these results together with the strongest non-OMG-VLM baseline on each dataset. Overall, OMG-VLM remains effective across backbone families and model scales. The QwenVL-based version achieves the best results on most datasets, including 95.30 on MoviesLP , 99.10 on RedditSLP , and 92.45 on RedditMLP , substantially outperforming the strongest corresponding baselines of 91.51, 90.10, and 72.30. The alternative backbones also remain competitive: LLaVAv1.6-7B-based variant reaches 91.36 on ArtsNC and 93.12 on MoviesLP , while Qwen3-VL-4B-based variant achieves 91.78 on PubMedNC , and 40.88 on CDsNC . These results suggest that OMG-VLM is not tied to a particular pretrained VLM implementation. Instead, its graph adapters can operate on the native textual and visual representations of different VLM backbones without backbone-specific redesign. This further shows that the gains mainly come from the proposed graph adapters rather than the VLM backbone itself, validating the effectiveness of our adapter design.
of OMG-VLM across tasks and modality schemas.
D
Complete Sensitivity Results
Computational Cost Analysis
We report the computational cost of representative multimodal baselines in Table 5. All results are measured on a single NVIDIA H100 80GB GPU under comparable configurations. Since OMGVLM is built on a VLM backbone, the most meaningful efficiency comparison is with VLM-based graph reasoning methods rather than LLM-only or GNN-based baselines, whose computational profiles are fundamentally different. We therefore focus our discussion on GraphVLM, which represents an intuitive VLM-based solution that incorporates neighborhood information by directly flattening graph context into the prompt. Compared with GraphVLM, OMG-VLM achieves lower training time, inference time, and prompt length. The main reason is that GraphVLM serializes graph neighborhoods directly into the textual prompt, so its input length grows with the number of neighbors and induces a higher attention cost. In contrast, OMG-VLM compresses graph-derived context into a fixed number of tokens before passing it to the VLM. This bounded context reduces prompt expansion while retaining VLM-based multimodal reasoning capacity. 13
Dataset
Text Pool
ArtsNC arXivNC MoviesLP RedditSLP PubMedNC RedditMLP VideoGamesLP CDsNC CoraLP
Image Pool
Text/Image Pool Mean Attn.
Mean
Attn.
Mean
Attn.
92.43 (-0.07) 67.38 (-0.47) 92.93 (-2.37) 97.80 (-1.30) 59.25 (-19.30) 87.15 (-5.30) 76.57 (-1.42) 26.34 (-6.22) 51.00 (-0.80)
92.20 (-0.30) 67.09 (-0.76) 92.30 (-3.00) 97.80 (-1.30) 60.41 (-18.14) 86.85 (-5.60) 76.99 (-1.00) 31.53 (-1.03) 50.50 (-1.30)
92.30 (-0.20) 67.84 (-0.01) 93.56 (-1.74) 76.25 (-22.85) 78.27 (-0.28) 79.85 (-12.60) 78.26 (+0.27) 30.89 (-1.67) 51.49 (-0.31)
91.25 (-1.25) 67.35 (-0.50) 85.33 (-9.97) 74.95 (-24.15) 78.32 (-0.23) 82.30 (-10.15) 65.57 (-12.42) 27.21 (-5.35) 54.50 (+2.70)
92.20 (-0.30) 67.12 (-0.73) 90.75 (-4.55) 76.30 (-22.80) 57.13 (-21.42) 80.70 (-11.75) 69.75 (-8.24) 26.34 (-6.22) 50.90 (-0.90)
Full Model
91.67 (-0.83) 67.09 (-0.76) 73.81 (-21.49) 74.70 (-24.40) 56.25 (-23.30) 76.45 (-16.00) 65.49 (-12.50) 30.76 (-1.80) 50.60 (-1.20)
92.50 67.85 95.30 99.10 78.55 92.45 77.99 32.56 51.80
Table 6: Ablation on neighborhood aggregation modules. Mean replaces the corresponding adapter with mean pooling over neighbor representations, while Attn. uses center-query attention pooling, where the center node representation attends to neighbors. Text/Image Pool replaces both adapters. Relative differences are computed against the full OMG-VLM model.
10 ArtsNC MoviesLP RedditSLP RedditMLP
92.50 95.30 99.10 92.45 VideoGamesLP 77.99 CDsNC 32.56
91.5
w/ Compr. 40 70
100
10
w/o Compr. 40 70
100
91.17 94.32 98.39 92.40 78.22 33.46
91.84 95.17 98.49 93.10 78.95 31.85
92.64 93.85 98.04 89.35 78.72 33.58
92.16 94.51 98.54 91.30 77.65 33.98
91.86 94.32 98.59 90.85 77.61 34.57
91.88 94.45 98.64 89.60 78.37 33.58
91.20 94.19 98.24 91.05 78.87 34.69
Accuracy (%)
Dataset
ArtsNC arXivNC MoviesLP RedditSLP PubMedNC RedditMLP VideoGamesLP CDsNC CoraLP
90.94
91
L=1
L=2
L=3
Figure 7: Sensitivity to the number of visual aggregation layers L. Results are reported on ArtsNC with separate models trained and evaluated for each value of L.
Co-Optimization Two-Stage 92.50 67.85 95.30 99.10 78.55 92.45 77.99 32.56 51.80
91.2
90.5
Table 7: Ablation on per-neighbor image compression across visual neighborhood sizes. Compressed visual token length is fixed to MI =32. Dataset
91.45
prompting; for example, it decreases performance on ArtsNC (62.09 vs. 69.39), MoviesLP (50.05 vs. 61.15), and PubMedNC (72.61 vs. 77.32). These results indicate that simply exposing a VLM to a serialized graph context is insufficient. In contrast, OMG-VLM benefits from trained graph adapters that aggregate neighborhood information within the VLM’s native representation space.
92.14 66.65 93.97 96.94 90.42 85.70 76.23 31.63 66.20
Table 8: Performance of OMG-VLM under end-to-end co-optimization and two-stage training.
E.4
E.5
VLM Zero-Shot Ablations
We further evaluate whether the gains of OMGVLM can be obtained by prompting the base VLM without graph-adapter training. Specifically, we use the pretrained Qwen-VL backbone under two zero-shot settings: a non-structural prompt, which uses only the available node attributes and task instruction, and a structural prompt, which additionally includes serialized neighborhood information in the prompt. No parameters are updated in either zero-shot setting. Results are shown in Table 11. OMG-VLM consistently outperforms both zeroshot controls. The gap is prominent on datasets such as arXivNC (67.85 vs. 8.50), MoviesLP (95.30 vs. 61.15), RedditSLP (99.10 vs. 62.68), and RedditMLP (92.45 vs. 68.60). Structural prompting does not consistently improve over non-structural
Layer L Sensitivity Analysis
We analyze the sensitivity of OMG-VLM to the number of center-conditioned visual aggregation layers L in the Graph-Aware Visual Adapter. For efficiency, we conduct a targeted sensitivity analysis on ArtsNC , training and evaluating separate models with L ∈ {1, 2, 3}, rather than repeating the full mixed-training protocol for each setting. Figure 7 reports the results. The results suggest that performance is stable across the tested values of L. All variants remain within a narrow range, with the best result obtained at L = 1. Since deeper aggregation does not yield consistent improvement in this analysis, we use a small value of L = 1 in the main model to reduce unnecessary computation. 14
Setting
In-Domain
Transfer
Dataset
Text Tokens (MT ) MI = 32, K = 10
Image Tokens (MI ) MT = 1, K = 10
Neighbors (K) MT = 8, MI = 32
1
8
16
32
8
32
64
10
40
70
100
ArtsNC arXivNC MoviesLP RedditSLP
92.73 68.14 94.70 98.95
92.50 67.85 95.30 99.10
92.36 67.98 94.45 98.64
92.04 68.16 94.76 98.19
92.36 68.06 94.98 98.69
92.73 68.14 94.70 98.95
92.85 67.67 94.67 98.90
92.50 67.85 95.30 99.10
91.17 67.91 94.32 98.39
91.88 67.89 94.45 98.64
91.84 67.93 95.17 98.49
PubMedNC RedditMLP VideoGamesLP CDsNC CoraLP
77.28 79.10 78.14 32.49 56.30
78.55 92.45 77.99 32.56 51.80
77.87 83.80 74.08 32.39 50.50
77.05 89.85 78.49 35.22 51.00
78.60 91.55 77.72 32.10 52.00
77.28 79.10 78.14 32.49 56.30
77.08 86.95 79.41 34.60 52.70
78.55 92.45 77.99 32.56 51.80
76.17 92.40 78.22 33.46 52.30
78.72 89.60 78.37 33.58 61.90
76.98 93.10 78.95 31.85 55.70
Table 9: Sensitivity analysis to textual token length (MT ), visual token length (MI ), and number of aggregated neighbors (K). Backbone Model
ArtsNC
arXivNC
MoviesLP
RedditSLP
PubMedNC
CDsNC
RedditMLP
VideoGamesLP
Best Baseline OMG-VLM(LLaVA-v1.6-7B) OMG-VLM(Qwen3-VL-4B) OMG-VLM(Qwen-VL-7B)
89.63 91.36 89.00 92.50
64.59 66.30 67.67 67.85
91.51 93.12 93.66 95.30
90.10 89.05 83.31 99.10
72.13 88.97 91.78 78.55
31.10 34.73 40.88 32.56
72.30 81.50 76.65 92.45
70.30 76.93 76.29 77.99
Table 10: Additional backbone ablation results. We instantiate OMG-VLM with different VLM backbones and compare them with the strongest non-OMG-VLM baseline on each dataset. Bold indicates the best result, and underline indicates the second-best result. Dataset ArtsNC arXivNC MoviesLP RedditSLP PubMedNC RedditMLP VideoGamesLP CDsNC CoraLP
F.2
Non-Struct. Struct. OMG-VLM 69.39 2.89 61.15 49.90 77.32 68.60 50.08 28.73 50.00
62.09 8.50 50.05 62.68 72.61 38.95 50.00 29.20 50.50
92.50 67.85 95.30 99.10 78.55 92.45 77.99 32.56 51.80
We show three qualitative examples from different datasets, each including the constructed input, retrieved graph context, model response, and groundtruth answer. Amazon-Arts. Figure 8 shows a MAG node classification example where OMG-VLM predicts the product category from target attributes and graphaware multimodal neighborhood context. ogbn-arXiv. Figure 9 shows a TAG node classification example where OMG-VLM predicts the paper category from textual attributes and graph-aware textual neighborhood context. RedditS. Figure 10 shows an IAG link prediction example where OMG-VLM predicts whether two image-attributed posts are connected using their visual attributes and neighborhood context.
Table 11: Zero-shot ablation results using the base Qwen-VL backbone without graph-adapter training. Non-Struct. uses only node attributes and task instructions, while Struct. additionally serializes neighborhood information into the prompt. Bold indicates the best result, and underline indicates the second-best result.
F
Prompt Templates and Qualitative Examples
F.1
Prompt Templates
Qualitative Examples
Table 12 and Table 13 summarize the prompt templates used for node classification and link prediction, respectively. All prompts follow the ChatML format, where the system message is set to “You are a helpful assistant.” (Bai et al., 2023). The placeholders <anchor text> and <anchor image> denote the textual and visual attributes of the anchor node, while <text neighborhood> and <image neighborhood> denote modality-specific neighborhood context incorporated into the input. 15
Dataset
Type
User Prompt Content
Arts
MAG
Given the target product information on Amazon: Picture: <anchor image> Image neighborhood: <image neighborhood> Title and description: <anchor text> Text neighborhood: <text neighborhood> Question: Based on the target product’s picture, title, description, and neighborhood context, which category does the target product belong to? Choose from the following options: Knitting & Crochet; Beading & Jewelry Making; Painting, Drawing & Art Supplies; Crafting; Model & Hobby Building; Sewing; Scrapbooking & Stamping.
arXiv
TAG
Given the target paper information on arXiv: Title and abstract: <anchor text> Text neighborhood: <text neighborhood> Question: Based on the target paper’s title, abstract, and neighborhood context, which of the following categories does the target paper belong to? Options: arxiv cs ai, arxiv cs ar, arxiv cs cc, arxiv cs ce, arxiv cs cg, arxiv cs cl, arxiv cs cr, arxiv cs cv, arxiv cs cy, arxiv cs db, arxiv cs dc, arxiv cs dl, arxiv cs dm, arxiv cs ds, arxiv cs et, arxiv cs fl, arxiv cs gl, arxiv cs gr, arxiv cs gt, arxiv cs hc, arxiv cs ir, arxiv cs it, arxiv cs lg, arxiv cs lo, arxiv cs ma, arxiv cs mm, arxiv cs ms, arxiv cs na, arxiv cs ne, arxiv cs ni, arxiv cs oh, arxiv cs os, arxiv cs pf, arxiv cs pl, arxiv cs ro, arxiv cs sc, arxiv cs sd, arxiv cs se, arxiv cs si, arxiv cs sy.
PubMed
TAG
The following is the title and the abstract of a paper: Title and abstract: <anchor text> Text neighborhood: <text neighborhood> Question: Based on the paper’s title, abstract, and neighborhood context, which case of diabetes does the paper involve? Choose from the following options: Type 1 diabetes; Type 2 diabetes; Experimentally induced diabetes.
CDs
MAG
Given the target CD/Vinyl product information on Amazon: Picture: <anchor image> Image neighborhood: <image neighborhood> Title and description: <anchor text> Text neighborhood: <text neighborhood> Question: Based on the target product’s picture, title, description, and neighborhood context, which category does the target product belong to? Choose from the following options: Pop; Today’s Deals in Music; Rock; Indie & Alternative; Classic Rock; Country; International Music; Jazz; Metal; R&B; Classical; Rap & Hip-Hop; Christian & Gospel; Blues; Dance & Electronic.
Table 12: Prompt templates for node classification datasets. Each prompt explicitly separates anchor-node attributes from modality-specific neighborhood context.
16
Dataset
Type
User Prompt Content
RedditS
IAG
Given two nodes from a social media graph, the information of the first node is as follows: Picture: <anchor image> Image neighborhood: <image neighborhood> and the information of the other node is: Picture: <anchor image> Image neighborhood: <image neighborhood> If the connections between nodes represent the co-comment relationships between posts, are these two central nodes connected? Give me a direct answer of “yes” or “no”.
Movies
MAG
Given two nodes from the Amazon movies product graph, the information of the first node is as follows: Picture: <anchor image>; Image neighborhood: <image neighborhood>; Title and description: <anchor text> Text neighborhood: <text neighborhood> and the information of the other node is: Picture: <anchor image>; Image neighborhood: <image neighborhood>; Title and description: <anchor text> Text neighborhood: <text neighborhood> If the connections between nodes represent the co-purchased or co-reviewed relationships between products, are these two central nodes connected? Give me a direct answer of “yes” or “no”.
Cora
TAG
Given two nodes from the Cora citation graph of machine learning papers, the information of the first node is as follows: Title and abstract: <anchor text> Text neighborhood: <text neighborhood> And the information of the other node is: Title and abstract: <anchor text> Text neighborhood: <text neighborhood> If the connections between nodes represent citation relationships, i.e., one paper cites the other as a reference in its work, are these two central nodes connected? Give me a direct answer of “yes” or “no”.
RedditM
MAG
Given two nodes from a social media graph, the information of the first node is as follows: Picture: <anchor image>; Image neighborhood: <image neighborhood>; Text content: <anchor text> Text neighborhood: <text neighborhood> and the information of the other node is: Picture: <anchor image>; Image neighborhood: <image neighborhood>; Text content: <anchor text> Text neighborhood: <text neighborhood> If the connections between nodes represent the co-comment relationships between posts, are these two central nodes connected? Give me a direct answer of “yes” or “no”.
VideoGames
MAG
Given two nodes from the Amazon video games product graph, the information of the first node is as follows: Picture: <anchor image>; Image neighborhood: <image neighborhood>; Title and description: <anchor text> Text neighborhood: <text neighborhood> and the information of the other node is: Picture: <anchor image>; Image neighborhood: <image neighborhood>; Title and description: <anchor text> Text neighborhood: <text neighborhood> If the connections between nodes represent the co-purchased or co-reviewed relationships between products, are these two central nodes connected? Give me a direct answer of “yes” or “no”.
Table 13: Prompt templates for link prediction datasets. Each prompt presents the two anchor nodes together with their modality-specific neighborhood context.
17
MAG Amazon-Arts: Node Classification
Target Product
Title and Description. Origami Paper - Traditional Prints - 8 1/4” 49 Sheets: Tuttle Origami Paper: Large Origami Sheets Printed with 6 Different Patterns: Instructions for 6 Projects Included. From the Back Cover About the Author Tuttle Studio draws......
Retrieved Neighborhood Context
[Additional neighbors omitted for brevity] Neighbor 1: LuPro Japanese LPAT-4006 Origami Folding Paper Case Box......
Neighbor 2: Lion Brand Yarn Lion Brand Bonbons 680 Crayons......
Prompt Given the target product information on Amazon: Picture: <anchor image> Title and description: <anchor text> The target product is associated with the following graph neighborhood context: Neighbor pictures: <image neighborhood> Neighbor titles and descriptions: <text neighborhood> Question: Based on the target product’s picture, title, description, and graph neighborhood context, which category does the target product belong to? Options: Knitting & Crochet, Beading & Jewelry Making, Painting, Drawing & Art Supplies, Crafting, Model & Hobby Building, Sewing, Scrapbooking & Stamping. OMG-VLM Output
Ground Truth
Crafting
Crafting
Figure 8: Qualitative example for MAG Amazon-Arts node classification. Full text information is omitted for brevity.
18
TAG ogbn-arXiv: Node Classification Target Paper Title and Abstract. learning spectral spatial temporal features via a recurrent convolutional neural network for change detection in multispectral imagery Change detection is one of the central problems...... Retrieved Neighborhood Context Neighbor 1. multitask learning for large scale semantic change detection Change detection is one of the main problems in remote sensing, and is essential to the accurate processing and understanding...... Neighbor 2. deeplab semantic image segmentation with deep convolutional nets atrous convolution and fully connected crfs In this work we address the task of semantic image segmentation with Deep Learning...... [Additional neighbors omitted for brevity] Prompt Given the target paper information on arXiv: Title and abstract: <anchor text> The target paper is associated with the following graph neighborhood context: Neighbor titles and abstracts: <text neighborhood> Question: Based on the target paper’s title, abstract, and graph neighborhood context, which of the following categories does the target paper belong to? Options: arxiv cs ai, arxiv cs ar, arxiv cs cc, arxiv cs ce, arxiv cs cg, arxiv cs cl, arxiv cs cr, arxiv cs cv, arxiv cs cy, arxiv cs db, arxiv cs dc, arxiv cs dl, arxiv cs dm, arxiv cs ds, arxiv cs et, arxiv cs fl, arxiv cs gl, arxiv cs gr, arxiv cs gt, arxiv cs hc, arxiv cs ir, arxiv cs it, arxiv cs lg, arxiv cs lo, arxiv cs ma, arxiv cs mm, arxiv cs ms, arxiv cs na, arxiv cs ne, arxiv cs ni, arxiv cs oh, arxiv cs os, arxiv cs pf, arxiv cs pl, arxiv cs ro, arxiv cs sc, arxiv cs sd, arxiv cs se, arxiv cs si, arxiv cs sy. OMG-VLM Output
Ground Truth
arxiv cs cv
arxiv cs cv
Figure 9: Qualitative example for TAG ogbn-arXiv node classification. Full text information is omitted for brevity.
19
IAG RedditS: Link Prediction Target Node Pair Node A
Node B
Retrieved Neighborhood Context Neighbor of Node A
Neighbor of Node B
[Additional neighbors omitted for brevity] Prompt Given two nodes from a social media graph, the information of the first node is as follows: Picture: <anchor image of Node A> The first node is associated with the following graph neighborhood context: Neighbor pictures: <image neighborhood of Node A> The information of the second node is as follows: Picture: <anchor image of Node B> The second node is associated with the following graph neighborhood context: Neighbor pictures: <image neighborhood of Node B> If the connections between nodes represent the co-comment relationships between posts, are these two central nodes connected? Give a direct answer of “yes” or “no”. OMG-VLM Output
Ground Truth
yes
yes
Figure 10: Qualitative example for IAG RedditS link prediction.
20