Conceptio › Archive › arXiv CS

arXiv CSopen access

S2Aligner: Pair-Efficient and Transferable Pre-Training for Sparse Text-Attributed Graphs

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

S2Aligner: Pair-Efficient Transferable Pre-Training for Sparse Text-Attributed Graphs

arXiv:2605.18579v1 [cs.LG] 18 May 2026

Yuhan Wang2 , Haopeng Zhang1 , Yibo Ding1 , Jiaqi Yu1 , Xinyu Zhao1 Yuhang Liu3 , Ziwei Zhang1 , Xiao Wang1 , Ruijie Wang1∗ 1 School of Computer Science and Engineering, Beihang University 2 School of Computer Science and Engineering, Beijing University of Technology 3 School of New Media and Communication, Tianjin University {24373469,dingyibo,24373465,25371032,zwzhang,xiao_wang,ruijiew}@buaa.edu.cn [email protected], [email protected]

Abstract Pre-training on text-attributed graphs (TAGs) is central to building transferable graph foundation models, where LLM-as-Aligner methods align graph and text representations through the semantic knowledge of large language models. However, these methods usually assume that node texts provide sufficient and reliable supervision, an assumption often violated in real-world sparse TAGs. When textual anchors are missing, noisy, or uneven across domains, graph structures must be aligned with weak semantic evidence, leading to unreliable structure–semantics correspondence and sparsity-induced transfer bias. This paper presents S2 Aligner, a Sparsity-aware and Structure-enhanced LLM-as-Aligner framework for graph– text pre-training on sparse TAGs. The key idea is to decouple semantic alignment from structural modeling, allowing topology-aware signals to enhance alignment without contaminating the shared semantic space. Specifically, S2 Aligner decomposes graph–text representations into semantic and structural components, uses structure-oriented reconstruction with consistency control to inject reliable topology cues into text representations, and suppresses inconsistent structural signals under textual sparsity. Moreover, S2 Aligner introduces sparsity-aware cross-domain risk balancing, which calibrates domain risks through a global-domain density ratio and downweights unreliable sparse samples via graph reliability estimation. Theoretical analysis shows that this objective reduces cross-domain generalization gaps by controlling domain risk discrepancy. Extensive experiments across diverse graph domains, sparsity levels, and downstream tasks demonstrate that S2 Aligner consistently outperforms existing baselines.

Introduction

Graph foundation models (GFMs) have recently emerged as a promising paradigm for inductive graph learning. They aim to distill transferable structural and semantic priors from large-scale graph corpora, reducing the need for task-specific supervision and costly fine-tuning [18, 12, 27]. This paradigm is particularly suitable for text-attributed graphs (TAGs), common in citation networks, social platforms, and e-commerce systems [6, 7, 4, 3], where node texts provide rich semantics and edges capture relational structure. Recently, cross-modal graph-text alignment has become a central direction for TAG pretraining [37, 43, 19], aligning graph or subgraph embeddings with textual embeddings, often leveraging large language models (LLMs) as aligners — the so-called LLM-as-Aligner paradigm (Figure 1). ∗ Corresponding author.

Preprint.

100

Full Text

77.5%

83.5%

86.6%

80 70

6.36

4 3.16

60 Markers / 1K tokens Uncertain summaries

57.4%

Full

10%

Graph-to-Text MRR (%)

Markers / 1K tokens

9.30 90.4%

Figure 1: Aligner.

9.33

Uncertain summaries (%)

10.38

+5.7%

Semantic +Struct

81.00

76.66

-4.4% 65.52

62.62

60 50

Full

Sparse

T2N Retrieval (%)

100

Sparse Text 93.97

86.47

96.45

80.42

80 60 40

38.85

44.50

48.88

33.63

20 0

MRR

R@1

R@5

R@10

LLM-as- (a) Uncertainty vs. Sparsity (b) Structural supplementation (c) Full vs. sparse retrieval Figure 2: (a) Summary uncertainty increases. (b) Naive structural supplementation becomes harmful. (c) Full vs. Sparse alignment.

Despite its promise, the LLM-as-Aligner paradigm relies on a strong assumption: textual evidence should be sufficient to provide reliable supervision for graph-text alignment [37, 43, 19]. This assumption is often violated in real-world TAGs. Node texts can be missing, short, noisy, or highly uneven across domains. In this case, the text view becomes an incomplete anchor for the graph view. Rich graph structures must be aligned with weak and ambiguous semantic signals, making the learned structure–semantics correspondence unreliable. The problem is further amplified by recent methods that use LLM-generated subgraph summaries as supervision [43, 19]. When textual contexts are sparse, LLMs have to infer missing semantics from incomplete evidence, which may introduce uncertainty and bias into the alignment signal. This raises a fundamental question: how can graph-text pretraining learn transferable structure–semantics correspondence when the cross-modal supervision itself is sparse and unreliable? This paper studies LLM-as-Aligner pretraining on sparse TAGs, preserving transferable pretraining while reducing dependence on dense textual supervision. It introduces two coupled challenges: Challenge 1: Unreliable structure–semantics alignment. In sparse text-attributed graphs, textual anchors are incomplete or ambiguous, while the graph structure remains rich. This imbalance can cause the model to misalign structural patterns with noisy text, failing to learn faithful structure– semantics correspondences. Figure 2 illustrates this issue: (a) text sparsity increases semantic uncertainty, weakening alignment anchors; (b) structural supplementation helps under dense text but may amplify mismatches under sparse text; (c) sparse-text graph–text retrieval performs substantially worse than full-text retrieval, highlighting the limited robustness of LLM-as-Aligner models. Challenge 2: Sparsity-induced transfer bias. Sparse texts also weaken zero-shot generalization. Different domains require different densities of alignment signals, since some domains can transfer with coarse semantic anchors while others depend on finer graph–text correspondences. Text sparsity makes such reliability uneven across domains. Standard pretraining may therefore overfit to domains with sufficient anchors or learn biased correlations from noisy ones, causing imbalanced domain risks and unstable transfer to unseen sparse TAGs. Thus, robust cross-domain generalization requires sparsity-aware risk balancing to extract domain-invariant representations under sparse supervision. To address the above challenges, we propose S2 Aligner, a sparsity-aware and structure-enhanced LLM-as-Aligner framework for graph–text pretraining on sparse TAGs. The overall idea is to decouple semantic alignment from structural modeling, so that sparse and uncertain structural cues can be used to enhance graph–text alignment without directly contaminating the shared semantic space. For Challenge 1, S2 Aligner separates each graph–text representation into semantic and structural components. The semantic components define the main alignment space, while the structural components are modeled through structure-oriented reconstruction. By using reconstruction consistency to control structural enhancement, S2 Aligner injects reliable topology-aware signals into text representations and suppresses inconsistent structural cues, thereby mitigating structural negative transfer under textual sparsity. For Challenge 2, S2 Aligner introduces a sparsity-aware cross-domain risk balancing mechanism. It calibrates domain risks with a global-domain density ratio and further uses graph reliability to reduce the impact of unreliable sparse samples. This encourages the model to focus on domain-invariant graph–text correspondences rather than sparse domain-specific correlations. Theoretical analysis suggests that this objective can reduce the cross-domain generalization gap by controlling domain risk discrepancy. Our contributions are summarized as follows:

• We propose S2 Aligner, a sparsity-aware and structure-enhanced LLM-as-Aligner framework for sparse TAGs. It transforms uncertain topological signals into verifiable structural enhancements and incorporates sparsity-aware risk balancing for better cross-domain generalization. • We design a sparsity-aware cross-domain risk balancing mechanism. It filters noisy samples using graph reliability evaluation and calibrates cross-domain risks via a global-domain density ratio, reducing domain-biased alignment and improving generalization. • Extensive experiments validate our method across sparsity levels, graph domains, and downstream tasks. Notably, even with only 10% textual attributes, it surpasses baselines trained with full attributes, demonstrating superior sparsity generalization and data efficiency.

Preliminary

LLM-as-Aligner for graph problems. Following GraphCLIP [43], we define a text-attributed graph (TAG) as G = (V, E, T ), where V and E denote the node and edge sets, and T = {(gi , ti )}M i=1 denotes a collection of graph-text pairs. Here, gi can be a node, a sampled subgraph, or an entire graph, and ti is the associated textual information, such as a textual attribute, structural summary, or label prompt. Given a graph encoder fG and a text encoder fT , each pair is encoded as zg,i = READOUT(fG (gi )) ,

zt,i = READOUT(fT (ti )) ,

(1)

For a mini-batch T , graph and text representations are aligned by a symmetric contrastive loss: " # X 1 exp (sim(zg,i , zt,i )/τ ) exp (sim(zg,i , zt,i )/τ ) Lalign (T ) = − log P − log P , 2|T | j exp (sim(zg,i , zt,j )/τ ) j exp (sim(zg,j , zt,i )/τ ) (gi ,ti )∈T

(2) where sim(·, ·) denotes cosine similarity and τ is a temperature parameter. This objective treats the matched graph–text pair as the positive pair and all other pairs in the mini-batch as negatives. Sparse text-attributed graph problem. In real-world TAGs, textual attributes are often missing, incomplete, or unreliable. We define a sparse TAG as Ge = (V, E, Te ), where Te ⊆ T denotes the observed sparse graph–text pair set. A straightforward solution is to optimize the graph and text encoders over the sparse pairs: min Lalign (Te ), (3) θG ,θT

where θG and θT are the parameters of fG and fT , respectively. However, directly aligning sparse graph–text pairs can be unstable, since missing or unreliable textual attributes provide weak supervision and may introduce noisy cross-modal correspondences. Although the above formulation is general, we instantiate gi as a sampled subgraph in practice to exploit local structural context. After pre-training, zero-shot prediction is performed by converting each candidate label c ∈ Y into a textual description tc and selecting the label with the highest graph–prompt similarity: ŷ = arg max sim (fG (g), fT (tc )) . (4) c∈Y

Method

In this section, we present S2 Aligner, a sparsity- and structure-enhanced LLM-as-Aligner framework (Figure 3) for pretraining on sparse text-attributed graphs. First, we decouple graph–text representations into semantic and structural components, separating semantic alignment from structural modeling (Sec. 3.1). Next, structure-oriented reconstruction injects reliable structural signals while suppressing inconsistent cues (Sec. 3.2). Finally, Sparse-aware Cross-domain Risk Balancing reweights source-domain risks using cross-domain support and graph reliability to enhance zero-shot cross-graph generalization (Sec. 3.3). Together, these components enable robust graph–text alignment under sparse supervision and enhance transferability to unseen domains. 3

Inputs

Sec.3.1&3.2 Structure Reconstruction 𝒛𝒄𝒈,𝒊 (content)

Graph Encoder (GNN)

D1 Academic

𝒛𝒔𝒈,𝒊 (structure)

…

D2 E-commerce

𝒛𝒄𝒕,𝒊 (content)

Text Encoder (LLM)

𝒛𝒔𝒕,𝒊 (structure)

𝓛𝒓𝒆𝒄 Reconstruction Head

Φ𝑟𝑒𝑐

… Uncertainty

Dk Web

…

High Uncertainty Low-quality LLM summary (example)

Samples

After Before (uneven risks) (balanced risks) Re R′e

𝒛෤ 𝒄𝒕,𝒊

… 0 𝑫𝟏𝑫𝟐

𝜿 𝒛ො 𝒔𝒈,𝒊 , 𝒛ത 𝒔𝒈,𝒊

… 𝑫𝒌

0 𝑫𝟏𝑫𝟐

𝑫𝒌

𝑲

𝟏 ഥ )𝟐 𝓛𝒓𝒊𝒔𝒌 = ෍(𝑹𝒆 − 𝑹 𝑲

Certainty

①

Reliability Weight structure-semantic text structural consistency 𝒄𝒊 stability𝒅𝒊 completeness 𝒂𝒊

𝒆=𝟏

𝑹⋆𝟏 , 𝑹𝟐⋆ , 𝑹⋆𝟑 , … , 𝑹⋆𝑲 .

Domain Balancing Weight

Global 𝒑𝟎

② ③ ⓝ

𝝆𝒊 = 𝒄𝒍𝒊𝒑 𝝀𝒅 𝒅𝒊 +𝝀𝒂 𝒂𝒊 + 𝝀𝒄 𝒄𝒊 , 𝟎, 𝟏

𝓛𝒅𝒆𝒏

Density Ratio

𝒓𝒊 =

aggregate sparse reliability 𝝆𝒊

…

(hedging words → unreliable)

𝒛𝒄𝒕,𝒊

Weight Estimation-→ 𝒘

Summary

Node n1 could be categorized into computer science, its neighbor n2 might belong to a related research field…

Domain Risk Balancing 𝒛෪𝒔𝒈

෩ 𝓛𝒕𝒐𝒕𝒂𝒍 = 𝓛+ 𝒂𝒍𝒊𝒈𝒏 𝓣; 𝝎 + 𝜶𝓛𝒓𝒆𝒄 + 𝝂𝓛𝒅𝒆𝒏 +𝝁𝓛𝒓𝒊𝒔𝒌

? ?

Cross-Domain Generalization

෩ 𝓛+ 𝒂𝒍𝒊𝒈𝒏 𝓣; 𝝎

𝒑𝟎 𝒛𝒊 𝒑𝒆 𝒛𝒊 + 𝝐

Domain 𝒑𝒆

Sec.3.3

Figure 3: The overall framework of S2 Aligner is shown in the figure above. It encodes sparse text-attributed graphs into content and structural components and applies latent reconstruction on the structural branch to reduce negative transfer from sparse text. We further introduce Sparseaware Cross-domain Risk Balancing, aligning multi-source domain risks via density estimation and reliability weighting to learn domain-invariant features for robust cross-domain generalization. 3.1

Content-Structure Factorization

In text-sparse cross-modal alignment scenarios, topological structure is crucial for capturing graph–text correspondences. However, structural information generated from text often exhibits high uncertainty, which may introduce noise and lead to negative transfer. To address this issue, we disentangle content semantics and structural information into two independent branches: • Semantic alignment branch: relies solely on deterministic content information, extracting stable semantics from node attributes while avoiding interference from high-uncertainty structural text. • Structural branch: kept independent and purified through a structure reconstruction task, extracting reliable topological information to fully leverage the positive transfer effect of structure. For each graph sample g, we construct two complementary text views: a semantic summary S that characterizes attribute semantics, and a structure-aware description S s that represents topological features. The semantic summary focuses on the content information at the node attribute level, while the structure-aware description captures neighborhood patterns, connection topology, and the structural roles of nodes. On the graph modality side, we learn and obtain the graph content embedding zcg,i and graph structure embedding zsg,i from the subgraph representation and topological structure, respectively. On the text modality side, the two types of text views are encoded into the text content embedding zct,i and text structure embedding zst,i , respectively. The content embedding pair (zcg,i , zct,i ) forms the main semantic alignment space; the structural embedding pair (zsg,i , zst,i ) is used for the topology-oriented structure reconstruction in Sec. 3.2, which completes the denoising and purification of structural text representations as well as the modeling of positive transfer. 3.2

Structure-Oriented Reconstruction

To filter out noise and redundancy in high-uncertainty structural text while extracting reliable topological information, we use the graph-side structural embedding as a gradient-detached topological 4

target and train the text-side structural embedding to reconstruct it: X X 1 1 1 − sim b zsg,i , z̄sg,i = 1 − sim ϕrec (zst,i ), detach(zsg,i ) , Lrec = e e |T | |T | (gi ,ti )∈Te

(gi ,ti )∈Te

(5) where b zsg,i = ϕrec (zst,i ) is the reconstructed structural embedding, z̄sg,i = detach(zsg,i ) is the gradientdetached topology target, and ϕrec (·) is a lightweight reconstruction head. We further use the reconstruction consistency as a reliability gate for structural injection. Instead of introducing a separate score equation, we directly define the enhanced text embedding as 1 + sim(a, b) . (6) 2 Here, κ(·, ·) ∈ [0, 1] measures how well the structure-aware text recovers graph topology. If the structural description is unreliable, the gate shrinks and the injected structural signal is suppressed. Thus, only recoverable, topology-consistent structural information is injected into the text anchor. s e zct,i = norm zct,i + κ b zsg,i , z̄sg,i b zg,i ,

κ(a, b) =

For each sample, we define the enhanced contrastive loss as # " c c c c e e sim z sim z exp , z exp , z /τ /τ 1 g,i t,i g,i t,i − log P . ℓ+ − log P i = c c 2 ec ec (gj ,tj )∈Te exp sim zg,i , zt,j /τ (gj ,tj )∈Te exp sim zg,j , zt,i /τ (7) To connect the reconstruction module with the risk balancing module, we write the batch-level alignment loss in a weighted form. Therefore, the alignment loss and base training loss is: P + (g ,t )∈Te ωi ℓi + e ; ω + αLrec , e T (8) , Lbase = L+ Lalign T ; ω = P i i align (gi ,ti )∈Te ωi where α controls the strength of structure-oriented reconstruction. Next, we introduce how to design ω to adaptively adjust each sample’s contribution to the overall loss and to guide the model to focus on invariant features across modalities and views. 3.3

Sparse-aware Cross-domain Risk Balancing

Sparse graph–text pre-training usually involves multiple source domains with different graph structures, textual sparsity levels, and noise patterns. Let {De }K e=1 denote K source domains, where each domain contains graph–text pairs (gi , ti ). Although the enhanced alignment loss in Eq. (8) improves sample-level robustness, directly minimizing the pooled empirical risk can still overfit domain-specific sparse patterns. To address this issue, we propose Sparse-aware Cross-domain Risk Balancing, which assigns each sample a reliability-aware density-ratio weight and further regularizes the risks across source domains. Shared-support density ratio. For each pair (gi , ti ) from domain De(i) , let zi denote representation of each sample. We estimate two densities over zi : a global density p0 (zi ) over all source domains and a domain-specific density pe(i) (zi ) within its source domain. The density ratio is defined as γ X p0 (zi ) 1 ri = , Lden = − log p0 (zi ) + log pe(i) (zi ) , (9) e pe(i) (zi ) + ϵ |T | e (gi ,ti )∈T

where ϵ is a small constant for numerical stability, γ controls the sharpness of density-ratio reweighting, and Te denotes the current mini-batch. Intuitively, ri upweights samples lying in the cross-domain shared support and downweights samples dominated by domain-specific structural bias. Sparse graph reliability. Density compatibility alone cannot identify unreliable sparse samples. We therefore introduce a sparse reliability score ρi to measure whether the graph–text pair provides trustworthy supervision. Specifically, we consider three complementary signals: structural stability, textual completeness, and structure–semantic consistency:   X 1 + sim zcg,i , zct,i 1 |Vt (gi )|   di = exp pi,j log pi,j , ai = , ci = . (10) log Ni |V(gi )| 2 j∈V(gi )

Here, pi,j denotes the probability of visiting node j in subgraph gi starting from the center node vc via a random walk. zsg,i,r is the structural representation of the r-th perturbed view of gi , Vt (gi ) denotes nodes with observed textual attributes, and V(gi ) denotes all nodes in the sampled graph object. The final sparse reliability score is ρi = clip(λd di + λa ai + λc ci , 0, 1)

(11)

where λd , λa , and λc control the contribution of the three reliability factors. A sample receives a higher reliability score when it exhibits more stable topology, more complete textual attributes, and stronger structure–semantic consistency. Adaptive sample weighting. We combine the density ratio and sparse reliability into a unified sample weight. For each mini-batch domain subset Tee = Te ∩ De , the normalized weight is |Tee(i) |ri ρi , (gj ,tj )∈Tee(i) rj ρj

ωi = P

(gi , ti ) ∈ Tee(i) .

(12)

This intra-domain normalization keeps the average weight within each source domain close to one, preventing domains with larger density values or higher textual completeness from dominating the training objective. Built upon the enhanced per-sample contrastive loss ℓ+ i derived in Eq. (7), we further formulate the weighted empirical risk for each domain De as P + K 1 X (gi ,ti )∈Tee ωi ℓi Re = P Re . (13) , R̄ = K e=1 (gi ,ti )∈Tee ωi Cross-domain risk balancing. To encourage the model to learn domain-invariant graph–text alignment rather than sparse domain-specific shortcuts, we penalize the dispersion of weighted risks across source domains: K 2 1 X Lrisk = Re − R̄ . (14) K e=1 Unlike pairwise domain constraints, this variance-style regularizer provides a compact objective for aligning all source-domain risks. Finally, the Sparse-aware Cross-domain Risk Balancing mechanism is plugged into Eq. (8) by using the weights ω = {ωi }(gi ,ti )∈Te . The overall objective is e Ltotal = L+ (15) align T ; ω + αLrec + µLrisk + νLden , where α, µ, and ν control structure-oriented reconstruction, cross-domain risk balancing, and density estimation, respectively. 3.4

Theoretical Analysis

We provide a theoretical justification for the risk balancing mechanism from the perspective of weighted risk equalization. Invariant learning methods [1, 23, 13] reduce out-of-domain generalization error by encouraging different source domains to share similar risks. However, when graph–text supervision is sparse, direct risk matching may still be biased by domain-specific structural distributions and unreliable text attributes. SCRB addresses this issue by using the global-domain density ratio to align shared structural support and using sparse reliability to suppress unreliable samples. Theorem 3.1 (Sparse-aware weighted risk equalization). Let D1 , . . . , DK be K source domains. For a graph–text pair (g, t) ∼ Pe , let z = ϕ(g) denote its structural key, where ϕ(·) corresponds to the graph-side structural representation used in Eq. (9). Let pe (z) be the marginal density of z in domain De , and let K X p0 (z) p0 (z) = πe pe (z), re (z) = , (16) pe (z) e=1 6

Table 1: Zero-shot node classification accuracy (%) on unseen target datasets under the 10% text sparsity setting. Results are averaged over five random seeds. Academic

Method

E-commerce

Web

Avg.

Cora

CiteSeer

Books-History

Ele-Computers

Ele-Photo

WikiCS

Instagram

SBERT Qwen3-0.6B

59.67 ± 1.76 41.48 ± 1.66

66.93 ± 1.82 51.13 ± 0.95

41.18 ± 0.56 39.31 ± 0.61

41.01 ± 0.26 12.35 ± 0.09

38.36 ± 0.44 32.87 ± 0.40

60.87 ± 0.01 40.94 ± 0.01

59.91 ± 0.73 64.47 ± 0.63

52.56 40.36

GraphGPT LLaGA OFA ZeroG

14.45±0.37 30.15±1.88 8.09±0.01 53.47±1.19

18.38±0.32 35.72±1.01 20.53±0.01 50.16±0.01

55.58±0.03 45.48±0.92 7.86±0.01 35.69±0.49

17.29±0.06 28.36±0.85 7.40±0.07 33.79±0.18

40.89±0.02 25.91±0.47 28.45±0.07 36.23±0.50

4.42±0.05 30.27±0.56 12.74±0.07 51.29±0.01

45.94±0.28 48.63±1.14 41.64±0.14 52.56±0.67

28.14 34.93 18.10 44.74

DGI GRACE BGRL GraphMAE G2P2

27.36±1.08 34.90±0.51 13.52±1.74 11.29±3.89 29.78±3.04

33.27±0.15 28.69±1.98 17.41±2.90 16.77±4.86 34.85±6.09

13.39±0.33 24.42±1.86 6.85±2.86 11.72±22.27 17.28±3.68

13.86±0.70 20.07±1.27 13.45±7.96 6.64±5.68 20.25±4.39

12.49±0.64 21.56±0.99 7.95±4.05 11.05±15.36 20.31±5.86

46.85±3.42 55.93±0.99 9.42±6.41 9.82±3.96 28.32±3.84

58.63±0.16 46.75±2.68 46.13±8.49 63.71±0.01 53.47±3.31

29.41 33.19 16.39 18.71 29.18

ADAligner GraphCLIP S2 Aligner

60.51±1.64 57.68±0.70 66.73±1.40

64.52±1.77 63.73±1.91 67.25±2.32

51.94±1.70 49.79±3.02 56.00±1.01

50.45±1.24 59.09±1.84 60.18±2.89

41.06±0.92 36.78±2.73 44.01±1.09

57.65±3.01 58.58±3.78 62.29±3.61

54.84±1.57 60.48±2.03 62.71±0.21

54.42 55.16 59.88

PK where πe > 0 and e=1 πe = 1. Assume that all domains share the same support over z, and that the conditional reliability-weighted loss is invariant across domains: E(g,t)∼Pe [ρ(g, t)ℓ(g, t) | z] = m(z), ∀e ∈ {1, . . . , K}. (17) Then the density-ratio weighted risks are equal across all source domains: ⋆ R1⋆ = R2⋆ = · · · = RK , Re⋆ = E(g,t)∼Pe [re (z)ρ(g, t)ℓ(g, t)] . (18) Proof. The detailed proof is provided in Appendix D.1.

Experiments

4.1

Experimental Setup

Datasets. We evaluate S2 Aligner on text-attributed graph datasets from academic, e-commerce, social, and Wikipedia domains. Following the graph–text pre-training setting, ogbn-ArXiv [11], ArXiv_2023 [9], PubMed [29], ogbn-Products [11], and Reddit [12] are used as source datasets, while Cora [29], CiteSeer [29], Books-History [40], Ele-Computers [40], Ele-Photo [40], WikiCS [22], and Instagram [12] are used as unseen target datasets. To simulate sparse TAG scenarios, we retain only 1%, 3%, 5%, or 10% of node textual attributes during pre-training. Unless otherwise specified, experiments are conducted under the 10% sparsity setting. Baselines. We compare S2 Aligner with three groups of baselines: (1) LLM-only methods, including SBERT [26] and Qwen3-0.6B [41], which use textual information without explicitly modeling graph structure; (2) representative TAG methods, including GraphGPT [31], LLaGA [5], OFA [17], ZeroG [16], ADAligner [19], and GraphCLIP [43]; and (3) self-supervised graph methods applied to TAGs, including DGI [34], GRACE [42], BGRL [32], GraphMAE [10], and G2P2 [37]. Evaluation and Implementation. We evaluate zero-shot node classification, link prediction, and cross-modal retrieval. We report accuracy for node classification, AUC for link prediction, and MRR/Recall@K for retrieval. All results are averaged over five random seeds and reported as mean ± standard deviation. For S2 Aligner, we use a graph transformer-based encoder and a frozen pretrained language encoder. The trainable modules are optimized with graph–text contrastive learning, structure-oriented reconstruction, and reliability-aware robust training. 4.2

RQ1: Zero-Shot Inference on Target Data

We first evaluate whether S2 Aligner can generalize to unseen target graphs without task-specific fine-tuning. After pre-training on sparse source TAGs, the model is directly applied to target datasets for zero-shot inference. We consider three downstream tasks—node classification, link prediction, and cross-modal retrieval—which respectively assess semantic label prediction, topology-aware transfer, and fine-grained graph–text alignment. Node Classification. Table 1 reports the zero-shot node classification accuracy on seven target datasets under the 10% text sparsity pre-training setting. S2 Aligner achieves the best overall performance among all compared methods and shows consistent improvements over strong graph–text 7

Table 2: Zero-shot link prediction perfor- Table 3: Cross-modal retrieval results under globallevel and category-level settings. mance measured by AUC. Cora 83.11 ± 0.41 58.90 ± 0.28 84.65 ± 0.94 55.72 ± 3.86 68.91 ± 2.85 77.44 ± 2.32 91.06±0.09

Acad. E-com.

Method

MRR

R@1

R@5

R@10

Global

GraphCLIP N2T T2N S2 Aligner N2T T2N

Task

7.56±1.92 38.85±1.78 9.84±3.64 95.62±0.27

4.62±1.31 33.63±1.77 5.74±2.69 93.13±0.45

10.07±2.71 44.50±1.81 13.47±4.58 98.64±0.23

13.70±3.36 48.88±2.14 18.23±5.64 99.24±0.14

Category GraphCLIP N2T T2N S2 Aligner N2T T2N

19.75±2.45 44.33±1.64 20.99±3.94 96.79±0.23

12.09±2.16 38.21±1.64 12.60±3.35 94.89±0.26

26.79±2.96 50.78±1.97 28.92±4.91 99.05±0.34

35.59±3.41 48.88±2.14 38.17±5.16 99.49±0.16

Acad. E-com.

Web

Avg. Acc. (%)

GraphCLIP Time Ours Time

Ours-1% > GraphCLIP-10%

+6.2 +4.2

+4.0 +3.8

10%

Small 23M

Avg. Acc. (%)

62.0

+5.0

GraphCLIP Acc. Ours Acc.

Time / Epoch (min)

63.7

+1.8

49.6

Web

w/o Web +1.1 60.9

62.8

60.9

48.5

+5.6 42.9

59.4

+1.9

56.3

Acad. E-com.

61.3

65.2

62.5

Web

w/o E-commerce +3.5

+2.4 +5.8

62.4

Acad. E-com.

70.66 ± 0.23 63.97 ± 0.13 83.61 ± 0.12 76.54 ± 0.43 89.00 ± 2.98 70.45 ± 2.52 91.76±0.12

Setting

Ours

50.5

53.4

48.5

59.5

+3.0 +4.8

History

w/o Academic

+10.2

55.0

Full

67.0

60.7

55.9

Accuracy (%)

GraphCLIP +6.3

78.91 ± 1.49 59.82 ± 0.12 85.18 ± 0.14 69.02 ± 0.47 66.02 ± 6.46 73.33 ± 1.85 87.15±0.11

60.6

SBERT OFA ZeroG GraphMAE GraphCLIP ADAligner S2 Aligner

WikiCS

54.6

Method

67.0

Mid 110M

Large 0.6B

67.0 67.1

62.5

60.9

62.8

54.2 54.8 53.4

55 50 45

Acad.

Com.

Web

Figure 4: Effect of source- Figure 5: Performance-efficiency Figure 6: Effect of language endomain composition on zero- trade-off under varying text spar- coder scale on zero-shot transfer. shot transfer. sity levels. alignment baselines. Compared with GraphCLIP, S2 Aligner improves the average accuracy from 55.16% to 59.88%, demonstrating stronger transferability under sparse textual supervision. The gains are especially evident on Cora, CiteSeer, Books-History, and Ele-Photo. We further observe that our learned node representations better capture the semantic distinctions among nodes compared with GraphCLIP, for a visual comparison, see Appendix G.2. Link Prediction. We further evaluate zero-shot link prediction for transferable structural information. The pre-trained model is applied to target datasets without task-specific fine-tuning. For link prediction, we report average AUC scores and standard deviations over five runs with different random seeds. As shown in Table 2, S2 Aligner achieves strong zero-shot performance. On Cora, WikiCS, and History, it obtains the best AUC scores, outperforming baselines. Cross-modal Retrieval.We further evaluate cross-modal retrieval under both global-level and category-level settings, including node-to-text (N2T) and text-to-node (T2N) retrieval on the WikiCS dataset. As shown in Table 3, S2 Aligner consistently outperforms GraphCLIP across almost all retrieval metrics in both settings. The improvement is particularly significant for T2N retrieval, where S2 Aligner achieves much higher MRR and Recall scores, indicating that the learned text representations can more accurately retrieve the corresponding graph instances. Under the category-level setting, S2 Aligner also maintains consistent gains, suggesting that the proposed sparse structural alignment improves not only global graph–text matching but also fine-grained category-aware retrieval.

4.3

RQ2: Explorations on Source Datasets

To investigate how source-domain composition affects transferability, we compare S2 Aligner with GraphCLIP under different source dataset configurations. Following GraphCLIP, we consider four settings: using all source datasets, removing academic datasets, removing e-commerce datasets, and removing web datasets. We report the average target accuracy grouped by target domain, including academic, e-commerce, and web domains. As shown in Figure 4, S2 Aligner consistently outperforms GraphCLIP across all source-domain configurations and target-domain groups. The gains remain clear even when one source domain is removed, indicating that S2 Aligner does not simply rely on domain-specific shortcuts from the source data. Instead, the proposed sparsity-aware structural alignment encourages more transferable graph–text representations that remain effective under reduced source-domain diversity. 8

4.4

RQ3: Zero-Shot Robustness & Efficiency Experiment

Varying Text Sparsity: Performance and Efficiency. We further evaluate S2 Aligner under text sparsity levels of 1%, 3%, 5% and 10%, aiming to explore whether promising graph-text alignment can be preserved when only limited node textual attributes are accessible during pre-training. As illustrated in Figure 5, S2 Aligner steadily outperforms GraphCLIP under all sparsity configurations; notably, the model trained merely with 1% textual attributes achieves even better performance than GraphCLIP trained with 10% textual attributes, demonstrating superior sparsity generalization and higher data efficiency. Apart from consistent performance gains, our method also achieves substantial computational acceleration.The proposed approach considerably reduces per-epoch training overhead, yielding markedly lower runtime cost compared with GraphCLIP. These observations confirm that our framework delivers competitive zero-shot transfer performance with limited textual attributes and lower computational overhead, making it well-suited for large-scale sparse text-attributed graph scenarios. Detailed time and space complexity analysis of S2 Aligner is provided in the Appendix H. Varying Language Model Scale. We evaluate the effect of language encoder scale by replacing the text encoder with MiniLM-L6 [36], E5 [35], and Qwen3-0.6B [41] (small, medium, and large). As shown in Figure 6, performance differences are minor, with the small encoder already competitive. While the large encoder slightly improves some domains, overall gains are limited, indicating that S2 Aligner’s transferability is primarily driven by its sparsity-aware, structure-enhanced alignment rather than encoder scale. Hyperparameter Sensitivity. S2 Aligner shows stable performance across a wide range of hyperparameter settings, including α (structural reconstruction), µ (cross-domain risk balance), and ν (density estimation), consistently outperforming GraphCLIP (detailed results in Appendix G.1). We also report the fixed choice of the three reliability factors, λd , λa , and λc . RQ4: Ablation Study

As shown in Figure 7, removing any key component leads to performance degradation, confirming that all modules contribute to the final results. Specifically, w/o Structural Reconstruction drops notably on the e-commerce and Web domains, highlighting the importance of topology-aware auxiliary supervision under sparse textual conditions; w/o Risk Balancing also reduces performance, indicating that sparseaware cross-domain risk balancing helps mitigate domainbiased alignment and improve transferability; w/o Summary consistently degrades performance, particularly on the academic domain, demonstrating that semantic summaries are essential for stable graph–text alignment.

)8// ZR6WUXFW5HF 70

Accuracy(%)

4.5

ZR5LVN%DODQFLQJ ZR6XPPDU\

45 40

Acad.

Com.

Web

Figure 7: The study results.

Related Work

Graph–Text Alignment. Inspired by CLIP [24], contrastive dual-encoder frameworks have become the dominant paradigm for graph–text alignment on text-attributed graphs. Methods such as GraphCLIP [43], G2P2 [37], and GRENADE [15] construct graph-text positive pairs and map them into a shared space. However, they rely on fixed one-to-one alignment, limiting their ability to capture many-to-many semantic relations. ConGraT [2] introduces graph similarity to generate soft alignment targets, and ADAligner [19] dynamically balances one-to-one and many-to-many objectives based on batch-level data quality. These approaches, however, assume reliable node texts and summaries; in sparse scenarios, missing textual attributes weaken semantic evidence and may mislead supervision. Cross-domain Generalization. Cross-domain generalization aims to learn models from multiple source domains that can transfer to unseen target domains. Its core objective is to reduce the model’s reliance on domain-specific correlations and capture stable predictive mechanisms across domains. Existing studies mitigate distribution shifts from different perspectives, including domainadversarial learning [8], distribution alignment [20, 30], invariant risk minimization [1], risk variance regularization [13], and distributionally robust optimization [28]. In graph learning, recent works have further investigated OOD generalization on graphs. For example, GIL [14] captures stable structural information through invariant subgraph learning, while EERM [39] improves graph OOD 9

generalization through environment exploration. However, sparse graph–text pre-training faces a more complex scenario, where textual attribute missingness, structural variation, and source-domain distribution shifts jointly affect graph–text alignment.

Conclusion

This paper studied graph–text pre-training on sparse text-attributed graphs and presented S2 Aligner, a sparsity-aware and structure-enhanced LLM-as-Aligner framework. S2 Aligner disentangled content and structural representations, introduced structure-oriented reconstruction, and applied sparse-aware cross-domain risk balancing to improve robust graph–text alignment. Experiments on zero-shot node classification, link prediction, and cross-modal retrieval demonstrated that S2 Aligner achieves strong transferability across sparse settings and unseen target graphs.

References [1] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019. [2] William Brannon, Wonjune Kang, Suyash Fulay, Hang Jiang, Brandon Roy, Deb Roy, and Jad Kabbara. ConGraT: Self-supervised contrastive pretraining for joint graph and text embeddings. In Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing, pages 19–39, Bangkok, Thailand, August 2024. Association for Computational Linguistics. [3] Hongxu Chen, Hongzhi Yin, Xiangguo Sun, Tong Chen, Bogdan Gabrys, and Katarzyna Musial. Multi-level graph convolutional networks for cross-platform anchor link prediction. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1503–1511, 2020. [4] Junru Chen, Yang Yang, Tao Yu, Yingying Fan, Xiaolong Mo, and Carl Yang. Brainnet: Epileptic wave detection from seeg with hierarchical graph diffusion learning. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 2741–2751, 2022. [5] Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. Llaga: Large language and graph assistant. arXiv preprint arXiv:2402.08170, 2024. [6] Zhikai Chen, Haitao Mao, Jingzhe Liu, Yu Song, Bingheng Li, Wei Jin, Bahare Fatemi, Anton Tsitsulin, Bryan Perozzi, Hui Liu, et al. Text-space graph foundation models: Comprehensive benchmarks and new insights. Advances in Neural Information Processing Systems, 37:7464– 7492, 2024. [7] Jiarui Feng, Hao Liu, Lecheng Kong, Mingfang Zhu, Yixin Chen, and Muhan Zhang. Taglas: An atlas of text-attributed graph datasets in the era of large graph and language models. arXiv preprint arXiv:2406.14683, 2024. [8] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of machine learning research, 17(59):1–35, 2016. [9] Xiaoxin He, Xavier Bresson, Thomas Laurent, and Bryan Hooi. Explanations as features: Llm-based features for text-attributed graphs. CoRR, abs/2305.19523, 2023. [10] Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 594–604, 2022. [11] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118–22133, 2020. [12] Xuanwen Huang, Kaiqiao Han, Yang Yang, Dezheng Bao, Quanjin Tao, Ziwei Chai, and Qi Zhu. Can gnn be good adapter for llms? In Proceedings of the ACM web conference 2024, pages 893–904, 2024. 10

[13] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International conference on machine learning, pages 5815–5826. PMLR, 2021. [14] Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35:11828–11841, 2022. [15] Yichuan Li, Kaize Ding, and Kyumin Lee. GRENADE: Graph-centric language model for self-supervised representation learning on text-attributed graphs. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2745–2757, Singapore, December 2023. Association for Computational Linguistics. [16] Yuhan Li, Peisong Wang, Zhixun Li, Jeffrey Xu Yu, and Jia Li. Zerog: Investigating crossdataset zero-shot transferability in graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1725–1735, 2024. [17] Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. arXiv preprint arXiv:2310.00149, 2023. [18] Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Yuan Fang, Lichao Sun, Philip S. Yu, and Chuan Shi. Towards graph foundation models: A survey and beyond. CoRR, abs/2310.11829, 2023. [19] Yuhang Liu, Minglai Shao, Zengyi Wo, Yunlong Chu, Bing Hao, Shengzhong Liu, Ruijie Wang, and Jianxin Li. Learning noise-resilient and transferable graph-text alignment via dynamic quality assessment. arXiv preprint arXiv:2510.19384, 2025. [20] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97–105. PMLR, 2015. [21] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [22] Péter Mernyei and Cătălina Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901, 2020. [23] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):947–1012, 2016. [24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. [25] Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35:14501–14515, 2022. [26] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLPIJCNLP), pages 3982–3992, 2019. [27] Xubin Ren, Jiabin Tang, Dawei Yin, Nitesh Chawla, and Chao Huang. A survey of large language models for graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6616–6626, 2024. 11

[28] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731, 2019. [29] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine, 29(3):93–93, 2008. [30] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 443–450. Springer, 2016. [31] Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 491–500, 2024. [32] Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Veličković, and Michal Valko. Bootstrapped representation learning on graphs. In ICLR 2021 workshop on geometrical and topological representation learning, pages 1–14. OpenReview. net, 2021. [33] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605, 2008. [34] Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. stat, 1050:21, 2018. [35] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022. [36] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776–5788, 2020. [37] Zhihao Wen and Yuan Fang. Augmenting low-resource text classification with graph-grounded pre-training and prompting. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 506–516, 2023. [38] Gina Wong, Joshua Gleason, Rama Chellappa, Yoav Wald, and Anqi Liu. Weighted risk invariance: Domain generalization under invariant feature shift. arXiv preprint arXiv:2407.18428, 2024. [39] Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. arXiv preprint arXiv:2202.02466, 2022. [40] Hao Yan, Chaozhuo Li, Ruosong Long, Chao Yan, Jianan Zhao, Wenwen Zhuang, Jun Yin, Peiyan Zhang, Weihao Han, Hao Sun, et al. A comprehensive study on text-attributed graphs: Benchmarking and rethinking. Advances in Neural Information Processing Systems, 36:17238– 17264, 2023. [41] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [42] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131, 2020. [43] Yun Zhu, Haizhou Shi, Xiaotang Wang, Yongchao Liu, Yaoke Wang, Boci Peng, Chuntao Hong, and Siliang Tang. Graphclip: Enhancing transferability in graph foundation models for text-attributed graphs. In Proceedings of the ACM on Web Conference 2025, pages 2183–2197, 2025.

Appendix Overview

The appendix is structured as follows: • Section B discusses limitations and future work. • Section C provides the broader impact statement. • Section D presents the theoretical analysis of the proposed global-domain density ratio weighting. • Section E details the experimental setup, including datasets and baselines. • Section F provides implementation details of S2 Aligner, including pre-training settings and zeroshot prompts. • Section G reports additional experimental results, including hyperparameter sensitivity and visualization analysis. • Section H analyzes the time and space complexity of S2 Aligner.

Limitations and Future Work

Despite the effectiveness of our proposed S2 Aligner in capturing topological cues and facilitating text-graph alignment via the structure reconstruction consistency constraint, the current structural enhancement paradigm still performs optimally on sparsely text-attributed graphs with regular and clear topological connections. When facing highly fragmented graph structures with numerous isolated nodes and ambiguous topological relationships, the useful structural cues can be easily contaminated by redundant noises, leading to the dilution of structural enhancement signals and thus degenerating the robustness of cross-modal alignment. To mitigate this limitation, future work will explore adaptive topology-aware modeling and structure denoising strategies to capture latent connections in fragmented graph structures, and dynamically adjust the strength of structural constraints according to the degree of fragmentation, thereby improving the alignment performance on disordered and complex topologies.

Impact Statements

This work contributes to the development of graph–text pre-training by improving robust graph– text alignment on sparse text-attributed graphs. The proposed framework can be applied to lowresource graph–text scenarios, such as academic network analysis, product graph understanding, recommendation systems, and social information analysis. It may help reduce the reliance on largescale manually annotated data and improve the usability and generalization ability of graph–text models in practical applications. Since this work mainly focuses on methodological and technical improvements, and the datasets used in our experiments do not involve sensitive personal information or privacy risks, we do not expect the proposed method to have obvious negative societal impacts.

Theoretical Analysis

D.1

Proof of Theorem 3.1

This proof follows the standard paradigm of weighted risk invariance and general position in invariant learning [38]. In contrast to the pairwise weighting strategy that uses a single reference domain, we propose a global-domain density ratio weighting derived from a unified global mixture distribution, which serves as an unbiased and stable anchor for multi-domain generalization. Notation • Z: Shared invariant structural component across all domains. • Ve : Domain-private spurious component of the e-th domain. • pe (Z): Marginal density of invariant feature Z in domain e. P • p0 (Z) = e πe pe (Z): Global mixture distribution. 13

• Our global weighting function: re (Z) ≜

p0 (Z) . pe (Z)

• Linear predictor: f (Z, Ve ) = w⊤ Z ⊤

Ve⊤

⊤ w = winv

⊤

• Parameter decomposition: ⊤ wspu

where winv corresponds to invariant features Z, and wspu corresponds to domain-private spurious features Ve . Assumptions

Let E denote the set of training domains. We assume:

1. Feature decomposition: inputs consist of invariant Z and domain-specific Ve . 2. Conditional independence: Y ⊥ ⊥ Ve | Z. 3. Invariant label condition: pei (Y | Z) = pej (Y | Z) for all ei , ej ∈ E. 4. Non-degenerate density: pe (Z) > 0 for all e ∈ E. Proposition 1 If the assumptions hold and the predictor f depends only on Z, then the weighted risks are equal across all domains. Proof. The weighted risk for domain e is: ZZZ Re = pe (Z, Ve , Y ) · ℓ(f (Z), Y ) · re (Z) dVe dZdY. YZVe

Marginalizing out Ve : ZZ ℓ(f (Z), Y ) · re (Z) · pe (Z, Y ) dZdY.

Re = YZ

Substituting pe (Z, Y ) = pe (Y | Z)pe (Z) and re (Z) = p0 (Z)/pe (Z): ZZ Re = ℓ(f (Z), Y ) · p0 (Z) · pe (Y | Z) dZdY. YZ

Since p0 (Z) and pe (Y | Z) are domain-agnostic, Re is identical for all e ∈ E. Let k be the number of domains such that Σi,j = EPi [Z, Ve ][Z, Ve ]⊤ · ri (Z) ,

Definition 1 (General Position)

k 2

> 2dspu . Define:

µi,j = 2 · EPi [[Z, Ve ]Y · ri (Z)] . Domains are in general position if for any scalar γ ∈ R and nonzero x ∈ Rdspu : dim span{(Σi,j − Σj,i )x + γ(µi,j − µj,i )x}i,j∈[k] = dspu . This definition follows the standard formulation in invariant learning [1, 38]. Theorem 1 (Elimination of Spurious Features) Under linear regression setting, if |E| > dspu and domains are in general position, then any linear predictor satisfying global weighted risk invariance must have: wspu = 0. Our global-domain density ratio weighting satisfies general position with probability 1. 14

Proof. For any two domains, risk invariance gives: ZZZ 2 w⊤ [Z, Ve ] − Y p1 r1 dVe dZdY ZZZ 2 = w⊤ [Z, Ve ] − Y p2 r2 dVe dZdY. Expanding leads to: w⊤ Σ12 w − w⊤ µ12 + c12 = w⊤ Σ21 w − w⊤ µ21 + c21 . By re (Z)pe (Z) = p0 (Z), invariant terms cancel, leaving constraints only on wspu . Under general position, the system has full rank, so wspu = 0. Following the measure-theoretic property for density ratio weights [38], our global weighting satisfies general position with probability 1.

Experimental Setup Details

E.1

Datasets

To comprehensively assess the transfer capabilities and sparsity tolerance of our S2 Aligner, we conduct cross-domain evaluations following the zero-shot protocol established by [43]. The empirical study incorporates 12 distinct Text-Attributed Graphs (TAGs). These networks are partitioned into a set of 5 source domains for the pre-training phase, and 7 target domains for downstream verification. Source Datasets (Pre-training): The pre-training corpus encompasses major online scenarios including academic literature, e-commerce retail, and social interactions. We adopt the complete topology for both academic and social networks, alongside a sampled subset of the retail graph to maintain domain balance. • ogbn-ArXiv [11]: Constructed from the Microsoft Academic Graph, this dataset forms a directed network mapping citations between computer science manuscripts. We utilize the available title and abstract information as raw features for each document. The downstream objective entails assigning these documents into 40 distinct topical categories. • ArXiv-2023 [9]: Acting as a temporally updated counterpart to the aforementioned dataset, this collection captures recent publication citations from the year 2023 onwards. Models are similarly tasked with predicting the correct subject class out of 40 possible options for every scholarly article. • PubMed [29]: This graph organizes medical literature concerning diabetes research. Node entities must be classified into three specific clinical domains, separating experimental methodologies from Type 1 and Type 2 diabetes studies. • ogbn-Products [11]: Sourced from Amazon’s product listings, this network links items frequently bought together by consumers. Algorithms must accurately predict which of the 47 high-level retail categories a specific merchandise belongs to. • Reddit [12]: This platform-derived graph captures online interactions, connecting individuals who have engaged in mutual conversation threads. Textual attributes originate from past user posts, and the core task involves predicting the popularity metric of the accounts. Sparsity Setting: To properly mimic the prevalent issue of missing information in real-world environments, we intentionally drop node attributes during the training stage. Under the default 10% sparsity configuration, algorithms are only granted access to the raw texts of a minor fraction (10%) of the total entities. The remaining vertices are strictly initialized with zero-vectors and ”unknown” text flags, forcing the framework to heavily rely on topological propagation to infer absent semantics. Target Datasets (Downstream Evaluation): We verify the robust transferability of our model using 7 varied networks. During this phase, zero-shot inference is strictly conducted without applying any gradient updates on the target structures. For clarity, we categorize these datasets into three groups: academic citation networks (Cora, CiteSeer), product networks (Amazon-Computers, Amazon-Photo, Amazon-History), and web networks (WikiCS, Instagram). During this phase, zero-shot inference is strictly conducted without applying any gradient updates on the target structures. 15

• Cora [29]: A widely adopted academic benchmark containing 2,708 machine learning papers. References connect the manuscripts, and the model must identify one of seven corresponding research subfields. • CiteSeer [29]: Containing 3,186 documents spanning six computer science domains, this collection challenges the model to infer the correct research category by analyzing abstracts alongside paper titles. • WikiCS [22]: Hyperlinks connect various computer science articles from Wikipedia to construct this structure. Algorithms leverage the main article texts to assign nodes into ten respective academic branches. • Amazon-Computers and Amazon-Photo [40]: Both datasets emerge from Amazon’s electronics inventory, where topological edges capture co-viewing or co-purchasing behaviors. Textual features incorporate highly-rated buyer reviews, demanding models to sort items into 10 and 12 distinct classes, respectively. • Amazon-History [40]: Extracted specifically from the books category, relationships here also reflect consumer co-purchasing trends. We rely on book summaries and titles to predict a 12-way thematic classification. • Instagram [12]: Nodes designate accounts on this social media platform, with following behaviors acting as structural edges. The core objective demands differentiating standard personal profiles from commercial entities. E.2

Baselines

To validate the effectiveness of S2 Aligner under extreme attribute scarcity, we conduct extensive comparisons against a robust suite of competitive algorithms. These existing solutions fall into four primary paradigms: 1. Text-only Language Models: These approaches independently process the raw sentences of each entity while completely ignoring topological connections. • SBERT [26]: We incorporate standard sentence embedding architectures to produce dense semantic representations, specifically evaluating the all-MiniLM-L6-v2 and multi-qa-distilbert-cos-v1 variants. • Qwen3 (0.6B) [41]: A highly capable large language model acting as a robust generative baseline, tested on its ability to infer node labels using solely isolated contextual texts. 2. Text-Attributed Graph (TAG) & LLM-based Methods: These recent strategies strive to fuse structural patterns with the semantic comprehension capabilities of language models. • GraphGPT [31]: This architecture maps topological properties into discrete tokens and employs a dual-stage instruction fine-tuning process to synchronize GNN outputs with an LLM’s semantic space. • LLaGA [5]: By restructuring graph components into sequential prompts, this framework projects network topologies directly into the embedding layers of language models for generalized processing. • OFA [17]: This prompting strategy textualizes diverse topological properties—including edges and neighbors—allowing a single LLM to handle varied graph-based tasks via natural language comprehension. • ZeroG [16]: To enable cross-domain generalization, this method converts structural tasks into text matching problems by integrating lightweight adapters alongside prompt-driven subgraph sampling. 3. Graph Self-Supervised Learning (SSL) Models: Focusing primarily on topology and dense features, these methodologies leverage traditional graph neural networks. • DGI [34]: A foundational self-supervised strategy that maximizes mutual information by distinguishing authentic node-graph representations from artificially corrupted counterparts. 16

• GRACE [42]: This contrastive learning approach generates augmented views of the same network, pulling identical node embeddings closer while pushing disparate nodes apart. • BGRL [32]: Drawing inspiration from BYOL architectures, this framework trains an online encoder against a momentum-updated target network, eliminating the necessity for negative sample pairs. • GraphMAE [10]: Functioning as a graph autoencoder, it deliberately conceals certain node attributes during the encoding phase and trains the decoder to accurately reconstruct those hidden features. • G2P2 [37]: To improve performance in data-scarce scenarios, this technique blends structural pre-training signals with prompt-based text classification mechanisms. 4. State-of-the-Art Graph-Text Aligners: Serving as our primary zero-shot competitors, these methods focus explicitly on synchronizing semantic and structural spaces. • GraphCLIP [43]: This framework relies heavily on contrastive alignment objectives. It synthesizes subgraph summaries to align embedding spaces, providing strong zero-shot graph–text alignment performance. • ADAligner [19]: ADAligner also employs contrastive alignment but integrates dynamic quality assessments to filter noisy signals during the alignment procedure, enhancing robustness against unreliable subgraph-text pairs.

Implementation Details of S2 Aligner

In this section, we provide the implementation details of S2 Aligner. We first describe the experimental setup during the pre-training phase, and then present the prompts used for zero-shot learning. F.1

Pre-training Phase

For S2 Aligner, only a small number of hyperparameters need to be tuned. In our main experiments, we adopt the AdamW [21] optimizer, with both the learning rate and weight decay set to 1 × 10−5 . The graph encoder is implemented with GraphGPS [25], which consists of 12 layers with a hidden dimension of 1024. For the text encoder, we use a fine-tuned MiniLM [36] model with 6 layers and a hidden dimension of 384. To align graph and text representations into a unified semantic space, we employ a projector to transform the 1024-dimensional graph representations into the same 384-dimensional space as the text representations. During pre-training, we optimize only the parameters of the graph encoder and the projector, while keeping the text encoder frozen to reduce training costs and mitigate catastrophic forgetting. The pre-training process is conducted for 10 epochs on an H100-80G GPU. F.2

Zero-shot Learning

Zero-shot Node Classification. For zero-shot learning, we follow the same prompt construction strategy as the baseline GraphCLIP. Specifically, label information is incorporated into label-specific sentences to maintain consistency with the graph–text alignment format used during pre-training. Table 4 presents the prompts designed for different datasets, where {class} denotes the label text of the target node, and {class_desc} denotes the descriptive sentence generated by a large language model to further elaborate the semantic meaning of the label. Link Prediction. For link prediction, the model computes the similarity between the source node embedding and the target node embedding. An edge is predicted to exist if the similarity score exceeds a threshold of 0.5. We evaluate the predictions using standard metrics such as AUC, F1, Precision, and Recall to quantify how well the predicted edges match the ground-truth connections in the graph. Node-Text Retrieval. For node-text retrieval, we perform both Node-to-Text and Text-to-Node tasks. - In Node-to-Text retrieval, each node embedding is used to rank all candidate text embeddings according to cosine similarity, and the top-k results are returned as predictions. - In Text-to-Node retrieval, each text embedding is used to rank all candidate node embeddings similarly. Metrics such 17

Table 4: Prompts used for zero-shot learning of S2 Aligner. Dataset Prompt Cora CiteSeer WikiCS Instagram Ele-Photo Computers History

this paper has a topic on {class} {class_desc} good paper of {class} {class_desc} it belongs to {class} research area {class_desc} {class} {class_desc} this product belongs to {class} {class_desc} is {class} category {class_desc} this book belongs to {class} {class_desc}

as Mean Reciprocal Rank (MRR) and Recall@k are reported to evaluate retrieval quality and the model’s ability to semantically align nodes with text.

Additional Results for Experiments

G.1

Hyperparameter Sensitivity Analysis

58.74

2.0

(a) α sensitivity

57.95

3.0

65.0 62.5 59.12 59.85 60.0 57.5 57.12 55.0 52.5 50.0 0.1 0.5 1.0

57.26

57.27

2.0

3.0

(b) µ sensitivity

Accuracy (%)

65.0 62.5 59.85 60.0 58.01 58.15 57.5 55.0 52.5 50.0 0.1 0.5 1.0

Accuracy (%)

We evaluate the sensitivity of our key hyperparameters α, µ and ν, which control the weights of structural reconstruction loss, cross-domain risk balance loss and density estimation loss, respectively. We vary each hyperparameter in the range {0.1, 0.5, 1.0, 2.0, 3.0} and report the results with standard deviation. 65.0 62.5 60.0 59.31 59.85 57.91 57.5 55.0 52.5 50.0 0.1 0.5 1.0

58.12

2.0

56.12

3.0

Figure 8: Hyperparameter sensitivity analysis of α, µ and ν. Fig. 8(a), Fig. 8(b) and Fig. 8(c) illustrate the sensitivity curves of α, µ and ν, respectively. It is observed that our method consistently surpasses the strong baseline GraphCLIP across all parameter configurations, demonstrating the superiority and robustness of our framework. The model achieves optimal performance at α = 1.0, µ = 1.0 and ν = 0.5. Either excessively small or large hyperparameters lead to performance degradation, since insufficient constraint or over-weighted loss breaks the representation learning balance. In addition, our method shows stable performance and small variance under different settings, which verifies that our model is insensitive to hyperparameter tuning and has good practicality. Similarly, the reliability coefficients λd , λa , λc used in computing the sparse reliability score ρi are all set to 0.5. Since the three factors—structural stability di , textual completeness ai , and structure–semantic consistency ci —are normalized and inherently comparable, equal weighting is a natural choice. This simple setting maintains interpretability and avoids introducing additional hyperparameters, while not affecting the reliability score’s ability to reflect sample quality. G.2

Visualization

As a supplementary study of model effectiveness, we visualize both node and class representations using our model S2 Aligner and the state-of-the-art baseline GraphCLIP. Specifically, we employ t-SNE [33] to map the Cora representations into two-dimensional vectors for visualization. Figure 9 shows that, compared to GraphCLIP, equipping our method S2 Aligner results in: (1) nodes of the same class (i.e., colors in the visualization) forming more cohesive clusters in the embedding space, and (2) node representations from different classes being more discriminative. 18

GraphCLIP

-10

-20

-30

-30 -20

-10

-40 -30

RuL

Ours

-40 -30

-20

-10

Figure 9: Embedding visualization of Cora. Circles (•) represent nodes, while stars (⋆) represent classes.

Complexity Analysis

In this section, we present the time and space complexity analysis of S2 Aligner. Considering that S2 Aligner is a self-supervised graph-text pre-training framework, we compare its complexity with other self-supervised graph learning methods. H.1

Time Complexity

The primary time overhead arises from four components: the graph encoder, the text encoder, the structure-oriented reconstruction module, the Sparse-aware Cross-domain Risk Balancing mechanism, and computing the pre-training loss. For simplicity, we assume the layer count and hidden size of the text encoder are the same as those of the graph encoder. The time complexity of the graph encoder is O(LN 2 D + LN D2 ), and similarly, the time complexity of the text encoder is O(LN 2 D + LN D2 ). The structure-oriented reconstruction and the alignment loss both operate on individual node-text pairs, leading to a time complexity of O(N D). The Sparseaware Cross-domain Risk Balancing mechanism, including density estimation, reliability weighting, and cross-domain risk balancing, involves only lightweight feed-forward networks and sample-wise operations, resulting in a time complexity of O(N D). The pre-training loss computation has a time complexity of O(N 2 D). Thus, the total time complexity of S2 Aligner is O(LN 2 D + LN D2 ), ignoring smaller terms, which is of the same order as GraphCLIP [43], O(LN 2 D). H.2

Space Complexity

Each layer of the graph encoder has a space complexity of O(N D + D2 ) for computing queries, keys, and values. The attention score calculation then incurs a space complexity of O(N 2 + N D), while obtaining the hidden states results in a per-layer space complexity of O(N 2 +N D +D2 ). Performing these operations across all layers leads to a cumulative space complexity of O(LN 2 + LN D + LD2 ). Similarly, the text encoder has a space complexity of O(LN 2 + LN D + LD2 ). The reconstruction head, the density networks, and the reliability scoring module add negligible additional space complexity of O(N D). Finally, the contrastive and risk-balancing losses add an additional space complexity of O(N 2 ). Consequently, the overall space complexity of S2 Aligner amounts to O(LN 2 + LN D + LD2 ), which is of the same order as GraphCLIP [43], O(LN 2 + LN D + LD2 ).

Record · ID 200508 · SHA-256 fa8a8df2ba329890
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.