Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphs Hussein Abdallah1 , Ibrahim Abdelaziz2 , Panos Kalnis3 and Essam Mansour1 1 Concordia University, 2 IBM, 3 KAUST [email protected], [email protected], [email protected], [email protected]
Abstract
arXiv:2604.13979v1 [cs.CL] 15 Apr 2026
Open-world Question Answering (OW-QA) over knowledge graphs (KGs) aims to answer questions over incomplete or evolving KGs. Traditional KGQA assumes a closed world where answers must exist in the KG, limiting real-world applicability. In contrast, openworld QA requires inferring missing knowledge based on graph structure and context. Large language models (LLMs) excel at language understanding but lack structured reasoning. Graph neural networks (GNNs) model graph topology but struggle with semantic interpretation. Existing systems integrate LLMs with GNNs or graph retrievers. Some support open-world QA but rely on structural embeddings without semantic grounding. Most assume observed paths or complete graphs, making them unreliable under missing links or multi-hop reasoning. We present GLOW, a hybrid system that combines a pre-trained GNN and an LLM for open-world KGQA. The GNN predicts top-k candidate answers from the graph structure. These, along with relevant KG facts, are serialized into a structured prompt (e.g., triples and candidates) to guide the LLM’s reasoning. This enables joint reasoning over symbolic and semantic signals, without relying on retrieval or fine-tuning. To evaluate generalization, we introduce GLOW-B ENCH, a 1,000-question benchmark over incomplete KGs across diverse domains. GLOW outperforms existing LLM–GNN systems on standard benchmarks and GLOW-B ENCH, achieving up to 53.3% and an average 38% improvement. GitHub code and data are available here.
1
Introduction
Open-World Question Answering (OW-QA) over knowledge graphs (KGs) aims to answer questions when relevant facts are missing or the KG is incomplete. This challenge arises in real-world domains like biomedicine, scientific research, and finance, where knowledge is often evolving, implicit, or in-
complete (Lu and Yang, 2022; Wang et al., 2024). For example, systems may need to infer undocumented drug interactions, latent collaborations, or causal relationships. In such cases, answers are not explicitly stored in the KG and must be predicted based on graph structure, entity semantics, and cues in the question. Traditional KGQA methods assume a closedworld setting, where all facts are known and retrievable. This limits their use in dynamic or incomplete environments. Open-world QA instead requires reasoning over both observed and missing information. Unlike standard retrieval or link prediction, it must integrate symbolic language with structural graph signals. This motivates hybrid approaches that combine the semantic flexibility of Large Language Models (LLMs) with the relational reasoning of Graph Neural Networks (GNNs). Most existing KGQA systems rely on structured or dense retrieval to find explicit paths between question and answer entities. Methods like G-Retriever (He et al., 2024), GNN-RAG (Mavromatis and Karypis, 2024), RoG (Luo et al., 2024), and ToG (Sun et al., 2024) retrieve graph fragments using semantic similarity. GCR (Luo et al., 2025) improves scalability but still assumes a complete graph. These methods often fail under open-world conditions, where answer paths are missing or poorly aligned with the question. This results in noisy retrieval and low accuracy. OW-QA instead requires predictive reasoning to infer plausible answers beyond observed facts. AskGNN (Hu et al., 2024) addresses this by using GNNs to guide LLM inference, but relies heavily on structural embeddings and lacks semantic grounding. This limits its ability to handle complex or multi-hop questions. To address this challenge, we propose GLOW, a novel hybrid approach that synergistically combines the strengths of both LLMs and GNNs. Our method uses a pre-trained GNN to predict the top-k possible answers based on the graph
Table 1: Average accuracy (%) on OW-QA benchmarks by reasoning depth. GLOW-GN leads on both 1- and 2-hop questions, while GCR drops sharply under OWA. All use Qwen3-8B; scores for existing datasets are averaged over arxiv2023, ogbn-arxiv, and ogbn-products.
Method LLMOnly GCR GoG AskGNN GLOW-GN
Existing Datasets 1-Hop 25.7 7.8 44.6 53.7 71.7
GLOW-Bench (ours) 1-Hop 30.0 34.1 48.3 79.4 83.3
2-Hop 18.9 15.3 29.5 34.3 42.4
or returning incorrect answers. GoG hallucinates answer path generation as it overlooks the underlying KG schema and structural constraints, resulting in semantically inconsistent or invalid paths. Our benchmark highlights these issues and shows the need for models that can reason jointly over symbolic and structural signals. In summary, our contributions are: • We propose GLOW, a novel OW-KGQA system that combines a GNN and an LLM to jointly reason over structured and unstructured knowledge. • We present GLOW-B ENCH, a 1,000-question benchmark for open-world KGQA with multi-hop reasoning over incomplete, cross-domain KGs.
topology. These answers, along with a serialized • We demonstrate that GLOW outperforms state-ofsubgraph of KG facts relevant to the question, are the-art LLM–GNN QA and KGQA systems across then incorporated into the LLM prompt using a standard benchmarks and GLOW-B ENCH, with controlled and structured format (e.g., relational up to 53.3% and an average of 38% improvement. triples, top-ranked candidate answers), enabling 2 Related Work the LLM to reason jointly over language and structure. Rather than relying on retrieval or Our work connects to research in KG completion, pretraining alone, our model dynamically bridges LLM-based QA, GNN–LLM hybrid models, and gaps in the KG by augmenting the LLM with open-world KGQA. on-the-fly, GNN-driven prompts. Knowledge Graph Completion. EmbeddingWe evaluate GLOW on the AskGNN (Hu based methods (TransE, RotatE, ComplEx) (Rossi et al., 2024) open-world QA benchmark, which is et al., 2021a) and recent semantic models (Strucbased on ogbn-arxiv, ogbn-products, and arxiv2023 turE, HopfE, DensE) (Ge et al., 2024) infer datasets. AskGNN used these datasets to reflect missing links via latent representations. Rulerealistic KG incompleteness and avoid the closed- based systems (AnyBURL (Rossi et al., 2021b), world assumption in existing benchmarks, such as SAFRAN (Ott et al., 2021)) generalize KG patterns WebQSP or CWQ (Luo et al., 2025). To test gen- with logical rules. While useful for KG augmeneralization, we introduce GLOW-B ENCH, a new tation, these approaches operate independently of benchmark of 1,000 natural language questions QA. CBR-iKB applies case-based reasoning with across diverse domains. Each question requires KGEs but is computationally expensive and limreasoning over incomplete KGs, with the correct ited to transductive settings. In contrast, GLOW answer deliberately removed. Unlike AskGNN’s integrates KG completion into QA through GNNdataset, which covers only single-hop questions in guided prompting. less diverse domains, GLOW-B ENCH includes 1LLMs for KGQA. Fine-tuning LLMs on KG and 2-hop questions across multiple KGs. triples can improve domain adaptation (Wang et al., We compare our method against state-of-the-art 2021; Shu et al., 2024; Jiang et al., 2024), but baselines, including AskGNN (Hu et al., 2024), requires extensive training and risks catastrophic GCR (Luo et al., 2025), a KG-grounded QA forgetting (Xia et al., 2024; Zhao et al., 2021). pipeline, GoG (Xu et al., 2024),KGQA over incom- Our approach avoids fine-tuning by injecting both plete KG, and LLMs such as GPT-4o-mini (Ope- graph-structured data and GNN outputs into LLM nAI et al., 2024) and Qwen3-8B (Yang et al., 2025). prompts, enabling semantic and structural reasonAs shown in Table 1, LLMs perform poorly on ing without modifying LLM weights. deeper reasoning tasks, often failing to retrieve or GNN–LLM Hybrid Models. GNN-based organize relevant knowledge. AskGNN performs QA systems (Yasunaga et al., 2021; Zhu et al., better but is limited by its structural focus and lack 2023; Abdallah et al., 2024) support multi-hop of semantic flexibility. GCR struggles under in- reasoning, but often ignore language semancomplete KGs, often hallucinating answer paths tics. Retrieval-augmented methods, such as
GNN-RAG (Mavromatis and Karypis, 2024), G-Retriever (He et al., 2024), STaRK (Wu et al., 2024), and RoG (Luo et al., 2024), embed graph fragments for LLMs to reason over. However, they assume complete graphs and rely on retrieving full answer paths, which breaks under missing links (Zhou et al., 2025). Our method differs by using a GNN to predict candidate answers and relevant subgraphs, which are serialized into prompts for LLM reasoning. Open-World QA with In-Context Learning. AskGNN (Hu et al., 2024) enhances retrieval using GNN-based Structure-Enhanced Retrieval (SERetriever) to select in-context examples. However, it relies on joint LLM–GNN training using openweight LLMs and scales poorly with the model and graph size. It may also bias predictions toward dominant classes. GLOW avoids these issues by using lightweight GNNs for candidate generation and prompt construction, without requiring fine-tuning, and generalizing to various LLMs. GoG (Xu et al., 2024) addresses KGQA over incomplete KGs by deliberately removing randomly answer path predicates. While effective, this approach cannot guarantee the complete elimination of all answer paths and their associated edges, and generates on the fly LLM-based triples that do not conform to the KG schema, hence causing answer hallucination. Dense KGQA via Path Retrieval. Methods like G-Retriever, GNN-RAG, ToG (Sun et al., 2024), and RoG (Luo et al., 2024) retrieve semantically similar paths, assuming the answer exists in the KG. GCR (Luo et al., 2025) improves on RoG by fine-tuning LLMs to extract answers from retrieved paths. However, these systems fail in incomplete KG settings where the answer path is missing from the KG (Zhou et al., 2025). Unlike them, GLOW supports predictive reasoning by prompting the LLM with GNN-predicted candidates and structured facts, even when no complete path exists. Benchmarks. Existing KGQA benchmarks (e.g., WebQuestionsSP (Yih et al., 2016), LCQuAD (Trivedi et al., 2017), MetaQA (Zhang et al., 2018), STaRK (Wu et al., 2024)) assume closed-world settings with guaranteed answer paths. AskGNN (Hu et al., 2024) introduced open-world benchmarks but is limited to single-hop reasoning in narrow domains. We introduce GLOW-B ENCH, a benchmark of 1,000 open-world questions requiring single- and multi-hop reasoning across diverse KGs, where gold answers are explicitly removed to test generalization under incompleteness.
Algorithm 1 GET P ROMPT: Generate GLOW Prompt from Input OWA Question over KG Require: Q:User input question, GLOWv : System variation 1: function GET P ROMPT(Q,GLOWv ) 2: Qn , Qe , KG ← entityExtraction(Q) ▷ extract the question’s node and edge 3: KGSc ← getKGSchema(KG) ▷ Load KG schema 4: vt , et ← ER-Linking(Q, KGSc , Qn , Qe ) ▷ Link the question’s node and edge crossponding KG’s node and edge URIs 5: RC q ← TextToSPARQL(KGSc , vt ) ▷ generate RC equivalent SPARQL query 6: RC T riples ← execSPARQL(RC q , KG) ▷ Execute the RC SPARQL Query 7: L ← getPossibleLabels(vt , et ) ▷ Extract the set of possible labels 8: RC ← RC-Serialization(RC T riples ) ▷ Serialize the RC triples into text. 9: GN NAns ← GNNPredict(vt , et ) ▷ predict the top-K GNN answers for vt and et 10: P ← getPrompt(Q, vt , et , L, RC, GN NAns , GLOWv ) ▷ generate the GLOW’s variation prompt 11: return P 12: end function
OW-KGQA vs. GNN Node Classification. Our task fundamentally differs from GNN node classification (NC); it takes natural language questions as input and infers answer nodes using KG structure and semantics, whereas NC operates on graph inputs with fixed labels and no linguistic reasoning.
3
The GLOW Approach
GLOW1 is a hybrid approach for open-world QA on KGs, combining graph-based reasoning with LLMs. Beyond In-Context Learning (ICL), GLOW introduces In-Structure Learning, where textual and KG signals jointly guide reasoning. A pretrained GNN predicts top-k candidates and retrieves a relevant KG subgraph based on the question’s entity. These are serialized into a structured prompt with relational triples and GNN predictions, enabling the LLM to reason over linguistic and structural cues without additional fine-tuning. 3.1
GLOW Pipeline Overview
We develop three GLOW variants to explore different ways of integrating structured signals: GLOW-G (graph context), GLOW-N (GNN predictions), and GLOW-GN (combined). The full architecture is shown in Figure 1 and Algorithm 1. The pipeline proceeds in four stages: Question Understanding & Linking, Retrieval, Augmentation, and Generation. Each contributes build a GNN-guided LLM prompt. 1
Graph-LLM for Open-World QA
OWA-Q: What is the Species of the protein Q9LTJ2 in the BioKG knoweldge graph ? Entity-Relation Linking QNode URI: http://www.biokg.com/protein/Q9LTJ2 QNodeType URI: http://www.biokg.com/protein QEdge URI : http://www.biokg.com/protein-property/SPECIES
??
1
Protien
GNN Prediction (Top-K) Answer1 Answer..
RC Triples
TextToSPARQL select * from <KG> where { select <KG Node> ?p ?o where {<KG Node> ?p ?o.} union select ?s ?p <KG Node> where { ?s ?p <KG Node>.}} LLM API
Answer
A) GLOW-G
GNN Answers Serialization
GNN Prompt
LLM API
Answer
B) GLOW-N
GNN-Graph Prompt
LLM API
Answer
C) GLOW-GN
Augmentation
Q9LTJ2
4
Figure 1: Overview of GLOW’s four stages: question understanding, retrieval, augmentation, and generation. GLOW builds GNN-guided prompts using KG context and/or GNN predictions, with three variants: (a) KG facts (GLOW-G), (b) GNN predictions (GLOW-N), and (c) both (GLOW-GN).
Question Understanding & Linking: Given a question (e.g., "What is the species of the protein Q9LTJ2 from BioKG"?), we extract the main entity node (Qn ), its type, the target relation (Qe ), and the KG name at algorithm 1 lines 2 and 3. The linking step at line 4 maps these elements to KG schema types using an LLM-based prompt (see Appendix A.2) over RDF metadata. For example, species maps to predicate URI http://www.biokg.com/property/SPECIES, denoted et , and protein maps to a schema node type URI. We resolve the question entity to node vt via a SPARQL query over name or label fields. Unlike exact matching, this step uses semantic signals from attributes (name, label, description, URI) and schema-aware prompts to identify the node and relation types robustly. For two-hop questions, the second edge is parsed and appended to the SPARQL query. GLOW generalizes to multihop reasoning through extended semantic parsing.
which may be biased toward dominant GNN classes, we train a GNN NC model per question pattern (e.g., Protein→Species). These GNNs are trained independently of the LLM and queried via an API at inference time to return top-k candidate answers at line 9. See the GNN technical training/inference details in appendix A.4. These predictions guide the LLM to correlate graphderived candidates with the textual semantics in RC, improving robustness. Unlike AskGNN, if the GNN underperforms due to poor structure, the LLM can still rely on RC. This decouples model performance from GNN reliability.
KG Retrieval: LLMs often lack domain-specific facts (e.g., in BioKG) to answer questions about vt . However, neighboring KG nodes provide rich textual and structural context (RC), which aids in inferring the missing label along et . The possible labels L for edge et are retrieved via SPARQL and provided to constrain generation at lines 5,6 and 7. We retrieve 1-hop triples connected to vt via a SPARQL query generated at line 5 (Text-ToSPARQL) that accounts for schema and namespace details and serialized into triples format at line 8. Instead of retrieving top-K similar question entity examples like AskGNN (Hu et al., 2024),
Variant’s Examples are provided in Appendix A.1.
Augmentation: Each prompt contains the question Q, node vt , edge et , and possible answers L, optionally augmented with RC and GNN predictions at line 10. The prompt takes the form: Ŷ = f (Q, vt , et , L, RC, GNNAns )
(1)
Answer Generation: The LLM predicts a label for each node. Predictions are evaluated using an LLM-as-a-judge module described in §4. 3.2
GLOW Pipeline Variants
We study how structured prompts influence QA using three In-Structure Learning variants. Graph-Context Prompt(GLOW-G): This variant adds the neighborhood RC of vt , serialized into text via verbalization strategies (Baek et al., 2023) as shown in Figure 1.A. While this provides
Retrival
Graph Prompt
Ty pe
RC Serialization
Question Understanding
2
Generation
3
Species
RC
Entity Extraction QNode: Q9LTJ2 QNodeType: Protien QEdge : Species KG: BioKG
grounded semantic context, it may inflate prompt size if neighborhoods are large. See Figure 1 TextTo-SPARQL query. AskGNN RC comprises the top-k similar question nodes as ICL examples, whereas GLOW-G RC includes a subgraph of attributes and neighboring nodes connected to the question node and serialized to offer contextual grounding to the LLM for effective reasoning. A GNN-Guided Prompt (GLOW-N): As shown in Figure 1.B, this variant injects top-k GNN predictions for node vt as soft guidance. It provides structural signals without needing full KG serialization, but depends on GNN accuracy. We train a GNN model for each question pattern using GraphSAINT. The training subgraphs are extracted using KGTOSA (Abdallah et al., 2024), excluding benchmark nodes. This improves scalability and yields diverse, task-specific subgraphs. At inference time, the GNN model returns top-k candidates via API. See GNN details in the appendix. Hybrid Graph-GNN Prompt (GLOW-GN): This variant combines GLOW-G and GLOW-N by injecting both RC and GNN predictions. As shown in Figure 1.C, it enables the LLM to reason jointly over semantic and structural cues. If GNN confidence is low, the LLM can still rely on the verbalized KG context.
4
An Open-World Benchmark for KGQA
We present GLOW-Bench, a benchmark for multihop reasoning across diverse domains. It includes 25 open-world question templates based on four real-world KGs. Each template spans one of four dimensions: reasoning depth, knowledge domain, target entity type, and multiple-choice answer count.Table 2 summarizes the templates by (see Appendix A.3 for details): 1) Reasoning Hops (RH): 1–2 steps from target to answer; 2) Target Entity Type: ranging from general (e.g., people, works) to domain-specific (e.g., drugs, proteins); 3) Knowledge Domain (KD): Generic (G, YAGO4 (Tanon et al., 2020)), Entertainment (E, LinkedMDB (Hassanzadeh and Consens, 2009)), and DomainSpecific (DS, BioKG (Walsh et al., 2020), CrunchBase (Färber et al., 2018)); 4) Multiple Choice Count (MCC): 2–32+ candidates, one correct. While GLOW-Bench builds on existing KGs, it introduces a new benchmark for OW-QA and multihop reasoning, which current KGQA datasets lack. All questions are designed with answers absent
from the KG, enabling realistic evaluation under incompleteness, and are grounded in real KGs but formulated for OW-QA. Task Formulation: Each template defines a node classification task. For example, template #1 classifies drugs by structure (Organic vs. Non-Organic) using BioKG. Given a target node vt , the KG context is retrieved while excluding the gold answer (if present) from RC. The model selects the correct answer from the candidate set. Answer Evaluation: Evaluating LLM predictions is challenging due to linguistic variation, making exact matching insufficient. We adopt the LLMas-a-Judge framework (Gu et al., 2024), where an auxiliary LLM compares outputs to gold answers in two modes: 1) Exact Match (EM): Identical or semantically equivalent terms (e.g., Actor vs. Film Star); 2) Hierarchical Match (HM): Synonyms or subtypes (e.g., Athlete and Player). We use GPT4o-mini as the evaluation judge (Tan et al., 2025).
5
Experiments
5.1
Experimental Setup
Datasets and Metrics: In addition to our GLOWBench, we also evaluate GLOW on three existing datasets (Hu et al., 2024); arxiv2023, ogbn-arxiv, and ogbn-products. These datasets are originally node classification datasets, but AskGNN (Hu et al., 2024) adopted it for OW-QA, where each dataset is converted into a QA dataset using predefined question templates. In all experiments, we report the average across two runs. Baselines: We compare GLOW against recent methods in 3 categories: LLM-Only, Open-world KGQA, and closed-world KGQA. LLM: we use commercial models like GPT4o-Mini (OpenAI et al., 2024) and DeepSeekV3 (DeepSeek-AI et al., 2025), open-weight models including Qwen3-8B (Yang et al., 2025), DeepSeek-R1-Distill-Qwen-7B (DeepSeekAI, 2025), and IBM Granite 3.3-8B-Instruct (Saon et al., 2025). Open-world KGQA: AskGNN (Hu et al., 2024) is an approach that integrates LLMs and GNNs for QA over homogeneous graph datasets. We adapt AskGNN for the OW-QA setting over KGs and evaluate its performance on more challenging benchmark datasets. Recently GoG (Xu et al., 2024) has been developed as a KGQA system designed for incomplete KGs where the answer path is intentionally partially removed.
Table 2: Summary of Open-World Question Template (OW-QT) across four KG-based features: (1) Knowledge Domain (KD): Domain-Specific (DS), Entertainment (E), or Generic (G); (2) Target Entity Type (ET); (3) Reasoning Hops (RH); and (4) Multiple Choice Count (MCC). KG
KD #OW-QT
YAGO4 G BioKG DS LinkedMDB E CrunchBase DS
10 6 5 4
Target ET(s)
RH #Class MCC
Label Types
Person & Creative Work 1–2 3–102 2–32+ Nationality, Publisher, Occupation Drug & Protein 1–2 2–29 2–32 Kingdom, Class, SPECIES, R.Keyword Film 1–2 7–39 4–32+ Language, Producer, Genre Investor 1 6–14 4–16 Country, InvestRegion, Company
Open-Weight LLMs
Dataset RH LLM-Only AskGNN GCR GoG GLOW-G GLOW-N GLOW-GN Arxiv2023 1 31 62 4 53 64 66 81 ogbn-arxiv 1 12 70 14 57 71 60 77 Qwen3-8B ogbn-product 1 34 29 5 44 45 51 57 (Yang et al., 2025) GLOW-Bench 1 30 79 34 48 60 80 83 GLOW-Bench 2 19 34 15 29 44 37 42 Arxiv2023 1 24 48 7 28 55 59 67 ogbn-arxiv 1 13 50 15 35 30 61 63 DeepSeek-R1Distill-Qwen-7B ogbn-product 1 27 19 12 44 38 43 51 (DeepSeek-AI, 2025) GLOW-Bench 1 21 75 36 20 58 78 81 GLOW-Bench 2 12 31 16 13 27 32 39 Arxiv2023 1 14 63 14 53 32 63 77 ogbn-arxiv 1 5 66 21 56 31 55 73 Granite-3.3-8BInstruct ogbn-product 1 22 30 13 47 45 52 54 (Saon et al., 2025) GLOW-Bench 1 24 79 47 42 58 80 82 GLOW-Bench 2 14 35 19 27 39 32 42
Commercial LLMs
Table 3: Exact Match Accuracy (%) of GLOW vs. baselines across four OWA-QA datasets. GLOW-GN consistently outperforms AskGNN, GCR, GoG, and LLM-only setups across all LLMs and datasets, showing strong generalization and effective use of both textual and structural signals, even with smaller LLMs.
Arxiv2023 ogbn-arxiv GPT-4o-mini ogbn-product (OpenAI et al., 2024) GLOW-Bench GLOW-Bench Arxiv2023 ogbn-arxiv DeepSeek-V3 ogbn-product (DeepSeek-AI, 2024) GLOW-Bench GLOW-Bench
LLM-Model
1 1 1 1 2 1 1 1 1 2
35 50 37 29 28 17 58 36 34 30
N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
5.2 Closed-world KGQA: GCR (Luo et al., 2025) is a method designed for scalable KGQA over large KGs. It outperforms RoG (Luo et al., 2024) via fine-tuning LLMs to extract answers from retrieved answer paths. Evaluation Setup: GNN training was performed on an Ubuntu VM with dual 32-core Intel Xeon 2.4GHz CPU, 250GB RAM, and V100D-8C 16G vGPU. The GNN models were trained for the node classification tasks using GraphSAINT (Zeng et al., 2020) and ShaDowGNN (Zeng et al., 2021) with the task-oriented sampling method in (Abdallah et al., 2024). KGs were hosted on Virtuoso 07.20.32 with default settings. The GCR 8B models are fine-tuned using Colab A100 GPUs with 40G of VRAM.
N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
41 58 48 57 31 44 61 51 60 47
59 63 43 68 53 57 62 45 69 54
48 55 51 78 35 61 63 50 78 34
62 67 59 82 49 73 65 57 84 53
Experimental Results
Benchmark Results: As shown in Table 3, GLOW consistently outperforms all baselines across LLMs and datasets, showing strong generalization and effective use of textual and structural cues. Unlike AskGNN, which depends on GNN-based ICL examples, GLOW retrieves the question’s entity context and combines GNN predictions with neighborhood text for better performance. For example, using Qwen3-8B, GLOW-GN outperforms AskGNN by 18% on average across the AskGNN datasets, and by 4% and 8% on 1-hop and 2-hop GLOWBench questions, respectively. It also surpasses GCR by (64%, 49%, and 27%) and GoG by (20%, 35%, and 13%) . Similar trends hold across other open-weight (e.g., DeepSeek-R1-Distill-Qwen-7B, Granite-3.3-8B-Instruct) and commercial (e.g.,
N/A N/A
65 50 35 20 5
N/A N/A
Accuracy (%)
LLM-Only GCR GoG AskGNN GLOW-GN
-mini ek-V3 ek-R1 .3-8B 3-8B GPT-4o DeepSe DeepSe Granite3 Qwen
Figure 2: Average Exact and Hierarchical Match Accuracy (%) across datasets using different methods and LLMs. Dark bars show Exact Match; lighter segments show added Hierarchical Match gains. GCR and AskGNN are inapplicable to commercial LLMs.
GPT-4o-mini, DeepSeek-V3) LLMs. AskGNN’s performance varies and declines when either the GNN or LLM underperforms, showing its reliance on both components. GCR depends entirely on the presence of correct answer paths; when missing, it generates incorrect paths and answers. GoG relies entirely on generating missing paths, which are not constrained by the underlying KG schema, often leading to the creation of misleading or semantically invalid answer paths. In contrast, GLOW remains robust even with smaller LLMs, thanks to effective retrieval and contextualization. More detailed results per template and LLM are shown in Appendix A.5. For generic questions like Person → Nationality, textual attributes (e.g., name, residence, education) often suffice for LLMs to infer the answer. But for domainspecific tasks (e.g., Protein → Family or Protein → Species), textual cues are limited. Structurally similar nodes help the GNN yield better predictions. Combining both inputs in GLOW-GN consistently boosts accuracy—up to 26 points on ogbnarxiv and ogbn-products, and 6 points on GLOWBench templates like creative-work→country. GLOW-G performs best on 2-hop GLOW-Bench with GPT-4o-mini and DeepSeek-V3 due to weaker GNN results on dense subgraphs. For example, in creative-work→genre, the GNN reached only 22% vs. 33% by GLOW-G. In contrast, for drug→class, GNN accuracy was higher (68% vs. 45%). Exact vs. Hierarchical Match Accuracy: LLMs paraphrase answers or return semantically related concepts rather than producing exact matches. For instance, the occupation "Singer" may be returned in place of "Artist", its superclass—potentially acceptable in some contexts. Figure 2 analyzes this phenomenon by comparing Hierarchical-Match and Exact-Match accuracies, where GPT-4o-mini
is used as a judge for the Hierarchical-Match. The LLMOnly pipeline, which relies purely on pretraining without structural grounding, is particularly prone to generating such approximate answers. On average, its Hierarchical-Match accuracy exceeds Exact-Match accuracy by 12.5%, reflecting this tendency. This gap narrows to 3.3% with AskGNN, which supplements the LLM with contextual graph signals but still lacks fine-grained control over output specificity. In contrast, GLOWGN demonstrates minimal reliance on hierarchical leeway—showing only a 1.1% gain—indicating its robustness in steering the LLM toward precise answers. The integration of textual and structural semantics helps the LLM disambiguate fine-grained targets, improving accuracy and consistency across datasets and LLM architectures. 5.3
Effect of Domain, Graph, and Question
Knowledge Domain (KD): We analyze model performance across question domains using GLOWBench. As shown in Figure 3.A, GLOW-GN consistently outperforms all baselines. Gains are most notable in domain-specific (DS) areas such as pharmaceuticals and proteins, where GLOW-GN significantly exceeds models like Qwen3-8B. These results reflect LLMs’ difficulty in handling finegrained, specialized knowledge. By integrating textual and structural signals, GLOW-GN enables stronger generalization. Even in general domains like Generic (G) and Entertainment (ET), where entities are likely seen during pretraining, GLOWGN maintains an edge—often rivaling AskGNN without requiring additional fine-tuning. KG Structure: We assess the effect of KG structure by comparing performance across KGs used in Amazon-Product, Arxiv, and four GLOWBench KGs (BioKG, CrunchBase, LinkedIMDB, YAGO4). Figure 3.B shows that GLOW-GN outperforms baselines on all domain-specific KGs and performs well on YAGO4, a generic KG. The gap narrows only on LinkedIMDB, likely due to high entity overlap with LLM training corpora. Answer Choices (MCQ Format): We evaluate robustness under varying MCQ settings. Prior work (Zheng et al., 2024) showed LLM accuracy drops as the number of choices increases, and Figure 3.C confirms this. Still, GLOW-GN retains its edge, especially beyond two choices, underscoring the value of graph-based disambiguation under higher decision complexity.
LLM-Only GCR AskGNN GLOW-GN
DS ET G
A):The KD Effect
80 60 40 20 0
Accuracy(%)
Accuracy(%)
Accuracy(%)
80 60 40 20 0
zonP Arxiv BioKG nchBaseedIMDBYAGO4 Cru Link
Ama
B):The KG Effect
80 60 40 20 0
1-2 2-4 4-8 8-16 16-32 32+
C):The MCC Effect
60 50 40 30 20 10 0
GraphSAINT AskGNN(GS) GLOW-GN(GS) ShaDowGNN AskGNN(SH) GLOW-GN(SH) 40.4 41.4 33.4 31.8
29.8 30.1
45.8 46.3
43.6 42.8 33.4 32.9 35.6 35.5
DeepSeek-R1-7B Granite3.3-8B
33.4 33.7 35.3 35.9
Qwen3-8B
Figure 4: The GNN error propagation using GraphSAINT(GS) and ShaDowGNN(SH) models on questions with accuracy score < 50%. GLOW boosts accuracy by up to 12% over weak GNNs using textual cues. AskGNN closely follows GNN performance.
5.4
GNN Models and Answer Selection
GNN Error Propagation Analysis: AskGNN’s performance depends heavily on GNN quality. When the GNN is weak, AskGNN provides little to no gain and may even underperform the GNN itself. Figure 4 shows overall QA accuracy on OWA questions on which their corresponding GraphSAINT and ShaDowGNN models (used by AskGNN) scored below 50%. Across several LLMs, AskGNN typically matches or lags behind the GNN baseline. In contrast, GLOW-GN combines textual semantics and retrieval-based reasoning to outperform weak GNNs, with gains up to 12%. For example, on Amazon-product with Qwen3-8B, GraphSAINT scored 45%, AskGNN dropped to 29%, while GLOW-GN reached 51%. On ProteinKeyword, AskGNN achieved 35%, GraphSAINT 42%, and GLOW-GN improved to 57%. Varying GNN Top-K Answers: Increasing top-K GNN answers confuses the LLM, lowering accuracy. Figure 5 shows GLOW-GN performs best at K = 3 across five LLMs, while K = 4 or 5 reduces performance. Large closed-weight LLMs are less sensitive to higher K values.
Accuracy(%)
Accuracy(%)
Figure 3: Effect of GLOW-Bench characteristics on the GLOW answer accuracy(%) with Qwen3-8B. The effects are grounded by A) Knowledge Domain (KD), B) Knowledge Graph (KG) , and C) Multiple Choice Count (MCC).
75 72 69 66 63 60
GPT-4o-Mini DeepSeek-R1-7B Qwen3-8B DeepSeek-V3 Granite3.3-8B
1
2 3 4 GNN Top-K Answers
5
Figure 5: Impact of GNN top-K answer count on GLOW-GN’s average accuracy (%) across different LLMs. GLOW-GN achieves the best performance at top-K=3, while higher values tend to mislead the LLMs.
5.5
System Efficiency Analysis
Training Time: AskGNN’s joint GNN–LLM training grows with graph size, structural complexity (e.g., node/edge types), and LLM scale, averaging 23.4 hours across tasks. GCR, fine-tuned on WebQSP, takes about 6 GPU hours for 500 epochs. In contrast, GLOW-N and GLOW-GN decouple GNN and LLM training, requiring just 1.7 hours for GNN training (Table 4). This modular design enhances scalability and avoids dependence on LLM size, easing adaptation to new domains. Token Count: We analyze average token consumption per question across all pipelines. GLOW-N incurs the lowest token usage at 0.43K tokens per prompt, as its retrieved context (RC) consists solely of GNN-predicted answers and candidate labels, mirroring AskGNN’s structure but with reduced verbosity. GCR incurs the highest token cost due to XML-formatted triples. GoG incurs the highest token cost due to performing
Table 4: The average training time in Hours, Answer tokens count per question in (K-Tokens) and answer time per question in Seconds using Qwen3-8B LLM.
AskGNN GCR GoG GLOW-G GLOW-N GLOW-GN
Training Time (H) 23.4 6 N/A N/A 1.7 1.7
Tokens Count (K) 0.79 0.84 1.5 0.78 0.43 0.65
Answer Time (Sec) 11.2 13.4 15.7 12.6 8.2 11.5
an agentic chain-of-thought rounds for missing triple generation. AskGNN averages 0.79K tokens, driven by 20 ICL examples; fewer examples significantly reduce performance. GLOW-G and GLOW-GN offer a balanced cost, with prompt length shaped by KG density and the number of neighbors connected to the question node. Question Answering Time: In Table 4, GLOW-N is the fastest, averaging 8.2s per question due to its compact prompt. GCR is slower, as it relies on retrieved paths—often missing in OW-QA—causing hallucinations and longer processing. GoG is slowest, performing agentic chain-of-thought rounds for missing triple generation. AskGNN follows at 11.2s, hindered by reasoning over ICL examples. GLOW-GN shows moderate latency, combining GNN outputs with subgraph context. All experiments use vGPUs, not A100/H100; faster hardware would likely reduce runtime. Exact-Match
Hierarchical-Match Manual Verification
66.6%
17.6%
21.5%
33.4%
78.5%
Correct
82.4%
Incorrect
Figure 6: Human evaluation results versus Exact-match and Hierarchical-match. Hierarchical matching closely aligns with human evaluation.
Human Based Evaluation: To evaluate our LLM-as-a-Judge prompts, we manually validated the Qwen3:8B GLOW-GN answers across the GLOW-Bench question patterns, using 5 randomly selected questions per pattern. As shown in Figure 6, The Exact-match accuracy reached 66.6%, Hierarchical match 78.5%, and Manual verification is 82.4%. Notably, Hierarchical matching recovered 95.2% of manually verified correct answers, demonstrating the robustness of
our prompts. Some responses were marked as non-matches hierarchically despite being valid (e.g., (CEO, Co-Founder), (20th Century Studios, Paramount Pictures), (Catalan, Spanish)), while a few cases were incorrectly flagged as non-exact matches (e.g., (American English, English)).
6
Conclusion
This paper introduces GLOW, a system for openworld QA on KGs that integrates LLMs with GNNs. GLOW uses GNN-predicted candidates and relevant subgraphs as structured context to enhance multi-hop reasoning over incomplete KGs. By combining structural and textual semantics, GLOW overcomes key limitations of closed-world and retrieval-based KGQA. It consistently achieves strong performance across question types, domains, and LLMs, even when component quality varies. On standard open-world benchmarks and our new GLOW-Bench dataset, GLOW shows significant improvements in exact and semantic accuracy. These results highlight the need for hybrid approaches tailored to open-world settings and confirm GLOW’s robustness and generalizability.
7
Limitations
This work has three main limitations: First, data quality issues, such as sparse KGs with limited node and edge descriptions, impair both textual and structural semantics, reducing performance, especially for large LLMs. Second, our approach depends on high-performing GNN models for effectiveness. Third, all questions are currently framed as node classification tasks, though some may be better suited to link prediction, requiring prior evaluation and task-specific formulation.
References Hussein Abdallah, Waleed Afandi, Panos Kalnis, and Essam Mansour. 2024. Task-oriented gnns training on large knowledge graphs for accurate and efficient modeling. In ICDE, pages 1833–1846. IEEE. Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. CoRR, abs/2306.04136. DeepSeek-AI. 2024. Deepseek llm: Scaling opensource language models with longtermism. arXiv preprint arXiv:2401.02954.
DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. DeepSeek-AI, Aixin Liu, and et.al. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437.
Andrea Rossi, Denilson Barbosa, and et.al. 2021a. Knowledge graph embedding for link prediction: A comparative analysis. ACM Trans. Knowl. Discov. Data, 15(2):14:1–14:49.
Michael Färber, Carsten Menne, and Andreas Harth. 2018. A linked data wrapper for crunchbase. Semantic Web, 9(4):505–515.
Andrea Rossi, Denilson Barbosa, Donatella Firmani, Antonio Matinata, and Paolo Merialdo. 2021b. Knowledge graph embedding for link prediction: A comparative analysis. ACM Trans. Knowl. Discov. Data, 15(2):14:1–14:49.
Xiou Ge, Yun Cheng Wang, Bin Wang, C-C Jay Kuo, and 1 others. 2024. Knowledge graph embedding: An overview. APSIPA Transactions on Signal and Information Processing, 13(1).
George Saon, Avihu Dekel, and et.al. 2025. Granitespeech: open-source speech-aware llms with strong english asr capabilities. Preprint, arXiv:2505.08699.
Jiawei Gu, Xuhui Jiang, and et.al. 2024. A survey on llm-as-a-judge. CoRR, abs/2411.15594. Oktie Hassanzadeh and Mariano P. Consens. 2009. Linked movie data base. In WWW2009 Workshop on Linked Data on the Web, volume 538 of CEUR Workshop Proceedings. CEUR-WS.org. Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. In NeurIPS. Zhengyu Hu, Yichuan Li, Zhengyu Chen, Jingang Wang, Han Liu, Kyumin Lee, and Kaize Ding. 2024. Let’s ask GNN: empowering large language model for graph in-context learning. In Findings of the Association for Computational Linguistics: EMNLP, pages 1396–1409. Pengcheng Jiang, Lang Cao, Cao (Danica) Xiao, Parminder Bhatia, Jimeng Sun, and Jiawei Han. 2024. KG-FIT: knowledge graph fine-tuning upon openworld knowledge. In NeurIPS. Jiaying Lu and Carl Yang. 2022. Open-world taxonomy and knowledge graph co-learning. In AKBC. Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2024. Reasoning on graphs: Faithful and interpretable large language model reasoning. In International Conference on Learning Representations. Linhao Luo, Zicheng Zhao, Chen Gong, Gholamreza Haffari, and Shirui Pan. 2025. Graph-constrained reasoning: Faithful reasoning on knowledge graphs with large language models. In Forty-second International Conference on Machine Learning. Costas Mavromatis and George Karypis. 2024. Gnnrag: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139. OpenAI, Josh Achiam, and Steven Adler et.al. 2024. Gpt-4 technical report. Simon Ott, Christian Meilicke, and Matthias Samwald. 2021. SAFRAN: an interpretable, rule-based link prediction method outperforming embedding models. In AKBC.
Michael Sejr Schlichtkrull and et al. Thomas N. Kipf. 2018. Modeling relational data with graph convolutional networks. In ESWC, volume 10843, pages 593–607. Dong Shu, Tianle Chen, Mingyu Jin, Chong Zhang, Mengnan Du, and Yongfeng Zhang. 2024. Knowledge graph large language model (KG-LLM) for link prediction. In ACML, volume 260 of Proceedings of Machine Learning Research, pages 143–158. PMLR. Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, HeungYeung Shum, and Jian Guo. 2024. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Sijun Tan, Siyuan Zhuang, and et.al. 2025. Judgebench: A benchmark for evaluating llm-based judges. In ICLR. OpenReview.net. Thomas Pellissier Tanon, Gerhard Weikum, and Fabian M. Suchanek. 2020. YAGO 4: A reasonable knowledge base. In The Semantic Web - 17th International Conference, ESWC, volume 12123 of Lecture Notes in Computer Science, pages 583–596. Springer. Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017. Lc-quad: A corpus for complex question answering over knowledge graphs. In International Semantic Web Conference, pages 210–218. Springer. Brian Walsh, Sameh K. Mohamed, and Vít Nováček. 2020. Biokg: A knowledge graph for relational learning on biological data. In Proceedings of the 29th ACM Conference on Information Knowledge Management, page 3173–3180, New York, NY, USA. Association for Computing Machinery. Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Ying Wang, and Yi Chang. 2021. Structure-augmented text representation learning for efficient knowledge graph completion. In WWW, pages 1737–1748. ACM / IW3C2.
Chengrui Wang, Qingqing Long, and et.al. 2024. Biorag: A RAG-LLM framework for biological question reasoning. CoRR, abs/2408.01107. Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. Ioannidis, Karthik Subbian, James Zou, and Jure Leskovec. 2024. Stark: Benchmarking llm retrieval on textual and relational knowledge bases. In Advances in Neural Information Processing Systems, volume 37, pages 127129–127153. Curran Associates, Inc. Yuchen Xia, Jiho Kim, and et.al. 2024. Understanding the performance and estimating the cost of LLM finetuning. In IISWC, pages 210–223. IEEE. Yao Xu, Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Guang Liu, Jun Zhao, and Kang Liu. 2024. Generate-on-graph: Treat LLM as both agent and KG for incomplete knowledge graph question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18410–18430. Association for Computational Linguistics. An Yang, Anfeng Li, and et.al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. In North American Chapter of the Association for Computational Linguistics (NAACL). Wen-tau Yih, Matthew Richardson, Chris Meek, MingWei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2), pages 201–206, Berlin, Germany. Association for Computational Linguistics. Hanqing Zeng, Muhan Zhang, Yinglong Xia, and et.al. 2021. Decoupling the depth and scope of graph neural networks. In NeurIPS, pages 19665–19679. Hanqing Zeng, Hongkuan Zhou, and et.al. 2020. Graphsaint: Graph sampling based inductive learning method. In ICLR. , GitHub Code: https:// github.com/snap-stanford/ogb/blob/master/ examples/nodeproppred/mag/graph_saint.py. Yuyu Zhang, Hanjun Dai, Alexander J Smola, and Le Song. 2018. Variational reasoning for question answering with knowledge graph. In AAAI. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR. Chujie Zheng, Hao Zhou, and et.al. 2024. Large language models are not robust multiple choice selectors. In ICLR. OpenReview.net.
Dongzhuoran Zhou, Yuqicheng Zhu, Yuan He, Jiaoyan Chen, Evgeny Kharlamov, and Steffen Staab. 2025. Evaluating knowledge graph based retrieval augmented generation methods under knowledge incompleteness. Preprint, arXiv:2504.05163. Zhaocheng Zhu, Xinyu Yuan, Michael Galkin, LouisPascal A. C. Xhonneux, Ming Zhang, Maxime Gazeau, and Jian Tang. 2023. A*net: A scalable path-based reasoning approach for knowledge graphs. In NeurIPS.
A
Appendix
A.1
GLOW Prompt Examples
User Prompt: Predict the chemical kingdom for the drug Yohimbine from the BioKG knowledge graph. Answer:
Question Understanding Prompt: <system>: You are an an expert Entity-Extraction NLP system. <user>: Given the following question, identify 1-The question main entity type, 2- the main entity, 3- the prediction label and 4- the KG name. Question: {} Answer: 1-Question main entity: 2-Main Entity: 3-Prediction label: 4-KG name:
Entity/Relation Linking Prompt: <system>: You are an an expert knowledge graph entity-relation Linking NLP system. <user>: Given the following KG Schema in the basic graph pattern CSV format, one predicate per line: node type, relation, node type. ———– KG Schema: {} ———— 1- What is the node type in the schema that corresponds to {main entity}? Return only the name. 2- Choose from the schema the BGP (node type, relation, node type) that describes the {main entity type} value {main entity}. Return only the BGP. 3- Choose from the schema the BGP (node type, relation, node type) that describe the {main entity type } {prediction label}. Return only the BGP. ———— Answer: 123-
Question To SPARQL Prompt: <system>: You are an an expert text-To-SPARQL translation system. <user>: Given the following KG Schema in the basic graph pattern CSV format, one predicate per line: node type, relation, node type. ———– KG Schema: {} ———— Write a SPARQL query that selects the {main entity type} that satisfy the following BGPs. 1- {question node type} ,[label/name/titile] ,{question node} 2- {Other BGPs} graph prefix: {KG Prefix} ——————– Answer: SPARQL Query Do not return any explanation or reasoning details.
SPARQL Query Example: PREFIX biokg: <http://www.biokg.com/> SELECT ?drug as ?vt ?kingdom as ?vl WHERE { VALUES ?name { "Yohimbine" } ?drug biokg:NAME ?name . ?drug biokg:KINGDOM ?kingdom .}
Basic Prompt: <system>: You are an expert open world question answering system. <user>: What is the {Prediction Label} of the {question entity type} {question entity} from {KG details} knowledge graph. - Do not return any context or analysis. - Help: The possible list of {Prediction Label Type}s are: [{ Labels List}] Answer
To generate an instance of this template, replace the question node type, KG, and prediction label with values from one of the queries in table 5. An example prompt for OWA-Q #1: <system>: You are an expert open world question answering system. <user>: What is the Kingdom of the Drug Yohimbine from the BioKG , a Biomedical knowledge graph. - Do not return any context or analysis. - Help: The list of Kingdoms are: [Organic,Non-Organic] Answer:
GLOW-G Prompt: <system>: You are an expert open world question answering system. <user>: What is the Kingdom of the Drug Yohimbine from the BioKG , a Biomedical knowledge graph. - Do not return any context or analysis. - Help: The possible list of Kingdoms are: [Organic,Non-Organic] - The Drug associated triples. [("Yohimbine","DDI", "DB13677"), ..] Answer:
GLOW-N Prompt: <system>: You are an expert open world question answering system. <user>: What is the Kingdom of the Drug Yohimbine from the BioKG , a Biomedical knowledge graph. - Do not return any context or analysis. - Help: The possible list of Kingdoms are: [Organic,Non-Organic] - Verify the following list of GNN Answers: [ Organic,Non-Organic, ..] Answer:
GLOW-GN Prompt: <system>: You are an expert open world question answering system. <user>: What is the Kingdom of the Drug Yohimbine from the BioKG , a Biomedical knowledge graph. - Do not return any context or analysis. - Help: The possible list of Kingdoms are : [Organic,Non-Organic] - Verify the following GNN Answer: [ Organic] - The Drug associated triples. [("Yohimbine","DDI", "DB13677"), ..] Answer:
A.2
LLM as-a-Judge Prompt
<system>: You are an expert LLM-as-a-Judge system. <user>: Given the following list of predicted and true pairs of values. -Rank the predicted value against the true value using two metrics. 1- Exact Match Rule: you compare the two strings after normalization and remove any special characters. report 1 if both values are literally and semantically equal and 0 otherwise. 2- Hierarchical/Categorical Match Rule: report 1 if the predicted value is under a subcategory or hierarchically belongs to the true value or is a synonym and 0 otherwise. - Example: List of pairs: [[music, art], [painter, artist],[ football player, soccer player], [ lawyer, judge], [lawyer, player]] Answer: [[0,1],[0,1],[1,1],[0,1],[0,0]] - Question: -List of pairs: {ListOfPairs} -Note: refine each pair and return Answer for exactly {length(ListOfPairs)} pairs without explanation. -Finally: make sure you return only {length(ListOfPairs)} pair of answers. Answer:
A.3
Full Details of Our GLOW-Bench
Table 5: Our GLOW-Bench benchmark characterizes each Open-World Question Template (OW-QT) across four KG-based features: (1) Knowledge Domain (KD): Domain-Specific (DS), Entertainment (E), or Generic (G); (2) Target Entity Type (ET); (3) Reasoning Hops (RH); and (4) Multiple Choice Count (MCC).
16 17 18 19 20 21 22 23 24 25
A.4
BioKG
11 12 13 14 15
Drug Drug Drug Protein Protein Protein
1 1 2 1 2 1
2 13 29 18 15 4
2–4 8–16 16–32 16–32 8–16 2–4
Kingdom Superclass Class SPECIES R.Keyword Family
Kingdom Superclass RelatedPubMed → Class SPECIES RelatedPubMed → R.Keyword Family
CrunchBase
7 8 9 10
DS DS DS DS DS DS
DS Investor DS Investor DS Investor DS Investor
1 1 1 1
12 14 6 11
8–16 8–16 4–8 8–16
Country InvestRegion PositionTitle Company
Country InvestRegion PositionTitle Company
LinkedMDB
1 2 3 4 5 6
Reasoning Path
E E E E E
Film Film Film Film Film
1 1 2 2 1
15 27 39 7 28
8–16 16–32 32+ 4–8 16–32
Language Country Producer Genre Subject
Language Country Sequel → Producer Sequel → Genre Subject
YAGO4
OW-QT KG KD Target ET RH #Class MCC Label Type
G G G G G G G G G G
CreativeW CreativeW CreativeW CreativeW CreativeW Person Person Person Person Person
2 2 1 2 1 1 1 1 1 1
13 14 25 11 6 3 91 4 102 8
8–16 8–16 16–32 8–16 4–8 2–4 32+ 4–8 32+ 4–8
ProdComp Publisher PublishLang Genre Country GivenAward Nationality GraduateOfOrg Occupation SpokenLang
byArtist → ProdComp isBasedOn → Publisher Author ProdComp → Genre Country GivenAward Nationality GraduateOfOrg Occupation SpokenLang
The GNN Training/Inference Technical Details.
• For each question pattern, we train a node classification GNN using GraphSAINT with RGCN as GNN convolutional layer to support heterogeneous KG subgraphs. The initial node embeddings are iteratively aggregated with embeddings received from neighboring nodes connected to a specific relation until the embeddings of all nodes converge. The final embedding of a vt , our main question entity, is obtained through two aggregations: an outer aggregation over each relation type and an inner aggregation over neighbouring nodes RC of a specific relation and defined by RGCN (Schlichtkrull and Thomas N. Kipf, 2018) as follows: X X 1 (l+1) (l) (l) (l) hi = σ( Wr(l) hj + W0 hi ) (2) c i,r r r∈R j∈Ni
(l+1)
where l is an RGCN layer, hj is the hidden embedding of node j at layer l + 1, σ is element-wise r activation function, Ni denotes the set of neighbour indices of node i under relation r ∈ R, ci,r is a normalization constant that can either be learned or chosen in advance (such as ci,r = |Nir |), (l) Wr is the weight matrix for relation r at layer l, and W0l is the initial weight matrix at layer l. • To construct the training set, we exclude the GLOW-Bench question entity nodes from the training set and extract 1-hop in and out neighbor nodes subgraphs using the KGTOSA sampler (Abdallah et al., 2024). This ensures scalability to large KGs and yields task-relevant and diverse subgraphs for effective training.
• GraphSAINT Training HyperParameters: We set the input dimension (D = 128), hidden channels dimension=64, and number of layers (L = 2) of our GNN module with dropout rate 0.5 applied to each layer. We train the model with the Adam optimizer, learning rate from 5e-4, 5e-3, 1e-3, 2e-3. • Subgraph Sampling Hyperparameters: batch-size=20000, walk-length=2, num-steps=10. • At inference time, the node classification model M is resolved using the KG name, the question’s main entity type, and the target edge. The model M predicts the top-k answer classes based on the Log-Likelihood for each class and feeds them to the LLM prompt. The trained GNN model is hosted via an inference API. At runtime, this API returns the top-K predicted answer nodes, which are then passed to the GLOW pipeline as supportive evidence for the LLM’s reasoning. A.5
The OWA-QA Detailed Results:
Table 6: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from (Hu et al., 2024) and our developed benchmark GLOW-Bench. All results below use Qwen3-8B as the underlying LLM. Best results are marked in bold. LLM-Model
Qwen3:8b
Dataset Arxiv2023 ogbn-arxiv ogbn-product drug-superclass drug-kingdom protein-SPECIES protein-FAMILY film-country film-subject film-language person-nationality parson-graduateOfOrg person-occupation person-spokenLang person-givenAward CWork-PublishedLang CWork-country person-title person-Company person-InvestementRegion person-InvestementCountry drug-class protein-keyword film-genre film-producer CWork-ProductionCompany CWork-Genere CWork-publisher
RH 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2
AskGNN 62 70 29 88 98 75 67 55 77 87 86 51 48 86 93 75 72 79 97 98 98 46 35 54 35 20 20 31
GraphSAINT 41 59 45 84 96 89 70 55 81 83 89 53 46 83 91 80 51 72 95 94 95 68 42 51 21 22 22 28
LLM-Only 31 12 34 29 88 8 9 40 15 70 27 3 7 70 7 50 64 5 3 7 38 11 5 50 7 10 16 33
GCR 4 14 5 3 94 16 13 4 7 13 38 25 12 61 50 60 43 41 93 46 14 15 7 25 7 29 7 17
GLOW-L 59 63 41 50 91 19 35 40 50 73 45 27 15 83 75 55 62 63 30 95 94 33 33 60 14 21 22 58
GLOW-G 64 71 44 77 98 71 39 60 30 80 89 36 38 89 93 55 75 47 21 87 55 45 53 62 24 31 33 62
GLOW-N 66 60 51 85 97 91 69 57 82 85 91 54 46 83 93 85 48 77 96 96 97 70 44 51 21 25 19 27
GLOW-GN 81 77 57 92 96 92 71 55 80 86 90 55 53 85 93 95 79 83 97 100 97 73 57 52 28 28 27 32
Table 7: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from (Hu et al., 2024) and our developed benchmark GLOW-Bench. All results below use GPT-4o-Mini (OpenAI et al., 2024) as the underlying LLM. Best results are marked in bold. LLM-Model
GPT-4o-Mini
Dataset Arxiv2023 ogbn-arxiv ogbn-product drug-superclass drug-kingdom protein-SPECIES protein-FAMILY film-country film-subject film-language person-nationality parson-graduateOfOrg person-occupation person-spokenLang person-givenAward CWork-PublishedLanguage CWork-country person-title person-company person-InvestementRegion person-InvestementCountry drug-class protein-keyword film-genre film-producer CWork-ProductionCompany CWork-Genere CWork-publisher
RH 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2
AskGNN N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
GraphSAINT 41 41 45 84 96 89 70 55 81 83 89 53 46 83 91 80 51 72 95 94 95 68 42 51 21 22 22 28
LLM-Only 35 35 N/A 8 31 6 11 47 15 80 43 19 7 70 6 65 62 5 3 9 41 8 7 68 20 10 27 62
GLOW-L 58 58 N/A 47 100 10 53 47 73 80 56 22 23 81 68 60 64 38 6 7 44 27 28 65 17 28 27 70
GLOW-G 59 59 N/A 57 100 77 43 71 38 80 89 33 30 91 96 65 77 55 30 96 100 45 44 70 20 42 72 79
GLOW-N 48 48 N/A 73 98 92 69 57 76 86 92 55 46 83 93 83 48 74 96 100 97 69 40 50 21 25 19 25
GLOW-GN 62 67 N/A 87 99 93 72 58 75 88 87 57 52 89 91 87 83 77 98 96 97 67 45 53 31 32 44 75
Table 8: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from (Hu et al., 2024) and our developed benchmark GLOW-Bench. All results below use DeepSeek-V3 (DeepSeekAI, 2024) as the underlying LLM. Best results are marked in bold. LLM-Model
DeepSeek-V3
Dataset Arxiv2023 ogbn-arxiv ogbn-product drug-superclass drug-kingdom protein-SPECIES protein-FAMILY film-country film-subject film-language person-nationality parson-graduateOfOrg person-occupation person-spokenLang person-givenAward CWork-PublishedLanguage CWork-country person-title person-company person-InvestementRegion person-InvestementCountry drug-class protein-keyword film-genre film-producer CWork-ProductionCompany CWork-Genere CWork-publisher
RH 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2
AskGNN N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
GraphSAINT 41 59 45 84 96 89 70 55 81 83 89 53 46 83 91 80 51 72 95 94 95 68 42 51 21 22 22 28
LLM-Only 17 58 N/A 3 9 2 21 57 23 81 72 32 19 79 3 47 68 41 3 7 44 7 3 53 10 28 18 70
GLOW-L 50 60 N/A 32 88 8 76 65 80 86 81 48 34 79 56 50 75 41 24 21 58 21 40 60 30 53 22 79
GLOW-G 57 62 N/A 48 100 79 87 70 38 86 87 43 34 82 58 50 87 63 33 100 100 39 59 58 50 57 27 87
GLOW-N 61 63 N/A 73 98 92 69 57 80 86 91 52 46 83 93 84 48 74 96 89 97 69 49 50 14 25 16 16
GLOW-GN 73 65 N/A 86 96 93 81 71 83 93 93 54 53 85 94 90 79 77 97 100 100 70 52 62 34 53 28 70
Table 9: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from (Hu et al., 2024) and our developed benchmark GLOW-Bench. All results below use DeepSeek-R1-DistillQwen:7B (DeepSeek-AI, 2025) as the underlying LLM. Best results are marked in bold. LLM-Model
DeepSeek-R1Distill-Qwen:7B
Dataset Arxiv2023 ogbn-arxiv ogbn-product drug-superclass drug-kingdom protein-SPECIES protein-FAMILY film-country film-subject film-language person-nationality parson-graduateOfOrg person-occupation person-spokenLang person-givenAward CWork-PublishedLanguage CWork-country person-title person-company person-InvestementRegion person-InvestementCountry drug-class protein-keyword film-genre film-producer CWork-ProductionCompany CWork-Genere CWork-publisher
RH 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2
AskGNN 48 50 19 53 98 71 62 56 78 86 85 50 47 82 89 71 62 73 93 92 93 41 31 53 30 18 17 25
GraphSAINT 41 59 45 84 96 89 70 55 81 83 89 53 46 83 91 80 51 72 95 94 95 68 42 51 21 22 22 28
LLM-Only 24 13 27 20 60 1 7 17 8 73 22 5 3 35 8 50 37 5 3 3 11 4 3 35 7 10 11 12
GCR 7 15 12 5 90 12 23 34 8 32 41 17 12 56 23 44 42 51 90 33 56 13 8 18 9 23 17 23
GLOW-L 44 30 36 20 90 4 30 44 11 73 30 19 15 41 50 50 41 44 30 64 72 15 18 50 7 17 11 20
GLOW-G 55 30 38 21 100 62 35 52 34 66 79 36 31 79 70 55 70 52 19 96 94 24 22 56 7 21 16 40
GLOW-N 59 61 43 69 98 88 68 57 76 86 91 53 46 81 90 77 48 80 96 100 97 69 32 50 14 25 16 16
GLOW-GN 67 63 51 68 100 90 69 64 82 88 89 54 54 83 92 81 66 82 92 100 97 71 38 57 24 32 27 26
Table 10: The detailed accuracy (%) of GLOW compared to baseline models on existing question answering datasets from (Hu et al., 2024) and our developed benchmark GLOW-Bench. All results below use IBM Granite-3.3-8Binstruct (Saon et al., 2025) as the underlying LLM. Best results are marked in bold. LLM-Model
Granite-3.3:8BInstruct
Dataset Arxiv2 ogbn-arxiv ogbn-product drug-superclass drug-kingdom protein-SPECIES protein-FAMILY film-country film-subject film-language person-nationality parson-graduateOfOrg person-occupation person-spokenLang person-givenAward CWork-PublishedLanguage CWork-country person-title person-company person-InvestementRegion person-InvestementCountry drug-class protein-keyword film-genre film-producer CWork-ProductionCompany CWork-Genere CWork-publisher
RH 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2
AskGNN 63 66 30 86 97 75 67 56 76 84 86 52 47 85 93 81 72 78 97 99 99 47 35 54 35 22 21 32.7
GraphSAINT 41 59 45 84 96 89 70 56 81 83 89 53 46 83 92 80 51 72 96 95 96 68 42 51 21 22 22 28
LLM-Only 14 5 22 0 8 2 0 48 12 87 52 6 8 48 3 45 58 14 3 4 39 0 2 35 5 7 16 33
GCR 14 21 13 9 98 18 33 60 11 46 88 14 23 75 25 35 69 55 96 29 69 18 10 15 5 18 22 46
GLOW-G 32 31 45 22 95 45 36 66 27 80 79 31 23 52 63 80 75 64 42 82 78 27 36 52 30 21 50 58
GLOW-N 63 55 52 73 98 93 69 58 81 87 91 56 46 83 91 81 59 80 96 98 96 70 9 54 23 25 17 27
GLOW-GN 77 73 54 71 99 90 72 55 85 88 92 57 50 88 94 94 69 83 97 100 97 70 38 57 29 32 33 36