PATH ISE: Learning Informative Path Supervision for Knowledge Graph Question Answering
arXiv:2605.10791v1 [cs.AI] 11 May 2026
Shengxiang Gao
Chao Lei Jey Han Lau Jianzhong Qi The University of Melbourne {shengxiang, clei1}@student.unimelb.edu.au {laujh, jianzhong.qi}@unimelb.edu.au
Abstract Knowledge Graph Question Answering (KGQA) aims to answer user questions by reasoning over Knowledge Graphs (KGs). Recent KGQA methods mainly follow the retrieval-augmented generation paradigm to ground Large Language Models (LLMs) with structured knowledge from KGs. However, training effective models to retrieve question-relevant evidence from KGs typically requires high-quality intermediate supervision signals, such as question-relevant paths or subgraphs, which are time- and resource-intensive to obtain. We propose PATH ISE, a novel framework for learning high-quality intermediate supervision from answerlevel labels. PATH ISE introduces a lightweight transformer-based estimator that estimates the informativeness of relation paths to construct pseudo path-level supervision. This supervision is then distilled into an LLM path generator, whose generated paths are grounded in the KG to provide compact evidence for inductive answer reasoning. ExtensiveISE experiments on three KGQA benchmarks show that PATH ISE achieves competitive or state-of-the-art KGQA performance, and provides reusable supervision signals that can enhance existing KGQA models, without relying on costly LLM-refined supervision signals. Our source code is available at https://anonymous.4open.science/r/PathISE-2F87.
1
Introduction
Knowledge Graph Question Answering (KGQA) aims to answer users’ natural language questions by reasoning over Knowledge Graphs (KGs) [16, 31]. Modern large-scale KGs, such as Freebase [1], Wikidata [36], and WikiMovies [32], store abundant factual knowledge in structured formats. KGQA provides a user-friendly interface for accessing such knowledge [9], and has been widely applied in various domains, including search engines [5], fact-checking [7], and recommender systems [38]. Recent advances in Large Language Models (LLMs) have opened new opportunities for KGQA. Pretrained on large-scale corpora, LLMs such as GPT-4 [30], Gemini [4], LLaMA [8], and Qwen [42] possess extensive world knowledge and strong capabilities in natural language understanding and generation. With advanced generalizability and in-context learning abilities, LLMs have achieved strong performance across a wide range of natural language processing tasks [39, 41], including summarization [15], code generation [17], and question answering [43]. These capabilities make LLMs a promising backbone for KGQA, where they can interpret complex questions, plan reasoning over structured knowledge, and generate answers in natural language. A prominent line of KGQA research augments LLMs with KG knowledge through RetrievalAugmented Generation (RAG) [18], commonly referred to as KG-RAG. Existing KG-RAG methods [6, 19, 22, 25, 29, 34, 44] retrieve question-relevant KG components (e.g., entities, triples, paths, or subgraphs), and incorporate them into the generation context. By grounding reasoning with structured KG knowledge, these approaches reduce hallucinations and improve answer faithfulness [31]. Preprint.
Question:
Which team is LeBron James’s son playing for?
Knowledge Graph
Los Angeles
teammate pla y_ for location_of
Path 1: LeBron Bronny James play_for
live_in
LeBron James
parent
Ours
Weak Supervision Signals
Path 2: LeBron Path 3: LeBron Path 4: LeBron
!"#$%&
Bronny
&$",,"&$ !'"(_*+# '-.$_-%
!'"(_*+#
Bronny
(a)
Lakers
Lakers
LA
'+/"&-+%_+*
(b)
Path 1 Path 2
PathISE
Path 3 Path 4
Lakers
Spurious Paths Informative Paths
Los Angeles Lakers
Lakers
!'"(_*+#
Path 1 (c)
Figure 1: An illustration of supervision signals for models in KG-RAG: (a) A training sample with input question (yellow box), question entity (blue box), and ground-truth answer (orange box). (b) Existing KG-RAG models trained with weakly supervised signals include spurious paths, i.e., incorrect/irrelevant paths that reach the ground-truth answer. (c) Our PATH ISE framework uses a novel MIL estimator to identify informative paths, yielding more precise path-level supervision. The effectiveness of KG-RAG methods largely depends on whether the retrieved KG components provide sufficient and relevant information for answering the question [19, 25]. Therefore, stateof-the-art (SOTA) KG-RAG methods typically train graph retrievers or path generators to obtain question-relevant KG knowledge. Graph retrievers score and rank KG components, such as triples or subgraphs, whereas path generators produce relation paths as plans for fetching KG evidence. However, training such models typically requires intermediate supervision, such as supporting triples, reasoning paths, or subgraphs. This exposes a fundamental challenge: the lack of high-quality intermediate supervision signals. Obtaining these annotations is time- and resource-intensive, making intermediate supervision rarely available in practice [9, 20, 22]. This has motivated recent KGQA research to explore alternative training paradigms without annotated intermediate supervision. A common alternative is to train retrievers or generators using weakly supervised paths [19, 24, 25, 27], where all paths connecting the entities mentioned in the question to the ground-truth answers in the training samples are treated as positive intermediate supervision signals. Although this strategy is straightforward to implement, the example in Figure 1 illustrates that many of these weakly supervised paths are spurious paths, namely paths that are irrelevant to the underlying reasoning process despite being connected to the ground-truth answer. These spurious paths introduce noise into the supervision signals, thereby degrading retrieval accuracy and ultimately compromising reasoning correctness. To reduce the noise introduced by spurious paths, recent studies [23, 40, 44] use LLMs to refine intermediate paths for training models. In these studies, LLMs are instructed to select questionrelevant informative paths from weakly supervised path candidates, or to search for informative paths through step-by-step reasoning over the KG. The resulting LLM-refined paths are then used as intermediate supervision signals for training models, thereby substantially reducing the noise in intermediate supervision. However, constructing LLM-refined supervision typically requires extensive LLM calls across training instances, which introduces substantial computational costs and limits scalability for large-scale KGQA datasets (analyzed in Section 5.3). To address the lack of informative and computationally scalable supervision signals in KGQA, we introduce PATH ISE (Path-level Informative Supervision Estimation), a novel framework for learning informative path-level intermediate supervision from answer-level labels. Instead of training models with noisy paths or relying on LLM-refined supervision, PATH ISE treats paths as latent intermediate supervision and estimates their informativeness through a Multiple Instance Learning (MIL) paradigm. The resulting high-informativeness paths serve as pseudo path-level supervision, mitigating the noise introduced by spurious paths without requiring costly LLM interventions. PATH ISE instantiates this idea with a lightweight transformer-based MIL estimator. The estimator scores candidate relation paths according to their estimated utility for answering a given question, and selects high-scoring paths as pseudo path-level supervision. This pseudo supervision can be applied to KGQA models that rely on intermediate supervision, including graph retrievers and path generators. In this work, we use it to train an LLM-based relation path generator, enabling the generator to produce more precise and informative relation paths from a question and its entities. At inference time, the generated paths are grounded in the KG to retrieve compact KG evidence, which is then provided to an LLM to support KG-grounded inductive reasoning without fine-tuning. 2
The main contributions of this work are summarized as follows: • We propose PATH ISE, a novel framework for learning high-quality path-level intermediate supervision from answer-level labels, avoiding both noisy supervision from spurious paths and costly LLM-based supervision refinement. • We introduce a lightweight transformer-based MIL estimator that learns relation path informativeness from answer-level labels to constructs high-quality path-level supervision. • We conduct extensive experiments on three KGQA benchmarks and show PATH ISE achieves strong performance compared with SOTA baselines. Further experiments demonstrate the applicability of PATH ISE-estimated supervision for enhancing existing KGQA models.
2
Related Work
KGQA. Recent KGQA methods can be broadly categorized into two groups: retrieval-based and agent-based. Retrieval-based methods first employ graph retrievers to retrieve question-relevant KG components, or leverage LLMs to generate intermediate reasoning paths, which are then grounded in the KG to fetch supporting KG evidence [19, 24, 25, 27, 29]. Following the RAG paradigm, the retrieved KG components or evidence, together with the input question, are fed into LLMs to produce the final answer. On the other side, agent-based methods treat LLMs as agents that iteratively perform step-by-step traversal over the KG to find the answer [2, 23, 34, 40]. Retrieval-based methods typically rely on high-quality intermediate supervision signals, which are time- and resource-intensive to obtain. Agent-based methods generally avoid additional training, but often require multiple rounds of LLM interactions during inference, leading to high latency and costs, and may still suffer from suboptimal exploration in large search spaces due to intermediate biases. Weakly Supervised KGQA. To alleviate the dependence on annotated intermediate supervision signals, existing KGQA methods typically adopt two strategies. One line of work [19, 24, 25, 27] trains models with weak supervision signals, where all paths connecting the question entities and the answer entities are used as training supervision. Such paths, or their constituent KG triples, are uniformly treated as positive supervision for training path generators or graph retrievers. Another line [23, 40, 44] constructs LLM-refined supervision by using LLMs to select informative paths from weakly supervised candidates or search for informative paths through step-by-step KG reasoning. Training with weakly supervised paths often degrades retrieval due to noisy supervision induced by spurious paths, hence leading to suboptimal reasoning performance. Meanwhile, constructing LLM-refined supervision requires extensive LLM calls to annotate training data, leading to substantial computational costs and limited scalability. In contrast, we propose a lightweight transformer-based model to estimate path informativeness from answer-level labels, hence enabling efficient construction of high-quality intermediate supervision signals and improving overall model performance.
3
Preliminary
Knowledge Graphs (KGs) are composed of relational facts in the form of a set of triples: G = {⟨eh , r, et ⟩|eh ∈ E, r ∈ R, et ∈ E ∪ L}, where E denotes a set of entities, R a set of relations, and L a set of literals, e.g., textual labels or numerical values. In each triple ⟨eh , r, et ⟩, eh ∈ E is a head entity, et ∈ E ∪ L is a tail entity or a literal, and r ∈ R represents the relation between eh and et . Knowledge Graph Question Answering (KGQA) is a reasoning task over KGs. Given a KG G and a natural language question q = (w1 , w2 , . . . , wn ), where wi denotes the i-th token, the objective is to learn a function f that maps (q, G) to an answer set Aq ⊆ E ∪ L, i.e., Aq = f (q, G). Relation Path is an ordered sequence of relations z = (r1 , r2 , . . . , rl ), where ri ∈ R denotes the i-th relation and l denotes the path length. It defines a compositional relation that connects a source entity to a set of target entities by following the relations in order (example provided in Appendix A). Formally, given an entity e and a relation path z, we denote the set of entities reachable from e via z: n o rl r1 r2 Ez (e) = el ∈ E | e −→ e1 −→ · · · −→ el in G . Multiple Instance Learning (MIL) is a weakly supervised learning paradigm commonly applied to binary classification problems. In MIL, labels for individual instances are unavailable during training; 3
PathMIL 𝑧"
question
𝑧'
paths
𝑧+
KG
𝑧)
Pseudo Path Supervision 𝒵!∗ = Top−𝑇(𝑆# )
𝑧" 𝑧' 𝑧)
Path Encoder 𝑞 𝑟" 𝑟'
𝑟(
ℎ$# ℎ$$
𝑧*
ℎ%&% ℎ%&&
Transformer distill
ℎ$! ℎ$"
ℬ"# ℬ'#
Attention-based MIL Aggregator 𝐻ℬ
𝑆!
Bag Classifier
𝑌ℬ
LLM Path Generator
LLM Inductive Reasoning
𝑃% = 𝒵 𝑞, ℰ! )
𝑃& = 𝒜' 𝑞, 𝒵, ℰ! , 𝒢)
Figure 2: Overview of PATH ISE. Given a training dataset, we first estimate informative relation paths from answer-level supervision and construct pseudo path supervision. The pseudo path supervision is then distilled into an LLM-based relation path generator, which then produces relation paths that are grounded in the KG for LLM-based inductive answer reasoning. In the KG example, the blue node denotes the question entity, and red nodes denote answer entities. instead, only the labels of bags, i.e., collections of instances, are observed [10, 12]. Formally, a bag is denoted as X = {x1 , x2 , . . . , xK }, where xk is the k-th instance and K may vary across bags. Under the standard MIL assumption, each instance has an unobserved binary label yk ∈ {0, 1}. A bag is labeled positive (Y = 1) if at least one instance is positive, and negative (Y = 0) only if all instances are negative: 1, if ∃k ∈ {1, . . . , K} such that yk = 1, Y = (1) 0, if yk = 0 for all k ∈ {1, . . . , K}. MIL enables instance-level signal estimation from bag-level supervision, thereby reducing the cost and complexity of collecting fine-grained instance-level labels [13].
4
Methodology
Following prior work [24], we formulate KGQA as reasoning over latent relation paths. Given a question q, question entities Eq , and KG G, answer prediction can be decomposed as: X P (Aq | q, Eq , G) = Pω (Aq | q, z, Eq , G)Pθ (z | q, Eq ), (2) z∈Zq
where Zq denotes the space of candidate relation paths considered for question q. Here, Pθ (z | q, Eq ) is a trainable relation path generator, and Pω (Aq | q, z, Eq , G) denotes answer prediction by an LLM after grounding z in the KG. Since informative relation paths are unobserved, directly learning Pθ (z | q, Eq ) is challenging. In contrast, ground-truth answers Aq are easier to obtain. They are typically available in existing benchmark datasets, or are easier to label (compared with ground-truth paths) if new training data is needed. We exploit such answer-level supervision to learn a path informativeness scoring function: Sϕ (z | q, Aq , Eq , G). (3) To learn Sϕ (·), we propose a transformer-based model trained with an MIL objective. We convert answer-level labels into bag-level labels by grouping candidate relation paths that reach the same answer entity into a positive bag. Consistent with the MIL assumption, a positive bag label indicates that at least one candidate path inside the bag provides informative evidence for deriving the answer. By optimizing bag-level prediction, the model estimates path-level informativeness according to each path’s contribution within the bag-level prediction, without requiring explicit path annotations. Overall, our framework PATH ISE consists of three stages (cf. Figure 2): (1) MIL-based Informative Path Estimation, where we estimate the informativeness of candidate paths; (2) Pseudo Supervision Distillation, where the estimated path supervision is distilled into an LLM path generator; and (3) KGgrounded Inductive Reasoning, where the paths generated by the path generator are grounded in the KG and provided to an LLM for answer reasoning. We detail these stages next. 4
4.1
MIL-based Informative Path Estimation
We propose a lightweight transformer-based MIL estimator. Given a question q, its answer set Aq , and a candidate path set Zq , the estimator learns a question-conditioned path informativeness scoring function from answer-level supervision: Sϕ (z | q, Aq , Eq , G). The top-T (hyperparameter) scored paths, denoted as Zq∗ , are used as pseudo path supervision for the subsequent path generator training. Zq∗ = Top -Tz∈Zeq+ Sϕ (z | q, Aq , Eq , G),
Zeq+ = {z ∈ Zq | ∃e ∈ Eq , Ez (e) ∩ Aq ̸= ∅}.
(4)
Here, Zeq+ denotes weakly supervised paths, and Zq∗ denotes the selected pseudo supervision. MIL Bag Construction. Starting from each question entity e ∈ Eq , we retrieve candidate relation paths Zq from G using breadth-first search with a maximum L-hop (hyperparameter) constraint. Each path z reaches a set of entities Ez (e). We then construct MIL bags from answer-level labels following the MIL assumption in Section 3. For each answer a ∈ Aq , the positive bag contains all paths (as retrieved above) that can reach a from at least one question entity. The retrieved paths that cannot reach any answer entity are treated as negative paths, i.e., Zq− = {z ∈ Zq | ∀e ∈ Eq , Ez (e) ∩ Aq = ∅}. A negative path z ∈ Zq− forms a singleton negative bag: + B(a) = {z ∈ Zq | ∃e ∈ Eq , a ∈ Ez (e)}, − B(z) = {z},
a ∈ Aq , (5)
z ∈ Zq− .
For scalability and training efficiency on large-scale KGs, we sample a subset of negative paths to form the negative bags during training, as detailed in Appendix B. This formulation is consistent with the MIL assumption: each positive bag is expected to contain at least one informative path. Each negative bag contains a path that does not reach any answer entity and is therefore treated as negative supervision during training. Question-conditioned Path Encoding. To model the semantic compatibility between the question and a relation path, we jointly encode them with a transformer encoder. For a path z = (r1 , . . . , rl ), we obtain the question embedding hq ∈ Rd and relation embeddings {hr1 , . . . , hrl | hri ∈ Rd } using a pre-trained text encoder. These embeddings are combined with learnable positional embeddings {e0 , . . . , el } and fed into a transformer to produce the path representation hz as: Hz = Transformer ([hq + e0 , hr1 + e1 , . . . , hrl + el ]) ,
hz = H(0) z .
(6)
(0) Here, Hz denotes the first token representation in Hz .
The trainable parameters include the transformer encoder and positional embeddings, while the pre-trained text encoder is frozen. Attention-based MIL Aggregation. For a positive bag B + = {z1 , . . . , zK }, we compute the bag representation following Attention-based MIL [12]: hB+ =
K X i=1
αi hzi ,
exp(si ) αi = PK , j=1 exp(sj )
si = w⊤ tanh(Vhzi ),
(7)
where w ∈ Rd and V ∈ Rd×d are trainable parameters. The normalized weight αi is used to aggregate instance representations within a bag for bag-level prediction, while the unnormalized score si is used as the estimated informativeness score for ranking candidate paths across bags: Sϕ (zi | q, B + ) = si .
(8)
− For a negative bag B(z) = {z}, the bag representation is its path embedding, i.e., hB− = hz . (z) Negative bags provide contrastive supervision for learning path patterns that fail to support the answer-level label. This contrastive signal improves the shared path representation, enabling the attention mechanism within positive bags to better distinguish informative paths from spurious ones.
The bag representation and question embedding are concatenated and fed into an MLP classifier: ŷB = σ (MLP(hB ∥hq )) . 5
(9)
Training Objective. The estimator is trained as a binary bag classifier with the following BCE loss: X X LMIL = − log ŷB − log(1 − ŷB ). (10) B∈B− q
B∈B+ q
Optimizing the bag-level classifier encourages the attention layer to emphasize paths with stronger evidence for the answer-level label and down-weight spurious paths in the same bag. We therefore use the unnormalized attention score si as the path informativeness score. Since these scores are intended for ranking rather than calibrated probability estimation, we use them to rank weakly supervised paths Zeq+ and construct model-agnostic pseudo path-level supervision Zq∗ , as defined in Eq. 4. 4.2
Pseudo Supervision Distillation
The PATH ISE-estimated path supervision is model-agnostic and can be used to train different KGQA models that require intermediate supervision. For example, it can supervise graph retrievers by using the constituent triples of the selected paths as positive evidence, or supervise path generators by using the selected paths as target path sequences. In PATH ISE, we use this supervision to train an LLM-based relation path generator Pθ (z | q, Eq ) that can produce relation paths given only the question and its entities. The generated relation paths provide a structured and logically coherent representation of the reasoning process, which can facilitate faithful reasoning by the downstream LLM-based reasoners [25, 44]. Following the KL-based path generator training objective in RoG [24], we define a pseudo target distribution for supervision distillation. Unlike RoG, which constructs the pseudo target distribution from weakly supervised paths, PATH ISE converts the MIL-estimated high-informativeness paths Zq∗ into a hard pseudo distribution by assigning uniform probability over Zq∗ : 1/|Zq∗ |, z ∈ Zq∗ , (11) 0, otherwise. We train the path generator by minimizing the KL divergence between this hard pseudo distribution and the learned path generator distribution: Ldistill = DKL QTϕ (z | q, Aq , Eq , G)∥Pθ (z | q, Eq ) 1 X (12) log Pθ (z | q, Eq ), ≃− ∗ |Zq | ∗ QTϕ (z | q, Aq , Eq , G) =
z∈Zq
where constants independent of θ are omitted. For autoregressive LLM-based path generators, the path likelihood is optimized with standard token-level next-token prediction. The detailed derivation and prompt template are provided in Appendices C and D, respectively. By optimizing Ldistill , the LLM-based path generator is trained with estimated informative paths rather than noisy weak supervision signals. This provides higher-quality intermediate supervision, enabling the generator to generate more precise and informative relation paths. 4.3
KG-grounded Inductive Reasoning
During inference, the trained path generator first produces the K (hyperparameter) relation paths via beam search conditioned on the question and question entities. Each generated path zi ∈ ZqK is then grounded in the KG, yielding grounded evidence of the form (e, zi , Ezi (e)). Each evidence (e, zi , Ezi (e)) represents a grounded reasoning trace from a question entity e to a set of target entities via zi in the KG. This format compactly groups entities that share the same starting entity and relation path, allowing LLMs to perform inductive reasoning over candidate answer sets rather than processing numerous individual KG triples. Furthermore, PATH ISE-estimated pseudo supervision guides the path generator towards informative relation paths, thereby reducing noisy context and mitigating hallucinations for reasoning over the input question. The overall process is formulated as: ZqK = Top -Kz Pθ (z | q, Eq ), WqK = {(e, zi , Ezi (e)) | e ∈ Eq , zi ∈ ZqK }, Â = LLM(q, WqK ). 6
(13)
The grounded evidence is verbalized and provided to an LLM for KG-grounded inductive reasoning, with the prompt template and an example provided in Appendix D and Appendix J, respectively.
5
Experiment
5.1
Experiment Setups
Datasets. Following prior work [24, 25, 27, 29], we first evaluate PATH ISE on WebQuestionsSP (WebQSP) [45] and ComplexWebQuestions (CWQ) [35], two multi-hop KGQA datasets built on Freebase [1]. WebQSP contains 1- to 2-hop questions from Google search logs with annotated SPARQL queries, while CWQ focuses on compositional and multi-constraint questions of up to 4 hops. To evaluate PATH ISE on another KG, we also conduct experiments on MetaQA [32], a largescale 1- to 3-hop KGQA dataset built on WikiMovies. Following prior work [26, 27], the MetaQA evaluation uses 200 questions for each different hop number. For fair comparison, all experiments follow the same dataset splits as the baselines, with detailed statistics provided in Appendix E. Evaluation Metrics. We adopt F1, Hit, and Hits@1 as the evaluation metrics, following our baselines. F1 measures answer set correctness by computing the harmonic mean of precision and recall. Hit assesses whether any ground-truth answer is presented in the predicted answer set. Hits@1 measures the proportion of questions for which the top-1 predicted answer is correct. Baselines. We compare PATH ISE against SOTA KGQA methods grouped by their supervision paradigms: (1) Prompting with in-context learning; (2) Training without intermediate supervision; (3) Training with weakly supervised paths; and (4) Training with LLM-refined supervision. Details of these baselines are provided in Appendix F. Implementation Details. We implement the MIL estimator as a two-layer Transformer and use gte-large-en-v1.5 [21] to encode questions and relations. We select the top-1 path ranked by the MIL estimator, i.e., T = 1, as pseudo path supervision, and fine-tune LLaMA3.1-8B-Instruct with LoRA [11] as the path generator. During inference, we generate relation paths using beam search with beam size K = 5 and use zero-shot prompting for KG-grounded inductive reasoning. Detailed configurations and parameter studies are provided in Appendices G and I. 5.2
KGQA Performance and Efficiency
KGQA Performance. As shown in Table 1, PATH ISE achieves competitive or best performance across WebQSP, CWQ, and MetaQA. The results for PATH ISE are averaged over three independent runs, with standard deviations reported in Appendix H. We follow prior work [27] to report available MetaQA baselines evaluated on the same subset. The result for PATH ISE are averaged over three independent runs, with standard deviations reported in Appendix H. We summarize three observations. (1) PATH ISE consistently outperforms weakly supervised path baselines, with F1 gains of at least 6.8% and 3.8% on WebQSP and CWQ, respectively. On MetaQA, PATH ISE improves over the best baseline DP by 1.4% in F1 and 3.1% in Hit. These gains suggest that reducing noisy supervision from spurious paths is crucial for KGQA reasoning accuracy. (2) Compared with LLM-refined supervision baselines, PATH ISE achieves comparable or better performance, including a 0.7% F1 gain on WebQSP and a 1.0% Hit gain on CWQ, while avoiding costly LLM-based supervision construction. (3) Methods trained with intermediate supervision generally outperform promptingbased methods and methods trained without intermediate supervision, demonstrating the importance of intermediate supervision signals in KGQA, especially for complex questions in CWQ. Overall, these results validate the effectiveness of PATH ISE for KGQA reasoning, with more fine-grained analyses provided in the following sections. Inference Efficiency. Table 2 shows that PATH ISE achieves efficient inference in terms of both the running time and the number of input/output tokens compared with representative baselines that are competitive in Table 1. PATH ISE and GCR have the lowest input token counts. PATH ISE benefits from PATH ISE-estimated supervision and KG-grounded inductive reasoning, which produce precise relation paths and compact KG evidence (hence the shorter input to the final LLM call) while 7
Table 1: Comparison of KGQA performance (%) across WebQSP, CWQ, and MetaQA. LLM denotes the model used for answer generation. We highlight the best performance in bold and underline the second-best. † indicates that the model is fine-tuned. ∗ indicates results taken from the original paper. WebQSP Method
LLM
CWQ
MetaQA
F1
Hit
Hits@1
F1
Hit
Hits@1
F1
Hit
Hits@1
50.9 68.7 56.1
78.5 86.2 91.2
72.1 81.4 66.9
42.3 51.9 47.5
56.9 59.3 58.4
51.8 54.4 43.0
86.3 93.3
80.1 98.3
87.7 90.4
70.8 72.7 -
76.4 -
75.7 77.4 83.2
47.8 49.5 -
52.9 -
52.1 53.6 61.7
-
-
98.8 -
72.2 67.0 69.2 69.8 76.1 75.8
77.2 81.9 81.1 86.9 89.9 88.9
78.3 76.2 82.9 84.3 81.6
49.4 53.6 58.3 57.1 59.2 57.3
51.2 59.2 65.3 64.6 67.3 63.2
53.7 60.5 59.1 60.7 60.1
88.1 94.9
92.7 96.8
99.4 90.3 95.5
GPT-4o GPT-4o
80.7 78.8
93.3 91.4
-
58.8 62.5
69.0 68.9
-
-
-
-
GPT-4o GPT-4.1
81.3 81.1
91.2 91.6
86.8 86.4
61.5 61.3
69.7 71.9
63.4 62.3
96.2 95.9
99.8 99.7
99.8 99.9
Prompting with in-context learning ToG [34] PoG [2] DoG [26]
GPT-4o GPT-4o GPT-4o
Training without intermediate supervision ReaRev [28] NuTrea∗ [3] KG-Hopper∗ [37]
Qwen-2.5-7B†
Training with weakly supervised paths UniKGQA∗ [14] RoG [24] GNN-RAG [29] GCR [25] SubgraphRAG (100 triples) [19] DP [27]
LLaMA2-Chat-7B† LLaMA2-Chat-7B† GPT-4o GPT-4o GPT-4o
Training with LLM-refined supervision RAPL∗ [44] ReG∗ [46] Ours PATH ISE PATH ISE
Table 2: Efficiency comparison on WebQSP and CWQ. All reported results are averages. Train Sup. denotes training supervision construction. Input and Output denote input and output token. Stage
Inference
Train Sup.
WebQSP
Method GCR [25] SubgraphRAG [19] PoG [2] DP [27]
CWQ
Runtime (s)
# Calls
# Input
# Output
Runtime (s)
# Calls
# Input
# Output
3.5 2.3 10.9 4.4
2.0 1.0 9.1 2.5
349.0 2,610.9 5,190.8 2,552.8
337.6 89.0 277.1 246.7
3.6 2.8 15.6 6.6
2.0 1.0 13.9 3.1
361.3 2,514.1 7,856.9 3,710.2
380.0 93.6 379.0 275.5
PATH ISE
2.1
2.0
277.0
154.5
2.9
2.0
472.9
97.4
RAPL [44] PATH ISE
4.7 6.7
1.0 -
1,792.2 -
330.2 -
5.2 7.5
1.0 -
2,943.8 -
415.9 -
maintaining strong accuracy with low cost and latency. GCR [25] achieves low input token counts by constrained decoding, although its KGQA accuracy is substantially lower than PATH ISE. SubgraphRAG [19] only takes one LLM call per question and has the lowest output token count, because it adopts a retriever to retrieve relevant triples without LLM intervention. PATH ISE takes 2.0 calls and slightly more output tokens, because it uses LLM-based relation generator to generate paths for fetching KG evidence. The other models take 2.0 or more calls, e.g., for 2.5 in the DP [27] model. The running times follow similar patterns to those of the number of LLM calls which is expected. 5.3
Pseudo Supervision Evaluation
We evaluate PATH ISE-estimated supervision using paths extracted from annotated SPARQL queries on WebQSP and CWQ as the ground-truth paths. Due to the cost of evaluating LLM-refined baselines, we randomly sample 200 training questions from each dataset and report averages over five runs. Pseudo Supervision Quality. Figure 3 compares different supervision construction methods, including PATH ISE, LLM-refined supervision from RAPL [44], and heuristic rankings based on F1, recall, and cosine similarity.1 For each method, we rank weakly supervised candidate paths and evaluate whether the top-T selected paths contain a SPARQL-extracted reference path (Hits@T ). 1 F1 and recall compare each path’s target entity set with the answer set; cosine similarity averages similarities between the
question embedding and relation embeddings along the path.
8
Compared with the best LLM-refined variWebQSP CWQ ant using GPT-4o, PATH ISE achieves comparable performance on WebQSP and gains up to 11% on CWQ. Since CWQ contains more complex 3- to 4-hop questions, its weakly supervised candidate set is larger and noisier, making LLM refinement and heuristic ranking more vulnerable to noise and long-context interference. In contrast, PATH ISE estimates path informativeness through MIL without LLM- Figure 3: Hits@T of SPARQL-extracted paths on Webased selection or hand-crafted heuristics, bQSP and CWQ train set. enabling more effective identification of informative paths under noisy candidates. These results demonstrate the effectiveness of PATH ISE in estimating pseudo supervision from answer-level labels. Supervision Construction Efficiency. Table 2 further reports the average cost of constructing intermediate supervision signals. For RAPL [44], this per-instance token cost arises from feeding all weakly supervised paths to LLMs for refinement. Thus, the total supervision refinement cost scales linearly with the training set size. In contrast, PATH ISE uses a lightweight MIL estimator to construct supervision without LLM intervention, substantially reducing resource cost and improving scalability. This validates our motivation to obtain high-quality intermediate supervision through a low-cost estimation approach. 5.4
Ablation Study
To study the contribution of PATH ISE-estimated Table 3: Ablation study on the WebQSP pseudo supervision and its applicability to other KGQA and CWQ. WebQSP CWQ models, we report KGQA performance of different Method variants on WebQSP and CWQ in Table 3. ReplacF1 Hit F1 Hit ing PATH ISE-estimated supervision with weakly superRoG 67.0 81.9 53.6 59.2 vised paths (w/ WSP Sup.) or cosine-similarity-ranked w/ PATH ISE Sup. 72.6 77.1 57.5 65.1 paths (w/ SIM Sup.) results in substantial performance SubgraphRAG 76.1 89.9 59.2 67.3 w/ PATH ISE Sup. 78.4 89.3 60.3 68.8 drops. We further apply PATH ISE-estimated supervision to a path generator baseline (RoG) and a graph rePATH ISE 81.3 91.2 61.5 69.7 w/ WSP Sup. 77.4 88.1 56.0 64.7 triever baseline (SubgraphRAG), both originally trained w/ SIM Sup. 72.1 79.0 51.3 54.7 with weakly supervised paths. Using our supervision improves both models, especially RoG, with relative F1 gains of 8.4% on WebQSP and 7.2% on CWQ. For SubgraphRAG, our supervision brings up to 3.0% F1 improvement. These results validate that PATH ISE-estimated supervision is model-agnostic, which can effectively improve overall KGQA performance across different models. Meanwhile, under the same supervision setting, PATH ISE consistently outperforms the corresponding RoG variants trained with either weakly supervised paths or PATH ISE-estimated supervision. This further verifies the effectiveness of our KG-grounded inductive reasoning design, which grounds generated relation paths into compact KG evidence for more accurate answer generation. We further report a case study, error analysis, and supplementary KGQA results in Appendices J to L.
6
Conclusion
We propose PATH ISE, a lightweight framework for learning high-quality intermediate supervision for KGQA from answer-level labels. PATH ISE identifies informative relation paths from noisy weakly supervised candidates through a transformer-based MIL estimator and distills the resulting pseudo supervision into an LLM path generator. The generated paths are then grounded in the KG to provide compact evidence for answer reasoning. Experiments on WebQSP, CWQ, and MetaQA show that PATH ISE achieves strong KGQA performance and provides reusable supervision that can enhance existing KGQA models, demonstrating an efficient and scalable alternative to noisy weak supervision and costly LLM-based supervision construction. 9
References [1] K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor. Freebase: A collaboratively created graph database for structuring human knowledge. In SIGMOD, pages 1247–1250, 2008. [2] L. Chen, P. Tong, Z. Jin, Y. Sun, J. Ye, and H. Xiong. Plan-on-graph: Self-correcting adaptive planning of large language model on knowledge graphs. NeurIPS, pages 37665–37691, 2024. [3] H. K. Choi, S. Lee, J. Chu, and H. J. Kim. NuTrea: Neural tree search for context-guided multi-hop KGQA. In NeurIPS, pages 35954–35965, 2023. [4] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. [5] D. Dessí, F. Osborne, D. Reforgiato Recupero, D. Buscaldi, and E. Motta. CS-KG: A large-scale knowledge graph of research entities and claims in computer science. In ISWC, page 678–696, 2022. [6] S. Fang, K. Ma, T. Zheng, X. Du, N. Lu, G. Zhang, and Q. Tang. KARPA: A training-free method of adapting knowledge graph as references for large language model’s reasoning path aggregation. In ACL, page 24724–24746, 2024. [7] S. Gong, R. O. Sinnott, J. Qi, C. Paris, P. Nakov, and Z. Xie. Multi-sourced, multi-agent evidence retrieval for fact-checking. arXiv preprint arXiv:2603.00267, 2026. [8] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [9] Y. Gu, S. Kase, M. Vanni, B. Sadler, P. Liang, X. Yan, and Y. Su. Beyond I.I.D.: Three levels of generalization for question answering on knowledge bases. In WWW, pages 3477–3488, 2021. [10] J. Hense, M. Jamshidi Idaji, O. Eberle, T. Schnake, J. Dippel, L. Ciernik, O. Buchstab, A. Mock, F. Klauschen, and K.-R. Müller. xMIL: Insightful explanations for multiple instance learning in histopathology. In NeurIPS, pages 8300–8328, 2024. [11] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. [12] M. Ilse, J. Tomczak, and M. Welling. Attention-based deep multiple instance learning. In ICML, pages 2127–2136, 2018. [13] J. Jang and H.-Y. Kwon. Are multiple instance learning algorithms learnable for instances? In NeurIPS, pages 10575–10612, 2024. [14] J. Jiang, K. Zhou, W. X. Zhao, and J.-R. Wen. UniKGQA: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph. In ICLR, 2023. [15] P. Laban, W. Kryściński, D. Agarwal, A. R. Fabbri, C. Xiong, S. Joty, and C.-S. Wu. SummEdits: Measuring LLM ability at factual reasoning through the lens of summarization. In EMNLP, pages 9662–9676, 2023. [16] Y. Lan, G. He, J. Jiang, J. Jiang, W. X. Zhao, and J.-R. Wen. Complex knowledge base question answering: A survey. IEEE Transactions on Knowledge and Data Engineering, 35(11):11196–11215, 2023. [17] C. Lei, Y. Chang, N. Lipovetzky, and K. A. Ehinger. Planning-driven programming: A large language model programming workflow. In ACL, pages 12647–12684, 2025. [18] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS, pages 9459–9474, 2020. [19] M. Li, S. Miao, and P. Li. Simple is effective: The roles of graphs and large language models in knowledge-graph- based retrieval-augmented generation. In ICLR, 2025. [20] T. Li, X. Ma, A. Zhuang, Y. Gu, Y. Su, and W. Chen. Few-shot in-context learning for knowledge base question answering. In ACL, pages 6966–6980, 2023. [21] Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, and M. Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023.
10
[22] Z. Li, S. Fan, Y. Gu, X. Li, Z. Duan, B. Dong, N. Liu, and J. Wang. FlexKBQA: A flexible LLM-powered framework for few-shot knowledge base question answering. In AAAI, pages 18608–18616, 2024. [23] H. Luo, H. E, Y. Guo, Q. Lin, X. Wu, X. Mu, W. Liu, M. Song, Y. Zhu, and L. A. Tuan. KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search. In ICML, 2025. [24] L. Luo, Y.-F. Li, G. Haffari, and S. Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. In ICLR, 2024. [25] L. Luo, Z. Zhao, C. Gong, G. Haffari, and S. Pan. Graph-constrained reasoning: Faithful reasoning on knowledge graphs with large language models. In ICML, 2025. [26] J. Ma, Z. Gao, Q. Chai, W. Sun, P. Wang, H. Pei, J. Tao, L. Song, J. Liu, C. Zhang, et al. Debate on graph: A flexible and reliable reasoning framework for large language models. In AAAI, pages 24768–24776, 2025. [27] J. Ma, N. Qu, Z. Gao, R. Xing, J. Liu, H. Pei, J. Xie, L. Song, P. Wang, J. Tao, and Z. Su. Deliberation on priors: Trustworthy reasoning of large language models on knowledge graphs. In NeurIPS, 2025. [28] C. Mavromatis and G. Karypis. ReaRev: Adaptive reasoning for question answering over knowledge graphs. In Findings of EMNLP, pages 2447–2458, 2022. [29] C. Mavromatis and G. Karypis. GNN-RAG: Graph neural retrieval for efficient large language model reasoning on knowledge graphs. In ACL, pages 16682–16699, 2025. [30] OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, Cai, and et al. GPT-4 technical report. arXiv preprint arXiv:2312.11805, 2024. [31] S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, and X. Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering, 36(7):3580–3599, 2024. [32] H. Puerto, G. Şahin, and I. Gurevych. MetaQA: Combining expert agents for multi-skill question answering. In EACL, pages 3566–3580, 2023. [33] Y. Sui, Y. He, N. Liu, X. He, K. Wang, and B. Hooi. FiDeLiS: Faithful reasoning in large language model for knowledge graph question answering. In ACL, 2025. [34] J. Sun, C. Xu, L. Tang, S. Wang, C. Lin, Y. Gong, L. M. Ni, H.-Y. Shum, and J. Guo. Think-on-Graph: Deep and responsible reasoning of large language model on knowledge graph. In ICLR, 2024. [35] A. Talmor and J. Berant. The web as a knowledge-base for answering complex questions. In NAACL, pages 641–651, 2018. [36] D. Vrandečić and M. Krötzsch. Wikidata: A free collaborative knowledgebase. Communications of the ACM, 57:78–85, 2014. [37] S. Wang and Y. Yu. KG-Hopper: Empowering compact open LLMs with knowledge graph reasoning via reinforcement learning. arXiv preprint arXiv:2603.21440, 2026. [38] S. Wang, W. Fan, Y. Feng, L. Shanru, X. Ma, S. Wang, and D. Yin. Knowledge graph retrieval-augmented generation for LLM-based recommendation. In ACL, pages 27152–27168, 2025. [39] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In ICLR, 2023. [40] Y. Wang, S. Fan, M. Wang, S. Gao, C. Wang, and N. Yin. Damr: Efficient and adaptive context-aware knowledge graph question answering with llm-guided mcts. In ICLR, 2025. [41] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chainof-Thought prompting elicits reasoning in large language models. In NeurIPS, pages 24824–24837, 2023. [42] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [43] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, pages 2369–2380, 2018.
11
[44] T. Yao, H. Li, Z. Shen, P. Li, T. Liu, and K. Zhang. Learning efficient and generalizable graph retriever for knowledge-graph question answering. arXiv preprint arXiv:2506.09645, 2025. [45] W.-t. Yih, M. Richardson, C. Meek, M.-W. Chang, and J. Suh. The value of semantic parse labeling for knowledge base question answering. In ACL, pages 201–206, 2016. [46] D. Zou, Y. Chen, M. Li, S. Miao, C. Liu, B. Han, J. Cheng, and P. Li. Weak-to-strong GraphRAG: Aligning weak retrievers with large language models for graph-based retrieval augmented generation. arXiv preprint arXiv:2506.22518, 2025.
A
Relation Path Example
We provide an illustrative example on how a relation path is grounded over a KG. Consider a relation path z = {parent → play_for}. Given a starting entity e = LeBron James, the reachable entity set Ez (e) contains all entities that can be reached by following the relations in z from e. Traversing the path z from the entity LeBron James can ground a KG reasoning path: parent
play_for
LeBron James −−−−→ Bronny James −−−−−→ Los Angeles Lakers. Thus, Ez (LeBron James) = {Los Angeles Lakers}.
B
Negative Path Sampling
Negative paths provide contrastive supervision for learning path patterns that fail to support the answer-level label. This contrastive signal improves the shared path representation, while spurious paths are mainly down-weighted through attention-based competition within positive bags. We therefore sample negative paths that are structurally or semantically close to weakly supervised paths, yielding harder contrastive examples for MIL training. Specifically, given a question q and its weakly supervised path set Zeq+ , we sample negatives from Zq− according to four types: • Truncated negatives. We select negative paths that match a prefix of a weakly supervised path. These negatives correspond to truncated weakly supervised paths, stopping before the full path reaches the answer. • Extended negatives. We select negative paths for which at least one prefix is a weakly supervised path. These negatives extend from a weakly supervised path with additional relations but no longer reach an answer entity. • Deviated negatives. We select negative paths that share a prefix with a weakly supervised path but then follow a different relation. These negatives correspond to deviated weakly supervised paths, following a plausible initial reasoning pattern but branching away before reaching an answer entity. • Other negatives. The remaining negative paths in Zq− are treated as other negative candidates. For scalability, we use a fixed budget to control the number of retained negative paths for each training sample. Specifically, we keep at most Nmax = 1000 paths, including weakly supervised paths and sampled negative paths. After reserving |Zeq+ | slots for weakly supervised paths, the remaining budget is allocated to negative paths: Nneg = max(0, Nmax − |Zeq+ |). We divide this negative budget among four types of negatives with fixed proportions: Ntrunc = ⌊ρtrunc Nneg ⌋,
Next = ⌊ρext Nneg ⌋,
Ndev = ⌊ρdev Nneg ⌋,
and assign the remaining budget to other negatives: Nother = Nneg − Ntrunc − Next − Ndev . In our implementation, we set ρtrunc = 0.1, ρext = 0.4, ρdev = 0.3, and ρother = 0.2. 12
If the number of candidates in a category exceeds its budget, we rank candidates by question–path similarity and retain the top-ranked paths. The question–path similarity of a path is computed by first measuring the similarity between the question embedding and each relation embedding along the path, and then averaging these relation-level similarities as the final path-level score. For other negatives, we additionally consider relation overlap with weakly supervised paths and fill the remaining slots by random sampling. Finally, we combine the selected truncated, extended, deviated, and other negatives to form the negative path set used for MIL training.
C
Derivation of Path Distillation
Given the selected informative path set Zq∗ , we define a hard pseudo distribution: 1/|Zq∗ |, z ∈ Zq∗ , QTϕ (z | q, Aq , Eq , G) = 0, otherwise.
(14)
The path generator is trained by minimizing: Ldistill = DKL QTϕ (z | q, Aq , Eq , G)∥Pθ (z | q, Eq ) = Ez∼QTϕ log QTϕ (z | q, Aq , Eq , G) − log Pθ (z | q, Eq )
(15)
= −Ez∼QTϕ [log Pθ (z | q, Eq )] + CONST. Since QTϕ is uniform over Zq∗ , we have: Ldistill = −
1 X log Pθ (z | q, Eq ) + CONST. |Zq∗ | ∗
(16)
z∈Zq
The constant term is independent of θ and is omitted in the final optimization objective. For an autoregressive LLM, the likelihood of a relation path is factorized as: log Pθ (z | q, Eq ) =
|z| X
log Pθ (zt | z<t , q, Eq ).
(17)
t=1
where zt denotes the t-th token in the serialized relation path, z<t denotes the preceding tokens, and |z| denotes the sequence length.
D
Prompt Template
In this section, we illustrate all prompt templates used in this work. Relation Path Generation Prompt. As shown in Figure 4, the prompt template for relation path generation takes a question and its topic entity as input, and instructs the LLM to generate a relation path consisting of a sequence of relations, i.e., z = {r1 , r2 , . . . , rl }. KG-grounded Inductive Reasoning Prompt. As shown in Figure 5, the prompt for KG-grounded inductive reasoning takes a question and K grounded KG reasoning paths as input. The LLM is instructed to select all possible answers from the end entities mentioned in the reasoning paths, where each path contains a topic entity, a relation sequence, and its corresponding end entities.
E
Datasets Statistics
Table 4 reports the overall scale of each dataset, including the associated KG, numbers of entities, relations, triples, training and test questions, and the maximum reasoning hop. Table 5 provides the distribution of answer set sizes for WebQSP and CWQ, which helps characterize the answer sparsity and complexity of these two datasets. 13
Relation Path Generation Prompt Instruction: Reasoning path is a sequence of relations in the Knowledge Graph that connects the topic entity in the question to answer entities. Given a question, please generate a reasoning path in the Knowledge Graph starting from the topic entity to answer the question. Input: Question: <Question> Topic entity: <Topic Entity> Output: <PATH> r1 → r2 → · · · → rl </PATH> Figure 4: The prompt template for relation path generation. KG-grounded Inductive Reasoning Prompt Instruction: You are a helpful and precise assistant for answering questions based on the provided reasoning paths on a knowledge graph. Please return all the possible answers from the entities mentioned in the reasoning paths. Please return each answer at a new line. Input: Reasoning Paths: [PATH1] Topic Entity: <Topic Entity>, Relation Path: r1 → r2 → · · · → rl End Entities: <End Entity1 > <SEP> <End Entity2 > <SEP> · · · [PATH2] ··· Question: <Question> Output: Answer1 Answer2 ··· Answern Figure 5: The prompt template for KG-grounded inductive reasoning.
F
Baselines
We compare PATH ISE against SOTA KGQA methods grouped into four categories: (1) prompting with in-context learning; (2) Training without intermediate supervision; (3) Training with weakly supervised paths; and (4) Training with LLM-refined supervision. The details of each baseline are described as follows.
Table 4: Statistics of the datasets. Dataset
KG
WebQSP CWQ MetaQA
Freebase Freebase Wiki-Movie
Entities
Relations
Triples
Train
Test
Max Hop
License
2,566,291 2,566,291 43,234
7,058 7,058 9
8,309,105 8,309,105 133,592
2,826 27,639 329,282
1,628 3,531 30,903
2 4 3
CC BY 4.0 -
14
Table 5: Statistics of number of answers on WebQSP and CWQ. Dataset #Ans = 1 2 ≤ #Ans≤ 4 5 ≤ #Ans≤ 9 #Ans≥ 10 WebQSP CWQ MetaQA
51.2% 70.6% 37.9%
27.4% 19.4% 29.6%
8.3% 6% 12.6%
12.1% 4% 19.9%
Prompting with In-context Learning. These methods treat LLMs as agents and guide them to reason step by step over the KG through few-shot in-context learning, without model training. • ToG [34] treats the LLM as an agent that performs beam search over the KG, iteratively expanding relations and entities until the retrieved evidence is judged sufficient for answering. • PoG [2] focuses on explicit knowledge paths rather than local triple expansion, using dynamic multi-hop path exploration with graph reduction and multi-step pruning. • FiDeLis [33] combines a training-free Path-RAG candidate retrieval with deductive verification beam search, where each reasoning step is verified against the KG before proceeding to the next step or producing the final answer. • DoG [26] is a training-free KGQA framework where multiple prompted LLM roles debate over candidate subgraphs, while an answer-trying module determines whether to stop or continue reasoning. Training without Intermediate Supervision. These methods train KGQA models using only answer-level supervision, without explicit intermediate supervision such as supporting paths, triples, or subgraphs. • ReaRev [28] formulates KGQA as adaptive instruction execution over a question-specific subgraph. It uses a Graph Neural Network (GNN) to emulate breadth-first reasoning and refines question instructions with KG-aware feedback. • NuTrea [3] models KGQA as neural tree search, combining forward expansion with subtree backup messages to inject broader KG context into a GNN reasoner. • KG-Hopper [37] trains a compact open LLM with reinforcement learning, enabling multihop traversal, backtracking, and answer selection within a single reasoning round rather than relying on an external multi-step controller. Training with Weakly Supervised Paths. These methods train KGQA models with weakly supervised paths, where all paths connecting question entities to answer entities are used as intermediate supervision signals. • UniKGQA [14] unifies retrieval and reasoning in a PLM-based architecture by combining question–relation semantic matching with matching-information propagation over the KG. • RoG [24] RoG adopts a planning–retrieval–reasoning pipeline, where an LLM first generates relation paths, retrieves matching KG reasoning paths, and then reasons over the retrieved paths for answer generation. • GNN-RAG [29] uses a GNN to score answer candidates over a question-specific subgraph and verbalizes shortest paths from question entities to predicted candidates for LLM-based reasoning. • GCR [25] incorporates KG structure into LLM decoding through a KG-Trie, using a KGspecialized LLM to generate KG-faithful reasoning paths and a general LLM to aggregate them for final answer prediction. • SubgraphRAG [19] retrieves compact evidence subgraphs by scoring triples with a lightweight model enhanced by directional distance encoding, and then prompts an LLM to reason over the retrieved triples. • DP [27] distills KG structural priors into an LLM through supervised fine-tuning and Kahneman-Tversky Optimization. It further uses reasoning introspection to verify constraint satisfaction and trigger backtracking when needed. 15
Table 6: Stability analysis of PATH ISE with different LLM backbones. Results are reported as mean values with standard deviations shown as subscripts. WebQSP
LLM GPT-4o GPT-4.1
CWQ
MetaQA
F1
Hit
Hits@1
F1
Hit
Hits@1
F1
Hit
Hits@1
81.3±0.3 81.1±0.3
91.2±0.4 91.6±0.4
86.8±0.7 86.4±0.6
61.5±0.5 61.3±0.3
69.7±0.3 71.9±0.5
63.4±0.6 62.3±0.4
96.2±0.2 95.9±0.2
99.8±0.2 99.7±0.1
99.8±0.1 99.9±0.1
Training with LLM-refined supervision. These methods use LLMs to construct refined intermediate supervision signals for training downstream KGQA models. • RAPL [44] trains a lightweight graph retriever with LLM-rationalized path labels, directed line-graph transformation, and triple-level message passing to retrieve compact KG evidence for downstream reasoning. • ReG [46] refines weak supervision with LLM feedback and reorganizes retrieved graph evidence into logic-preserving chains, improving the compatibility between weak retrievers and downstream LLM reasoners.
G
Implementation Details
All experiments are conducted on a machine with an NVIDIA A100 GPU and 120 GB RAM. Following prior work, we use gte-large-en-v1.5 [21], frozen during training, to generate question and relation embeddings for fair comparison. The MIL estimator consists of two Transformer layers with a hidden dimension of 128 and is trained with the AdamW optimizer using a learning rate of 1 × 10−4 for 600 epochs on WebQSP, 200 epochs on CWQ. For relation path retrieval, we follow prior work [19, 27] to set the maximum hop L = 2 on WebQSP and L = 3 on CWQ. For MetaQA, which consists of fixed-hop subsets, we set L to the corresponding hop number of each subset, i.e., L = 1, 2, 3 for the 1-hop, 2-hop, and 3-hop subsets, respectively. We select the top-1 relation path, i.e., T = 1, ranked by the MIL estimator from the weakly supervised paths as pseudo path supervision. This supervision is used to fine-tune the path generator, LLaMA3.18B-Instruct, with LoRA [11], where the LoRA rank is 16 and the LoRA alpha is 32. The path generator is fine-tuned for 5 epochs with a learning rate of 5 × 10−5 . During inference, we use beam search with a beam size of 5, i.e., K = 5, to generate relation paths. For KG-grounded inductive reasoning, we use zero-shot prompting with LLMs to generate the final answers.
H
Standard Deviation of KGQA Performance
Following prior work [27], we conduct three independent runs and report the mean and standard deviation in Table 6.
I
Parameter Study
We conduct parameter studies to investigate the impact of key hyperparameters. When varying one parameter, we keep all the other parameters at their default values, as described in Appendix G. Figure 6a reports the average answer recall and average generation time under different beam sizes K. As K increases, answer recall improves because more generated relation paths are explored, while generation time also increases due to the larger decoding space. To balance answer coverage and inference efficiency, we set K = 5 as the default beam size in our experiments. Figure 6b reports the impact of the number of selected pseudo supervision paths T on final KGQA performance, measured by F1 and Hit. On both WebQSP and CWQ, increasing T leads to higher Hit but lower F1. This is because selecting more paths improves coverage of potentially informative paths, which benefits Hit, but also introduces more noisy or spurious supervision signals, which can reduce answer precision and thus lower F1. Considering both performance and training efficiency, we set T = 1 as the default value. 16
WebQSP Gen. Time
CWQ Gen. Time
WebQSP F1 CWQ F1
2.5
90
75
85
70
80
65
75
0.5
2
3
4
5
6
7
K (Beam Search Size)
8
9
10
60
0.0
(a) Answer recall and generation runtime under different beam sizes K.
Hit (%)
80
1.5 1.0
1
WebQSP Hit CWQ Hit
2.0
F1 (%)
CWQ Recall
Avg. Gen. Time (s)
Avg. Ans. Recall
WebQSP Recall
0.875 0.850 0.825 0.800 0.775 0.750 0.725
1
2 T
3
70
(b) KGQA performance under different numbers of selected paths T .
Figure 6: Parameter study results. Table 7: Case study of intermediate supervision construction on a CWQ training question. Green denotes the SPARQL-consistent path, and red denotes an incorrect selected path. Question: What language would be spoken in the region where the Khorshid newspaper is circulated? Question Entity: Khorshid Weakly Supervised Paths: Path 0: book.periodical.language Path 1: book.periodical.language→language.human_language.countries_spoken_in→location.country.official_language Path 2: book.newspaper.circulation_areas→film.film_subject.films→film.film.language Path 3: book.newspaper.circulation_areas→location.country.languages_spoken Path 4: book.periodical.language→language.human_language.language_family→language.language_family.languages Top-1 LLM-refined Supervision (RAPL + GPT-4o): book.periodical.language Top-1 PATH ISE-estimated Supervision: book.newspaper.circulation_areas→location.country.languages_spoken
Case Study
We provide two case studies to illustrate how PATH ISE constructs intermediate supervision and performs KG-grounded inductive reasoning. Table 7 shows a CWQ training question together with its weakly supervised paths. The question asks for the language spoken in the region where the Khorshid newspaper is circulated. Although Path 0 directly reaches the newspaper language, it ignores the key semantic constraint “region where the newspaper is circulated” and is therefore inconsistent with the annotated SPARQL query path. In contrast, Path 3 first follows the newspaper circulation region and then retrieves the languages spoken in that region, matching the intended reasoning process.
0
Path Index
J
10
1
5
2
0 5
3
10
4 0 1 2 3 4 5 6 7 8 9
Epoch The LLM-refined baseline RAPL selects Path 0, which is a shortcut path that reaches a plausible answer but does not follow Figure 7: Attention heatmap of the question semantics. PATH ISE, however, selects Path 3 the MIL estimator over candidate as the top-1 supervision path, which is consistent with the paths for the case in Table 7 across SPARQL annotation. Figure 7 further visualizes the attention epochs. scores of the MIL estimator over training epochs. As training proceeds, the MIL estimator gradually assigns higher attention to the SPARQL-consistent path, showing that the MIL objective can identify informative paths from noisy weakly supervised candidates.
Table 8 further illustrates the effect of learned supervision on KG-grounded inductive reasoning. When the path generator is trained with weakly supervised paths, it produces noisy or less reliable paths. Although one generated path reaches the gold answer, it relies on a spurious parent–children relation chain that does not match the question semantics. Meanwhile, the second path captures the relevant sibling relation but is incomplete, resulting in unresolved Freebase IDs as predictions. 17
Table 8: Case study of PATH ISE KG-grounded inductive reasoning on a WebQSP test question. Question
Who were demeter’s siblings?
Answer
Zeus
PATH ISE w WSP
Generated Paths: [PATH 1] Start Entity: Demeter Path: fictional_universe.fictional_character.parents→fictional_universe.fictional_character.children End Entities: Hera, Poseidon, Hestia, Hades, Zeus [PATH 2] Start Entity: Demeter Path: fictional_universe.fictional_character.siblings End Entities: m.0gwhv5j, m.0j85m5t Predicted Answer: m.0gwhv5j, m.0j85m5t
PATH ISE
Generated Paths: [PATH 1] Start Entity: Demeter Path: fictional_universe.fictional_character.siblings→ fictional_universe.sibling_relationship_of_fictional_characters.siblings End Entities: Zeus Predicted Answer: Zeus
This shows that noisy weak supervision may preserve broad informative path coverage, but can still mislead the generator toward spurious or incomplete paths and degrade final answer accuracy. In contrast, PATH ISE generates a relation path that explicitly follows the sibling relation structure and grounds it to the correct answer entity. This example shows that PATH ISE-estimated supervision helps the path generator produce more semantically aligned relation paths, leading to cleaner grounded evidence and more accurate final answer generation.
K
Error Analysis Path Generation Error
Reasoning Error
200 150 100
14.8% 23.8% 85.2%
50
76.2%
0
PathISE
# Samples (hit = 0)
# Samples (hit = 0)
WebQSP
PathISE w/ WSP
CWQ
1400 1200 1000 800 600 400 200 0
30.9% 45.9% 54.1%
PathISE
69.1%
PathISE w/ WSP
Figure 8: Error analysis on failed cases with Hit = 0 on WebQSP and CWQ. Errors are categorized into path generation errors and reasoning errors. We further analyze failed cases on WebQSP and CWQ where the predicted answer set does not contain any ground-truth answer, i.e., Hit = 0. We categorize these errors into two types: path generation errors, where the generated relation paths fail to retrieve evidence containing the correct answer, and reasoning errors, where the grounded evidence contains useful information but the LLM fails to produce the correct answer. As shown in Figure 8, PATH ISE substantially reduces the number of failed cases compared with the variant trained with weakly supervised paths (PATH ISE w/ WSP). The proportion of path generation errors decreases from 85.2% to 76.2% on WebQSP and from 69.1% to 54.1% on CWQ. This indicates that PATH ISE-learned pseudo supervision effectively improves the path generator by reducing noisy or irrelevant generated paths. 18
Furthermore, the results show that reasoning errors are substantially more prominent on CWQ than on WebQSP, especially after path generation errors are reduced. This suggests that for complex multi-hop questions, even when the generated paths retrieve relevant evidence, the LLM may still fail to infer the correct answer due to reasoning hallucination, evidence misinterpretation, or conflicts with its parametric knowledge.
L
Supplementary KGQA Results Table 9: Breakdown of KGQA performance (%) over # hops. WebQSP-sub
Model
1
2
CWQ-sub Overall
1
≥3
2
Overall
F1
Hit
F1
Hit
F1
Hit
F1
Hit
F1
Hit
F1
Hit
F1
Hit
SubgraphRAG GCR
80.3 71.7
89.7 90.1
74.4 69.7
83.0 88.5
78.3 71.0
87.5 89.6
57.0 60.6
59.7 67.3
63.3 65.6
68.0 74.5
51.6 47.7
52.0 51.5
60.8 63.1
64.7 71.1
PATH ISE w Llama2-7B w Qwen3.1-8B
84.0 83.0 82.7
93.0 92.2 91.8
79.8 77.6 78.8
92.3 90.7 91.6
82.6 81.2 81.4
92.7 91.7 91.7
65.3 64.5 63.9
75.0 73.8 73.2
73.1 69.7 71.3
81.6 78.6 80.2
52.3 51.4 50.8
60.2 59.6 59.0
69.7 67.1 68.0
78.5 76.1 77.0
To further analyze the KGQA performance of PATH ISE across questions with different reasoning depths and different LLMs for path generation, we report the breakdown of KGQA performance by the number of hops in Table 9. Following SubgraphRAG [19], we evaluate on WebQSP-sub and CWQ-sub, where questions whose answer entities are absent from the KG are removed. We compare PATH ISE with representative competitive baselines, including SubgraphRAG [19] and GCR [25], and further report variants of PATH ISE with different LLMs used for relation path generation. The results show that PATH ISE consistently outperforms the baselines across different hop settings on both datasets. On WebQSP-sub, PATH ISE improves over SubgraphRAG and GCR on both 1-hop and 2-hop questions, indicating that the effectiveness of PATH ISE on both simple and multi-hop reasoning. On CWQ-sub, PATH ISE achieves substantial performance gaps over the baselines on 1-hop and 2-hop questions, and also improves performance on questions requiring three or more hops. Moreover, PATH ISE remains effective with different path generator backbones, showing that the learned path supervision is not tied to a specific LLM. These results further confirm the robustness of PATH ISE across reasoning depths and model backbones.
M
Limitations
While PATH ISE provides an efficient way to estimate path-level supervision from answer-level labels, it still has several limitations. First, the estimated supervision may still contain errors. As shown in Figure 3, PATH ISE improves the hit rate of SPARQL-consistent paths compared with several baselines, but it does not perfectly recover all gold reasoning paths. This indicates that the MIL estimator can still assign high scores to imperfect or spurious paths, especially when the weakly supervised candidate set is large and noisy. Second, PATH ISE still relies on accurate relation path generation during inference. As shown in the error analysis in Appendix K, path generation failures remain a major source of errors, even though PATH ISE reduces their proportion compared with the weakly supervised variant. When the generated paths fail to retrieve evidence containing the correct answer, the downstream LLM reasoner cannot recover the answer from the provided KG evidence. Finally, PATH ISE assumes that the KG contains answer-reaching paths from question entities during supervision construction. When the KG is incomplete, or when no valid path exists between the question entities and answer entities, both supervision estimation and path generation may be constrained. Future work may extend PATH ISE to more incomplete KG settings by incorporating missing-edge prediction, entity linking uncertainty, or hybrid KG-text evidence.
19