G-MARK: Grounded Multi-Agent Reasoning for Cooperative Driving via Knowledge Graphs

G-MARK: Grounded Multi-Agent Reasoning for Cooperative Driving via Knowledge Graphs Bhavya Gupta1 , Onat Gungor2 , Tajana Rosing1 1

University of California, San Diego, CA, USA West Virginia University, Morgantown, WV, USA {b5gupta, tajana}@ucsd.edu, onat.gungor@wvu.edu

arXiv:2608.19964v1 [cs.LG] 20 Aug 2026

2

the ego vehicle directly observed the object, or how much uncertainty remains in the object estimate. Recent language-based and vision-language driving systems study high-level driving reasoning through visual question answering, explanation generation, behavior planning, and language-conditioned control [8]–[11]. Cooperative driving systems such as V2V-LLM [12] and V2V-GoT [13] extend this paradigm to multi-agent reasoning. Yet, their intermediate states are often encoded as prompts, generated text, free-form rationales, or hidden multimodal features rather than structured object-level cooperative evidence. This limits traceability for safety-critical decisions that depend on objects outside the ego vehicle’s direct view but observed by a connected partner. We propose G-MARK, a grounded multi-agent reasoning framework that represents cooperative driving scenes as provenance-aware knowledge graphs. Operating above the perception layer, G-MARK consumes processed multi-agent observations, associates compatible observations into shared object hypotheses, and preserves weak or unmatched observations as uncertain candidates rather than discarding them. The resulting graph explicitly stores source-agent support, egoversus-partner visibility, uncertainty, cross-agent disagreement, I. I NTRODUCTION spatial relations, motion cues, and planning context. KGAutonomous driving systems must make safety-critical based task heads then query this shared KG to support decisions from incomplete and viewpoint-dependent observa- object grounding, occlusion reasoning, hidden-object discovery, tions [1]. Objects that are crucial for planning may be occluded, motion prediction, control selection, and trajectory forecasting. We evaluate G-MARK on V2V-GoT-QA, a cooperative drivoutside the ego vehicle’s field of view, or observable only by a connected partner vehicle [2]. Vehicle-to-vehicle cooperation ing benchmark spanning perception, prediction, and planning can mitigate this partial observability by sharing complementary tasks. Compared with V2V-GoT [13], G-MARK improves scene evidence across agents [3]. However, when the ego occlusion reasoning by 42.2%, hidden-object discovery by vehicle must reason about an object it has not directly observed, 12.3%, and control-selection error by 13.1%. For future the system should retain more than the object hypothesis itself. trajectory forecasting, G-MARK reaches near-parity in average It should also preserve its provenance: which agent observed L2 error while using approximately 25.6× lower structured the object, how reliable the observation is, and why the object communication. Our contributions are summarized as follows: matters for downstream planning [4]. • We introduce a provenance-aware cooperative KG that Existing cooperative perception methods improve aupreserves object hypotheses, source-agent support, visitonomous driving perception through early fusion, late fusion, bility, uncertainty, disagreement, spatial relations, motion and feature-level fusion [5]–[7]. These methods primarily cues, and planning context. aim to aggregate multi-agent observations into more accurate • We propose task-conditioned delayed evidence fusion, perception outputs, often by compressing shared evidence which retains weak partner-only observations so downinto latent feature maps or final detections. However, this stream tasks can decide which evidence is relevant. compression can discard the evidence structure needed for • We show that our approach can support object reasoning, downstream reasoning. Hence, downstream modules cannot motion prediction, control selection, and future trajectory infer which agent supported an object hypothesis, whether forecasting with compact structured communication. Abstract—Autonomous driving systems must operate under partial observability, where safety-critical objects may be occluded or visible only to neighboring connected vehicles. Vehicle-tovehicle cooperation can reduce this uncertainty, but existing cooperative driving methods often compress multi-agent evidence into latent features or hidden multimodal states. As a result, they obscure which agent observed each object, whether the object is visible to the ego vehicle, and how conflicting evidence affects downstream decisions. We propose G-MARK, a grounded multi-agent reasoning framework that converts cooperative objectcentric observations into explicit provenance-aware knowledge graphs (KGs). The resulting KGs preserve object hypotheses together with their source attribution, ego-versus-partner visibility, uncertainty, conflicts, spatial relations, and planning-relevant context. G-MARK then derives a shared feature representation from these KGs, enabling lightweight task heads to support object reasoning, motion prediction, control selection, and trajectory forecasting. Compared with the state-of-the-art baseline, GMARK improves occlusion reasoning accuracy by 42.2%, reduces control-selection error by 13.1%, and achieves comparable trajectory-planning accuracy with a 25.6× smaller structured communication payload. Our code is available at this repository. Index Terms—Cooperative Driving, Multi-Agent Knowledge Graphs, Vehicle-to-Vehicle Communication, Motion Planning

TABLE I C OMPARISON OF COOPERATIVE EVIDENCE EXPOSED BY RELATED METHOD FAMILIES .

Capability

Multi-agent evidence Object provenance Ego-versus-partner visibility Uncertainty / conflict Planning relevance Reusable across tasks Comm. efficiency

Cooperative Perception [6] / V2X Fusion [7]

Language / VLM Driving Reasoning [8]

Structured Scene and KG-based Reasoning [14]

Cooperative Language-Based Reasoning [13]

G-MARK (Ours)

✓ — — — △ — △

△ — △ △ ✓ ✓ △

— △ — △ △ △ —

✓ △ △ △ ✓ ✓ △

✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ Explicitly modeled

△ Partially exposed

II. R ELATED W ORK Prior work relevant to G-MARK spans cooperative perception and V2X fusion [5]–[7], language- and VLM-based driving reasoning [8]–[11], structured scene and KG-based reasoning [14], [15], and cooperative language-based reasoning [12], [13], [16], [17]. G-MARK targets the gap between these research families by constructing an explicit cooperative evidence representation from perception outputs. This representation preserves source support, ego-versus-partner visibility, uncertainty, disagreement, and planning context before downstream task inference. These properties are important because cooperative driving decisions depend not only on which objects are present, but also on who observed them, whether they are visible to the ego vehicle, and why they affect planning [4]. Table I compares how existing method families expose these evidence variables for downstream reasoning. Cooperative Perception and V2X Fusion. Cooperative perception improves scene understanding beyond a single ego viewpoint by sharing raw observations, object predictions, learned features, BEV representations, or sparse spatial messages across agents. Representative systems include Where2Comm [5], V2XViT [7], and CoBEVT [6]. These methods provide strong perception backbones, but their fused outputs rarely expose source support, ego-versus-partner visibility, uncertainty, or disagreement as reusable object-level evidence. Language and VLM-based Driving Reasoning. Language, vision-language, and multimodal large language models have recently been used for driving question answering, explanation, planning, and control [8]–[11]. These methods broaden autonomous-driving evaluation from perception accuracy to reasoning-oriented tasks, including object grounding, scene understanding, decision explanation, and planning. However, most are ego-centric or non-cooperative, motivating a layer that preserves source-agent support and cross-agent disagreement. Structured Scene and KG-based Reasoning. Structured scene representations and knowledge graphs encode entities, attributes, and relations in an explicit form for visual reasoning and decision making [15]. In autonomous driving, systems such

— Not explicitly exposed

as DriveLM [8] and KLDrive [14] show that structured scene facts can support interpretable, planning-oriented reasoning. However, existing driving KG approaches primarily focus on ego-centric scenes, language-grounded QA, or high-level facts rather than cooperative V2V evidence. G-MARK extends this direction with a per-scene cooperative KG that stores object hypotheses, source-agent support, visibility, uncertainty, disagreement, and planning relevance. Cooperative Language-Based Reasoning. Recent cooperative language-based methods extend language reasoning to multi-agent driving. LangCoop uses language for compact collaborative communication [16], CoLMDriver studies LLMbased negotiation [17], V2V-LLM formulates vehicle-to-vehicle reasoning as multimodal QA [12], and V2V-GoT organizes perception, prediction, and planning questions through graphof-thought reasoning [13]. These systems are closest to GMARK in task scope, but their intermediate states are typically language messages, generated answers, reasoning traces, or hidden multimodal activations rather than explicit object-level evidence. G-MARK instead uses a provenance-aware KG, allowing downstream predictions to be traced to source agents, visibility facts, uncertainty, and supporting observations. III. G-MARK F RAMEWORK Fig. 1 provides an overview of G-MARK, a cooperative reasoning layer that operates on top of a cooperative perception pipeline. Given processed multi-agent scene artifacts, including object detections or tracks, agent poses, confidence scores, and planning context, G-MARK constructs a provenanceaware KG for downstream reasoning. Importantly, G-MARK does not consume raw camera or LiDAR streams directly; instead, it abstracts already-processed cooperative perception evidence into a structured graph representation. The central design principle of G-MARK is to postpone irreversible fusion until the downstream reasoning task is specified. Rather than merging all agent observations into a single fused object list, G-MARK represents observing agents, local observations, and object hypotheses as linked graph entities. This structure

Fig. 1.

Overview of the G-MARK cooperative KG reasoning framework.

allows different reasoning heads, such as occlusion reasoning and motion prediction, to query the same cooperative evidence while emphasizing different task-relevant factors. G-MARK proceeds in four stages. First, each CAV’s processed perception output is instantiated as a local evidence KG containing an agent node, observation nodes, and objecthypothesis nodes before cross-agent association. Second, egoside conservative association links compatible local evidence across CAVs into shared object hypotheses, attaches observation nodes as support, and retains weak unmatched candidates. Third, each hypothesis is enriched with provenance, agentrelative visibility, uncertainty, disagreement, spatial relations, motion cues, and planning relevance. Finally, KG-based task heads query the enriched cooperative KG for object-level reasoning, motion prediction, and trajectory planning. A. Cooperative KG Generation

candidate from communicated CAV evidence or a shared hypothesis formed after cross-agent association. This separation preserves the source vehicle, supporting observation, and egoversus-partner visibility of each hypothesis. For example, if a partner observes a pedestrian hidden from the ego vehicle, the agent node identifies the partner, the observation node stores the local detection, and the hypothesis node represents the shared belief that the pedestrian exists. Let At = {a1 , . . . , aNt } denote the set of connected agents available at timestamp t, where Nt is the number of agents at that timestamp. Each agent ai ∈ At reports a set of Mi,t local object observations: (i)

(i)

(i)

Ot = {ot,1 , . . . , ot,Mi,t }. Here, Mi,t is the number of local observations from agent ai at time t, and j = 1, . . . , Mi,t indexes them. Each observation is represented as

G-MARK represents each CAV’s processed perception output, including local observations and V2V-communicated   (i) (i) (i) (i) (i) object hypotheses, as local KG evidence and assembles it into ot,j = xt,j , τt,j , ct,j , ai , t, η t,j , an initial cooperative KG, instead of collapsing it into a flat list of fused objects [14], [15]. A flat object list can indicate that an (i) where j denotes an individual observation, xt,j is the object object exists, but it does not preserve who observed it, whether (i) it is ego-visible or partner-only, how reliable the supporting position or bounding box in a shared coordinate frame, τt,j is (i) evidence is, or whether the object should influence planning. the semantic object type, such as car, truck, or pedestrian, ct,j is These factors are especially important under occlusion and the detection confidence, and ai records the source agent. The (i) partial observability, where safety-relevant evidence may be auxiliary vector η t,j stores optional observation-level attributes, available only through another connected agent [7], [18], [19]. such as velocity, track identity, source flags, or visibility labels. To preserve the evidence structure behind cooperative Graph-level quantities such as hypothesis-level provenance, perception outputs, each local evidence graph separates agent uncertainty, disagreement, and planning relevance are finalized (i) nodes, observation nodes, and object-hypothesis nodes. An after association. Each local observation ot,j is instantiated as O A agent node stores the identity, pose, timestamp, and ego-versus- an observation node vt,i,j ∈ VtO . One agent node vt,i ∈ VtA is partner role of the CAV that produced the local evidence. An created for each connected vehicle ai ∈ At . Each observation O A observation node represents one agent-local object observation node vt,i,j is linked to the agent node vt,i that generated it, (i) ot,j , while an object-hypothesis node represents an object making the source of every observation explicit in the KG.

The CAV-local evidence is then assembled into an initial cooperative KG: Gt = (Vt , Et ), with node set Vt = VtA ∪ VtO ∪ VtH . Here, VtA , VtO , and VtH denote agent nodes, observation nodes, and object-hypothesI removedis nodes, respectively. The relation set Et uses typed relations for source attribution, hypothesis support, spatial context, planning context, and evidence consistency. The source relation connects an agent node to each observation node it produces, while the support relation connects an observation node to the object-hypothesis node it contributes to. Spatial relations, such as front-of, behind, left-of, right-of, and near, are computed in the ego coordinate frame. Planning-context relations, such as near-trajectory and path-relevant, use distances to the ego vehicle’s planned or predicted trajectory. Evidence-consistency relations include cooperatively-supported, indicating support from multiple agents or from a partner agent, and low-conflict, indicating geometrically consistent support. This typed relation design follows standard entity–relation KG representations [15], while provenance is stored explicitly through the support set and provenance record [20]. An object-hypothesis node h ∈ VtH is associated with a support set of compatible observation nodes,

The output of this stage is an initial cooperative KG Gt containing agent nodes, observation nodes, object-hypothesis nodes, typed relations, and traceable evidence variables. The following stages resolve compatible candidate hypotheses through conservative association and enrich each hypothesis with provenance, visibility, uncertainty, disagreement, and planning relevance. B. Conservative Association After local evidence is assembled into the cooperative KG, G-MARK associates compatible candidate hypotheses across CAVs into shared cooperative object hypotheses. Observation nodes are attached as hypothesis support, while unmatched observations are retained as weak candidates. This avoids overmerging nearby but distinct objects while preserving partneronly evidence for occlusion and planning. Following standard tracking and data-association practice, we use a deterministic gating rule to decide which local evidence pairs are eligible for association [21]. For observation-level support attachment, let vpO , vqO ∈ VtO denote two observation nodes, with semantic types τp , τq and positions xp , xq inherited from their corresponding local observations. We define the association gate as   δt (vpO , vqO ) = 1 type match(τp , τq ) ∧ d(xp , xq ) < ϵτp ,τq ,

where d(·, ·) is the distance between observation centers and ϵτp ,τq is a class-dependent association threshold. The function type match(τp , τq ) returns true only when the two observaThus, S(h) contains observation nodes, each corresponding to tions have the same semantic class. We use exact semantic-class (i) an original local observation ot,j . It is represented as matching as a conservative design choice because false crossclass merges can create unsafe object hypotheses. For example, h = (xh , τh , ch , πh , S(h), uh , γh , rh ), nearby vehicle observations can be associated, but a pedestrian where (xh , τh , ch ) denote fused location, semantic type, and and a vehicle remain separate even if they are geometrically confidence; (πh , S(h)) store supporting agents and observa- close. For gated pairs, G-MARK assigns an interpretable distancetions; and (uh , γh , rh ) capture uncertainty, disagreement, and based affinity score, planning relevance.  The provenance record is 1 − d(xp , xq ) , if δ (v O , v O ) = 1, t p q O O O πh = {(ai , vt,i,j ) : vt,i,j ∈ S(h), vt,i,j is produced by ai }. ϵτp ,τq αt,pq =  0, otherwise. This record keeps each object hypothesis traceable to its supporting agents and observations [20]. The KG exposes The affinity score ranks gated pairs by spatial agreement, with cooperative evidence variables that are often lost after immedi- higher values assigned to closer pairs. The gating criterion ate fusion. Source support is computed from the number and follows standard data-association practice [21], while the identity of agents in πh . Ego-relative visibility is represented normalized distance score is an interpretable heuristic used in as an agent-specific relation, G-MARK. When an unmatched observation node is promoted to a candidate hypothesis, its representative state inherits vis(h, ai ) ∈ {visible, occluded, uncertain}, the observation’s semantic type and position, so the same rather than a global label. Downstream heads use this relation semantic and geometric compatibility rule is used for later to distinguish ego-visible hypotheses from partner-only or candidate-hypothesis merging. The association graph induced occluded hypotheses, especially for invisible-object retrieval, by δt (vpO , vqO ) = 1 determines which observation nodes attach occlusion reasoning, and planning-aware object selection. as support evidence. For candidate hypotheses, the same Uncertainty uh captures weak confidence, candidate status, semantic and geometric rule is applied to their representative and limited support; disagreement γh captures inconsistent states to identify candidates eligible for merging. The fused observations; and planning relevance rh measures proximity position xh is computed as a confidence-weighted average of to the ego vehicle’s future path. the observation positions in S(h). S(h) ⊆ VtO .

TABLE II C OOPERATIVE CONTEXT ENRICHMENT CATEGORIES IN G-MARK.

TABLE III KG- BASED TASK - HEAD FAMILIES IN G-MARK.

Category

Role in downstream reasoning

Head family

Role in downstream reasoning

Provenance and support

Stores source agents, support counts, and ego/partner/cooperative status to trace object evidence. Stores ego-relative and agent-relative visibility for occlusion and hidden-object reasoning. Marks weak, ambiguous, or candidate hypotheses so they are not treated as fully reliable. Records geometric or semantic inconsistency among agents to expose conflicting evidence. Stores trajectory proximity, motion cues, and control context.

Selection and visibility reasoning

Retrieves visible, occluded, hidden, or planning-relevant objects using KG evidence. Predicts object or agent behavior from state, motion, support, uncertainty, and path-relative geometry. Predicts controls or future waypoints from risk, planning relevance, geometry, and trajectory context.

Agent visibility

Uncertainty and reliability Cross-agent disagreement Planning relevance

Motion and interaction prediction Control and trajectory planning

Formally, for each object hypothesis h, G-MARK augments the node with an enrichment vector

ϕ(h) = [ϕprov (h), ϕvis (h), ϕunc (h), ϕdis (h), ϕplan (h)], For each object hypothesis, G-MARK summarizes confiwhere ϕprov (h), ϕunc (h), and ϕdis (h) summarize support, dence using a bounded noisy-or-style aggregation [22]: Y provenance, confidence, and association consistency from the ch = 1 − (1 − c(v O )), conservative association stage. The components ϕvis (h) and v O ∈S(h) ϕplan (h) encode ego-relative visibility and distance to the ego O vehicle’s planned or predicted trajectory. The final KG-derived where c(v ) is the confidence stored by observation node O feature vector ψ(h) concatenates object-state attributes, such v . We use ch as a bounded confidence summary rather than as position, semantic type, confidence, and motion cues, with as a calibrated probabilistic fusion model. Unlike immediate the enrichment vector ϕ(h). fusion, the association output is not only a fused object location. The output is an enriched cooperative KG in which downFor every hypothesis, G-MARK stores the support set S(h), stream modules can query each hypothesis through ψ(h) provenance record πh , confidence summary ch , candidate status, without recomputing evidence from raw observations. uncertainty, and disagreement attributes. If an observation node does not pass the association gate with any other observation D. Unified KG Task Heads node, G-MARK retains it as a candidate hypothesis with The enriched KG provides a shared evidence state for task lower support and higher uncertainty. Candidate retention is a heads with different output formats. As shown in Table III, safety-oriented design choice: partner-only or partially observed we group them into object selection and visibility reasoning, objects remain available for downstream inference, but are not motion and interaction prediction, and control and trajectory treated as fully reliable detections. planning, following the standard perception-prediction-planning This module outputs well-supported hypotheses with attached decomposition [23]. Each head operates on KG-derived features observation evidence, merged compatible candidates, and ψ(h) that combine object state, provenance, visibility, uncerretained weak candidates. The next module enriches them with tainty, disagreement, spatial relations, and planning-context provenance, visibility, uncertainty, disagreement, and planning attributes. relevance. For a task family k, we write the inference head as C. Cooperative Context Enrichment
yk = Rk Gt , {ψ(h) : h ∈ VtH } , After hypothesis formation, G-MARK derives context features for each object hypothesis from support, visibility, where Gt is the enriched cooperative KG, ψ(h) is the KGrelative geometry, and ego motion context. Unlike association, derived feature vector for hypothesis h, Rk is the task head, this stage describes how the belief was formed, who can observe and yk is the structured task output. Depending on the task, it, how reliable it is, whether agents disagree, and whether it is Rk is implemented as either a deterministic retrieval/ranking relevant to ego motion. We group these enrichments into five function or a lightweight learned predictor over ψ(h). Learned categories, summarized in Table II. Together, these enrichments heads support motion regression, scene-level prediction, control let downstream modules reason over object state, supporting selection, and trajectory forecasting without heavy multimodal inference, while preserving evidence traceability. Table III evidence, and ego-motion relevance [8]. These enrichments are needed because the same object can summarizes how the same enriched KG supports different matter differently across tasks: partner-only objects support task families through a common feature interface. This design hidden-object discovery, uncertainty affects prediction con- decouples representation construction from task inference: Gfidence, and path proximity affects control and forecasting. MARK builds one provenance-aware cooperative KG per

TABLE IV M AIN RESULTS ON V2V-G OT-QA. G-MARK IMPROVES MOST STRONGLY ON TASKS THAT REQUIRE EXPLICIT COOPERATIVE EVIDENCE , INCLUDING OCCLUSION REASONING AND HIDDEN - OBJECT DISCOVERY.

Task

Task family

Metric

Notable Objects (Q1) Occluding Objects (Q2) Invisible Objects (Q3) Planning Awareness (Q4) Object Motion (Q5) Agent Motion (Q6) Object Motion (Q7) Control Settings (Q8) Future Trajectory (Q9)

Visible-object grounding Occlusion reasoning Hidden-object discovery Planning-relevance reasoning Object motion prediction Agent motion prediction Object motion prediction Control selection Future trajectory forecasting

F1@0.5m ↑ F1@0.5m ↑ F1@0.5m ↑ F1@0.5m ↑ L2 avg. ↓ Accuracy ↑ L2 avg. ↓ Action L1 ↓ L2 avg. ↓

scene, and each head selects the evidence needed for its output. In our benchmark instantiation, these heads cover object grounding, visibility and occlusion reasoning, planning-aware object selection, motion prediction, control selection, and future trajectory forecasting.

G-MARK V2V-GoT [13] Improvement 0.586 0.428 0.494 0.614 3.822 0.905 3.822 0.076 2.710

0.525 0.301 0.440 0.608 8.050 0.874 7.610 0.088 2.620

+11.6% +42.2% +12.3% +0.9% +52.5% +3.5% +49.8% +13.1% -3.4%

V2V-GoT [13]. We restrict this analysis to future trajectory forecasting because this task directly links cooperative communication cost to downstream planning accuracy. These comparisons evaluate whether explicit cooperative evidence improves visibility-sensitive reasoning, generalizes across multiple task families, and offers a communication-efficient IV. E XPERIMENTAL A NALYSIS alternative to language-mediated cooperative reasoning. Implementation details. All experiments use the KG A. Experimental Setup construction pipeline over processed cooperative perception Benchmarks. V2V4Real provides synchronized real-world artifacts, including object boxes, confidence scores, agent poses, V2V scenes with connected vehicles, object annotations, agent trajectory context, and scene metadata. After KG construction, poses, timestamps, and processed perception artifacts. V2VG-MARK exports a shared KG-derived feature bank with three GoT-QA extends this setting with typed object-selection, typed views: object retrieval for Q1–Q4, motion regression for visibility-reasoning, motion-prediction, control-selection, and Q5/Q7/Q9, and scene-action prediction for Q6/Q8. trajectory-forecasting tasks. We use the official splits, with Object retrieval uses a shared logistic regression ranker with approximately 110K training and 31K validation questions, and task one-hot features and thresholds selected from the training evaluate on the validation split using the same task definitions split. Motion prediction uses regularized regression heads, with and reference answers as V2V-GoT [13]. a shared Q5/Q7 object-motion head and a separate Q9 trajectory Evaluation metrics. Object selection and visibility reasoning head. Action prediction uses lightweight classifiers for Q6 and tasks use localization-aware precision, recall, and F1-score, separate speed and steering heads for Q8. All heads are trained where predictions must match ground-truth coordinates within on the official training split, with validation labels used only the benchmark threshold. Motion and trajectory outputs use L2 for evaluation; experiments run on a Linux machine with 8 error, agent motion uses binary accuracy, and control selection CPU cores and 24 GB RAM. uses normalized action error. Higher is better for precision, recall, F1, and accuracy; lower is better for L2 and action error. B. Results Baselines. Our primary baseline is V2V-GoT [13], the closest recent cooperative reasoning method for this benchmark. Explicit cooperative evidence improves visibility-sensitive V2V-GoT extends V2V-LLM-style cooperative multimodal reasoning. Table IV compares G-MARK with V2V-GoT [13], QA [12] by incorporating graph-of-thought reasoning for a state-of-the-art cooperative reasoning baseline for V2V-GoTperception, prediction, and planning questions. It reports QA, across all task categories. We report Object Motion Q5 strong performance on future trajectory forecasting and other and Q7 separately because they are distinct benchmark entries, high-level cooperative driving tasks. We therefore compare although they share the same output family. The strong gains G-MARK with the reported V2V-GoT results across all on these entries likely reflect the structure of the output: objectbenchmark tasks, including object reasoning, motion prediction, motion prediction is a numeric regression problem, and Gcontrol selection, and trajectory forecasting. MARK directly exposes object state, recent motion cues, and Communication efficiency is an important requirement for path-relative geometry as compact KG-derived features. This cooperative driving, where agents must share sufficient scene representation is well suited to motion regression because ψ(h) evidence without incurring excessive bandwidth overhead. For directly encodes numeric object state, recent motion cues, and the communication–accuracy analysis, we compare G-MARK path-relative geometry. Language-mediated reasoning baselines with the future-trajectory forecasting results of No Fusion, must infer these quantities less directly before producing a AttFuse [3], V2X-ViT [7], CoBEVT [6], V2V-LLM [12], and numeric output.

Average Trajectory L2 Error (m)

6.5

Planning Accuracy vs. Communication Cost AttFuse

6.0

No Fusion

5.5

V2X-ViT CoBEVT

5.0

V2V-LLM

4.5

Representative G-MARK Latency Breakdown Notable Objects

14.2

Occluding Objects

12.3

1.3

Object Motion

4.0 3.5

Future Trajectory

3.0

G-MARK

2.5 0.0

0.1

0.2

0.3

Scene loading KG/task solver Other overhead

13.3

0

V2V-GoT

33.4

0.8

10

20

30

Average latency per sample (ms)

0.4

Communication Cost (MB)

Fig. 3. End-to-end latency breakdown of G-MARK by task.

Fig. 2. Planning accuracy versus communication cost.

Full G-MARK

Invisible Objects, F1 ↑

Action L1

F1

G-MARK improves over V2V-GoT on eight of the nine 0.494 benchmark entries, with the largest gains on tasks that require 0.5 0.443 0.396 explicit cooperative evidence. Occlusion reasoning improves by 0.4 42.2%, hidden-object discovery by 12.3%, and object motion 0.3 prediction by 49.8%. These gains support task-conditioned delayed evidence fusion: preserving provenance, visibility, 0.2 uncertainty, disagreement, and candidate status helps each task 0.1 head query relevant evidence rather than relying on a single 0.000 collapsed object state. The shared KG also supports prediction 0.0 Full Ego-only No Flat and control, with object motion benefiting from KG-derived G-MARK provenance non-KG KG motion cues and path-relative geometry, and control-selection error reduced by 13.1%. Future trajectory forecasting shows a Control Settings, Action L1 ↓ different pattern: G-MARK reaches near-parity with V2V-GoT, 0.213 with an average L2 error of 2.710m compared with 2.620m. 0.20 This suggests that the KG-conditioned representation captures 0.152 0.15 useful planning context, but longer-horizon forecasting remains sensitive to accumulated speed, steering, and curvature errors. 0.089 0.10 0.076 G-MARK approaches planning accuracy with substantially lower communication. We use future trajectory 0.05 forecasting because it tests whether communication preserves 0.00 planning-relevant evidence, including nearby objects, occluded Full Ego-only No Flat or partner-only hazards, and path-relative interactions. TrajecG-MARK provenance non-KG KG tory error therefore measures useful scene information retained per unit of communication. Fig. 2 compares G-MARK with Fig. 4. Diagnostic ablations on evidence-sensitive tasks. three categories of cooperative planning baselines: a noncooperative baseline (No Fusion), intermediate-fusion baselines including AttFuse [3], V2X-ViT [7], and CoBEVT [6], and and explicit evidence traceability. Structured KG reasoning adds minimal CPU-only overlanguage-mediated cooperative reasoning baselines including V2V-LLM [12] and V2V-GoT [13]. No Fusion has high trajec- head. Fig. 3 shows representative per-sample CPU runtimes aftory error, while intermediate-fusion and language-mediated ter upstream perception has produced processed scene artifacts. methods improve accuracy at roughly 0.4008–0.4068 MB per We group runtime into scene loading, KG/task solving, and sample. G-MARK achieves a different operating point: it other overhead for graph construction and feature extraction. uses only 0.0159 MB per sample, giving about 25.2× lower The reasoning head is lightweight: task-solver time remains communication than intermediate-fusion methods and 25.6× below 1.4 ms per sample and below 1 ms for six of the lower communication than V2V-GoT. This reduction comes eight task types. Current-frame tasks require 10.3–14.2 ms from transmitting compact KG evidence rather than dense total CPU latency, while temporal motion tasks require 32.4– feature-level or language-mediated payloads. Although V2V- 33.4 ms due to previous-frame context loading. Most runtime GoT achieves the lowest trajectory error, G-MARK attains comes from data access and scene preparation rather than KG comparable forecasting accuracy with far lower communication reasoning, indicating that the proposed reasoning layer is not

the primary bottleneck. Recent VLM-based cooperative driving systems report end-to-end latencies on the order of hundreds of milliseconds [24], [25]. Although not a direct system-level comparison, G-MARK requires only about 4–6% of these reported latencies for current-frame tasks, while preserving millisecond-scale CPU reasoning. C. Ablation Study We conduct construction-level ablations to evaluate which KG evidence components contribute to downstream reasoning. The task heads are kept fixed, while the KG evidence is selectively modified before inference. Fig. 4 summarizes the results for hidden-object discovery and control selection. Partner evidence is essential for hidden-object discovery. As shown in Fig. 4, removing partner observations drops Invisible Objects F1 from 0.494 to 0.000. This confirms that hidden-object discovery depends on cooperative evidence outside the ego vehicle’s direct view; an ego-only representation cannot recover objects visible only to connected vehicles. Provenance affects both reasoning and control. Removing provenance reduces Invisible Objects F1 from 0.494 to 0.396 and increases Control Settings Action L1 from 0.076 to 0.152, showing that source-agent support is useful for downstream decisions. KG-derived structure helps beyond object geometry. Replacing the KG with an unstructured object-level representation reduces Invisible Objects F1 from 0.494 to 0.443 and increases Control Settings Action L1 from 0.076 to 0.089. These results suggest that visibility, uncertainty, disagreement, and path-relevance features provide useful structure for evidencesensitive cooperative reasoning. V. C ONCLUSION We presented G-MARK, a grounded multi-agent reasoning framework that maps processed cooperative perception evidence into a provenance-aware KG for object reasoning, prediction, control selection, and trajectory forecasting. On V2V-GoT-QA, G-MARK improves evidence-sensitive tasks such as occlusion reasoning and hidden-object discovery, while achieving near-parity in future trajectory forecasting with 25.6× lower structured communication and millisecond-scale CPU overhead. These results demonstrate the potential of provenance-aware cooperative KGs as compact, traceable, and communicationefficient reasoning layers for cooperative driving. ACKNOWLEDGEMENTS This work has been funded in part by NSF, with award numbers #2112665, #2112167, #2003279, #2120019, #2211386, #2052809, #1911095 and in part by PRISM and CoCoSys, centers in JUMP 2.0, an SRC program sponsored by DARPA. R EFERENCES [1] Z. Zhang and J. F. Fisac, “Safe occlusion-aware autonomous driving via game-theoretic active perception,” arXiv preprint arXiv:2105.08169, 2021. [2] C. Zhang et al., “Occlusion-aware planning for autonomous driving with vehicle-to-everything communication,” IEEE Transactions on Intelligent Vehicles, vol. 9, no. 1, pp. 1229–1242, 2023.

[3] R. Xu, H. Xiang, X. Xia, X. Han, J. Li, and J. Ma, “Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-tovehicle communication,” in 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 2583–2589. [4] A. Kuznietsov et al., “Explainable ai for safe and trustworthy autonomous driving: A systematic review,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 12, pp. 19 342–19 364, 2024. [5] Y. Hu et al., “Where2comm: Communication-efficient collaborative perception via spatial confidence maps,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 4874–4886. [6] R. Xu, Z. Tu, H. Xiang, W. Shao, B. Zhou, and J. Ma, “Cobevt: Cooperative bird’s eye view semantic segmentation with sparse transformers,” in Conference on Robot Learning. PMLR, 2023, pp. 989–1000. [7] R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, and J. Ma, “V2x-vit: Vehicle-to-everything cooperative perception with vision transformer,” in European Conference on Computer Vision. Springer, 2022, pp. 107–124. [8] C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beisswenger, P. Luo, A. Geiger, and H. Li, “DriveLM: Driving with graph visual question answering,” in Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2024, pp. 256–274. [9] Z. Xu, Y. Zhang, E. Xie, Z. Zhao, Y. Guo, K.-Y. K. Wong, Z. Li, and H. Zhao, “DriveGPT4: Interpretable end-to-end autonomous driving via large language model,” IEEE Robotics and Automation Letters, 2024, also available as arXiv:2310.01412. [10] H. Shao et al., “LMDrive: Closed-loop end-to-end driving with large language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. [11] X. Tian, J. Gu, B. Li, Y. Liu, Y. Wang, Z. Zhao, K. Zhan, P. Jia, X. Lang, and H. Zhao, “DriveVLM: The convergence of autonomous driving and large vision-language models,” in Proceedings of the Conference on Robot Learning (CoRL), 2024. [12] H.-k. Chiu et al., “V2V-LLM: Vehicle-to-vehicle cooperative autonomous driving with multi-modal large language models,” 2025, accepted to ICRA 2026. [13] ——, “V2V-GoT: Vehicle-to-vehicle cooperative autonomous driving with multimodal large language models and graph-of-thoughts,” 2025, accepted to ICRA 2026. [14] Y. Tian, J. Zhang, Z. Wang, X. Ren, X. Yu, O. Gungor, and T. Rosing, “Kldrive: Fine-grained 3d scene reasoning for autonomous driving based on knowledge graph,” arXiv preprint arXiv:2603.21029, 2026. [15] A. Hogan et al., “Knowledge graphs,” ACM Computing Surveys, vol. 54, no. 4, pp. 1–37, 2021. [16] X. Gao et al., “Langcoop: Collaborative driving with language,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2025. [17] C. Liu et al., “Colmdriver: Llm-based negotiation benefits cooperative autonomous driving,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. [18] R. Xu et al., “V2v4real: A real-world large-scale dataset for vehicleto-vehicle cooperative perception,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 712–13 722. [19] Y. Li, D. Ma, Z. An, Z. Wang, Y. Zhong, S. Chen, and C. Feng, “V2XSim: Multi-agent collaborative perception dataset and benchmark for autonomous driving,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 10 914–10 921, 2022. [20] P. Groth and L. Moreau, “PROV-Overview: An Overview of the PROV Family of Documents,” W3C Working Group Note, Apr. 2013, world Wide Web Consortium (W3C). [Online]. Available: https://www.w3.org/TR/prov-overview/ [21] Y. Bar-Shalom and T. E. Fortmann, Tracking and Data Association. Academic Press, 1988. [22] D. Koller and N. Friedman, Probabilistic Graphical Models: Principles and Techniques. MIT Press, 2009. [23] E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” IEEE Access, vol. 8, pp. 58 443–58 469, 2020. [24] J. You et al., “V2x-vlm: End-to-end v2x cooperative autonomous driving through large vision-language models,” arXiv preprint arXiv:2408.09251, 2024. [25] J. You, P. Li, Z. Jiang, Z. Huang, R. Gan, H. Shi, and B. Ran, “V2x-realm: Vision-language model-based robust end-to-end cooperative autonomous driving with adaptive long-tail modeling,” arXiv preprint arXiv:2506.21041, 2025.

Related documents

• Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Driving#822096
• Context-aware knowledge graph reasoning for road maintenance decision-making#238062
• Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration#153072
• CoLMIN: LLM-based Multi-Decision Path Negotiation for Cooperative Autonomous Driving#971473
• LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios#1156205