M EM D REAMER: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanism
1 Ant Group, 2 Zhejiang University, 3 Central South University, 4 HKUST(GZ) ∗ Equal Contribution. † Corresponding Author.
Abstract Current Vision-Language Models struggle with hours-long videos because processing fulllength visual sequences induces prohibitive token explosion and attention dilution. To overcome this, we introduce M EM D REAMER to decouple perception and reasoning, shifting long-video understanding into an agentic exploration process. As a plug-and-play framework, it incrementally streams videos to construct a Hierarchical Graph Memory, a top-down three-tier architecture for semantic abstraction, anchored by a foundational graph capturing spatiotemporal and causal relations. During inference, the reasoning model employs agentic tool-augmented retrieval, navigating hierarchies, searching nodes, and traversing logical edges via an Observation-Reason-Action loop. Experiments show M EM D REAMER achieves SOTA results across four mainstream benchmarks, narrowing the gap with human experts to only 3.7 points. It constrains the reasoning context window to merely 2% of full-context ingestion while delivering a 12.5 point absolute accuracy gain. Furthermore, statistical analysis uncovers a strong positive linear correlation between an VLM’s performance on logic reasoning and long-video understanding benchmarks, establishing agentic capability scaling as a new paradigm for multimodal comprehension. Project: https://aim-uofa.github.io/MemDreamer/ ( ; m 1 _ v ĺ Ƒ Ə Ƒ Ɣ
Ɩ Ə
; l b m b Ŋ ƒ ĺ Ɛ Ŋ u o
; m ƒ ( Ŋ m v | u 1 |
ƕ Ѷ ĺ Ƒ ƕ Ɣ ĺ Ɩ
$ Ŋ Ɠ o ƕ Ə
b l b Ŋ Ƒ ĺ Ɣ
ƕ Ƒ ĺ Ə
ѵ ƕ ĺ Ɩ
ѵ ƕ ĺ ƕ ѵ ƒ ĺ ѵ
ѵ Ə
Ѵ - 7 ; Ŋ r v Ŋ Ɠ ĺ ѵ ѵ Ə ĺ Ɛ
m 7 Ŋ | o Ŋ m 7 ŐR Ʒ Ə ĺ ƕ Ə ķ p = 0.052 ő ; l u ; - l ; u ŐR Ʒ Ə ĺ Ɩ Ə ķ p = 0.003 ő
Ɣ Ə
Ɠ Ѷ ĺ Ɩ ƕ Ə
Ѷ Ə
Ɩ Ə
Ƒ Ə Ƒ Ɣ Ő ] ; m | b 1 ! ; - v o m b m ] ő
Ɛ Ə Ə
Ѷ Ə ĺ ƕ
Ƒ Ɠ Ə
ƕ Ƒ ĺ Ə
ƕ Ə ѵ ƒ ĺ ѵ
Ɠ Ɛ× v _ o u | ; u
Ɛ Ƒ Ɠ× v _ o u | ; u
Ƒ Ə Ə Ɛ Ə Ə
Ɠ ƒ× v _ o u | ; u Ƒ Ə
ѵ Ə Ɣ Ɣ Ɣ Ə
Ɛ Ɣ Ə Ə
Ƒ ѵ Ɣ ƕ Ѷ ĺ Ƒ
ƕ Ɣ ѵ Ɣ
Ɩ Ə ĺ ƕ
Ѷ Ɠ ĺ Ѷ
Ѷ Ɣ Ѷ Ə
m 7 Ŋ | o Ŋ ; m 7 | ; l u ; - l ; u Ő u v ő |
ƕ Ѷ Ɠ
Ɩ Ə
; l b m b Ŋ Ƒ ĺ Ɣ Ŋ u o
Ѷ Ə
ѵ Ə
m 7 Ŋ | o Ŋ ; m 7 1 1 ; l u ; - l ; u Ő u v ő 1 1
; m ƒ ( Ŋ $ _ b m h b m ] $ Ŋ Ɣ
Ɣ ĺ Ɩ
ѵ ĺ ƒ
ѵ ĺ Ƒ
; m ƒ Ŋ (
; l b m b Ŋ Ƒ ĺ Ɣ
; l b m b Ŋ ƒ ĺ Ɛ
Ɣ
o m | ; | ) b m 7 o " b ; Ő ķ o ] " 1 - Ѵ ; ő
l - m r ; u | Ő Ɩ Ɠ ĺ Ɠ ő
( ; m 1 _ 1 1 u - 1 Ő Ѹ ő
( ; m 1 _ " 1 o u ; Ő o m ] ( b 7 ; o & m 7 ; u v | - m 7 b m ] ő
arXiv:2606.07512v1 [cs.CV] 5 Jun 2026
Cong Chen1,2,∗ , Guo Gan2,∗ , Kaixiang Ji1,∗ , ChaoYang Zhang1 ,3 , Zhen Yang4 , Guangming Yao1 , Hao Chen2 , Jingdong Chen1 , Yi Yuan1 , Chunhua Shen1,2,† ,
Figure 1: (left) Decoupling perception unlocks reasoning ability for long video understanding. We evaluate the same backbone in two regimes: as an end-to-end VLM (gray) and as the reasoning model of M EM D REAMER over a pre-built hierarchical graph memory (red). End-to-end performance is insensitive to reasoning ability, as the model is overwhelmed by noisy long-context tokens. In contrast, M EM D REAMER exhibits a strong linear trend, confirming that decoupling facilitates the direct transfer of the backbone’s agentic reasoning ability into long video understanding performance. (right) M EM D REAMER only needs a 5-6K reasoning window, 41-124× smaller than end-to-end input.
1
1
Introduction
Long video understanding (Jin et al., 2024; Song et al., 2024) represents a core capability for driving VisionLanguage Models (VLMs) (Chen et al., 2024a; Google DeepMind, 2026; Hurst et al., 2024) toward embodied intelligence and open-world interaction. Despite breakthroughs in single (Liu et al., 2023; Chen et al., 2024b; 2025c) or multi-images (Wang et al., 2025a; Ye et al., 2025; Chen et al., 2025b; Gan et al., 2026) and short video analysis (Zhao et al., 2025), current VLMs (Li et al., 2024) struggle to scale to hours-long videos. This failure stems from the structural intricacy and extreme temporal redundancy inherent in long videos, which submerge critical reasoning signals beneath massive noise. Current VLMs (Lin et al., 2024; Zhang et al., 2024b; Li et al., 2025) typically adopt a tightly coupled strategy, relying on ultra-long context windows to process visual perception and logical reasoning simultaneously. This coupled paradigm flattens long videos into massive token streams via brute-force frame sampling, introducing two bottlenecks. In perception, it incurs an intractable token explosion: sampling a 2-hour video at 1 FPS generates over 1.6M tokens, drastically exceeding current context limits. In Reasoning, the influx of redundant tokens induces severe attention dilution (Xiao et al., 2024) and exacerbates the “lost in the middle” phenomenon (Liu et al., 2024), significantly degrading long-range reasoning capabilities. To overcome these bottlenecks, we introduce M EM D REAMER, a novel paradigm that decouples perception from reasoning. Specifically, our framework employs a perception model (Google DeepMind, 2026; Bai et al., 2025) to process videos in a streaming fashion, incrementally constructing a persistent memory bank. During inference, a separate reasoning model executes a search loop within this memory to retrieve task-relevant cues. While this streaming-and-retrieval mechanism circumvents context limits and token redundancy, its efficacy depends on the memory’s structural organization. Existing flat (Yang et al., 2025) or chunk-based (Zhang et al., 2026; Wang et al., 2025e) storage schemes obscure global perspectives and sever temporal-causal links, causing decoupled reasoning to degenerate into myopic, exhaustive retrieval. To bridge this gap, M EM D REAMER introduces a novel Hierarchical Graph Memory. Video content is inherently hierarchical (Chen et al., 2025a), mirroring how humans comprehend a long video coarse-to-fine, starting from the overall plot, then scene by scene, event by event, finally down to individual actions, rather than memorizing a flat, exhaustive sequence of moments (Pang & Wang, 2025; Long et al., 2025). Moreover, these fine-grained events and entities are far from independent: they are intertwined by spatiotemporal and causal relations (Edge et al., 2024) that no purely sequential representation can faithfully capture. To realize this hierarchical and topological intuition, we construct a three-tier memory architecture shown in Figure 2. The top tier is a Video Root that summarizes global context, which is progressively decomposed into Super Events at the intermediate layer and Macro Events at the bottom. At the Macro Event tier, information is instantiated into a local subgraph characterizing entities, events, and their logical relations. This representation suppresses irrelevant detail at the appropriate granularity while preserving the long-range dependencies that downstream reasoning relies on. To fully exploit this architecture, we propose a tool-augmented Agentic Retrieval Mechanism, superseding traditional full-context ingestion or similarity-based retrieval (Lewis et al., 2020; Gao et al., 2023). We construct a dedicated Agentic Tool Bank comprising three categories: Navigation for vertical hierarchical traversal, Search for rapid node localization, and Graph Traversal for tracking logical chains along topological edges. By engaging in an iterative Observation-Reason-Action loop (Yao et al., 2022), the model dynamically interacts with the memory. This design shifts long video understanding from passive token consumption to a multi-step active agentic exploration task. Extensive experiments validate the superiority of M EM D REAMER. On challenging hours-level benchmarks (Wang et al., 2025b; Wu et al., 2024; Fu et al., 2025), M EM D REAMER consistently achieves SOTA results (Mangalam et al., 2023), narrowing the performance gap with human experts to 3.7 points. Notably, our decoupled strategy achieves an absolute gain of 12.5 points over end-to-end coupled paradigms using only 2% of the context window on hours-long video bench LVBench (Wang et al., 2025b). Furthermore, M EM D REAMER reveals a critical observation: we empirically establish a strong positive correlation between an VLM’s intrinsic agentic capability and its long-video understanding performance, suggesting the scaling of agentic capacity as a new paradigm for future research. In summary, our contributions are threefold: 1. M EM D REAMER decouples perception and reasoning via persistent memory and agentic retrieval, bypassing long-context bottlenecks. To our knowledge, we reveal for the first time a positive correlation between longvideo performance and reasoning, suggesting a new optimization direction. 2
Macro Event
2
1
4 3
6 5
7
t
… t-1
Sub Graph Event Judy Intervene
Entity Judy
Entity Nick Entity Money Event Refuse Service
Event Enter Shop
Entity Finnick
Event Buy Popsicle
Entity Jerry
Entity Popsicle
Movie
Judy Hopps
SuperEvent2: Meter Maid Duty and the Fox Scam Time Range: 00:16:47-00:26:30 Description: Disappointed by her assignment as a parking enforcement officer, Judy encounters the cynical con-artist .…
SuperEvent1: From Bunnyburrow to ZPD Time Range: 00:07:36-00:16:47 Description: Judy Hopps overcomes physical limitations to graduate top of her class at the …
MacroEvent
Super Event
MacroEvent2.1: Parking Enforcement Duty Time Range: 00:16:47-00:17:39 Description: Judy reluctantly begins her assignment as a meter maid. Determined to prove her worth…
MacroEvent2.2: Pawpsicle Scam Discovery Time Range: 00:17:39-00:23:43 Description: Judy spots a suspicious fox, Nick Wilde, entering an elephant ice cream parlor…
Nick Wilde Dawn Bellwether Chief Bogo
Entity node: Main Characters
SuperEvent3: The ZPD Ultimatum Time Range: 00:26:30-00:33:58 Description: After arresting a thief against orders, Judy intercepts a missing mammal case, from …
MacroEvent2.3: Confrontation with Nick Time Range: 00:23:43-00:25:37 Description: Judy corners Nick to confront him about tax evasion, Nick uses legal loopholes…
Video Clip
Video Root
Summary: In Zootopia, rookie rabbit officer Judy Hopps and fox Nick Wilde conduct a cross-regional investigatio, spanning Tundratown, the Rainforest District, and the Cliffside Asylum, to trace a series of missing predator cases. Following a severe societal fracture caused by a biological misattribution of sudden predator savageness, the duo traces the true causal root. They ultimately intercept a biochemical conspiracy orchestrated by Assistant Mayor Bellwether using "Night Howler" toxins at the Natural History Museum, closing the global causal chain …
SuperEvent
Video Root
Hierarchical Graph Memory
Figure 2: M EM D REAMER organizes a long video as a three-tier coarse-to-fine hierarchy with cross-tier topological edges; at the leaf tier, each segment is a subgraph of entities and micro-events via attributive and causal edges.
2. M EM D REAMER achieves SOTA performance across four representative long-video benchmarks. Using identical base models, we achieve substantial gains over traditional end-to-end baselines, narrowing the gap toward human expert levels. 3. M EM D REAMER establishes foundational design principles for multimodal memory systems. Empirically, hierarchical graphs and agentic tool-use outperform flat chunk-based storage and naive full-context ingestion or semantic matching, providing a cornerstone for future architectures.
2
Related Work
2.1 Long Video Understanding Long video understanding remains a frontier for VLMs, marking the evolutionary leap to complex, real-world visual streams. To handle potentially infinite inputs, previous research has attempted to optimize visual encoders (Lin et al., 2024; Zhang et al., 2024b) and scale native context windows (Bai et al., 2025; Hurst et al., 2024; Team et al., 2023). However, even with leading models expanding context limits to 1M tokens like Gemini-2.5-Pro (Comanici et al., 2025), systems still rely on aggressive frame subsampling to accommodate lengthy videos. This brute-force paradigm leaves models vulnerable to “needle-in-a-haystack” retrieval failures (Wang et al., 2024a; 2026) and the “lost in the middle” phenomenon (Liu et al., 2024) on benchmarks like LVBench (Wang et al., 2025b). To bypass these bottlenecks, a more scalable paradigm structures video information within an external memory bank, transforming static perception into an agentic reasoning process. 2.2 Agentic Memory Systems While memory systems like MemGPT (Packer et al., 2023) and MemoryBank (Zhong et al., 2024) achieve remarkable success in LLMs, adapting them for VLMs remains challenging. Early video attempts (Song et al., 2024) rely on simple frame buffers, and subsequently M3-Agent (Long et al., 2025) and WorldMM (Yeo et al., 2025) encapsulate semantics into discrete bins. Yet, these flat designs lack top-down taxonomy and topological edges. Recently, VideoARM (Yin et al., 2025) and MM-mem (Lian et al., 2026) pioneered hierarchical memory architectures. However, they maintain these hierarchies as edge-less perceptual buffers or rely on passive, uncertainty-driven vertical drill-down. M EM D REAMER bridges this gap by coupling a three-tier hierarchical abstraction with a causal graph topology, encoding multi-granular semantics and localized details to ground subsequent retrieval. Furthermore, to decouple perception and reasoning for long videos, we construct a purely textual memory. Unlike previous leading methods (Zhang et al., 2026; Yin et al., 2025) that require additional perception models to revisit raw video frames during retrieval , our approach relies solely on interactions with text memory.
3
Hierarchical Graph Memory Construction Input Video Stream
00:28:02
00:18:02
00:00:00
(a) Streaming Adaptive Segmentation Video Segments (Variable Lengths)
Segment 1
(00:00:00-00:01:20)
Segment 2
(00:01:20-00:03:37)
Segment 3
(00:03:37-00:04:26)
01:22:10
(00:04:26-00:06:02)
01:48:28
(00:08:51-00:10:00)
1
Super Event Node
Vi Micro Event Node Nj
Entity Node
4
2
N0
Timeline Video Root Node
Navigation Tools
3
1.Get Video Summary
7 5
…
t
t-1
6
(b) Downward Subgraph Extraction
…
t
Agentic Tools Bank
1
Macro Event Node Relation Edge
V0
V1
N1 N2
User Query Where does the ice cream for the pawpsicles Nick sells come from?
Super Event
… Segment
Video Memory
Video Root
Macro Event Segment 4
Agentic Tool-Augmented Retrieve
(c) Upward Hierarchical Aggregation
V2
N3
N4
V3 N5
2.Get Super Events 3.Get Macro Events
Observations Relative information about scene and details
Actions
Agent
'Get super events' 'Search nodes'
Information enough for answer?
(Nick sells pawpsicles)
4.Get Event Subgraph
Search Tools 5. Search Nodes 6. Search By Time
Graph Traversal Tools 7. Get Relation Graph
No. Thoughts Look for the scene and details
Yes!
(where Nick sells the pawpsicles.)
Final Response
The ice cream (juice) comes from the JumboPop that Nick bought from Jumbeaux's Cafe.
Figure 3: Architectural workflow of M EM D REAMER. (Left) Memory construction comprises three phases: streaming adaptive segmentation, downward subgraph extraction and upward hierarchical aggregation. (Right) Three tool categories support an agentic retrieval mechanism driven by an Observation-Reason-Action loop.
2.3 Agentic Retrieval Mechanism Early information-seeking approaches (Luo et al., 2026; Jeong et al., 2025) rely on naive Retrieval-Augmented Generation (Lewis et al., 2020; Gao et al., 2023), using static dense embeddings to recall contexts. However, semantic similarity does not guarantee logical relevance. A visually similar clip may lack causal connection to the query, and these single-turn retrievals cannot dynamically self-correct. To address this, pioneering works like VideoAgent (Wang et al., 2024b) and DVD (Zhang et al., 2026) introduced agentic retrieval, enabling VLMs to actively search. Yet, constrained by flat clip databases, their exploration often degrades into blind trial-and-error. M EM D REAMER extends the agentic loop by coupling it with our graph memory. Through a tool-augmented Observation-Reason-Action loop, we unleash the agent’s reasoning capabilities to explore the memory across multiple dimensions and locate relevant information.
3
Method
We formulate long-video understanding as a decoupled paradigm comprising two stages: persistent memory construction and tool-augmented agentic retrieval. Given a video stream V, a perception model P first processes the visual inputs in a streaming fashion to construct a structured, purely textual Hierarchical Graph Memory, denoted as G. Subsequently, upon receiving a text query Q, a reasoning model R equipped with a tool bank T actively explores G through a multi-step Observation-Reason-Action loop. During inference, R operates only on this pure text memory, never raw video information. This iterative exploration extracts a concise set of task-relevant clues C, yielding the final answer A = R( Q, C ). 3.1 Hierarchical Graph Memory M EM D REAMER conceptualizes a long video stream as a coarse-to-fine three-tier semantic topology comprising the Video Root, Super Events and Macro Events layers. Beneath the foundational Macro Events layer, we formulate a local subgraph to characterize entities, micro-events, and their spatiotemporal and causal dependencies. To construct this memory representation, the pipeline first adaptively segments the streaming video to yield semantically self-contained Macro Events as anchors. subsequently, the system extracts finegrained local subgraph structures downward to preserve visual details, while aggregating nodes upward to distill macro-level hierarchical semantics. Streaming Adaptive Segmentation. We introduce a streaming adaptive segmentation mechanism driven by semantic boundaries, rather than relying on rigid, fixed-length chunking (e.g., 30-second intervals) common in prior works (Zhang et al., 2026). Specifically, at each iteration k, the system maintains a temporal sliding k k k window Wk = [tstart , tstart + τ ], where tstart denotes the initial timestamp of the current window and τ represents the maximum window duration. Within Wk , the perception model P executes temporal grounding to resolve a localized set of complete Macro Events, denoted as {e1 , e2 , . . . , e N }. The concluding boundary of 4
Symbol
Type
Hierarchy nodes (one per video) vR Video Root vS Super Event v Mc
Macro Event
Definition Global root: title, description, themes, key_entities, time_range. Narrative-phase: label, description, time_range, {v Mc } children, deduplicated key_entities. Per-episode summary: label, summary, time_range, child micro-event ids, top-k key_entities.
Subgraph nodes Vi = ViE ∪ ViM (one subgraph per Macro Event) ViE Entity Persistent referent: name, entity_type ∈ { P ERSON, O BJECT, L OCA TION , G ROUP }, structured attributes, description. ViM Micro-event Atomic action with explicit temporal extent: time_range, subject, object, action, description. Subgraph edges Ei , three topological categories Spatial-attribute ViE ↔ ViE Subject-object
ViE ↔ ViM
Temporal-causal
ViM → ViM
Cross-tier edges E H Hierarchical Macro causal / progression
v Mc → vS , vS → v R v Mc → v Mc , vS → vS
LOCATED_IN, NEXT_TO, ATTACHED_TO, PART_OF, IS_SAME_AS, ANNOTATES, static layout, composition, coreference, and on-screen attribute. PERFORMS, RECEIVES, USES_TOOL, HAPPENS_IN, CONTEXT_FOR, role assignment of entities to micro-events and contextual annotation of events. BEFORE, OVERLAP, CAUSES, PREVENTS, SUBEVENT_OF, chronology and causality among micro-events. SUBEVENT_OF edges that wire the three-tier topology. Inter-episode reasoning: CAUSES, PREVENTS, ENABLES (intra-Super), LEADS_TO, RESOLVES, CONTRASTS_WITH (cross-Super). BEFORE edges between consecutive Super Events are auto-generated.
Table 1: Schema of the Hierarchical Graph Memory G. The hierarchy spans three tiers (v R , vS , v Mc , and the Macro Event leaves whose contents are unfolded into local subgraphs gi ). Each subgraph factorises a Macro Event into entities, micro-events, connected by three edge categories that decouple layout, action role binding, and temporal-causal flow.
this final event e N directly serves as the starting point for the subsequent window. This paradigm yields two benefits: first, the adaptive semantic partitioning ensures that each Macro Event node remains semantically intact and self-contained, thereby avoiding the arbitrary truncation inherent in previous fixed-window strategies. Second, it caps the maximum video length of a single input to P within the horizon τ, bounding context window pressure during the perception phase. Downward Subgraph Extraction. For each segmented Macro Events ei , we construct a local spatiotemporal subgraph gi = (Vi , Ei ) to capture fine-grained visual details. While conventional LLM-based graph memories (Edge et al., 2024) encapsulate an entire event into a single static ⟨Subject entity, Relation edge, Object entity⟩ triplet, they struggle to model the highly dynamic temporal evolution and causal chains inherent in long videos. To address this limitation, we introduce a novel Video-centric Graph Ontology where the vertex set is formally defined as Vi = ViE ∪ ViM , incorporating micro-events (ViM ) as distinct nodes alongside standard entities (ViE ). These vertices are interconnected via a heterogeneous edge set Ei comprising three distinct topological relations: spatial-attribute edges (ViE ↔ ViE ) delineate the relative spatial layouts and static physical attributes among entities, subject-object edges (ViE ↔ ViM ) anchor discrete entities to the actions they initiate or undergo, and directed temporal-causal edges (ViM → ViM ) model the chronological progression and causal dependencies between micro-events. We use the perception model P to parse the video clip to extract this structured subgraph gi alongside a textual summary of the Macro Event. Upward Hierarchical Aggregation. Upon establishing the foundational graph layer, the system executes a bottom-up semantic abstraction to provide a global navigation backbone for subsequent retrieval. Specifically, the localized textual descriptions of all Macro Events are fed into the model P as the foundational leaf nodes of the three-tier topology. Guided by temporal adjacency and semantic affinity, P clusters and distills neighboring, highly correlated Macro Events into Super Events that span broader temporal scales, until ultimately converging at a single Video Root node. This coarse-to-fine topology structurally spans from micro-level actions to the overall narrative, empowering the retrieval agent with efficient global navigation and precise target localization. 5
Tool
Parameters
Hierarchical Navigation GetSummary – GetSuperEvent super_id GetMacroEvent
super_id or macro_ids
GetSubgraph
macro_id
Precise Search SearchNodes
query, top_k, node_types
SearchByTime
start_sec, end_sec
Graph Traversal GetRelationGraph
node_id
Output
Purpose
Video-level summary Super-event records: label, time_range, and child-macro. Macro-event records: label, time_range, and key_entities. Local subgraph of one macroevent: entities, micro-events, onscreen text, and key edges.
Retrieve the overview of video. Browse high-level narrative segments. Drill from a super-event into its constituent macro-events. Inspect fine-grained dynamics inside a single macro-event.
Top-k nodes ranked by embedding similarity, each annotated with its parent macro-event and one-hop ego-graph context. Macro-events whose time intervals overlap [ts , te ], paired with their parent super-event labels.
Dense semantic retrieval over entities, micro-events, and on-screen text within a unified embedding space. Time-anchored localization given a temporal range.
Outgoing and incoming edges of the target node, with relation types, labels, and rationales.
Multi-hop traversal along spatialattribute, subject-object, and temporal-causal edges.
Table 2: Specification of the seven tools available to the reasoning model R during the Observation-Reason-Action loop. The toolkit is organized into three categories aligned with the agent’s retrieval intent: navigating the three-tier hierarchy, executing semantic or temporal search, and traversing local graph neighborhoods.
3.2 Agentic Tool-Augmented Retrieval We introduce a tool-augmented agentic retrieval mechanism to exploit our structured, multi-granular graph memory. Equipping the reasoning model R with a graph-access toolbox across three functional dimensions and an Observation-Reason-Action loop, we transform static video understanding into a multi-step, proactive reasoning process. Multi-Dimensional Tool Bank. To facilitate deep retrieval and reasoning over the hierarchical graph memory, we construct a graph-access toolbox encompassing three functional dimensions. First, the hierarchical navigation tools, consisting of GetSummary(·), GetSuperEvent(·), GetMacroEvent(·), and GetSubgraph(·), allow the model R to progressively unpack the three-tier topology from a top-down, coarse-to-fine perspective, traversing from global narratives down to super-events, macro-events and localized subgraphs at varying granularities. Second, the precise search tools, including SearchNodes(·) and SearchByTime(·), support vectorbased semantic retrieval via text embeddings and chronological filtering within targeted temporal windows, enabling rapid localization of relevant nodes. Third, the local graph traversal tool GetRelationGraph(·) extracts the topological neighborhood of a target node, allowing the model to execute multi-hop tracing along spatial, temporal, or causal edges to resolve complex cross-spatiotemporal logic chains. Collectively, these three tool categories formulate an expressive action space, providing the retrieval agent with the capability to shift its perspective between the macroscopic semantic hierarchy and the underlying dense graph details. Agentic Observation-Reason-Action Loop. Our retrieval mechanism operates within an ObservationReason-Action loop driven by the model R. At each step t, the model evaluates the initial user query Q alongside the historical execution trajectory Ht−1 to determine the proper tool execution and corresponding parameters at : at = R( Q, Ht−1 ) (1) Executing the action at invokes the designated tool from our toolkit to yield a environmental observation ot from the graph memory G. To prevent multi-turn interactions from accumulating irrelevant textual noise, which induces the “lost in the middle” phenomenon within long contexts, we eschew direct concatenation of raw observational outputs. Instead, the model R distills ot to extract only the task-relevant evidence cues conditional on the query, denoted as c t = R( o t , Q ). (2)
6
Video-MME
EgoSchema
Avg.↑
ER↑
EU↑
Rea↑
KIR↑
Sum↑
TG↑
Avg.↑
Long↑
Long(w/o sub)↑
Val↑
End-to-End VLMs (Proprietary) GPT-4o GPT-4o Gemini-2.0-Flash Gemini-2.0-Flash Gemini-2.5-Pro Gemini-2.5-Pro Gemini-3.1-Pro Gemini-3.1-Pro OpenAI-o3 OpenAI-o3 Seed1.5VL Seed1.5VL-Thinking
48.9 48.6 72.0 78.2 57.1 64.6
48.9 47.4 71.5 78.1 57.6 65.4
49.5 48.5 71.1 76.7 56.4 63.4
50.3 44.4 67.7 74.1 50.8 68.0
48.1 56.8 80.0 86.3 62.9 53.6
50.0 41.4 63.5 70.7 67.2 63.7
40.9 39.3 69.1 81.4 46.8 46.6
66.7 − 71.0 78.6 66.7 −
60.9 45.7 68.6 77.0 60.6 74.4
65.3 63.0 75.9 80.3 64.7 −
70.4 71.2 72.8 76.4 63.2 −
End-to-End VLMs (Open-Source) Qwen3-VL Qwen3-VL-235B-A22B-Thinking Qwen3.5 Qwen3.5-35A3B GLM-4.6V GLM-4.6V-106B-A12B GLM-4.5V GLM-4.5V-106B-A12B InternVL2.5 InternVL2.5-78B InternVL3.5 InternVL3.5-30BA3B AdaRETAKE AdaRETAKE
63.6 71.3 59.5 53.4 43.6 44.4 53.3
63.7 72.2 58.8 55.3 43.8 42.7 53.0
62.6 69.4 59.2 53.1 42.0 44.1 50.7
63.1 74.4 69.7 57.5 51.0 48.3 54.7
62.6 68.9 57.8 55.3 37.9 46.4 62.2
65.5 67.5 61.8 39.6 42.1 36.2 37.9
59.6 66.2 53.9 49.9 36.8 40.9 45.5
71.4 − 67.6 66.4 − 62.9 67.0
67.1 − 58.7 54.1 − 52.7 −
71.2 62.6 66.0 64.7 62.6 64.1 65.0
75.9 − 68.8 68.4 − 86.8 −
Agentic Video Understanding Methods VideoTree VideoTree VideoAgent VideoAgent VCA VCA MR. Video MR. Video M3-Agent M3-Agent WorldMM-GPT GPT-5 MM-Mem MM-Mem DVD OpenAI-o3 VideoARM OpenAI-o3 VideoSeek GPT-5
28.8 29.3 41.3 60.8 49.3 61.9 − 74.2 79.7 68.4
30.3 28.0 43.7 59.8 − − − 73.4 − −
25.1 30.3 40.7 57.4 − − − 73.3 − −
31.9 28.0 46.2 57.7 − − − 70.7 − −
26.5 28.0 37.8 71.4 − − − 80.4 − −
25.5 36.4 27.3 50.0 − − − 74.1 − −
27.7 29.3 38.0 58.8 − − − 72.3 − −
− − − − − − − 71.6 78.0 −
− − − 61.6 − − − 68.6 76.4 73.5
− − − 61.8 61.8 76.6 66.1 67.3 81.2 60.9
67.0 63.2 73.6 73.0 − − − 76.6 76.2 −
M EM D REAMER (Ours, Plug-and-Play Agentic Framework) M EM D REAMER Qwen3-VL-235B-A22B-Thinking 84.8 (+21.2) M EM D REAMER Gemini-2.5-Pro 80.7 (+8.7) M EM D REAMER Gemini-3.1-Pro 90.7 (+12.5)
84.6 77.1 90.1
85.2 83.6 90.6
80.6 81.1 89.6
85.6 82.5 91.4
84.5 91.4 89.7
87.7 86.4 91.8
86.3 (+14.9) 78.6 (+7.6) 92.9 (+14.3)
83.2 (+16.1) 78.7 (+10.1) 91.0 (+14.0)
86.2 (+15.0) 85.0 (+9.1) 92.1 (+11.8)
87.4 (+11.5) 88.2 (+15.4) 87.8 (+11.4)
Human Expert
−
−
−
−
−
−
−
−
−
−
Methods
LVBench
Reason Model
Human
94.4
LongVideoBench
Table 3: Main results on four long-video understanding benchmarks. M EM D REAMER achieves state-of-the-art across all four benchmarks, with substantial gains over end-to-end baselines (green: improvement over the strongest end-to-end baseline using the same backbone).
Subsequently, the current action-clue pair is appended to the historical trajectory to transition the working memory state of the agent: Ht = Ht−1 ∪ {( at , ct )} (3) By obtaining raw observations and aggregating only relevant clues, this mechanism guarantees that the reasoning model R exclusively attends to high-quality evidence. Through this entire Agentic Loop, the model R can fully explore the memory space to gather the essential evidence required for the final response.
4
Experiments
4.1 Experiment Setup Evaluation Benchmarks. We use four challenging benchmarks: LVBench (Wang et al., 2025b) comprises 103 long videos of 30 mins to 2 hours and 1,549 QA pairs across six fine-grained dimensions. The LongVideoBench (Wu et al., 2024) validation set includes 753 videos and 1,337 questions, featuring 188 videos of 15 to 60 minutes. The long-video split of Video-MME (Fu et al., 2025) covers 300 videos (30 to 60 minutes) and 900 questions. EgoSchema (Mangalam et al., 2023) contains egocentric clips focusing on reasoning. Together, these benchmarks span diverse video durations, task granularities, and reasoning depths. Implementation Details. In the memory construction phase, the continuous video stream is segmented via a τ = 10min sliding window, and Gemini-3.1-Pro serves as the perception model to construct the hierarchical graph memory using raw video streams without any external transcripts. In the online reasoning phase, we evaluate Gemini-2.5-Pro (Comanici et al., 2025), Gemini-3.1-Pro (Google DeepMind, 2026), and the open-source Qwen3-VL-235B-A22B-Thinking (Bai et al., 2025) as the reasoning engines. We adopt Qwen3Embedding (Zhang et al., 2025) for semantic vector computing and cap the agent’s maximum tool-call budget at 12 steps. Baselines. We benchmark M EM D REAMER against two categories of long-video systems: (1) Vanilla Long Video VLMs: we compare it directly against its underlying reasoning engines, covering top closed-source
7
frontiers (Hurst et al., 2024; Google DeepMind, 2026; Guo et al., 2025) and representative open-source (Bai et al., 2025; Team, 2026; 2025; Zeng et al., 2025; Chen et al., 2024a; Wang et al., 2025c;d). (2) Memory-based Video LLMs: we evaluate our framework against established long-form video memory baselines (Wang et al., 2025e; 2024b; Yang et al., 2025; Pang & Wang, 2025; Long et al., 2025; Yeo et al., 2025; Lian et al., 2026) alongside the latest SOTA memory-driven systems (Zhang et al., 2026; Yin et al., 2025; Lin et al., 2026). 4.2 Main Results SOTA Performance Across Diverse Benchmarks. Table 3 summarizes the performance of M EM D REAMER across four benchmarks. Results demonstrate that M EM D REAMER establishes new SOTA across all evaluation metrics. Specifically, on LVBench, M EM D REAMER achieves a peak score of 90.7 with 12.5 points improvement over the strongest native closed-source baseline Gemini-3.1-Pro, significantly narrowing the gap to human expert performance to a 3.7 points. On LongVideoBench and Video-MME, our framework attains 92.9 and 92.1, yielding substantial improvements of 14.3 and 11.8 points. Furthermore, even on EgoSchema, which features egocentric perspectives, M EM D REAMER secures a top score of 88.2. These consistent improvements across both open-source and closed-source reasoning engines validate the generality and pronounced superiority of our plug-and-play architectural paradigm. Role
Model
Context Windows LVB
Vanilla Full Video Gemini-3.1-Pro End-to-End Gemini-2.5-Pro Qwen3-VL-235B-A22B-Thinking
265K 784K 240K
78.2 72.0 63.6
MemDreamer (Ours) Perception Gemini-3.1-Pro Gemini-3.1-Pro Reasoning Gemini-2.5-Pro Qwen3-VL-235B-A22B-Thinking
40.3K 6.2K 6.3K 5.9K
90.7 80.7 84.8
Table 4: Comparison of input context length requirements and overall performance on LVBench (LVB).
Mitigating Context Overload via Decoupled Reasoning. Table 4 and Figure 1 compare the context length of our decoupled paradigm against conventional native end-to-end full-context ingestion. The vanilla approach forces the vision-language model to ingest massive sequences ranging from 240K to 784K tokens, severely degrading reasoning accuracy due to spatial-attention dilution and the “lost in the middle” effect. Conversely, by segregating perception from reasoning, M EM D REAMER avoids stacking redundant visual tokens. During agentic reasoning, our tool-augmented navigation operates over a condensed topology, restricting the active context to 5.9K–6.3K tokens, delivering an approximate 40× reduction versus full-video ingestion. Reducing token noise and redundancy unlocks a 21.2-point leap for Qwen3-VL from 63.6 to 84.8, empirically showing that structured agentic reasoning is a superior alternative to brute-force token scaling. 4.3
Analysis Model Claude-Opus-4.6 Gemini-3.1-Pro Kimi-K2.5 GPT-5 Qwen3-VL-235B-A22B-Thinking Gemini-2.5-Pro Qwen3-VL-235B-A22B-Instruct gpt-4o-2024-11-20
LVBench
AIME2025 99.8 98.1 96.1 94.6 89.7 88.0 74.7 61.9
End-to-end
M EM D REAMER
67.9 78.2 75.9 60.1 63.6 72.0 67.7 48.9
85.1 90.7 79.2 84.4 84.8 80.7 71.5 68.5
0.702 0.052
0.897 <0.01
Statistical Correlation with Agentic Capability Pearson R ↑ p-value ↓
-
Table 5: Correlation between Agentic reasoning capabilities (AIME 2025) and long video understanding.
8
Paradigm Shift to Pure Agentic Reasoning. Table 5 uncovers the correlation between an LLM’s logical capacity and long-video understanding performance. We evaluate eight models (Team et al., 2026; Anthropic, 2026) on the widely used AIME2025 reasoning benchmark and compute the Pearson correlation (R) against their LVBench scores. Under the traditional end-to-end paradigm, the correlation is restricted to 0.702 with an insignificant p-value of 0.052, demonstrating that raw video ingestion creates perceptual barriers that hinder models from leveraging their reasoning capacity. Conversely, integrating M EM D REAMER scales this correlation to 0.897 with statistical significance (p < 0.01). This establishes a paradigm shift: supported by structured graph memory and agentic retrieval mechanism, long-video understanding bypasses the context-window limits, scales with the model’s logical reasoning capability.
Perception Model
Reasoning Model
LVBench
Gemini-2.5-Pro
Gemini-2.5-Pro Gemini-3.1-Pro Qwen3-VL-235B-A22B-Thinking
78.2 90.3 83.4
Gemini-3.1-Pro
Gemini-2.5-Pro Gemini-3.1-Pro Qwen3-VL-235B-A22B-Thinking
80.7 90.7 84.8
Table 6: Ablation study on different base models.
Robustness via Decoupled Base Models. Table 6 presents the ablation under varying combinations of perception and reasoning backbones. Because M EM D REAMER only requires the perception model to process localized short-form video clips (under 10 min), the capability gaps between different perception backbones are largely smoothed out. When driving the same downstream reasoning engine, either Gemini-3.1-Pro or Qwen3-VL, the final performance fluctuations caused by swapping the underlying perception model are merely 0.4 and 1.4 percentage points, respectively. This demonstrates that our decoupled framework exhibits high perceptual error tolerance, effectively alleviating the over-reliance on expensive, long-context perception for macroscopic video understanding.
]
є ѕ ķ Ə Ƒ ƍ ķ ƒ Ƒ Ə ķ ƒ
( - m b Ϲ Ϲ - l 0 ; 7 7 b m ] Ϲ Ϲ o m { ; { ] ; m { b 1 " ; - u 1 _ m Ϲ ] ; m { b 1 Ϲ Ϲ $ o o Ϲ v Ľ u v ľ
ƒ ƌ ѕ ķ Ə Ǝ ķ Ƌ
v o m ! ; -
!
ѕ є ķ ƍ Ƒ Ƒ ķ Ɛ
Ɛ ķ ƒ Ƒ ƍ ķ Ƒ Ƒ Ƒ є ķ є
ѕ ƒ ķ є
&
ƒ Ƌ ķ є Ƒ ƒ ķ Ə Ƒ ѕ ķ Ɛ
ѕ ƒ ķ Ƒ
&
Ƒ ķ ƌ ѕ Ɛ ķ ƍ ƒ Ƌ ķ є ѕ
Ə Ɗ
Ɛ Ə ķ є
Ǝ ѕ ƌ ķ Ǝ ѕ ƌ ķ
ѕ є ķ Ƌ
" l
ѕ ƒ ķ є
Ƒ Ƒ ķ Ɛ
$
Ƒ ƌ ķ Ƒ
ƍ ķ ƍ
!
ķ Ǝ ѕ Ƌ ķ Ǝ
є ƒ ķ Ǝ
Ƒ Ƌ ķ ƍ
ѕ ƒ ķ Ƒ
ƒ Ƌ ķ ƌ
ѕ ѕ Ə ķ Ɛ
" l
ķ Ə ѕ Ƒ ķ Ə ѕ є
ķ Ǝ Ƒ Ǝ ѕ Ɛ ķ Ƌ
Ɛ Ɗ
ѓ Ɗ
ƒ Ƌ ķ ƌ Ƒ
Ƒ Ɛ ķ Ǝ ѕ ƍ ķ Ə ѕ ѕ ķ Ɛ
Ƒ
ѕ є ķ
Ə Ɗ
Ƒ Ɗ є Ɗ
ѕ Ƌ ķ ķ ѕ Ƒ Ƒ
ƍ
ķ ƒ ѕ Ǝ
Ƒ є ķ
ƒ Ƌ ķ Ƒ
Ƒ ѕ ķ ƒ ѕ Ƌ ķ ƍ Ɛ Ƒ Ƌ ķ
!
Ɛ Ɗ ѓ Ɗ
ż Ϲ - { ķ _ m h b ; u - u 1 _ b 1 - Ϲ ķ _ m h ż Ϲ - { ķ u - r _ b ; u - u 1 _ b 1 - Ϲ u - r _ Ľ u v ľ
ƒ ƌ
ѕ ƒ ƌ ķ
є Ɗ
!
Ƒ Ƒ ķ Ə
ѕ Ə ķ ѕ
Ƒ Ɗ
$
ѕ
ƒ ƌ ķ
]
ѕ є ķ Ǝ ƒ Ƌ ķ Ƒ
o m
v ! ; -
D 0 H P R U \ & R Q V W U X F W L R Q $ E O D W L R Q E 5 H W U L H Y D O 0 H F K D Q L V P $ E O D W L R Q (a) Memory Construction (b) Retrieval Mechanism Figure 4: Ablations of different memory designs and retrieval strategies across LVBench sub-categories.
Hierarchy and graph contribute complementary gains. Table 7 and Figure 4 isolates the independent and joint contributions of the hierarchical topology (Hierarchy) and cross-node relational edges (Graph). Removing both components 1D Flat-Chunk yields the worst baseline performance of 77.4. Introducing topological edges alone 1D Flat-Graph drives a prominent 7.4-point performance gain to 84.8, showing that temporal and causal graph connections effectively preserve event continuity. Similarly, incorporating the bottom-up abstraction layer alone Hierarchical-Chunk raises the score to 86.3, validating that a multi-granular macro-skeleton prevents the retrieval agent from getting trapped in localized trivial details. Our full design 9
Memory Architecture
Components
LVBench
LVBench
Retrieval Strategy
Hierarchy Graph
Avg.
Sum.
Rea.
Flat-Chunk Flat-Graph Hierarchical-Chunk
✗ ✗ ✓
✗ ✓ ✗
77.4 84.8 86.3
Vanilla Embedding Similarity Full Memory Context Agentic Search Only
70.5 78.9 80.2
54.6 81.3 81.3
75.9 72.7 76.6
Hierarchical-Graph (Ours)
✓
✓
90.7
Agentic Full Tools (Ours)
90.7
89.7
89.6
Table 7: Ablation study on the memory design.
Table 8: Ablation study on memory retrieval strategies.
Hierarchical-Graph achieves the peak performance of 90.7. This compounding gain demonstrates a strong effect, where the macroscopic hierarchy and the underlying dense graph complement each other to optimize both global path navigation and localized target grounding. Tool-augmented retrieval beats embedding lookup. Table 8 and Figure 4 evaluates distinct memory access mechanisms to verify the necessity of our tool-augmented agentic retrieval. Our strategy Agentic Full Tools achieves an score of 90.7, outperforming the traditional static match baseline Vanilla Embedding Similarity by 20.2 points. This demonstrates that multi-step tool invocation captures significantly more precise cues than single-turn passive recall. While feeding the entire graph skeleton Full Memory Context benefits summarization with a score of 81.3, its performance on reasoning sharply degrades to 72.7. This confirms that passing the context window with unfiltered, graph structures introduces severe topological noise that disrupts critical logical deduction. Furthermore, restricting the agent to semantic retrieval alone Agentic Search Only yields a suboptimal accuracy of 80.2, demonstrating that the completeness of our multi-dimensional toolkit is critical for efficient long-form video understanding. Tmax
LVBench
Avg. rounds
Tokens/round
8 10 12 15
88.7 89.3 90.7 90.2
2.87 3.00 3.06 3.07
5,990 6,141 6,181 6,222
Table 9: Ablation on the round budget Tmax with R=Gemini-3.1-Pro and the full toolkit. Avg. rounds is the mean number of rounds actually taken before R emits an answer. Tokens/round is the mean input length per tool-call round.
Round Budget: Tmax Sweep. We finally study the round budget Tmax that hard-caps the loop. With R=Gemini-3.1-Pro and the full toolkit, we sweep Tmax ∈ {8, 10, 12, 15} and, alongside accuracy, record the mean number of rounds actually used and the mean input length per round. The latter two probe whether a larger budget is exercised passively (cap exhaustion) or actively (more reasoning only on hard queries). Three trends are visible in Table 9. First, accuracy improves steadily from 88.7 to 90.7 as Tmax grows from 8 to 12, then slightly regresses at 15 (90.2). A larger budget therefore does help, but with diminishing returns once the cap is no longer binding. Second, the average number of rounds actually used barely changes (2.87 → 3.07), indicating that R self-terminates as soon as it has accumulated enough evidence and only spends extra rounds on queries that genuinely require them, rather than passively exhausting the cap. Third, the mean input tokens per round remain nearly flat (5,990 → 6,222, +3.9%), showing that the selective compression of raw information keeps per-round context size bounded as Tmax grows, scaling the round budget therefore comes at negligible incremental cost. top-k
LVBench
Avg. rounds
5 10 (default) 15
87.1 88.7 87.2
2.91 2.87 2.90
Table 10: Ablation on the search top-k. We sweep the number of results returned by the Precise Search tools (SearchNodes, SearchByTime) per query, with R=Gemini-3.1-Pro and Tmax =8 held fixed.
Search Breadth: top-k Sweep. The Precise Search tools (SearchNodes, SearchByTime) return the top-k neighbours of a query embedding. A natural follow-up question is whether returning more candidates per call yields strictly better evidence coverage. We sweep k ∈ {5, 10, 15}, holding the rest of the configuration (R=Gemini-3.1-Pro, Tmax =8, full toolkit enabled) fixed. The relationship between k and answer accuracy 10
is non-monotonic and peaks at k =10 (Table 10). Both k =5 (87.1) and k =15 (87.2) underperform the default. We read the drop at k =5 as insufficient evidence coverage, relevant nodes occasionally fall outside the top-5 neighbour set, and the drop at k =15 as dilution: returning more candidates forces R to spend reasoning capacity disambiguating loosely related neighbours, which makes the truly informative cues harder to isolate. Crucially, the average number of rounds is essentially flat across the sweep (2.87–2.91), so k shapes the quality of each retrieval call rather than how many calls the agent chooses to issue. Tool category
Setup Full Memory (no tools) + Search + Graph Traversal + Hier. Nav. (full toolkit)
LVBench
Search
Graph
Hier.
– ✓ ✓ ✓
– – ✓ ✓
– – – ✓
78.9 80.2 86.8 90.7
Table 11: Ablation on the retrieval toolkit. We progressively enable the three tool categories of Table 2 starting from a tool-free baseline that dumps the full textual memory into R’s context. Reasoning model: Gemini-3.1-Pro, round budget Tmax =12, search top-k =10.
Effect of the Retrieval Toolkit. We next isolate the contribution of each of the three tool categories defined in Table 2, holding the reasoning model (R=Gemini-3.1-Pro), the round budget (Tmax =12), and the search top-k (= 10) fixed. As a tool-free baseline (Full Memory), we serialise the entire textual memory into R’s context in a single pass and ask it to answer without any tool call. We then progressively enable Precise Search, add Graph Traversal, and finally enable Hierarchical Navigation to recover the full toolkit. Three observations follow from Table 11. First, naively concatenating the full memory underperforms even singlestep search (76.9 vs. 78.2): without selective retrieval the relevant evidence is diluted by surrounding episodes. Second, adding Graph Traversal yields the largest single jump (+6.6), confirming that multi-hop chains along entity, role, and causal edges unlock the reason-over-events queries that flat semantic retrieval cannot resolve. Third, layering Hierarchical Navigation on top contributes a further +3.9, indicating that the macro-/super-/root-tier summaries provide global context that local search and graph walks .
5
Conclusion
In this work, We propose M EM D REAMER, a long-video understanding paradigm integrating a Hierarchical Graph Memory with tool-augmented agentic retrieval. Our framework establishes new SOTA across multiple challenging benchmarks. To our knowledge, we reveal for the first time a strong positive correlation between long-video performance and agentic reasoning capabilities. This paradigm bypasses context limits and mitigates attention dilution, offering a promising scaling direction for future multimodal comprehension.
Acknowledgements This work is supported by the Ant Group Reseach Internship Program.
References Anthropic. Claude 4.6 Opus, 2026. URL https://www.anthropic.com/news/claude-opus-4-6. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. Cong Chen, Ziyuan Huang, Cheng Zou, Muzhi Zhu, Kaixiang Ji, Jiajia Liu, Jingdong Chen, Hao Chen, and Chunhua Shen. Hieratok: Multi-scale visual tokenizer improves image reconstruction and generation. arXiv preprint arXiv:2509.23736, 2025a. Cong Chen, Kaixiang Ji, Hao Zhong, Muzhi Zhu, Anzhou Li, Guo Gan, Ziyuan Huang, Cheng Zou, Jiajia Liu, Jingdong Chen, et al. Gui-shepherd: Reliable process reward and verification for long-sequence gui tasks. arXiv preprint arXiv:2509.23738, 2025b. Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, and Chunhua Shen. Perturbollava: Reducing multimodal hallucinations with perturbative visual training. arXiv preprint arXiv:2503.06486, 2025c.
11
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024a. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024b. URL https://arxiv.org/abs/ 2312.14238. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130, 2024. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24108–24118, 2025. Guo Gan, Yuxuan Ding, Cong Chen, Yuwei Ren, Yin Huang, and Hong Zhou. Android coach: Improve online agentic training efficiency with single state multiple actions. arXiv preprint arXiv:2604.07277, 2026. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, Haofen Wang, et al. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1):32, 2023. Google DeepMind. Gemini 3.1 pro, 2026. URL https://deepmind.google/models/gemini/pro/. Accessed: 2026-05-20. Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062, 2025. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. Soyeong Jeong, Kangsan Kim, Jinheon Baek, and Sung Ju Hwang. Videorag: Retrieval-augmented generation over video corpus, 2025. URL https://arxiv.org/abs/2501.05874. Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13700–13710, 2024. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledgeintensive nlp tasks. Advances in neural information processing systems, 33:9459–9474, 2020. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. Science China Information Sciences, 68(10):200102, 2025. Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323–340. Springer, 2024. Niu Lian, Yuting Wang, Hanshu Yao, Jinpeng Wang, Bin Chen, Yaowei Wang, Min Zhang, and Shu-Tao Xia. From verbatim to gist: Distilling pyramidal multimodal memory via semantic information bottleneck for long-horizon video agents. arXiv preprint arXiv:2603.01455, 2026. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 conference on empirical methods in natural language processing, pp. 5971–5984, 2024.
12
Jingyang Lin, Jialian Wu, Jiang Liu, Ximeng Sun, Ze Wang, Xiaodong Yu, Jiebo Luo, Zicheng Liu, and Emad Barsoum. Videoseek: Long-horizon video agent with tool-guided seeking. arXiv preprint arXiv:2603.20185, 2026. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics, 12:157–173, 2024. Lin Long, Yichen He, Wentao Ye, Yiyuan Pan, Yuan Lin, Hang Li, Junbo Zhao, and Wei Li. Seeing, listening, remembering, and reasoning: A multimodal agent with long-term memory. arXiv preprint arXiv:2508.09736, 2025. Yongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, et al. Video-rag: Visually-aligned retrieval-augmented long video comprehension. Advances in Neural Information Processing Systems, 38:168008–168033, 2026. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36: 46212–46244, 2023. Charles Packer, Vivian Fang, Shishir_G Patil, Kevin Lin, Sarah Wooders, and Joseph_E Gonzalez. Memgpt: towards llms as operating systems. 2023. Ziqi Pang and Yu-Xiong Wang. Mr. video:" mapreduce" is the principle for long video understanding. arXiv preprint arXiv:2504.16082, 2025. Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18221–18232, 2024. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. Qwen Team. Qwen3.5: Towards native multimodal agents, 2026. URL https://qwen.ai/blog?id=qwen3.5. Accessed: 2026-02-16. Z.ai Team. Glm-4.6v: Open source multimodal models with native tool use, 2025. URL https://z.ai/blog/ glm-4.6v. Accessed: 2026-02-16. Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, et al. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning. arXiv preprint arXiv:2509.02544, 2025a. Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, et al. Lvbench: An extreme long video understanding benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22958–22967, 2025b. Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. Needle in a multimodal haystack. Advances in Neural Information Processing Systems, 37:20540–20565, 2024a. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025c. Xiao Wang, Qingyi Si, Shiyu Zhu, Jianlong Wu, Li Cao, and Liqiang Nie. Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 5417–5432, 2025d.
13
Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision, pp. 58–76. Springer, 2024b. Zhaowei Wang, Wenhao Yu, Xiyu Ren, Jipeng Zhang, Yu Zhao, Rohit Saxena, Liang Cheng, Ginny Wong, Simon See, Pasquale Minervini, et al. Mmlongbench: Benchmarking long-context vision-language models effectively and thoroughly. Advances in Neural Information Processing Systems, 38, 2026. Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3272–3283, 2025e. Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828– 28857, 2024. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, volume 2024, pp. 21875–21895, 2024. Zeyuan Yang, Delin Chen, Xueyang Yu, Maohao Shen, and Chuang Gan. Vca: Video curious agent for long video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20168–20179, 2025. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, et al. Mobile-agent-v3: Fundamental agents for gui automation. arXiv preprint arXiv:2508.15144, 2025. Woongyeong Yeo, Kangsan Kim, Jaehong Yoon, and Sung Ju Hwang. Worldmm: Dynamic multimodal memory agent for long video reasoning. arXiv preprint arXiv:2512.02425, 2025. Yufei Yin, Qianke Meng, Minghao Chen, Jiajun Ding, Zhenwei Shao, and Zhou Yu. Videoarm: Agentic reasoning over hierarchical memory for long-form video understanding. arXiv preprint arXiv:2512.12360, 2025. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471, 2025. Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024a. URL https://arxiv.org/abs/2407.12772. Xiaoyi Zhang, Zhaoyang Jia, Zongyu Guo, Jiahao Li, Bin Li, Houqiang Li, and Yan Lu. Deep video discovery: Agentic search with tool use for long-form video understanding. Advances in Neural Information Processing Systems, 38:89863–89895, 2026. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava-video: Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024b. Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, et al. Mmvu: Measuring expert-level multi-discipline video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8475–8489, 2025. Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 19724–19731, 2024.
14
Appendix 6
Reproducibility
M EM D REAMER inference configuration. At retrieval time, R runs with a maximum output length of 16K tokens and temperature 0.7 across all backbones. We evaluate Gemini-3.1-Pro, Gemini-2.5-Pro, and Qwen3-VL under these defaults. Closed-source baseline evaluation. Numbers in the main table are taken from each model’s technical report or prior benchmarks where reported. The remaining cells are produced in-house. Hour-scale videos in our four benchmarks exceed the multi-GB ceiling of direct API uploads. We therefore re-encode each video before sending it to the closed-source APIs. For Gemini-family models, we re-encode at FPS=1 and stage the artefact on Google Cloud Storage, which lifts the size restriction. For Claude-Opus-4.6, we uniformly sample 128 frames per video. For the GPT family, we uniformly sample 50 frames, which is the maximum the GPT API accepts. Default temperature for closed-source models is 0.7. For GPT-5, the reasoning_effort is set to high to activate extended thinking. For Claude-Opus-4.6, we set temperature to 1.0, which is the API-mandated value for activating thinking mode. Open-source baseline evaluation. We run open-source baselines with the lmms-eval (Zhang et al., 2024a) framework. For Qwen3-VL we follow the official recipe with max tokens 240k and the official prompt template, remaining open-source models use the framework’s default frame-sampling policy. All in-house reproduction runs use temperature 0.7 and a maximum output length of 8192 tokens. To illustrate the inference details, we report the computational costs of evaluating different models on the LVBench dataset. All tests are conducted using NVIDIA H800 GPUs: Qwen3VL-235B requires 29 hours on 32 GPUs, whereas InternVL takes 18 hours on 4 GPUs. With the vLLM framework, Qwen3.5-35B (4 GPUs), GLM-4.5V (8 GPUs), and GLM-4.6V (8 GPUs) all complete the evaluation in approximately 3 hours. Details of Token Calculation claimed in Introduction. In the Introduction, we claim that a 2-hour 720P video requires over 1.6M tokens. To clarify, for a 2-hour video sampled at 1 FPS (7200 frames) with a standard resolution of 1280 × 720 pixels, we employ the Qwen3-VL visual tokenizer with a patch size of 32. Additionally, a spatial 2 × 2 token merging strategy is applied, which compresses every 4 adjacent tokens into 1. Consequently, the total number of visual tokens input to the LLM is calculated as: Ntokens = ×720 7200 × 1280 32×32×4 = 7200 × 225 = 1, 620, 000 (1.62M). The License For Artifacts. The artifacts in this academic work primarily consist of four open-source video understanding benchmarks (Wang et al., 2025b; Wu et al., 2024; Fu et al., 2025; Mangalam et al., 2023). Additionally, the visualization cases are from other existing open-source benchmarks (Song et al., 2024; Wang et al., 2025b). All usage strictly complies with their respective licenses.
7
Qualitative Case Studies
We trace two LVBench questions through the loop and contrast our system with two end-to-end Gemini baselines that consume the same video directly. The reasoning model is Gemini-3.1-Pro in all settings. For our system, retrieval reads only the textual hierarchical graph and no frames are accessed. 7.1 Direct Drill-Down Figure 5 traces the question “Why does the worker pour the small fish from the blue basket into the large container?”. From the bootstrap state, the agent localises the question to super_08, identifies macro_0020 from the round-1 search hits, and pulls its subgraph in round 2. The subgraph makes the pour → boil → spread → mesh-tray drying chain explicit, so the agent answers (A) Drying them. Both end-to-end Gemini baselines see the pour shot at correct timestamps but, without the downstream causal chain, mistake it for rinsing and answer (B) Cleaning them. Perception is not what they miss, the multi-step causal context that the textual subgraph encodes as explicit edges is. 7.2 Multi-Round Reformulation Figure 6 traces a harder question on a Nigerian news broadcast: “In which topic does a man wearing a beige cowboy hat, standing in front of a green podium, appear while talking to a red microphone?”. Round 1 bundles all four cues into one query and returns no node that satisfies them simultaneously. Rather than commit to a weak hit, the agent logs the failure to useful_info. Round 2 drops the cowboy-hat detail, which the textual descriptions are unlikely to record verbatim, and retrieves a node mentioning a green podium inside 15
Gemini-3.1-Pro ORA Loop
User Query: Why does the worker pour the small fish from the blue basket into the large container? (A) Drying them (B) Cleaning them (C) Cooking them (D) Mincing them Ground Truth: A
Observation ## Video Summary The Ultimate Japanese Culinary Journey: Street Food, Seafood, and Premium Wagyu… ## Found 17 super events [super_01] Tokyo Food Tour and 7-Eleven Ramen Tasting [0:00-4:59] The video follows a male host and Shizuka Anderson on a vibrant culinary journey through Tokyo. It begins with a dynamic food tour montage, showcasing the hosts dining at narrow restaurant counters, sampling unique dishes like thinly. … [super_02] Tokyo Ramen Tasting Tour [4:59-14:27]… [super_03] Sendai Travel and Yakiniku Kotora Beef Sushi [14:27-18:31]… Reasoning ## Reasoning The question asks about the purpose of a specific action: a worker pouring small fish from a blue basket into a large container. Based on the video summary, this likely happens in super_08 (Enoshima Seafood Tour), which mentions the processing of fresh shirasu (tiny silver fish). I will search for this specific event to find the causal relationship or description that explains why the fish are being poured.. ## Can Answer False ## Useful Info Looking for the reason why a worker pours small fish from a blue basket into a large container. This is likely related to the shirasu (whitebait) processing in Enoshima (super_08)... ## Information Need Search Nodes(A worker pours small fish from a blue basket into a large container.) Action ## Call Tools: Search Nodes(A worker pours small fish from a blue basket into a large container) Observation Found 10 matching nodes: In <Macro Event> macro_0020 "Shirasu Whitebait From Catch to Tasting"<Match Node 0> (Event) [42:06-42:10] packs fish into containers A young man wearing a light blue shirt and white gloves uses a small scoop to carefully pack boiled white shirasu into clear plastic clamshell containers<[SEMANTIC:PERFORMS] (Entity) "Young Man": A young male worker wearing a light blue long-sleeved shirt who helps unload coolers, pack fish… In <Macro Event> macro_0000 "Japanese Food Tour Montage"<Match Node 1> (Event) [1:28-1:30] dumps and spreads A worker dumps a massive quantity of tiny white fish from a metal container onto a mesh-bottomed tray and spreads them evenly with a wooden tool…. In <Macro Event> macro_0020 \“Shirasu Whitebait From Catch to Tasting”<Match Node 2> (Event) [45:24-45:27..
Reasoning ## Reasoning The previous search didn't find the exact event of pouring fish from a blue basket into a large container, but it did find several events related to processing small fish (shirasu) in macro_0020 ('Shirasu Whitebait From Catch to Tasting'). I will retrieve the full subgraph for macro_0020 to find the specific event involving the blue basket and the large container, and see what happens immediately after to determine the purpose (drying, cleaning, cooking, or mincing). ## Can Answer False ## Useful Info macro_0020 ('Shirasu Whitebait From Catch to Tasting') contains events related to processing small fish (shirasu). Need to find the specific event where a worker pours fish from a blue basket into a large container to determine the reason (drying, cleaning, cooking, or mincing). ## Information Need Get EventSubGraph(macro_0020)
## Call Tools: Get EventSubGraph(macro_0020)
Action
Observation === Subgraph: Shirasu Whitebait From Catch to Tasting=== Time: 40:09-44:05 Description: The video begins with a fast-paced montage of various scenes. First, a male host wearing a dark blue shirt and a white patterned headband tied around his head points towards a white fishing boat approaching a pier. Next to him is a female host wearing a red... --- Entities (18) ---: Male Host (PERSON) A male presenter wearing a dark blue shirt and a distinctive white patterned headband tied around his head.… --- Events (35) ---: point towards a boat [0:00-0:08] The video begins with a fast-paced montage showing a male and female host pointing towards … -- Key Relations (40) --- (Cooler Boxes) -[SEMANTIC:RECEIVES]-> (unload coolers onto the pier) … Reasoning
## Reasoning The question asks why the worker pours the small fish from the blue basket into the large container. Based on the typical processing of shirasu shown in the video (and the provided options), the fish are poured into a large metal vat filled with vigorously boiling water in order to cook them. This boiling process turns the translucent raw shirasu into opaque white boiled shirasu. ## Can Answer True Answer: A
Gemini-3.1-Pro End-to-End Answer
## Reasoning: Between 40:14 and 40:16, the video shows a worker pouring small fish from a blue container into a larger blue barrel filled with water and ice. This action is a washing or rinsing process to clean the fish before they are further processed or dried. ## Answer: B
Gemini-2.5-Pro End-to-End Answer ## Reasoning: The video shows a worker pouring small fish from a blue basket into a larger container filled with water at timestamp 43:13. The subsequent shots show the fish being swirled around in the water, which is a common method for cleaning or rinsing food items to remove impurities. The other options are incorrect: the fish are being put into liquid, not dried; the water does not appear to be hot, so they are not being cooked; and the fish remain whole, so they are not being minced. ## Answer: B
Figure 5: Observation-Reason-Action loop trace on a Japanese culinary travelogue of LVBench.
macro_0010. Round 3 pulls the subgraph, where an entity description and an OCR overlay reading R IVERS P OLITICS together yield the answer (A) River Politics. The trace shows that the loop survives an unhelpful first round by reformulating its query instead of committing.
16
Gemini-3.1-Pro ORA Loop
User Query:
In which topic does a man wearing a beige cowboy hat, standing in front of a green podium appear while talking to a red microphone? (A) River Politics (B) Electricity Tarrif Hike (C) Lagos Market Fire Aftermath (D) Drug Trafficking
Ground Truth: A
Observation ## Video Summary Channels TV News at 10: National Crises, Politics, and Global Updates… ## Found 10 super events [super_01] Channels TV News at 10 Opening and Headlines [0:00-2:44The segment captures the opening sequence and headline rundown for Channels TV's "NEWS at 10" broadcast. It begins with a dynamic 3D animated intro featuring the Channels TV logo. … [super_02] Dosumu Market Fire Disaster News Report [2:44-6:20], … [super_03] Rivers State Politics News Broadcast [6:20-12:40]… Reasoning ## Reasoning The question asks for the topic of a news segment featuring a man wearing a beige cowboy hat, standing in front of a green podium, and talking to a red microphone. The options correspond to different news topics covered in the broadcast: River Politics, Electricity Tariff Hike, Lagos Market Fire Aftermath, and Drug Trafficking. I will search for this specific visual description to find the corresponding event and its topic. ## Can Answer False ## Useful Info Need to find a man wearing a beige cowboy hat, standing in front of a green podium, talking to a red microphone. Options are River Politics, Electricity Tariff Hike, Lagos Market Fire Aftermath, and Drug Trafficking.. ## Information Need Search Nodes(A man wearing a beige cowboy hat stands in front of a green podium and talks into a red microphone.) ## Call Tools: Search Nodes(The Pied Piper sleeps or spends the night)
Action
Observation Found 10 matching nodes: In <Macro Event> macro_0003 "Rivers Politics Report"<Match Node 0 (Entity) [1:21-1:26] PERSON: A man wearing a white hat and shirt speaking into a microphone, shown on the digital display. Related events and entities: ->[SPATIAL:LOCATED_IN (Entity) "OBJECT: A large digital display screen behind the anchor. ->[SPATIAL:NEXT_TO (Entity) "GROUP:A seated audience shown on the digital display graphic In <Macro Event> macro_0037 "Lagos Professionals Conference Highlights"<Match Node 1 (Event) [42:09-42:24] …
Reasoning ## Reasoning The previous search returned a node in macro_0002 ('Lagos Market Fire Aftermath Report') describing an older man wearing a wide-brimmed hat speaking before multiple microphones in a graphic discussing 'another twist in Rivers politics'. However, the description didn't explicitly mention a 'beige cowboy hat', 'green podium', or 'red microphone'. I will search specifically for 'green podium' and 'red microphone' to find the exact match and confirm the topic. ## Can Answer False ## Useful Info Searched for the man with the beige cowboy hat, green podium, and red microphone. Found a potential match in macro_0002 where a man in a wide-brimmed hat speaks into microphones with an OCR graphic 'ANOTHER TWIST IN RIVERS POLITICS'. Need to confirm if this or another node explicitly mentions the green podium and red microphone to be certain of the topic. ## Information Need Search Nodes(A man stands at a green podium and speaks into a red microphone.)
Action ## Call Tools Search Nodes(A man stands at a green podium and speaks into a red microphone)
Observation Found 10 matching nodes: In <Macro Event> macro_0037 "Lagos Professionals Conference Highlights"<Match Node 0 (Event) [42:09-42:24] addresses the audience A man wearing a blue suit, a matching blue baseball cap, and glasses stands behind a white podium, holding a microphone and gesturing with his right hand as he addresses the audience. Related events and entities… In <Macro Event> macro_0061 "Return to News Studio"<Match Node 1> (Event) [60:19-60:20]… ## Reasoning Reasoning The search results show a 'bright green podium equipped with microphones' in macro_0010, which is titled 'Rivers Politics Odili Backs Fubara Report'. This strongly points to option (A) River Politics. To be absolutely certain, I will retrieve the subgraph for macro_0010 to confirm the presence of the man wearing a beige cowboy hat and talking to a red microphone. ## Can Answer False ## Useful Info: Found a 'bright green podium' in macro_0010 ('Rivers Politics Odili Backs Fubara Report'). Need to check the subgraph of macro_0010 to confirm if the speaker (Peter Odili) is wearing a beige cowboy hat and talking into a red microphone.. ## Information Need Get EventSubGraph(macro_0010) ## Call Tools: Get EventSubGraph(macro_0010)
Action
=== Subgraph: Rivers Politics Odili Backs Fubara Report === Observation Time: 6:20-9:04 Description: he video begins with an aerial shot of a densely packed urban area where smoke is rising from several buildings, suggesting recent damage or fire. A transition effect blends the aerial city view into a studio shot of a news anchor... --- Entities (13) ---: LOCATION: An aerial view of a densely packed urban area showing buildings with damage and rising smoke… --- Events (16) ---: shows a damaged city [0:00-0:01] The video starts with an aerial view of a densely packed… -- Key Relations (8) --- (PERSON: A man, wearing an orange traditional outfit and glasses, holding a black umbrella.) -[SEMANTIC:USES_TOOL]-> (OBJECT: A black umbrella held over Peter Odili to provide shade during his speech.)…
Reasoning ## Reasoning In macro_0010, Peter Odili is described as wearing a beige hat and standing at a bright green podium equipped with microphones. The OCR text clearly displays the topic as 'RIVERS POLITICS'. This perfectly matches the description in the question and aligns with option (A). ## Can Answer True Answer: A
Figure 6: A harder LVBench question on a Nigerian news broadcast where the first-round retrieval is uninformative. The agent logs the negative result, reformulates the query in round 2, and finally drills into the subgraph of macro_0010 in round 3. 17
Memory Streaming Segmentation Prompt """You are an Expert Visual Story Segmenter. [YOUR CORE OBJECTIVE] Watch the video and segment it into "Major Story Episodes". Each episode MUST be a complete, self-contained logical event, typically lasting between 3 to 8 minutes. [STRICT SEGMENTATION RULES] 1. WHAT CONSTITUTES AN EPISODE: An episode is a full narrative arc for ONE subject or ONE specific phase. For example, a single contestant's entire segment (walking on stage -> singing -> judges' visual reactions -> judges talking to them -> leaving stage or screen fading/cutting) is ALL ONE EPISODE. Do not split it. 2. RELY ON PURE VISUAL CUES: Look for major visual resets to mark a new episode: - A completely new contestant appearing on the main stage. - Distinct broadcast graphics (e.g., a new name tag or lower-third graphic appearing). - A major, permanent location change (e.g., shifting from the main stage to a long backstage interview segment). 3. ANTI-FRAGMENTATION (DO NOT CUT): - DO NOT cut when the camera simply switches angles (e.g., cutting from the singer to a reacting judge, or to the audience). - DO NOT cut for flashing stage lights or brief visual effects if the core subject on stage remains the same. 4. BOUNDARY PRECISION (NO BLEEDING): - The `end_time` of an episode MUST NOT include the appearance of the next episode's subject. - Stop the clock the exact second BEFORE the new contestant walks in, BEFORE the new stage is shown , or BEFORE the new graphics appear. - Every episode must be 100% exclusive to its own subject. [OUTPUT FORMAT] Use your internal video timestamp perception to determine the time boundaries. Output ONLY a valid JSON array of objects. Each object must have: - `start_time`: "HH:MM:SS" (based on video timestamp) - `end_time`: "HH:MM:SS" (based on video timestamp) - `episode_title`: A concise name (e.g., "Contestant_01_Full_Segment", "Host_Opening_Rules"). - `visual_evidence`: Briefly explain the visual cue that justified this boundary, confirming no bleed-over. Output raw JSON only. Do not wrap in markdown format. """
Figure 7: Streaming Segmentation Prompt.
18
Memory Subgraph Construction Prompt You are a video knowledge-graph annotator. Given a segment of video, output a single JSON object that describes its subgraph G_i = (V_i, E_i) with three node groups (V^E entities, V^M micro-events) and the edges among them. Watch the video carefully before producing the output. # Task overview Produce, for the given segment, an exhaustive but de-duplicated list of: 1. micro-events — every meaningful action, scoring play, transition, or salient happening, time-stamped at second precision. 2. entities — every PERSON, OBJECT, LOCATION, or GROUP that participates in or appears alongside those events, anchored to the moment they are most clearly visible. 3. edges — directed labelled relations among the three node groups, covering subject/object roles, spatial-attributive ties, and explicit causal links. # Node schema - micro_events: {event_id, event_type, time_range:[start_sec,end_sec], subject, object, action, description} Rules: * event_type is free text (e.g. "scoring play", "foul", "timeout"). * subject/object MUST be specific enough to match a single entity. * action describes the precise motion. * description is the retrieval payload: full sequence, with cause when applicable. * Do NOT create a micro_event for static on-screen graphics. - entities: {entity_id, name|null, entity_type in {PERSON,OBJECT,LOCATION,GROUP}, attributes, description, visual_grounding} Rules: * name is the real name only if explicitly identified by broadcast graphics, jersey, or caption. Never invent. * attributes: PERSON{jersey_number,jersey_color,team,role}; OBJECT{type,color,location}; LOCATION/GROUP{name,type}. * description fuses attributes + visual cues into one sentence. * visual_grounding = {primary_time:"MM:SS", primary_time_sec:int, distinctive_features:[2-4 unique cues], spatial_hint}. Rules: * text is the verbatim transcription. * If the same overlay persists, emit ONE node spanning its full time_range. * description interprets the text in context. # Edge schema Each edge: {source_id, target_id, relation_label, relation_type, description}. Allowed labels and their relation_type are: - Entity -> Event: PERFORMS, RECEIVES, USES_TOOL (SEMANTIC) - Event -> Entity: HAPPENS_IN (SEMANTIC) - Event -> Event: CAUSES, PREVENTS (CAUSAL) SUBEVENT_OF (HIERARCHICAL) - Entity <-> Entity: PART_OF, LOCATED_IN, NEXT_TO, ATTACHED_TO (SPATIAL) IS_SAME_AS (IDENTITY) # Output format Return ONLY one valid JSON object with the following top-level keys: { "micro_events": [ {event_id, event_type, time_range:[start,end], subject, object, action, description}, ... ], "entities": [ {entity_id, name, entity_type, attributes, description, visual_grounding}, ... ], "edges": [ {source_id, target_id, relation_label, relation_type, description}, ... ] } Output the final JSON now.
Figure 8: Subgraph Construction Prompt.
19
Memory Hierarchical Aggregation Prompt You are a video narrative analyst. Given a chronologically ordered list of Macro events {macro_id, label, time_range, summary, key_entities, event_types, ocr_texts}, fold them upward into a hierarchy V^Mc -> V^S -> v^R and emit a single JSON object. # Task overview Produce four things in one pass: 1. super_events — 10-20 Super Events that partition all Macro events; each captures one narrative arc. 2. macro_relations — strong logical edges between Macro events within the same Super Event. 3. super_relations — high-level edges between Super Events (not restricted to adjacent ones). 4. root — a video-level summary v^R synthesised from all Super Events. # Clustering rules (Macro -> Super) A Super Event is a contiguous run of Macro events that share (i) temporal adjacency, (ii) scene/topic , and (iii) a common goal. A boundary fires when any of the three breaks. Use as cluster signals: - key_entities overlap between adjacent Macros (strong group signal); - event_types continuity (e.g. a sustained run of "shot_made" macros); - ocr_texts cues (scoreboard increments stay inside a phase; broadcast graphic changes mark boundaries). Every Macro MUST be assigned to exactly one Super Event; each Super Event holds 3-8 Macros on average ; a singleton is allowed for a self-contained segment. # Edge schema - Macro -> Macro: CAUSES (A directly triggers B), PREVENTS (A blocks B), ENABLES (A supplies a prerequisite for B). Stay within the same Super Event. Each edge needs a one-sentence reason. Omit when uncertain — never use BEFORE/AFTER. - Super -> Super: LEADS_TO (phase A naturally progresses to B), RESOLVES (B resolves a tension introduced in A), CONTRASTS_WITH (sharp reversal in outcome/tone). Cross-distance edges are allowed. # Naming and description - super_event.label: noun phrase under 10 words; include the central entity when one exists ("LeBron 's Third-Quarter Run"). - super_event.description: 100-200 words covering the arc, key entities (woven in, not listed), and any salient OCR milestones. - key_entities are deduplicated within a Super Event AND canonicalised across the whole video: variants of the same real-world entity ("LeBron", "King James", "LeBron James") collapse to one canonical name with a consistent type. When in doubt, keep them separate. - root.title: under 15 words, specific. root.description: 3-5 sentences synthesising ALL Super Events. root.themes: 3-5 short tags. root.key_entities: 5-10 canonical names. root.emotional_tone: 2-3 adjectives. # Output format Return ONLY one valid JSON object, no markdown fences, no commentary: {
"super_events": [ {super_id, label, description, sub_macro_ids:[macro_id,...], time_range:[start, end], key_entities:[{name,type},...]}, ... ], "macro_relations": [ {source, target, type, reason}, ... ], "super_relations": [ {source, target, type, reason}, ... ], "root": {title, description, themes:[...], key_entities:[...], emotional_tone} } Hard constraints: every Macro id appears exactly once across all sub_macro_ids; super_event. time_range = [min(start), max(end)] of its Macros; every relation references ids that exist in the output; emit only confident causal/structural edges. Output the final JSON now.
Figure 9: Consolidated hierarchical aggregation prompt used by the reasoning model during memory construction. The prompt fuses the clustering, refinement, entity-unification, and global-synthesis stages of Phase 3 into a single end-to-end specification that lifts Macro Events V Mc to Super Events VS and the Video Root v R , with the resulting hierarchy mirroring Table 1.
20
Agentic Retrieval Prompt You are the reasoning model R in an Observation-Reason-Action loop over a hierarchical video memory. At each round you read the current observation, decide whether you can answer, and either emit a final letter A/B/C/D or call exactly one tool. # Memory hierarchy The memory is a 4-tier graph: - Root v^R — video-level title, description, themes, key entities. - SuperEvent V^S — 10-20 narrative phases with time_range, key entities, description. - MacroEvent V^Mc — detailed events inside a Super, with time_range and key entities. - Subgraph G_i — per-Macro detail: Entity / micro-Event and their edges. Initial context contains ONLY Root + the Super Event list. Macro descriptions, subgraph contents, and edges are NOT pre-loaded — you MUST drill down with tools. # Toolkit (call at most one per round) Hierarchical Navigation: - get_summary() - get_super_events() — list Supers - get_macro_events(super_id) — list Macros under a Super. - get_subgraph(macro_id) — full Entity/Event/edges of a Macro. Information-richest tool; never call it without first identifying the macro_id. Precise Search: - search_nodes(query, top_k, node_types) - search_by_time(start_sec, end_sec)
— semantic search over Entity/Event embeddings. — Macros covering a time window.
Graph Traversal: - get_relations(node_id, direction) — outgoing/incoming/both edges of any node, typed PROGRESSION / CAUSAL / TEMPORAL / HIERARCHICAL / SEMANTIC / SPATIAL. # Strategy Default workflow: search -> identify macro_id -> get_subgraph for full detail. Macro labels are summaries; do NOT answer whodid-what or exact-score questions from them alone. Pick the tool from the question pattern: time references -> search_by_time; person + action -> search_nodes; causal chain -> get_relations. # Query construction (for search_nodes and search_semantic) - Write a DESCRIPTIVE STATEMENT, not a question — embeddings match event descriptions, not interrogatives. - Use information SHARED across answer options as the retrieval signal; stay neutral on details where options disagree. - Good: "A foul is committed, leading to free throws." Bad: "Australia #7 fouled" (commits to one option) or "Who fouled?" ( interrogative). # Round memory (Extract operator) Previous tool observations are NOT carried forward — only your `useful_info` field persists. Distil into it every fact, ID, and reasoning thread you will need next round (super_ids, macro_ids found, time ranges, score state, what you have already searched, what to try next). The Exploration History block lists prior queries and which Macros remain unexplored — use it to avoid repeating searches and to drill into unexplored macros instead of re-searching. # Output (strict JSON, no markdown fences, no commentary) { "reasoning": "What does the question ask? What do I already have? What is missing?", "can_answer": true | false, "answer": "A" | "B" | "C" | "D" | null, "useful_info": "All facts / IDs / reasoning threads — the only state carried to the next round.", "information_need": null OR { "type": "browse_macros | get_macro_details | search_nodes | search_time | get_relations", "target": "...", "search_hints": [...], "reason": "..." } } Termination: when can_answer=true emit `answer` and set information_need=null; otherwise emit one tool call via information_need; the loop is hard-capped at T_max rounds.
Figure 10: Consolidated agentic retrieval prompt used by the reasoning model R inside the Observation–Reason–Action loop.
21
System Prompt for End-to-end Methods """Please watch the video carefully and answer the following multiple choice question. {question} Instructions: 1. First, provide your step-by-step reasoning about the question based on the video content. 2. Then, choose the correct answer from the options. 3. Output your response as a JSON object with two fields: - "reasoning": your step-by-step analysis - "answer": the letter of your chosen answer (e.g., "A", "B", "C", "D", or "E") Example: {{ "reasoning": "The video shows a person cooking pasta. At 2:30, they add tomatoes to the sauce, which corresponds to option C.", "answer": "C" }} Output ONLY the JSON object, no additional text."""
Figure 11: System prompt used for Evaluating End-to-end methods with thinking mode.
22