EgoEverything: A Benchmark for Human Behavior–Inspired Long-Context Egocentric Video Understanding in AR Environment Jieyu Lin, Ziyun Li, Barbara De Salvo Meta Reality Labs
arXiv:2604.08342v1 [cs.LG] 9 Apr 2026
Abstract
…
Streaming egocentric video AR HMD
Long-context egocentric video understanding has recently attracted significant research attention, with augmented reality (AR) highlighted as one of its most important application domains. Nevertheless, the task remains highly challenging due to the need for reasoning over extended temporal contexts and diverse, unstructured activities. Although several benchmarks exist, most egocentric datasets rely on human-worn cameras and focus mainly on visual content, with limited consideration of underlying user behavior when forming videorelated queries. EgoEverything is a benchmark that explicitly considers human behavior by leveraging human attention signals, abstracted from gaze data, when generating questions. It comprises over 5,000 multiple-choice question–answer pairs, spanning more than 100 hours of video. By integrating human attention signals during question generation, it more faithfully captures natural human behavior and offers a realistic evaluation setting for longcontext egocentric video understanding in AR.
1
Sai Qian Zhang New York University
…
Qiance Tang, Ziqi Wang * New York University
… T=10
T=1
(a)
Red lobster Coffee bags
What was the name of the restaurant I saw 10 minutes ago?
Cup
Coffee machine
(b)
Figure 1: (a) An example of a real-life AR LEU scenario. (b) An example illustrating how user attention varies.
tunities for long-context ML models to capture correlations among user attention, behavior, and the environment over extended timescales, enabling more effective everyday assistance. As illustrated in Figure 1 (a), consider an AR head-mounted display (HMD) used for driving navigation. The device continuously encodes multimodal inputs through ML models. When the user briefly glances at a restaurant, this gaze event must be accurately linked to the corresponding visual features. Later, answering a query such as “What was the name of the restaurant we passed 10 minutes ago?” requires an episodic memory module that can index temporal embeddings, retrieve the relevant instance, and integrate it with a machine learning model (e.g., Vision-Language Model (VLM)) for analysis. This paradigm transforms AR from a passive data collector into an intelligent personal assistant powered by long-context ML. In everyday life, AR could enable superhuman memory, helping users retrieve details such as where they left their keys. Building toward this vision, long-context egocentric video understanding (LEU) has become an increasingly active area of research in the ML community. Long egocentric recordings capture extended daily activities and interactions, requiring models to reason over temporal dependencies and multimodal signals spanning minutes or hours. To evaluate progress, a growing set of benchmarks has been introduced (Perrett et al., 2025; Chan-
Introduction
Augmented Reality (AR) is emerging not only as a novel user interface but also as a machine learning (ML) platform that integrates embodied experiences such as sensing, perception, memory, and speech. By aligning digital and physical realms, AR enables real-time information extraction and contextual decision-making, transforming domains including healthcare (Gerup et al., 2020; Chengoden et al., 2023), education (Westin et al., 2022; Al-Ansi et al., 2023), and industry (Chidsin et al., 2021; Jo et al., 2021). Beyond immersive use, AR devices continuously generate rich multimodal data from cameras, eye and hand tracking, and motion sensors. While challenging to model, these streams offer unique oppor* Qiance and Ziqi contributed equally to this work.
1
drasegaran et al., 2024; Zhou et al., 2025; Wu et al., 2024), each designed to test how well models can recall, integrate, and reason over such extended sequences. While these benchmarks represent important advances, they largely emphasize generic video-based queries and fall short of capturing the human-centric and attention-guided nature of real AR usage, where questions are often grounded in what the user was attending to at a given moment. These limitations can be summarized as follows: Questions Not Reflecting Human Attention: Existing benchmarks rarely consider user attention when designing queries, creating a mismatch with real-world usage. In practice, people tend to ask about objects or events they have looked at, fully or partially, where their attention was directed. Current datasets instead emphasize generic questions about visual details or scene overviews, and fail to reflect human inquiry patterns. Questions Not Framed in Natural Language: Prior benchmarks often rely on rigid, templatebased question generation that does not align with authentic human questioning. For example, prompts such as “Is the light off in the video?” frequently appear, but they are uncommon in daily use. In contrast, real users are more likely to ask context-specific, attention-driven questions such as “Did I forget to turn off the lights?” Questions Not Aligned with The Moment of Interaction: Most benchmarks restrict questioning to occur before or after a clip has ended. However, users typically pose questions during ongoing interactions, requiring real-time reasoning over partially observed streams. To address these limitations, we present EgoEverything, a benchmark for LEU that simulates real-life interactions with AR glasses. Collecting egocentric video in realistic AR scenarios and manually authoring multiple choice questions is both labor and time intensive. Annotators must repeatedly review videos, verify fine grained details, craft challenging queries, and refine phrasing to approximate natural user language. As a result, many prior works resort to template based question generation (Perrett et al., 2025; Xiao et al., 2021; Li et al., 2023; Zhou et al., 2025; Wu et al., 2024; Mangalam et al., 2023), which lowers cost and improves consistency but fails to capture how AR users actually ask questions. In contrast, EgoEverything is constructed through a novel Visual Question Answering (VQA) generation pipeline that leverages multiple AI agents to produce questions aligned
with authentic human questioning patterns. We further introduce an attention inspired sampling strategy that selects question targets based on simulated gaze, enabling the benchmark to include both attention driven queries and detail oriented ones outside the user’s focus. This design raises task difficulty while more closely matching real world AR query behavior. Finally, we incorporate comprehensive human review to enhance question quality and reliability. Specifically, our contribution can be summarized as follows: • To incorporate human attention into question generation for LEU benchmark, EgoEverything involves an attention-inspired sampling strategy that selects targets based on simulated gaze, enabling both attention-driven and detail-oriented queries. • We propose a VQA pipeline with multi-agent question generation and attention inspired sampling based on simulated gaze, producing both attention driven and detail oriented queries. Rule based filtering and human review further ensure quality and reliability. • EgoEverything comprises over 5,000 multiplechoice question–answer pairs across more than 100 hours of video. Evaluation on several cutting-edge VLMs reveals consistently lower performance on EgoEverything, highlighting current limitations of VLMs in handling reallife AR LEU scenarios.
2
Background and Related Work
2.1
Spatial Dynamics of Human Attention
Human perception is inherently selective, as the brain cannot process the entire visual field with equal fidelity. Prior research has described the attention field as a “spotlight” (Eriksen and St. James, 1986; Posner, 1980), often approximated by gaze location. Attention strength typically decays spatially, commonly approximated by a 2D Gaussian (Ioannides and Poghosyan, 2010), resulting in high fidelity at the focus point that gradually diminishes with distance (Desimone et al., 1995; Carrasco, 2011). This attentional pattern strongly influences the types of questions an AR user is likely to ask in real scenarios. As illustrated in Figure 1 (b), a user may focus on a cup while walking in the kitchen. Since attention is concentrated on the cup, the user 2
Outer camera
is more likely to later issue a query about this object (e.g., Where did I put the cup?). In contrast, nearby items, shown with lighter bounding boxes in Figure 1 (b), receive weaker attention and are therefore less likely to become the subject of subsequent queries. Similar findings have been reported in cognitive psychology and vision science, where gaze serves as a reliable predictor of future memory recall and questioning behavior (Yarbus, 2013; Land and Hayhoe, 2001).
(a)
(c)
Coffee bags
Vision Language Models
Attention level
Figure 2: (a) Front and (b) inner views of the Meta Quest Pro headset, which functions as both an AR and VR (Virtual Reality) device. (c) Meta Aria glasses equipped with multiple cameras. Cup
Coffee machine
Cup
Coffee bags
Dist.
(d) context egocentric video understanding (Perrett et al., 2025; Chandrasegaran et al., 2024; ZhouCoffeeCupmachine et al., 2025; Wu et al., 2024; Mangalam et al., 2023; Coffee bags Coffee machine Xiao et al., 2021; Li et al., 2023). However, these Time (e) (c) datasets primarily emphasize generic video-based questions and overlook the human-centric nature of real AR usage. They also restrict questioning to occur only before or after a clip or fixed segment has concluded. In practice, users tend to ask questions anchored to where their attention was directed, often indicated by gaze during recording, yet existing benchmarks fail to capture this critical dimension. Consequently, they fall short of simulating realistic AR scenarios in which attentional focus fundamentally shapes memory retrieval and contextual reasoning. Empirical evidence further supports this view, as incorporating gaze has been shown to significantly improve grounding in egocentric retrieval and natural language query (NLQ) tasks (Lin et al., 2025).
Contemporary Vision–Language Models (VLMs) (Team and et al., 2024; OpenAI and et al., 2024; Zhang et al., 2025a; Deitke and et al., 2024; Bai et al., 2025) extend language-guided foundation models to additional modalities and demonstrate strong, general capabilities. Trained at scale with spatial data (Chen et al., 2024), they achieve high accuracy in spatial understanding and encode rich human preferences and priors. They support perception, reasoning, instruction following, and dialogue, enabling AR assistants that ground semantics in what the user sees and points to. 2.3
(b)
Attention level
2.2
Eye-tracking Eye-tracking Outer cameras camera camera
Long-context Egocentric Video Understanding
Long-context egocentric video understanding has recently gained significant attention in the machine learning community, particularly in extended first-person recordings that capture daily activities and interactions. Such representations enable AR systems to support timely, context-aware assistance by recalling and reasoning over past events in real-world environments. Recent advances in long-context information representation increasingly emphasize structured approaches (Yang and Ren, 2025; Wang et al., 2023; Yang et al., 2025; Arnab et al., 2021; Baradel et al., 2018; Brendel and Todorovic, 2011; Cong et al., 2021). In the egocentric vision domain, studies have explored structured video representations by grouping video segments into activity threads (Price et al., 2022; Fan et al., 2024; Yang and Ren, 2025) or constructing egocentric scene graphs to model object–user relationships (Goletto et al., 2024; Rodin et al., 2024; Huang et al., 2025). The stored memory entries can later be queried by the user, and these entries are then provided as input to a machine learning model (e.g., VLM) to generate an answer. Alongside these advances, numerous benchmarks have been introduced to evaluate long-
2.4
AR System
Figure 2 illustrates typical AR device hardware configurations. These systems feature multiple front- and side-facing cameras that capture the user’s field of view and gaze position. Outwardfacing cameras generate high-resolution imagery (e.g., 1408 × 1408 on the Meta Aria glasses (Meta, 2023a)), while inward-facing cameras capture lower-resolution monochrome images of the eyes. Combined, these sensing mechanisms enable gaze tracking (Meta, 2023b; Microsoft, 2023; que), typically through either analytical methods such as pupil–corneal reflection modeling or machine learning approaches (Liu et al., 2025). These modalities provide critical signals of user attention and interaction, making them foundational for many AR applications. 3
STEP 1 Video Stream Summary and Clustering (VSSC)
Cluster 1
Action Summary
Cluster 2
VLM
Cluster 3 Chatting with two people …
Playing card with friend …
0.0-40.1: Playing card with friend in the living room. 40.1-50.3: Chatting with two people in the bedroom. 50.3-110.6: Walking with people in the woods.
Walking …
VLM
VLM
STEP 2 Gaze-Oriented Target Sampling(GOTS)
Target object & frame book
Object detector & ReID
Random Sampling by distance
card
Gaze point
probability
jar Distance to gaze point
All Frames
card book jar
STEP 3 Question Generation and Manuel Curation (QGMC)
8
Generation Agent
Action Summery
1 I can see the title of the book is … I need another
Target object & frame
3 I have confirmed the title at 35s. I can give the
video
2 Accept
Target Frame
Action Summery Review Checklist
Reject
7
Generation Agent Thanks for the review. I will rewrite the QA
5
Review Agent
4 I need to check the title of this book
Review Agent
Q: I forgot the title of that book on my coffee table. Can you tell me what it is? A: [A … B … C … D … E …], correct: B Evidence: [35s, 32.5s] From these frame I can clearly see the title is …
QA now Q: I forgot the title of the book. Can you tell me what it is? A: [A … B … C … D … E …], correct: B Evidence: [35s, 32.5s] From these frames I can clearly see the title is …
Dist. to gaze
Pass. QA is good to go!
35s
frame to confirm <Tool calling Get_Frame 35s>
Target Object Name: Book Target Object bbox: [95,476,133,520] Target Frame timestamp: 32.5s
34s
<Tool calling Get_Frame 34s>
video
Review Agent
Review
6 Answer is correct. But you need to be more specific about which book you are referring to
Figure 3: Data generation pipeline for EgoEverything data. Step 1 Video Stream Summary and Clustering (VSSC) combines clustering, summary generation, and manual inspection to acquire high quality video description. Step 2 Gaze-Oriented Target Sampling, first detect all objects from each frame and calculate their distance to gaze, then applies ReID on each object to acquire group labels for target groups, lastly samples target objects based on distance. Step 3 Question Generation and Manual Curation uses self-feedback Synthesizer Agent and Validator Agent to produce MCQs related to given target object.
3
Data Collection Procedure
3.1
Overview
ception Sampler (PS), which adaptively adjusts its statistical distribution according to the gaze location in the current frame. The selected target objects are then passed to the subsequent stage for question generation. For QGMC, illustrated in Step 3 of Figure 3, we deployed two VLM agents to handle question synthesis and validation. The Synthesizer Agent (VA) creates multiple-choice questions (MCQs) centered on the selected target object, whereas the Validator Agent (VA) examines these questions and delivers feedback. Following VQA generation, the dataset undergoes additional refinement through over 400 hours of human review combined with rule-based screening.
The collection process of EgoEverything consists of three steps: (1) Video Stream Summary and Clustering (VSSC), (2) Gaze-Oriented Target Sampling (GOTS), (3) Question Generation and Manual Curation (QGMC). During VSSC, the VLM is prompted with an egocentric video clip to generate a summary of the user’s action over the given time span, as shown in Step 1 of Figure 3. Manual inspection is then applied to remove errors in the generated action summaries. Based on the temporal scope of each summary, the egocentric video stream can be categorized accordingly. During GOTS, highlighted in Step 2 in Figure 3, each clustered video tile is examined to sample the objects appearing within it. Sampling is guided by the Per-
3.2
Video Stream Summary and Clustering
Our dataset EgoEverything draws from real egocentric videos with gaze annotations, specifically the 4
Aria Everyday Activities (AEA) (Lv et al., 2024) and Nymeria (Ma et al., 2024) datasets. The former consists of 143 clips (7.3 hours total) capturing annotated daily activities across five indoor environments with synchronized gaze data. The latter contains approximately 300 hours of recordings from nearly 50 indoor and outdoor scenes with real gaze traces. While the Nymeria dataset provides rich, timealigned narration text describing major events and interacted objects, the AEA dataset lacks such annotations. To address this gap, we cluster consecutive frames within each video clip and generate action summaries, following the procedure outlined in Step 1 of Figure 3. We cluster consecutive frames within each video clip using k-means on ResNet-50 (He et al., 2016) features and generate action summaries (see Appendix ?? for details). After clustering, the majority of AEA segments span 2–8 seconds. Each segment is then passed to a VLM-based captioning model, which generates a textual summary along with the segment’s start–end timestamps. Concatenating these segment-level summaries yields the Action Summary, as illustrated in Step 1 of Figure 3. Finally, the summaries are manually reviewed to eliminate obvious errors. 3.3
then sample a single object from the Target Frame as the Target Object based on the PS Sθ (·). We parameterize Sθ (·) as a 2D Gaussian distribution in its basic form, expressed as: ∥o − f ∥2 (1) Sθ (·) ∝ exp − 2θ2 Here f denotes the gaze position, and Sθ (·) determines the selection probability at object centroid o. This probability diminishes as the separation ∥o − f ∥ increases, with θ modulating the decline rate. Multiple Target Objects are then drawn from this distribution and forwarded to the question generation pipeline described below. 3.4
Question Generation and Manual Curation
3.4.1
Iterative Question Refinement
The Synthesizer Agent (SA) generates questions based on the Target Object, mimicking natural human inquiry patterns. Given the Target Frame at time t0 , we first sample a questioning timestamp tq ∈ [t0 + ∆, T ], where T is the video end and ∆ is termed recall interval. This randomization avoids trivial overlap with the Target Frame while ensuring diverse temporal coverage. Prior work in cognitive science (Roediger III and Karpicke, 2006; Carpenter and DeLosh, 2005; Zacks et al., 2007) shows that varying the delay between stimulus and questioning can improve memory and comprehension, and that humans flexibly recall events across different temporal spans. Uniform sampling provides a practical and behaviorally plausible baseline for determining when to ask questions. We employ a pretrained VLM as the Synthesizer Agent (SA), fine-tuned to invoke external tools through specific APIs. To reduce computational costs, SA does not process the full video directly. Instead, it interacts with two APIs: GetFrame, which retrieves a high-resolution frame at a specified timestamp for static detail analysis, and GetSegment, which provides a downsampled video clip over a selected time span for verifying dynamic activities. Guided by the system prompt, SA constructs an MCQ about the Target Object, following Step 3 of Figure 3, with multiple sub-steps. In sub-step 1, SA analyzes the Action Summary and uses the Target Frame to locate the Target Object. The timestamp of the Target Frame provides contextual information about the activity in the Action Summary, while the associated bounding box for
Gaze-Oriented Target Sampling
Using the video segments obtained from VSSC, we introduce the GOTS framework, which simulates human attention mechanism described in Section 2.1 (Step 2 in Figure 3) to questions. GOTS begins by detecting all objects within each video segment, leveraging a VLM to extract their bounding boxes and labels. Because adjacent frames are often nearly identical, this step generates many redundant detections of the same object over time, which diminishes the diversity of potential questioning targets. To mitigate this, we incorporate a lightweight re-identification (ReID) stage. Specifically, each detected object is cropped and encoded using the visual encoder of a pretrained CLIP model (Radford et al., 2021). Detected objects with highly similar CLIP embeddings are then consolidated, ensuring only one representative instance of each object is retained across the sequence. Subsequently, we measure the Euclidean distance between each object’s bounding box centroid and the corresponding gaze position on a per-frame basis. Using this information, we first randomly sample a Target Frame from the video segment, and 5
what
oth
er
6%
Others 267 (5 .9%
)
didere w hen w was what
other when how man y how what wohtihcer d h wasid wh at wh en
where
wh at whic h w a whe s othern
ere
36%
l atia l-Sp ) pora 23.0% Tem1032 (
wh
Ap 657p(1ea4rance .6%)
cation 46% Direct Lo1% 276 (6. )
ence Pres %) Item14 (13.7 6 56%
45%
ing Fu part rn s Cl iture oth Sto ing Be rage dd in Ot g h Ele De er ctr cor Ligonics ht Sp T Pl ing Fo orts able ants & od w Ma & B Hob are b j Sm or A evera ies all ppl ges Ap ian pli ces K a Ba itch B nces thr en oo oo Fix ks m tu Su res p Ki Cle plies tch an en ing Ba thr Co Too oo ok ls m wa F r Sta ixtur e tio es ne T ry Pe La ools rso un Liv nal dry ing Ca R re Ou oom tdo or
ild Bu
where
0%
other
Figure 4: Distribution of Target Object Categories. The x-axis represents object categories, the y-axis counts the number of MCQs generated using the target object in a given category. The length of each colored bar segment indicates the number of questions of that type, revealing the reasoning associated with each target object.
Figure 5: The middle ring shows the proportion of each category and the number of MCQs in it. The outer ring shows the most frequent interrogative words in that category. The inner circle shows the average accuracy of all VLM models from Table 1 for each category.
object detection guides SA in identifying the visual features of the Target Object. In sub-step 2, SA first drafts a daily life question about the Target Object, reflecting natural human routines, framed in a natural, human-like style. Then, it identifies the required additional information and iteratively invokes tools and evaluates new evidence until sufficient context is collected to construct an MCQ. The SA then submits the MCQ with supporting evidence to the Validator Agent (VA) for review (sub-steps 3–4). Similar to the SA, the VA has access to the same Action Summary and may also invoke tools to inspect portions of the video during its evaluation. Unlike the SA, the VA does not receive the Target Object or Target Frame. Its task is to verify factual accuracy, identify ambiguities, evaluate question clarity, and provide at least one additional piece of evidence to enhance the credibility of the MCQ (sub-step 5). The VA returns feedback to the SA (sub-step 6), which refines and resubmits the MCQ to the VA. If all checks pass, the VA finalizes the MCQ. MCQs failing after two review rounds are discarded. 3.4.2
S 527tate Ver (11.7 ify %) 5
52%
0
whe n wer whate is did other
55%
s
Sp 730atia (16l-Sp .3%atia ) l4
# Questions
100
wa
erify nt V %) Eve87 (8.6 3
200
did
300
Temporal-Spatial Spatial-Spatial Appearance Item Presence State Verify Event Verify Direct Location Others
Did s wa her ot
400
can describe other
500
generation, producing questions that fail to align with typical AR user query patterns. To mitigate these issues, we first apply rulebased filtering to exclude Target Objects that are unsuitable for questioning (e.g., walls, ceilings, floors, or the camera wearer’s body parts). We further discard MCQs that violate typical AR user questioning patterns, such as those referencing timestamps or explicitly mentioning the word "video." Next, we conduct human review. Annotators are presented with each MCQ together with the corresponding video. Without access to the correct answer, they are asked to select one choice from five options. If minor issues are observed, annotators may refine the MCQ; for major flaws, they mark the item as invalid. After the review process, we retain only those MCQs whose pseudo answers are consistent with the annotators’ selections. Finally, we apply large language model (LLM)based blind filtering: the LLM is prompted to guess the correct answer without access to the video, and we retain only those MCQs it answers incorrectly. This ensures that the retained questions cannot be solved through simple logical reasoning or textual cues alone, thereby preventing the possibility of answering them in LEU without actually referring to the video.
Manual Filtering and Labeling
After generating the question, we obtain highquality MCQs; however, certain failure modes may still lead to low-quality outputs. The most common case arises when the Target Object in the Target Frame is ambiguous due to factors such as distance, occlusion, inadequate lighting, or viewpoint distortion. In such cases, the object detector may misclassify the Target Object as another item. A second failure mode arises when the Agent misinterprets spatial layouts, resulting in view-dependent or incorrect spatial descriptions. Finally, certain Target Objects are inherently unsuitable for MCQ
4
Evaluation
EgoEverything is built on the AEA (Lv et al., 2024) and Nymeria (Ma et al., 2024) datasets (details in Appendix ??), comprising over 5,000 MCQ pairs 6
Table 1: Performance comparison of each method over different VLMs on EgoEverything. "NA" means not available. The human annotators can achieve an average accuracy of 83.5%. Model
FR
AD
GC
GM
VMP
AMEGO
Videollama3-7b (Zhang et al., 2025b) Videollama3-2b Gemini 1.5 pro (Team et al., 2023) LongVA (Zhang et al., 2024a) Llava-Video (Zhang et al., 2025b)
49.1 46.5 63.1 34.6 42.6
46.1 44.9 58.4 31.6 36.6
42.4 40.5 52.7 28.9 32.9
35.2 34.5 37.7 22.2 26.0
20.2 21.3 33.2 NA NA
19.5 19.4 18.3 NA NA
Refrigerator
Table 2: VLM model performance under blind setting. Model Accuracy (%)
Orange Juice
Did I have orange juice in the refrigerator? Options: A. Yes, you had orange juice in your refrigerator. It was on the top shelf, next to the milk. B. No, there was no orange juice in your refrigerator. C. Yes, you had orange juice in your refrigerator. It was on the bottom shelf, behind vegetables. D. Yes, you had orange juice in your refrigerator. It was in the door compartment. VLM E. Yes, you had orange juice in your refrigerator. It was next to a carton of eggs. GT
Pile of Clothes
Gemini 1.5 pro
LongVA
22.9
35.9
21.8
processing baselines: Gaze Crop (GC), crops each frame around the gaze fixation location using a square bounding box covering roughly 10% of the original frame size; Gaze Mask (GM), retains the complementary regions outside GC; Average Downsampling (AD), uniformly downsamples each frame to 10% of its original resolution; and Full Resolution (FR) uses original frames. In addition, we evaluate two recent methods for LEU tasks, AMEGO (Goletto et al., 2024) and VideoMindPalace (VMP) (Huang et al., 2025), which construct egocentric scene graphs to capture key object–user relationships while filtering redundant information, achieving strong performance on standard LEU benchmarks such as EgoSchema (Mangalam et al., 2023) and NExT-QA (Xiao et al., 2021). We apply AMEGO and VMP over Videollama3 and Gemini. Our goal is to examine how these methods perform on this real-life, human attention-driven LEU dataset.
Question:
Black Garment
Videollama3-7b
Preparation for the Suitcase
Question: Did I pack the black garment I was holding before I placed other items into the suitcase? Options: GT A. No, you put the black clothes on the bed. B. No, you put the black garment back on the pile of clothes next to the suitcase. C. No, you left the black garment on the floor. VLM D. Yes, but you took it out again later to examine it more closely. E. No, you packed a different black garment.
Figure 6: Video frames and keywords in the MCQ are highlighted in different colors. Orange indicates the target objects, Blue highlights contextual entities mentioned in the question or answer, and Purple denotes regions where the target objects cannot be located. The GT label refers to the ground truth option, and the VLM label refers to the option selected by the model.
Question Categories We classify questions into eight categories (Item Presence, Appearance, Event/State Verification, Spatial–Spatial, Direct Location, Temporal–Spatial, and Others). Detailed definitions are provided in Appendix ??. Figure 5 shows the distribution across categories.
spanning more than 100 hours of video. Dataset examples are illustrated in Figure 6. In the generation stage, we use a 2D Gaussian distribution as the PS, shown in Equation 1, with the standard deviation θ = 400 pixels and recall interval ∆ = 3 minutes. For annotation, we developed a web-based labeling tool with 12 trained annotators, who collectively labeled about 21,600 questions over 400 hours. The MCQ adoption rate during annotation was approximately 70%, while rule-based and blind filtering yielded acceptance rates of around 50%. We evaluate several recent VLMs on EgoEverything, including Videollama3 (Zhang et al., 2025b), Gemini (Team et al., 2023), LongVA (Zhang et al., 2024a), and Llava-Video (Zhang et al., 2024b). Since our MCQs are sampled using a Gaussianbased PS, content near the gaze location tends to be more relevant for solving MCQs than distant information. To validate this property, we design several pre-
Target Object Category: As described in Section 3.3, our GOTS framework selects Target Objects when generating MCQs. Selected Target Objects are divided into 28 categories and show the distribution in Figure 4, where the y-axis denotes the number of associated MCQs and the stacked colors represent the proportions of different question categories within each Target Object category. 4.1
Accuracy Evaluation on EgoEverything
As shown in Table 1, among the VLMs, under the full-resolution setting where the entire input frame is provided for processing, Gemini achieves the highest accuracy of 63.1% across the MCQs, while the other models perform even lower. Nevertheless, all VLMs remain far behind human performance, as our human annotators reach an average accuracy of 83.5% across 12 participants. This gap highlights the clear performance deficiency of current 7
40 0
200
400 600 o f (pixel)
Dist. to Gaze || (a)
||
55.0
50
52.5
45
Accuracy %
50
Accuracy %
Accuracy %
VLMs on the EgoEverything dataset. All input processing methods, including GC, GM, AD, VMP, and AMEGO, present a degradation in MCQ prediction accuracy compared to FR. Among them, GC performs better than GM because the MCQs are generated according to the PS centered at the gaze fixation location, and this method preserves the most critical information. In contrast, GM suffers a substantial performance drop since it excludes these key details regarding human attention specified by the gaze fixation location. Interestingly, AD outperforms GC by preserving not only the gaze-centered visual content but also peripheral information that remains important for answering MCQs. Finally, both VMP and AMEGO achieve the lowest performance among the evaluated methods. This is because neither approach infers directly from raw video. Instead, they convert the video into structured text via object detection and use LLM for pure text reasoning. Both methods only detect interacted objects and overlook many activity-irrelevant objects in our benchmark. In addition, to test whether the MCQs within EgoEverything can be solved by solely inspecting the textual information from the question, we conduct a text-only evaluation. This setting is undesirable because it does not allow the model to leverage visual information during processing, and thus may encourage reliance on linguistic shortcuts instead of true multimodal reasoning. To achieve this, only the textual input within each data of MCQ is delivered to VLM and the video frames are eliminated. As indicated by Table 2, this will greatly degrade the accuracy for Videollama3-7b (Zhang et al., 2025b), Gemini (Team et al., 2023) and LongVA (Zhang et al., 2024a), confirming that our questions cannot be answered from text alone.
50.0 47.5 45.0 100 500 2k 5k 10k 50k 100k
BBox area (pixel2) (b)
49.0%
40 35 30
33.3% top 10% bottom 10%
Recall Interval tq t0 (c)
Figure 7: Impact of dataset setting on LEU performance.
consistency under human-like diverse questioning times, demonstrating that variation in questioning time directly impacts LEU accuracy and thereby validating our benchmark’s design choice to introduce randomized tq . Impact of Object-Gaze Distance During generation, we sample a target object according to its distance from the gaze point ∥o − f ∥, as defined in Equation 1. This better simulates human attention and produces questions that align with user focus. To examine the impact of object–gaze distance, we report Videollama3-7b’s accuracy on our MCQs grouped by ∥o − f ∥, as shown in Figure 7 (a). The results showed that Target Objects located at the periphery of the visual field produce more challenging MCQs than Target Objects near the center of attention, with accuracy decreasing from 55% to 30% as distance increased. These results indicate that current models struggle with MCQs related to peripheral information. Unlike prior benchmarks, our attention-aware generation produces both challenging peripheral questions and user-focused questions. Impact of Object Size To examine whether VLMs attend to objects of different scales in a comparable manner, we measure bounding box sizes and analyze accuracy as a function of target object area (in pixels2 ). Area thresholds ranging from 1×100 to 1×105 pixels2 were applied. As shown in Figure 7 (b), accuracy consistently increases with larger bounding box thresholds. Across the evaluated models, performance is biased toward larger objects, while smaller and less salient objects are frequently overlooked.
Impact of Questioning Time tq As described in Section 3.4, when generating MCQs we mimic reallife AR scenarios by introducing randomized questioning times tq , whereas in other LEU datasets questioning typically occurs only after a clip or fixed segment has ended. To study how variation in tq impacts accuracy, we select the 10% of MCQs with the largest recall intervals (∆) and the 10% with the smallest intervals, and then measure the accuracy of Videollama3-7B over EgoEverything. As shown in Figure 7 (c), the results reveal that accuracy drops markedly as the recall interval increases, from 49% to 33.3%. This indicates that current models exhibit substantial performance in-
5
Conclusion
We present EgoEverything, a benchmark for longcontext egocentric video understanding that incorporates human attention into question generation. Via attention-guided sampling, multi-agent generation, and layered filtering, the benchmark provides over 5,000 high-quality MCQs grounded in real AR 8
scenarios. Evaluations show that current VLMs struggle on this task, underscoring the need for more efficient and attention-aware LEU datasets.
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. 2024. Hourvideo: 1-hour video-language understanding. Preprint, arXiv:2411.04998.
Limitations Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. 2024. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. Preprint, arXiv:2401.12168.
Our benchmark assumes gaze location as a proxy for user attention, which, while grounded in cognitive science and widely adopted in AR/VR research, does not capture the full multimodal complexity of human attention (e.g., auditory input, cognitive states). The underlying videos are drawn from AEA and Nymeria datasets covering primarily daily activities, lacking specialized domains such as industrial or medical AR scenarios, and all questions are in English. Additionally, our Perception Sampler uses a fixed 2D Gaussian parameterization (θ = 400 pixels) that may not generalize across all tasks and individuals. Future work could incorporate additional modalities, adaptive attention modeling, and extend the benchmark to openended question formats and multilingual settings.
Rajeswari Chengoden, Nancy Victor, Thien Huynh-The, Gokul Yenduri, Rutvij H Jhaveri, Mamoun Alazab, Sweta Bhattacharya, Pawan Hegde, Praveen Kumar Reddy Maddikunta, and Thippa Reddy Gadekallu. 2023. Metaverse for healthcare: a survey on potential applications, challenges and future directions. IEEE Access, 11:12765–12795. Woranipit Chidsin, Yanlei Gu, and Igor Goncharenko. 2021. Ar-based navigation using rgb-d camera and hybrid map. Sustainability, 13(10):5585. Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. 2021. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16372–16382. Matt Deitke and Christopher Clark et al. 2024. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. Preprint, arXiv:2409.17146.
References Meta quest pro.
Robert Desimone, John Duncan, and 1 others. 1995. Neural mechanisms of selective visual attention. Annual review of neuroscience, 18(1):193–222.
Abdullah M Al-Ansi, Mohammed Jaboob, Askar Garad, and Ahmed Al-Ansi. 2023. Analyzing augmented reality (ar) and virtual reality (vr) recent development in education. Social Sciences & Humanities Open, 8(1):100532.
Charles W. Eriksen and James D. St. James. 1986. Visual attention within and around the field of focal attention: A zoom lens model. Perception & Psychophysics, 40(4):225– 240.
Anurag Arnab, Chen Sun, and Cordelia Schmid. 2021. Unified graph structured models for video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8117–8126.
Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. 2024. Videoagent: A memory-augmented multimodal agent for video understanding. In European Conference on Computer Vision, pages 75–92. Springer.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13923.
Jaris Gerup, Camilla B Soerensen, and Peter Dieckmann. 2020. Augmented reality and mixed reality for healthcare education beyond surgery: an integrative review. International journal of medical education, 11:1.
Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. 2018. Object level visual reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 105–121.
Gabriele Goletto, Tushar Nagarajan, Giuseppe Averta, and Dima Damen. 2024. Amego: Active memory from long egocentric videos. In European Conference on Computer Vision, pages 92–110. Springer.
William Brendel and Sinisa Todorovic. 2011. Learning spatiotemporal graphs of human activities. In 2011 International Conference on Computer Vision, pages 778–785. IEEE.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
Shana K Carpenter and Edward L DeLosh. 2005. Application of the testing and spacing effects to name learning. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition, 19(5):619– 636.
Zeyi Huang, Yuyang Ji, Xiaofang Wang, Nikhil Mehta, Tong Xiao, Donghyun Lee, Sigmund Vanvalkenburgh, Shengxin Zha, Bolin Lai, Licheng Yu, and 1 others. 2025. Building a mind palace: Structuring environment-grounded semantic graphs for effective long video analysis with llms. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24169–24179.
Marisa Carrasco. 2011. Visual attention: The past 25 years. Vision research, 51(13):1484–1525.
9
Andreas Ioannides and Vahe Poghosyan. 2010. The early spread of spatial and non-spatial attentional effects in human visual cortex. In Proceedings of the Frontiers in Neuroscience Conference, volume 4, pages –.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR.
Ye-Joon Jo, Jun-Seok Choi, Jin Kim, Hyo-Joon Kim, and Seong-Yong Moon. 2021. Virtual reality (vr) simulation and augmented reality (ar) navigation in orthognathic surgery: a case report. Applied Sciences, 11(12):5673.
Ivan Rodin, Antonino Furnari, Kyle Min, Subarna Tripathi, and Giovanni Maria Farinella. 2024. Action scene graphs for long-form understanding of egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18622–18632.
Michael F Land and Mary Hayhoe. 2001. In what ways do eye movements contribute to everyday activities? Vision research, 41(25-26):3559–3565.
Henry L Roediger III and Jeffrey D Karpicke. 2006. Testenhanced learning: Taking memory tests improves longterm retention. Psychological science, 17(3):249–255.
Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. 2023. Intentqa: Context-aware video intent reasoning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11963–11974.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
Wei-Cheng Lin, Chih-Ming Lien, Chen Lo, and Chia-Hung Yeh. 2025. Gazenlq @ ego4d natural language queries challenge 2025. Preprint, arXiv:2506.05782. Wenxuan Liu, Budmonde Duinkharjav, Qi Sun, and Sai Qian Zhang. 2025. Fovealnet: Advancing ai-driven gaze tracking solutions for efficient foveated rendering in virtual reality. IEEE Transactions on Visualization and Computer Graphics.
Gemini Team and Petko Georgiev et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530. Ying Wang, Yanlai Yang, and Mengye Ren. 2023. Lifelongmemory: Leveraging llms for answering queries in longform egocentric videos. arXiv preprint arXiv:2312.05269.
Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, and 1 others. 2024. Aria everyday activities dataset. arXiv preprint arXiv:2402.13349.
Thomas Westin, José Neves, Peter Mozelius, Carla Sousa, and Lara Mantovan. 2022. Inclusive ar-games for education of deaf children: Challenges and opportunities. In European Conference on Games Based Learning, volume 16, pages 597–604.
Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, and 1 others. 2024. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In European Conference on Computer Vision, pages 445–465. Springer.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828–28857.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212–46244. Meta. 2023a. Meta aria glasses. projectaria.com/glasses/. Meta. 2023b. Meta quest 3. quest/quest-3/.
https://www.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786. Yanlai Yang and Mengye Ren. 2025. Memory storyboard: Leveraging temporal segmentation for streaming selfsupervised learning from egocentric videos. arXiv preprint arXiv:2501.12254.
https://www.meta.com/
Microsoft. 2023. HoloLens 2 Specs.
Yanlai Yang, Zhuokai Zhao, Satya Narayan Shukla, Aashu Singh, Shlok Kumar Mishra, Lizhu Zhang, and Mengye Ren. 2025. Streammem: Query-agnostic kv cache memory for streaming video understanding. arXiv preprint arXiv:2508.15717.
OpenAI and Josh Achiam et al. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, Jacob Chalk, Zhifan Zhu, Rhodri Guerrier, Fahd Abdelazim, Bin Zhu, Davide Moltisanti, Michael Wray, Hazel Doughty, and Dima Damen. 2025. Hd-epic: A highly-detailed egocentric video dataset. Preprint, arXiv:2502.04144.
Alfred L Yarbus. 2013. Eye movements and vision. Springer. Jeffrey M Zacks, Nicole K Speer, Khena M Swallow, Todd S Braver, and Jeremy R Reynolds. 2007. Event perception: a mind-brain perspective. Psychological bulletin, 133(2):273.
Michael Posner. 1980. Orienting of attention. Q J Exp Psychol, 32:3–25.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. 2025a. Videollama 3: Frontier multimodal foundation models for image and video understanding. Preprint, arXiv:2501.13106.
Will Price, Carl Vondrick, and Dima Damen. 2022. Unweavenet: Unweaving activity stories. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13770–13779.
10
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, and 1 others. 2025b. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106. Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024a. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024b. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713. Wenqi Zhou, Kai Cao, Hao Zheng, Xinyi Zheng, Miao Liu, Per Ola Kristensson, Walterio Mayol-Cuevas, Fan Zhang, Weizhe Lin, and Junxiao Shen. 2025. X-lebench: A benchmark for extremely long egocentric video understanding. arXiv preprint arXiv:2501.06835.
11