EventHOI: Event-Driven Resource-Efficient Hand-Object Interaction Logging on Smartglasses

EventHOI: Event-Driven Resource-Efficient Hand-Object Interaction Logging on Smartglasses Le Zhang1 , Kailai Cui1 , Hao Chen2 , Vlad Roznyatovskiy2 , Jianzhong Zhang2 , Anhong Guo1 , Kang G. Shin1 , Ke Sun1 1 University of Michigan

2 Samsung Research America

zle@umich.edu

HOI Understanding

arXiv:2611.09089v1 [cs.DC] 6 Nov 2026

Abstract Hand-object interactions (HOIs) can be captured by smartglasses to learn how people manipulate objects in daily life. Automatic logging of these interactions from an egocentric perspective provides time-stamped evidence of daily actions for applications like memory assistance. However, continuous RGB capture and wireless transmission exceed the energy and bandwidth budgets of smartglasses. We present EventHOI, an event-driven sensing and compression system that couples selective RGB capture with HOIaware encoding to enable egocentric logging over Bluetooth Low Energy (BLE) while preserving downstream HOI logging performance. Our key insight is to use low-power event sensing both to determine when RGB capture is needed and to identify where visual detail must be preserved. A lightweight IMU-based egomotion cancellation pipeline suppresses egomotion-induced events so that residual HOI motion can guide RGB triggering and spatial compression. We build a smartglasses prototype and conduct an end-to-end evaluation of EventHOI on a 50-hour egocentric benchmark with 97 subjects and a 20-hour real-world dataset we collect from 10 participants, together covering a broad range of daily-life scenarios and environments. Compared to RGB-only baselines, EventHOI reduces always-on backend power consumption by 5.8–10.4× and uplink bandwidth by 2.9×, with minimal HOI recognition accuracy loss. The resulting RGB uplink of less than 130 kbps fits within commodity BLE bandwidth budgets, making stable BLE video streaming feasible for egocentric HOI logging. Demo video and project website: https://eventhoi.github.io.

1

Introduction

Hand–Object Interaction (HOI) is an atomic unit of everyday activities. Logging HOIs provides time-stamped evidence of daily actions, i.e., what objects the hands interacted with, what actions occurred, and when, enabling immediate assistance in daily life. This capability is important because everyday memory slips are both common and consequential: diary-based studies show that over 60% of reported memory failures involve forgetting intended actions [1]. As a result, HOI logging can naturally support reminders (e.g., “Did I lock the door?”), verification (e.g., “Did I take my medication?”), and rapid retrieval (e.g., “Where did I leave the charger?”), which correspond to frequent types of day-to-day lapses. Beyond memory augmentation, HOI logs can also strengthen health-routine summarization: the World Health Organization (WHO) reported that adherence to long-term therapies for chronic diseases averages only ∼50% in developed countries, motivating systems that can passively verify routine actions without relying on self-report [2]. Recent camera-equipped smartglasses have increased interest in egocentric HOI logging by providing a wearable, always-available viewpoint that naturally captures the wearer’s hands and the objects

(2) HOI Encode

BLE

Reduce 2.9x BW

Low-power Event Sensing EVS

Monochrome

EventHOI

Uniform Sampling & WiFi Streaming High-power RGB Sensing

(1) Detect & Trigger

Save 10.4x Power

RGB

Baselines

Figure 1: EventHOI uses low-power event sensing to (1) detect and selectively trigger the high-power RGB camera and (2) compress RGB uplink for HOI understanding. they manipulate. This trend is reflected in the rapid growth of largescale egocentric HOI datasets and advances in computer vision (CV) algorithms [3–5]. Despite this momentum, a key bottleneck remains: prior work often implicitly assumes away the on-device system cost of continuous sensing on smartglasses. In practice, egocentric HOI logging demands (1) continuous RGB recording and (2) sustained video uplink to edge/cloud resources for CV processing. For continuous RGB recording, Ray-Ban Meta glasses cap each video clip at up to 3 minutes, due in part to overheating during sustained use [6]. Even research prototypes face tight energy budgets: Meta Aria Gen 1 uses a 2.5 Wh battery and supports only ∼1.5 hours of continuous local recording [7]. Since larger batteries compromise ergonomics and user comfort of wearable devices, energy-efficient RGB sensing is essential to reduce power consumption and enable all-day recording. Prior research such as modality-driven RGB triggering using thermal cameras or microphones incurs high energy overhead or lacks robustness in noisy environments [8, 9], while event camera only solutions focus on close-set gesture classification [10], making real-world application and deployment infeasible. Additionally, smartglasses rely on wireless connectivity to offload RGB video for cloud processing. Wi-Fi video offloading is too energy-intensive for all-day operation on smartglasses [11]. BLE offers a lower-power alternative, but its throughput can be as low as 0.3 Mbps, below the bitrate required by our RGB-streaming baseline to preserve HOI recognition accuracy (§3.3) [12]. Traffic from microphones, IMUs, and other sensing modalities further reduces the available bandwidth. Recent semantic video compression approaches can reduce bitrate substantially, but they typically rely on heavy on-device computation, such as learning-based saliency estimation, making them difficult to deploy on resource-constrained smartglasses [13–15]. These techniques are also rarely evaluated on egocentric streams, where camera motion and scene dynamics change continuously with the wearer’s actions. In this paper, we design EventHOI to reduce the on-device system cost of egocentric HOI logging on smartglasses on sensing,

Zhang et al.

computation, and communication. Our key idea is to design an event-driven low-power sentinel, using low-power event sensing to detect HOI-induced motion and selectively control high-power RGB camera recording and compression. As shown in Fig. 1, we envision future smartglasses equipped with a low-power modality, such as an event camera or a monochrome camera, running continuously to produce event frames. A lightweight on-device pipeline processes these event frames as an always-on HOI detector that can (i) localize HOI timestamps and (ii) estimate an HOI saliency region. These cues then steer the RGB camera to record only when an HOI is detected and enable saliency-guided spatial compression, so that the compressed RGB stream can be uploaded directly to an edge/cloud device over commodity Bluetooth Low Energy (BLE), without requiring network-layer modifications. The uploaded RGB clips are then analyzed by powerful CV models, e.g., vision-language models (VLMs), on the edge/cloud to produce accurate HOI logs. EventHOI must address the following three key challenges. Challenge 1: How can we design a lightweight algorithm to cancel egomotion in event frames? Egocentric event streams are often dominated by wearer-induced motion (e.g., head turns and walking), which can easily overwhelm the subtler dynamics introduced by HOIs. To make always-on HOI detection practical, we leverage the on-board 6-axis IMU available on most smartglasses to drive a two-stage global event cancellation pipeline. First, a lightweight model estimates instantaneous egomotion speed from IMU signals, avoiding drift from long-term integration, and filters out large translational motion such as walking. Second, EventHOI uses gyroscope measurements to estimate the rotation-induced head-motion field and subtract the corresponding global events from the event frame, mitigating interference from head rotations. Challenge 2: How can we reliably detect HOIs and estimate an HOI saliency region from the event frames for selective RGB triggering and spatial compression? Even after egomotion cancellation, HOI signatures in event frames remain sparse, short-lived, and diverse across actions, objects, and environments. Residual events may also include non-HOI motion, such as moving bystanders or background changes, causing spurious triggers. To address this, EventHOI designs an event-driven HOI detector that exploits a simple egocentric prior: during HOIs, the wearer’s hands and forearms occupy a sufficiently large and consistently active region in the field of view. Building on this, we model the spatiotemporal structure of residual events and aggregate evidence over a short temporal window to localize HOI timestamps robustly. Given a detected HOI window, we estimate an HOI saliency region by identifying consistently activated event clusters and refining them into a compact, temporally stable region of interest (RoI), enabling reliable saliency-guided spatial compression. Challenge 3: How can we design a minimal video uplink pipeline that supports VLM-based egocentric HOI logging? This challenge arises because SoTA egocentric HOI understanding increasingly relies on VLMs [16]. However, existing VLM-based pipelines typically assume access to continuous RGB video. In contrast, EventHOI produces disconnected clips that are both spatially cropped (RoIcentric) and temporally sparse, to reduce on-device sensing and communication costs. The key question is how to preserve HOI logging accuracy under this constrained uplink without fine-tuning

the VLM, while minimizing bandwidth. To answer this, we systematically benchmark VLM-based HOI logging across frame rate, resolution, compression level, and temporal continuity, quantifying how each factor affects recognition accuracy. Guided by these results, we design an HOI-aware video encoder that selects the minimal set of RoI content, at the lowest feasible resolution and compression rate, that still retains the essential visual cues needed by off-the-shelf VLMs, to achieve performance comparable to that of streaming continuous high-frame-rate, high-resolution video. We build a smartglasses prototype with a low-power event or monochrome camera, an RGB camera, and an IMU, and implement the EventHOI pipeline on a Raspberry Pi Zero with commodity BLE uplink. We first evaluate EventHOI on the 50-hour Nymeria egocentric RGB benchmark [3] with 97 subjects across diverse daily-life scenarios, simulating event frames from RGB frames. We assess HOI recognition and localization accuracy, average RGB frame rate, and uplink bandwidth requirements. We then collect a 20-hour real-world dataset from 10 participants under IRB approval using our EventHOI prototype, comprising synchronized RGB, event camera, and IMU recordings. On this real-world testbed, we evaluate the same metrics and additionally measure system energy consumption and end-to-end latency. Compared with continuous RGB recording and on-device YOLO-based hand detection at 1 FPS, EventHOI reduces always-on backend power consumption by 5.8× and 10.4×, respectively, with minimal HOI recognition accuracy loss across downstream tasks and VLM models. EventHOI reduces uplink bandwidth by 2.9× to below 130 kbps, making stable BLE video streaming feasible for egocentric HOI logging. EventHOI incurs negligible accuracy loss compared to uniform sampling across daily activities from HOI-intensive cooking to HOI-sparse leisure, and across motions from subtle HOIs such as pouring water to large HOIs such as sweeping the floor. We summarize our contributions as follows: • We present EventHOI, the first event-driven smartglass system for resource-efficient egocentric HOI logging, using an always-on event or low-cost monochrome camera and IMU. • We design a lightweight on-device pipeline to use event frames for egomotion cancellation, HOI detection, and event-guided RGB triggering and uplink compression. • We implement EventHOI on resource-constrained embedded hardware with BLE uplink and evaluate it through prototyping, multiuser multi-scenario real-world testbed, and large-scale dataset benchmarks. Demos and examples are at https://eventhoi.github.io. • Comprehensive experiments show that EventHOI reduces power and minimizes uplink bandwidth while maintaining HOI understanding accuracy.

2

Background and Related Work

Egocentric HOI Understanding. Egocentric HOI understanding uses head-mounted cameras to recognize how users manipulate and interact with objects. It is important for modeling user behavior and enabling assistive systems to understand daily activities and intent. Despite recent progress, this task remains challenging. First, it requires reasoning about fine-grained hand-object relationships, subtle interaction cues, and diverse open-set objects, which cannot be captured by conventional closed-set detection alone [17, 18]. Frequent head motion and the limited field of view further complicate

Uniform

Thermal THOR [9]

High-power Camera

EventHOI

10 0

Monochrome NeuriCam [25]

EVS

Low-power Camera

Mic

EgoTrigger [8]

IMU 10 1

10 2 10 3 Log Power (mW)

10 4

Workload

Power Battery (W) Life (hrs)

Use Cases and Examples

Off Off 0.1 7.5 No video capability, no stream, on-device only BLE Off 0.15 (1.5× ) 5.0 (67%) Low-resource modality stream (image, audio, etc.) Off RGB 0.7 (7× ) 1.1 (15%) No real-time video stream WiFi RGB 1.1 (11× ) 0.7 (9%) Video stream (Ray-Ban Meta [26], Project Aria [27])

robust perception in egocentric videos [16, 19]. Second, SoTA HOI understanding often relies on computationally intensive models, such as ResNet-based backbones, transformers, and VLMs [5, 20– 24]. These models are difficult to deploy on resource-constrained wearables and often require transmitting data to the cloud, incurring substantial computation and communication overhead. In contrast, EventHOI addresses egocentric HOI understanding from a systems perspective by reducing sensing, computation, and communication costs on wearable devices. Event-based Computer Vision. Event cameras, also known as event vision sensors (EVS) [28, 29], are low-power bio-inspired sensors that asynchronously record changes in pixel intensity rather than capturing full image frames at fixed rates. Because they naturally respond to motion, they have been widely used in motioncentric tasks such as image deblurring, video reconstruction, drone perception, and autonomous driving [30–34]. More recently, they have also been explored in wearable systems for gesture recognition, object detection, and OCR [10, 35, 36]. Most prior systems rely on event frames, which aggregate asynchronous events into frame-like representations. However, existing work mainly focuses on spatial perception, using events to recover appearance or semantic cues. Their temporal dynamics remain underexplored for continuous reasoning tasks such as egocentric HOI understanding. EventHOI instead exploits the motion sensitivity of event frames to trigger the RGB camera selectively and enable lightweight HOI detection and HOI-aware video compression. Semantic Video Compression. Semantic video compression reduces bandwidth by prioritizing task-relevant content, such as salient objects, important regions, or meaningful events [13]. Existing approaches typically rely on additional saliency estimation modules, such as caches or neural networks, which introduce nontrivial computation, memory, and energy overhead [13, 14, 37, 38]. This overhead is particularly undesirable on wearable devices. Recent multimodal compression systems further improve efficiency by combining heterogeneous sensors (Fig. 2). THOR [9] uses an always-on thermal camera to detect the presence of hands but still incurs high energy consumption (∼150mW) [39]. EgoTrigger [8]

(a) Frame Rate

79.1

80

74.7

75 70 65 60

77.2

Qwen3

Qwen3.5

78.7

Gemma-4

70.6

Uniform YOLO

0.25 0.5 1 Frame Rate (FPS)

Non-visual Sensor

Figure 2: Comparisons of sensors in baseline methods used to reduce RGB camera usage. Table 1: Power consumption and expected battery life. Red indicates comparisons to the first-row idle state. Radio Sensor

Recognition Acc (%)

Less to More Informative Modality

RGB

2

Recognition Acc (%)

EventHOI : Event-Driven Resource-Efficient Hand-Object Interaction Logging on Smartglasses

(b) Bandwidth

80

75 73.2 70 65 60

Qwen3

75.9 75.6

77.2

77.0

Qwen3.5

Gemma-4

70.1 Target Bandwidth

JPEG Hands23

0.1 0.15 0.25 0.35 Bandwidth (Mbps)

0.5

Figure 3: Preliminary study on VLM-based HOI understanding for frame rate and uplink bandwidth. applies sound cues as indirect non-visual evidence to trigger RGB samples but is prone to failure in noisy environments. The closeset prediction model is also difficult to generalize to the diverse real-world acoustic environments. NeuriCam [25] continuously captures a low-resolution grayscale stream and periodically acquires high-resolution color key frames, from which it reconstructs high-resolution color video at the receiver. However, such designs still depend on periodic RGB capture and therefore cannot fundamentally reduce sensing cost. In contrast, EventHOI uses event motion derived from an ultra-low-power IMU and an ultra-lowpower event camera (<10mW for each) to trigger RGB capture selectively and guide lightweight HOI-aware encoding, reducing sensing and uplink overhead.

3 Motivation and Preliminary Study 3.1 HOI Logging Pipeline and Constraints Most wearable egocentric HOI logging systems follow the same high-level pipeline. First, a head-mounted RGB camera continuously captures egocentric video. Second, the wearable device processes and compresses the frames. Third, the compressed stream is transmitted to a smartphone or edge/cloud backend. Finally, a backend CV-based model performs HOI understanding and converts the results into structured activity logs. While backend inference can be offloaded, the front-end stages, i.e., RGB sensing and wireless transmission, must run continuously on the smartglasses and therefore dominate their energy and thermal budget. We therefore focus on two coupled bottlenecks: (1) Camera sensing/recording energy. Modern smartglasses such as Ray-Ban Meta [26] and Project Aria [27] support high-quality video capture, but continuous RGB sensing remains power-hungry. Commodity RGB sensors typically consume 200–600 mW depending on resolution, frame rate, and encoding settings [40, 41]. Given the limited battery capacity of smartglasses (e.g., ∼1–2.5 Wh), continuous RGB recording only supports short sessions (tens of minutes to ∼1.5 hours) rather than day-long operation [26, 27]. (2) Wireless uplink energy and bandwidth. Many smartglasses stream video at 720p and 15 FPS [7, 26] by default, which typically requires 2–5 Mbps depending on compression quality. Such bitrates often require Wi-Fi rather than low-power radios such as BLE, adding substantial communication energy overhead. It makes continuous video offloading impractical under the battery budget of smartglasses. These tradeoffs motivate our preliminary studies.

3.2

Study 1: On-device Power Breakdown

Existing smartglasses prototypes do not expose fine-grained power traces. To understand which front-end components dominate the energy budget, we conduct an on-device power study using an

Zhang et al. Egomotioninduced events HOI-induced events

(a) RGB frame

(b) Event voxel

(c) Event frame

(d) Event frame (grayscale)

Light and reflection

Bystander movement

(e) Background distractor after cancellation

Figure 4: (b) Asynchronous event voxels are aggregated to (c) an event frame (red for positive and blue for negative events). (d) Egomotion-induced and HOI-induced events. (e) Background lighting and bystander distractors after egomotion cancellation. ESP32-S3 with an OV5640 camera [42], representative of community smartglasses prototypes such as OpenGlass [43]. Specifically, we use an nRF-PPK2 [44] to measure power under four configurations: (i) idle with no workload, (ii) BLE workload on an nRF52840 at 0.8 Mbps using 1M PHY for comparison [42], (iii) RGB camera active at 720p and 1 FPS, and (iv) Wi-Fi enabled to offload the video stream at 0.8 Mbps. Tab. 1 reports the measured power breakdown and the estimated battery life for a 200 mAh LiPo battery. Finding (1): Continuous RGB recording is the dominant on-device energy bottleneck. Enabling continuous RGB recording increases power by ∼0.6 W. Finding (2): Wi-Fi uplink adds non-negligible overhead. Streaming video over Wi-Fi further adds about 0.4 W, reducing estimated battery life to only ∼0.7 h. These results suggest that day-long HOI logging on smartglasses requires (i) reducing RGB camera energy via selective capture; and (ii) lowering bandwidth enough to use low-power radios (e.g., BLE) instead of continuous Wi-Fi streaming.

3.3

can be compressed to about 0.35 Mbps with little accuracy loss, whereas stronger compression substantially degrades performance. In contrast, our empirical measurements of smartglasses-tosmartphone BLE uplink show that the available bandwidth can be as low as 0.3 Mbps on iPhones. Concurrent transmission of other multimodal signals may further constrain the bandwidth available for RGB streaming: for example, empirical measurements on RayBan Meta Gen 1 glasses indicate an AAC-encoded audio bitrate of 0.12–0.13 Mbps per channel. These observations motivate HOI-aware RGB compression methods that minimize uplink bandwidth while preserving comparable HOI recognition accuracy.

4

System Design

Our key insights lead to a concrete systems question: how can smartglasses determine when an HOI is likely and where it occurs, without paying the cost of continuous RGB sensing and Wi-Fi uplink? EventHOI answers this question with an always-on, low-power visual sentinel that uses event frames to gate RGB capture and guide compression.

Study 2: Frame Rate and Uplink Bandwidth

Study 1 shows that continuous RGB recording and Wi-Fi streaming are prohibitively expensive for day-long operation. A natural alternative is to trigger the RGB capture selectively and transmit a much smaller visual stream over low-power links such as BLE. However, HOI logging is task-driven: excessive reduction may remove the fine-grained hand-object cues and critical temporal transitions required by the backend model. We therefore study the minimum visual stream required to preserve HOI understanding performance. Starting from a dense-video baseline (720p, 2 FPS), we reduce the stream along three axes: (i) frame rate reduction (from 2 FPS to 0.25 FPS), (ii) compression strength, resulting in different uplink bitrates, and (iii) content-aware selection. Because our frame rate is very low, we compress each frame independently using JPEG rather than video codecs that rely on inter-frame prediction. For content selection, we consider an oracle that uses CV-based hand detection to keep only frames likely containing HOIs, using YOLO [45] and Hands23 [46] as described in §6.1. For each configuration, we evaluate HOI logging using the off-the-shelf backend VLM Qwen3VL-8B [47], which provides strong HOI understanding performance and is used primarily for evaluations in this paper, Qwen3.5-9B [48], and Gemma-4-12B-it [49]. We report HOI recognition accuracy (see §6.1) on a 10-hour subset of the Nymeria dataset [3]. Fig. 3 shows two key observations. (i) Content-aware selection is highly effective. Careful content-aware selection preserves similar HOI recognition accuracy while significantly reducing frame rate and bandwidth. (ii) Moderate temporal/spatial reduction is tolerable. Performance remains largely stable at 1 FPS, but drops sharply beyond that point on all tested models. At 1 FPS and 720p, the stream

4.1

Low-Power Event Frame Capture

As shown in Fig. 4, an event frame is a pixel-wise summary of recent visual changes, constructed by aggregating asynchronous intensitychange events over a short time window Δ𝑡 [50]. It highlights where and how strongly motion and scene dynamics occur while largely suppressing static background content. This makes event frames well suited for HOI gating: static scenes produce sparse signals, whereas HOIs generate localized events around motion boundaries. A straightforward way is to use low-power vision sensors to produce event frames, including event cameras and monochrome cameras. An event camera aggregates asynchronous events over a short window to generate an event frame [51]. When event cameras are not available, a monochrome camera can be a practical fallback, since recent monochrome sensors support an “event-like” mode that subtracts consecutive frames with similarly low energy cost [52]. Why is event-frame capture is more energy-efficient than RGB capture? Event cameras typically consume less than 10 mW, over 60× lower than OV5640 RGB cameras [51]. This stems from their change-driven readout: the sensor reports only per-pixel brightness changes rather than full-frame images, so readout and downstream processing scale with scene dynamics rather than fixed frame rate and resolution. Monochrome sensors still perform frame readout, but avoid the costly RGB ISP pipeline, such as demosaicing and color correction. In “event-like” mode, they can operate at around 35 mW, achieving roughly 17× lower power than OV5640 RGB cameras [52]. Since EventHOI only needs low-resolution event frames for gating and localization, the always-on sensing cost remains low.

EventHOI : Event-Driven Resource-Efficient Hand-Object Interaction Logging on Smartglasses

1FPS Trigger

RGB camera

§4.4 HOI motion detector

Egomotion fields u

Residual Flow

Spatial aggregate

§4.3 Egomotion cancellation

HOI encoder

Event Frame H Global optical flow f

HOI detector

Integrate

EVS OR Mono Camera

Distractor filter

Body motion detector

IMU

Low-FPS Fallback

Compress

§4.4 HOI-aware video encoder

Figure 5: EventHOI system overview.

4.2

Challenges and End-to-End Overview

An event-frame sentinel is necessary, but not sufficient. On headmounted devices, non-HOI motion can dominate the event stream and cause frequent false triggers. Fig. 5 summarizes how EventHOI addresses the following challenges. C1: Egomotion-induced background events. Head motion during walking or gaze shifts produces strong global changes across the field of view. As shown in Fig. 4(d), these events can overwhelm HOI cues and keep the RGB camera unnecessarily active. EventHOI uses the head-mounted IMU to estimate egomotion and cancel the corresponding global events in the event-frame stream (§4.3). C2: Non-HOI dynamic distractors. Motion from bystanders, background, and moving background objects (Fig. 4(e)) can also generate concentrated events, causing additional false triggers. EventHOI applies a lightweight HOI motion detector to identify localized residual motion patterns and suppress false triggers from bystanders and background dynamics (§4.4). C3: Bandwidth-efficient uplink without losing HOI cues. Following selective RGB triggering, we aim to compress the captured frames to minimize BLE uplink traffic while preserving fine-grained HOI details required by downstream visual models. This enables sustained RGB streaming over BLE alongside data from other sensing modalities. EventHOI uses an HOI-aware video encoder for saliencyguided spatial compression, reducing uplink bandwidth while preserving end-to-end HOI understanding performance (§4.4).

4.3

Egomotion Cancellation

Egomotion denotes the camera motion relative to the surrounding environment [53]. On head-mounted smartglasses, natural head and body movements can generate substantial background events, which interfere with HOI detection and may unnecessarily keep the RGB camera active. As shown in Fig. 4(d), motions such as walking and head turns can produce dense non-HOI events that dominate the event frame. Thus, the goal of this section is to design a lightweight, on-device method to (1) estimate egomotion and (2) cancel the events induced by egomotion from the event frame. 4.3.1 Egomotion Estimation. Accurate egomotion estimation is challenging on resource-constrained, battery-powered devices. A head-mounted IMU available on most smartglasses offers a lowpower option (<10 mW) and has been widely used for human motion estimation [54, 55]. In principle, linear displacement and angular

Task 2: HOI Localization When I opened fridge last time?

Task 1: HOI Recognition Pick up the milk bottle

BLE Uplink (<0.3Mbps)

VLM

LLM

Retrieval Store

Memory Storage

§6–7 Downstream evaluation

rotation can be obtained by integrating linear acceleration and angular velocity, respectively. However, integration error is the key challenge: displacement estimation requires double integration of acceleration, which amplifies sensor noise and bias and accumulates into large position drift over time [56, 57]. Existing systems typically address this issue either by fusing camera and IMU signals, as in VIO and SLAM [55, 58], or by using periodic zero-velocity references (ZUPT) [56, 57]. However, camera fusion requires pixellevel matching on high-FPS video (e.g., 20–30 FPS) [7, 58], adding sensing and compute overhead that conflicts with EventHOI’s lowpower goals. On the other hand, ZUPT works well for foot-mounted IMUs with stance phases, but is unreliable for head-mounted IMUs that rarely become stationary [56, 57]. Our key idea is to leverage prior knowledge about when informative HOIs are most likely to occur to guide egomotion estimation. We observe that meaningful HOIs are rarely associated with large translational motion of the user, which often reflects whole-body movement rather than interaction-relevant behavior. For example, activities such as walking can generate substantial global motion but seldom correspond to informative HOIs. By contrast, HOIs typically occur when the body is relatively stable, as successful object interaction requires coordinated eye, head, and hand movements [59]. Based on this observation, we decompose egomotion into two categories: (1) translational body motion, such as walking, which is often accompanied by few HOIs, and (2) rotational head motion, such as small head turns while standing or sitting, which commonly co-occurs with HOIs. Step 1: Translational Body Motion Detector. We first identify periods of large translational body motion, during which informative HOIs are unlikely [59]. Based on our empirical study, we define such motion as linear velocity above 0.5 m/s, since head motion alone rarely exceeds this threshold. Rather than directly integrating IMU signals, we train a lightweight 2-layer LSTM binary detector that takes 6-axis IMU data sampled at 50 Hz over a 0.5-second window as input. We train the model using the large-scale egocentric Nymeria dataset, collected with Meta Aria smartglasses [3, 7]. Ground-truth labels are derived from SLAM-estimated egomotion velocities, with windows above 0.5 m/s labeled positive. To maintain class balance, we use a 5-hour indoor subset for training. The detector runs with a default duty cycle of 0.5. When large translation is detected, EventHOI directly disables RGB recording. We adopt binary detection rather than precise velocity estimation because exact translation is unnecessary for HOI triggering and too costly

Zhang et al.

lds 0 n fie motio o r e Z

R

Top-K, absolute filters: flow too small Connect component: flow isolated

Egom otion field su

Keep for valid HOI regions

T'

T

Figure 6: Frame-to-frame egomotion fields 𝑢 (red arrows) derived from the rotation matrix 𝑅. for on-device processing. Our model contains only 37k parameters (~37 KB of int8 weights) and requires ~1.8 MFLOPs per 0.5-second window, making it practical for low-power MCU deployment. Despite this, our model shows a high prediction accuracy of 95.7% and a recall of 0.96 for no-translation-period recognition, which minimizes the likelihood of skipping HOIs. Step 2: Rotational Head Motion Estimation using Gyroscope. After gating out large translational body motion, the remaining egomotion is primarily head-rotation-dominated [59]. We estimate this component directly from gyroscope measurements. Specifically, we integrate the 3-axis angular velocity between two consecutive event-frame timestamps 𝑇 and 𝑇 ′ to obtain a frame-to-frame rotation matrix R, as illustrated in Fig. 6. We then project R into a 2D egomotion field u, which models the per-pixel displacement induced solely by rotational egomotion. Formally, given two consecutive event frames 𝐻𝑇 and 𝐻𝑇 ′ and interleaving gyroscope samples {𝝎 𝑡1 , 𝝎 𝑡2 , · · · , 𝝎 𝑡𝑛 }, where 𝑇 ≤ 𝑡 1 < · · · < 𝑡𝑛 < 𝑇 ′ , we compute the frame-to-frame motion field u as follows. Step 2.1: Gyroscope Reprojection. Raw gyroscope signals sampled at 50 Hz are first low-pass filtered with a 10 Hz cutoff to suppress high-frequency IMU noise. The filtered angular velocities are then re-projected into the camera coordinate frame using the extrinsic calibration matrix E provided by the manufacturer as 𝝎 𝑟𝑡 = E𝝎 𝑡 . Step 2.2: Angular Velocity Integration. Each re-projected angular velocity 𝝎 𝑟𝑡 is integrated over the IMU sampling interval Δ𝑡 to obtain an incremental rotation vector 𝜽 𝑡 = 𝝎 𝑟𝑡 Δ𝑡. The incremental rotation is converted to a quaternion via the exponential map 𝑞𝑡 = i h ∥𝜽 𝑡 ∥
∥𝜽 𝑡 ∥
𝜽𝑡 , cos sin . 2 2 ∥𝜽 𝑡 ∥ Step 2.3: Rotation Integration. Then, we compose the incremental quaternions to obtain the frame-to-frame rotation: → Ö 𝑄𝑇𝑇 ′ = 𝑞 𝑡 = 𝑞 𝑡 1 ⊗ 𝑞 𝑡 2 ⊗ · · · ⊗ 𝑞 𝑡𝑛 . 𝑇 ≤𝑡𝑖 <𝑇 ′

The resulting quaternion 𝑄𝑇𝑇 ′ is finally mapped to the frame-toframe rotation matrix R𝑇𝑇 ′ . Step 2.4: Motion Field Generation. Given R𝑇𝑇 ′ , we compute the corresponding 2D rotational motion field. For each pixel p = (𝑥, 𝑦) ⊤ with homogeneous coordinate p̃ = (𝑥, 𝑦, 1) ⊤ , the rotation-induced pixel location p′ is obtained via the homography p̃′ ∼ Hp̃, H = KR𝑇𝑇 ′ K−1 , where K denotes the camera intrinsic matrix. After dehomogenization from p̃′ to p′ , the motion field at pixel p is u(p) = p′ − p, which represents the pixel displacement induced solely by rotational egomotion, as shown by the red arrows in Fig. 6. 4.3.2 Egomotion Cancellation. Next, we use the egomotion field u(p) to suppress rotation-induced events in the raw event frame. Step 3: Optical Flow Estimation. Event frames encode per-pixel intensity changes but not explicit pixel displacements over time.

Figure 7: Example of distractor filters to residual optical flow. Boxes in the left figure indicate the verdicts of filters. We therefore estimate global motion between consecutive event frames using optical flow. Since this runs on resource-constrained smartglasses, it must remain lightweight. We adopt the classical lightweight Lucas–Kanade (LK) sparse optical flow method [60]. For each tracked pixel p, LK estimates the optical flow vector f (p) by minimizing the photometric error over a local window Ω(p): 2 ∑︁  f (p) = arg min 𝐼 (q + v, 𝑇 ′ ) − 𝐼 (q, 𝑇 ) v

q∈Ω (p)

where 𝐼 (q,𝑇 ) denotes the event-frame intensity at pixel q and time 𝑇 , and 𝑇 ′ is the timestamp of the next frame. We choose the sparse LK algorithm for two reasons. First, event frames are highly sparse, with only 5–10% of pixels active, making sparse flow estimation more suitable than dense methods [61–64]. Second, we find that aggressively downsampled event frames still yield reliable global motion estimates, enabling sparse point tracking and substantially reducing computation. We empirically choose to track fewer than 500 points, which incurs minimal loss in motion accuracy. Step 4: Egomotion Cancellation. Finally, for each pixel p, we combine the gyroscope-derived egomotion direction u(p) with the measured optical flow vector f (p). Since u(p) encodes only the direction of rotation-induced displacement, we suppress egomotion by removing from f (p) its component along u(p): fres (p) = f (p) − (f (p) ⊤ u(p))u(p). The resulting residual flow fres (p) suppresses motion consistent with egomotion while preserving unexplained motion, which in egocentric settings is largely associated with HOIs. The residual optical flow fres (p) is used for the next step.

4.4

HOI Motion Detector & HOI-Aware Encoder

After egomotion cancellation, the residual optical flow contains not only HOI-related motion but also non-HOI dynamic distractors such as background dynamics and bystander motion. We aim to denoise residual flows into HOI saliency maps and use them as a lightweight HOI detector to trigger sampling of RGB frames. Step 5: Spatial HOI Saliency Map Generation. We derive the HOI saliency map from fres (p) based on two observations from egocentric videos. First, hand-induced motion usually has larger flow magnitude than background noise because hands are closer to the head-mounted camera. Second, HOIs typically occupy a non-trivial image region due to the physical size of the hands and the typical hand-camera distance (e.g., 20–80 cm). Guided by these observations, we apply two denoising steps to suppress clutter and spurious motion. First, we remove small residual flows with a magnitude-based filter: a pixel is retained only if its residualflow magnitude exceeds an absolute threshold 𝛼 and falls within the frame-adaptive top-𝑘 percentile. Second, we apply a spatial connectivity filter to remove tiny isolated clusters. Specifically, we compute connected components on the binary mask and suppress components whose area is smaller than 𝐶% of the image. We refer to

EventHOI : Event-Driven Resource-Efficient Hand-Object Interaction Logging on Smartglasses RGB Camera 3D-Printed frame (IMX708)

𝐶 as the HOI-to-noise threshold. The resulting binary HOI saliency map 𝑠 at pixel p is   1,    𝑠 (p) =    0, 

if ∥fres (p)∥ 2 ≥ 𝛼, ∥fres (p)∥ 2 ∈ top-𝑘, and p belongs to a connected component ≥ 𝐶, otherwise.

Unless otherwise specified, we set 𝛼 = 0.3 and 𝑘 = 80%, which empirically provide a reasonable separation between meaningful HOI regions and background noise. We illustrate an example of this process in Fig. 7. We set 𝐶 = 5% by default, as determined by the ablation study in §6.2.7. Step 6: Temporal HOI Detection. We further exploit the temporal pattern of the HOI saliency map to avoid triggering RGB capture for visually uninformative periods. In particular, stationary and lowmotion HOIs, such as holding, resting, and pouring, often exhibit little motion and therefore contribute limited new visual information. Thus, we design an HOI detector to only detect non-stationary HOI occurrences to achieve more efficient RGB frame sampling for HOI understanding. Based on the denoised HOI saliency map 𝑠, we adopt a hysteresis-based sampling strategy to detect non-stationary hand motions as meaningful HOIs. Specifically, we monitor whether 𝑠 (p) = 1 for any pixel p within the past 𝑇 = 5 seconds to enable RGB capture. Our strategy enables efficient temporal-domain frame selection by mitigating the impact of momentary noise or brief motion dropouts. This prevents frequent on–off toggling of the RGB camera and, consequently, reduces temporal discontinuities that could confuse HOI understanding of backend VLMs. Step 7: Adaptive Low-FPS Fallback Sampling. Step 6 designs HOI detector for non-stationary HOIs. To capture stationary and low-motion HOIs that may not trigger the motion-based detector, e.g., a user holding a remote control for a long time, we add a 0.1 FPS fallback sampling mechanism if the detector is not triggered within a 10-second period. These frames will not be compressed by the encoder described in the following section. We evaluate the impact of EventHOI on this in §6.2.3. Step 8: HOI-aware Video Encoder. We first aggregate the HOI saliency maps 𝑠 over the past 𝑇 seconds and define the spatial cropping ROI as the smallest bounding box enclosing their union. To avoid losing critical scene context, we enforce a minimum square RoI size of 𝑆% of the original frame. If the resulting RoI is smaller than this threshold, we center it at the RoI centroid and symmetrically expand it until it reaches size 𝑆% on both dimensions. This design is motivated by two properties of HOI video understanding. First, existing video understanding models typically operate on uniformly sampled low-FPS frames and are sensitive to missing spatial context. Cropping too tightly around the hands may exclude the manipulated object or surrounding scene cues needed to distinguish fine-grained interactions. Second, because our motion-driven saliency map 𝑠 captures both hand and object motion, the resulting RoI naturally covers the most interaction-relevant regions. This improves robustness to the temporal uncertainty of sparse RGB sampling, where captured frames may miss the exact contact moment. By preserving both hand and object context, the encoder helps the downstream model infer the correct HOI even when RGB frames are sampled coarsely. We use the same 𝑇 = 5 s as Step 6 since it can provide more spatial context by including the trajectory

Raspberry Pi 4 (Data collector)

IMU Event Camera (LSM6DS3) (GENX320) (a) Prototype

Battery

(b) Front view

(c) Side view

Figure 8: EventHOI prototype for data collection.

of hand movements for HOIs with a source or destination (e.g., from, to), like “pick up milk bottle from the fridge”, which cover over 12% of our benchmark in §6.1. We set the minimum retained frame size to 𝑆 = 75% by default based on the ablation study in §6.2.8. These threshold-based solutions are extremely lightweight (32k FLOPs per second of input), allowing us to take advantage of egocentric perspective to select important information without using computationally heavy machine learning models like nano-size YOLO at 1 FPS (1.4G FLOPs per second of input) [45].

5

System Implementation

We implement the EventHOI prototype on a 3D-printed smartglasses frame as shown in Fig. 8. We employ two different implementations for (1) continuous RGB, event, and IMU data collection (§7) since continuous RGB capture is necessary for fair baseline comparisons and accuracy evaluations, and (2) minimized on-device system consumption measurements for reflecting real-world EventHOI efficiency on selective RGB capture and compression (§8). (1) Data Collection Prototype. This setup is used to record complete RGB streams and event frames for the real-world testbed in §7, so we prioritize comprehensive data collection over minimizing ondevice resource consumption. We use an IMX708 RGB camera [65] at 720p and 30 FPS. We use a GENX320 event camera [51] in histogram mode to sample 10-FPS 320×320 event frames. The two sensors are co-located for better alignment. The positions of the RGB and event cameras are further calibrated using a checkerboard target (with a toggling checkerboard for the event camera) and stereo rectification in OpenCV [66]. The cameras are connected to a Raspberry Pi 4B [67] for data collection. We use an nRF52840 MCU equipped with an LSM6DS3TR-C 6-DoF IMU [68] for headmounted IMU data collection [42]. We use elastic headbands to securely mount the smartglasses prototype on the wearer’s head. The smartglasses prototype weighs 46 grams, comparable to the weight of Ray-Ban Meta glasses [26]. A sling bag carries the portable battery and Pi to minimize interference from the wearer’s hand and body movements. Flex cables are used to connect the sensors on the smartglasses to the Raspberry Pi for data collection. (2) On-device Implementation. This setup is optimized for measuring the actual on-device system consumption of EventHOI in §8. We implement all baselines on a resource-constrained Raspberry Pi Zero 2W SoC with a quad-core Cortex-A53 processor running at 600 MHz and 512 MB of SDRAM [69], following [9]. An ESP32-S3 microcontroller paired with an OV5640 RGB camera [42] operates continuously in a low-power standby mode and wakes up only when triggered by the HOI detector running on the Raspberry Pi Zero. Other settings match (1) data collection prototype.

Zhang et al.

2% (0.3 h) 10% (1.3h)

) .7h % (0

(a) Script Selection

h)

19.8h )

24% (3.4h)

(b) Daily Time Use (Awake)

18% (3.5h)

(1.6

12% (2.4h)

2h

(1 . 9%

12% (6.0h)

8%

)

5

15% (7.5h)

34% (4.7h)

14h

.6 h

20% (10.0h)

15% (2.9h)

(1

50h

17% (2.4h)

8%

.5h)

7% (3

28% (14.0h)

18% (9.0h)

39% (7.8h)

6

Large-Scale Dataset Benchmark Evaluation

We evaluate whether the RGB uplink produced by EventHOI preserves downstream HOI understanding quality while substantially reducing the number of recorded frames and the stream transmission cost using a large-scale smartglasses dataset Nymeria [3].

6.1

Task 1: Recognition VLM

Episodic memory logs

(c) Our Testbed

Figure 9: Nymeria [3] sequence selection for evaluation, daily time-use survey results [71] (excluding sleeping time), and the recording time of our real-world testbed (§7).

Benchmark Evaluation Setup

6.1.1 Evaluation Setups. Dataset. We build our benchmark on the Nymeria dataset [3], which contains egocentric RGB videos and head-mounted IMU recordings captured with Project Aria glasses [27] across diverse scripted daily activities. Because Aria does not include an event camera, we simulate event frames at 10 FPS using the event simulator v2e [70] on RGB videos. We study the simulation-to-real gaps to show the effectiveness of simulations in §6.2.9. The corresponding IMU signals are downsampled to 50 Hz. These synchronized RGB, simulated event, and IMU streams are then used as inputs to EventHOI and all baselines. To better reflect continuous daily-life logging, we construct a representative 50-hour subset from Nymeria with 151 recording sequences and 97 subjects in 26 locations. Specifically, we group sequences into 6 scenario categories (leisure, working, eating, cooking, housework, and recreation) following the 14-hour daily timeuse distribution reported by the U.S. Bureau of Labor Statistics (Fig. 9(b)) [71]. We then weight the sequences of each category by its percentage share of the 14-hour daily time-use distribution. Fig. 9(a) summarizes the resulting benchmark composition. We consider two downstream HOI understanding tasks, shown in Fig. 10: HOI recognition and HOI localization. These two tasks correspond to two core functionalities of egocentric memory systems: generating text-based semantically meaningful HOI logs from RGB inputs and retrieving relevant moments from those logs. Task 1: HOI Recognition evaluates whether an algorithm-optimized RGB uplink stream can support accurate transcription of user HOIs into natural-language narrations. Benchmark Construction. Nymeria provides 5-second atomic activity annotations, but these labels are broader than the HOI-focused narrations needed here. We therefore use Gemini-2.5-Pro [72] as an annotation oracle to convert each atomic label into an HOIoriented natural-language narration, such as “pick up a water bottle” or “hold a cup under the dispenser.” To avoid annotation drift or hallucination, we manually verify the oracle outputs and use them as ground truth. We curate more than 22,000 HOI annotations from the 50-hour benchmark. For each annotation, we use the corresponding 5-second time window from Nymeria for evaluation. Evaluation Protocol. For each method, we feed its resulting RGB uplink stream into a state-of-the-art off-the-shelf VLM to generate

What did I do at 2pm?

When did I drink water?

Leisure Working Eating Cooking Housework Recreation Other

Memory Storage

t = 6:00 t = 4:00 t = 2:00 t = 2:30 take a mug start coffee pour a cup clean and rinse mug of milk from cabinet machine

Task 2: Localization LLM

User

Example Q&A

Q: When did the user <start the coffee machine>? A: [2:00-2:30], B: [2:45-3:15], C: [4:00-4:30], D: [6:30-7:00]

Figure 10: Examples of HOI recognition and localization tasks for memory logging and retrieval. Table 2: Comparison of baselines and EventHOI ablations. Baselines

EventHOI Ablations

Uniform YOLO [45] Hands23 [46] Vanilla EVS Detector only EventHOI Reduce RGB frames? Can compress frames? On-device feasible?

No No Yes

Yes∗ No Yes

No Yes No

No No Yes

Yes No Yes

Yes Yes Yes

∗ We consider YOLO as a CV-based oracle. It does not reduce RGB frames in reality.

HOI narrations. We then compare each generated narration against the corresponding ground-truth HOI annotation. Task 2: HOI Localization evaluates whether timestamped naturallanguage HOI logs from Task 1 can support episodic retrieval, i.e., whether an LLM can localize when a queried HOI occurred. Benchmark Construction. For each ∼20-minute video, we use GPT5.2 [73] to group the 5-second ground-truth annotations from Task 1 into 15–30-second activity summaries and generate 10–20 temporal retrieval queries. Following standard video-QA protocols [74–76], each query is paired with one correct time window and three distractors from the same video, yielding 1,533 multiple-choice questions. Evaluation Protocol. For each question, the answering LLM is given the full-video memory logs, i.e., the VLM narrations generated from sampled frames in Task 1, and selects the correct window from the four choices. This evaluates whether the sampled RGB frames preserve enough semantic evidence for accurate HOI localization. 6.1.2 Evaluation Metrics. Metric 1: HOI recognition accuracy. Qwen3-VL-8B [47] VLM is primarily used in the HOI recognition evaluations according to its strong performance shown in §3. We further supply results on the state-of-the-art Qwen3.6-27B [48] and Gemini-3-Flash [77] VLMs to show the generalization of EventHOI. For Task 1, we compute the sentence similarity between each generated narration and its ground-truth annotation using the BGE-M3 sentence embedding [78]. Following prior semantic matching practice [79], we binarize the score with a threshold of 0.65 [80, 81]. Metric 2: HOI localization accuracy. For Task 2, we report accuracy over all 1,533 multiple-choice QAs, using both GPT-5.2 [73] and Gemini-2.5-Pro [72] with medium-level thinking as the answering LLMs to ensure robustness across retrieval models. Metric 3: Effective Frame Rate. To quantify temporal RGB recording reduction, we define effective frame rate as the average number of RGB frames sampled per second after temporal gating. Unless otherwise specified, the default frame rate at triggering is 1 FPS based on our preliminary studies (§3), and at a fallback of 0.1 FPS otherwise (§4.4). Metric 4: Uplink Bandwidth Requirement. To measure actual transmission cost, we report the average uplink bandwidth in Mbps. Unless otherwise specified, the default resolution is 720p (§3).

EventHOI : Event-Driven Resource-Efficient Hand-Object Interaction Logging on Smartglasses

60 40

0

60

e sc at oo sl p ic op e e ha n ng fo m ld ar ov ra e ng e ge t pu pl ll ac ta e ke ch lift e to ck u th ch ro w w ip cl e os ho e dr ld in po k in ur se rt re pu m t ov w e a pr sh e ca ss sp rry re pi a ck d up so rt

Improvement (%)

20

40

ph

0

on e co nt ba ai ll ne c co h r ffe b ips e ott m le a to ke m r la ato pt sn op a m ck a s ha rke ng r p e pa ap r ck er ag sh e el eg f g te a re ba m g dr ote a cl we ot r he s tr a bo y o ca k bo rd ut ard e bl ns an il k bo et ok s be pl d at e so fa

20

Figure 11: Top-30 EventHOI (0.5 FPS effective frame rate) improvements over uniform sampling (0.5 FPS) on hand motions (blue) and object recognition (green). Red is the average. 6.1.3 Baselines. We compare EventHOI against three RGB sampling and uplink-stream baselines and two ablations (Tab. 2): (1) Our solution (EventHOI). (2) Uniform Sampling (Uniform): Frames are sampled at a fixed rate and transmitted uniformly. This is the default solution but incurs high RGB sampling and transmission cost. (3) YOLO Hand Detection Oracle (YOLO) [45]: This baseline assumes an oracle-like on-device hand detector running continuously. We fine-tune YOLOv8n on 2,000 manually annotated hand bounding boxes from Nymeria, then apply the same HOI motion detector as EventHOI (§4.4) on the detected RoIs. YOLO serves as an oracle for temporal hand localization, but requires uniformly sampled RGB frames to detect hands and therefore cannot avoid the RGB sampling cost in reality. Moreover, the hand-only detector does not localize interacting objects and thus cannot directly support our HOI-preserving uplink compression. (4) Hands23 HOI Detection (Hands23) [46]: This stronger oracle baseline requires uniform RGB sampling and predicts both hand masks and interacted-object regions. It therefore can compress frames but cannot reduce RGB sampling. Also, its large ResNet-101 backbone disallows resource-constrained on-device deployment. (5) EventHOI w/o Egomotion Cancellation (Vanilla EVS). This ablation removes the egomotion cancellation module (§4.3) while retaining the HOI motion detector (§4.4) and HOI-aware video encoder (§4.4). The raw, unprocessed event frames are used to trigger RGB camera capture and encode the RGB frames. (6) EventHOI Detector Only. This ablation disables the HOI-aware video encoder (§4.4). We do not include a variant without the HOI motion detector alone (§4.4) since the encoder depends on the detector outputs.

6.2

Benchmark Evaluation Results

6.2.1 Comparison to Uniform Sampling. We first compare EventHOI against uniform sampling to determine whether its gains come merely from recording fewer frames or from selecting more informative ones non-uniformly. We use HOI recognition accuracy to quantify downstream task performance, effective frame rate to quantify sensing cost, and uplink bandwidth to quantify the transmission cost. To better isolate the benefit of EventHOI, we exclude HOIs that are already correctly recognized by uniform sampling at 0.25 FPS (discussed in §3), as these are predominantly long-duration interactions that remain observable even when the RGB camera captures only one frame every four seconds. This results in a subset

of 6,644 challenging HOI queries. (1) Accuracy at the same effective frame rate. For fairness, we set the HOI-to-noise threshold of EventHOI to 𝐶 = 5% (see ablation in §6.2.7) to achieve the same 0.5 FPS effective frame rate. We observe a 9.0% HOI recognition accuracy improvement on average across different scenarios compared to 0.5 FPS uniform sampling. (2) Accuracy at the same uplink bandwidth. We further observe a 12.2% average accuracy improvement compared to 0.35 FPS uniform sampling at the same 0.12 Mbps BLE uplink bandwidth. This result demonstrates the strong benefits of EventHOI’s detector and encoder over uniform sampling at the same effective frame rate and BLE uplink bandwidth. 6.2.2 Analysis on Improvements to Uniform Sampling. To further understand the benefits of EventHOI compared to 0.5 FPS uniform sampling (the same effective FPS as EventHOI), we use the LLM-asJudge method to analyze the HOI queries for which EventHOI and uniform sampling produce different predictions. Using GPT-5.2 [73], we characterize whether each method correctly identifies (1) the object and (2) the hand motion. We present the improvements for the top-30 objects and hand motions in Fig. 11, each of which appears in at least 1% of the dataset. The results show that (1) EventHOI improves object recognition by +22.1% on average, with especially large gains (over +30%) for small objects such as smartphones, containers, markers, and hangers, which are prone to occlusion and require multiple frames to understand; and (2) EventHOI improves hand motion recognition by +17.8% on average, with above-average gains (over +35%) for short motions such as “open (doors or cabinets)”, “scoop or eat (food)”, and “hang (clothes)”, which require high temporal resolution to track. Overall, EventHOI is particularly effective for recognizing small objects and short-term HOIs. This observation is consistent with its ability to better temporally localize HOIs that are vulnerable to uniform downsampling. 6.2.3 Analysis on Stationary and Low-motion HOIs. Beyond capturing fast motions, we evaluate whether EventHOI can retain evidences for stationary or low-motion actions, such as “hold”, “type”, “write”, “pour”, etc. We use GPT-5.2 LLM to identify stationary HOIs by instructing the LLM to select queries of stationary or low-motion actions like above. This filtering yields a subset of 6,117 stationary HOI queries. At an effective frame rate of 0.5 FPS, EventHOI achieves similar HOI recognition accuracy (+1.9%) compared to uniform sampling at the same frame rate, benefiting from the 0.1 FPS fallback sampling that captures near-stationary HOIs whose motion falls below the HOI detector threshold. This result demonstrates EventHOI’s robustness in capturing subtle and stationary HOIs. 6.2.4 Comparison with CV-based Oracle Baselines. Next, we compare EventHOI against CV-based oracle baselines without considering on-device compute constraints. The detected hand regions of YOLO is used for selective RGB sampling only since it cannot detect HOI-related objects, while Hands23’s HOI regions are used for HOI-aware compression strategy as EventHOI (§6.1.3). As shown in Fig. 12, Hands23 requires continuous RGB input and thus operates at the same effective frame rate as uniform 1 FPS sampling. YOLO Oracle detects hand appearances at an effective frame rate of 0.67 FPS on average. In contrast, EventHOI triggers RGB sensing only when HOI motion is detected, reducing the effective frame rate to 0.5 FPS on average. In uplink bandwidth, EventHOI substantially outperforms YOLO, since YOLO detects only hands and cannot

Better

0.17 0.12

0.33

0.32

0.22 0.16

0.33 0.24 0.16 0.33 0.26 0.19 72.8 74.9 69.9 72.0 72.3 72.0

0.38 0.30 0.23 0.38 0.28 0.19

0.35 0.27 0.19

0.36 0.28

0.18

75.0 75.7 73.1 74.5 73.0 73.2

0.19 0.14

0.37

0.38

0.26 0.21

0.33

0.1

0.29 0.20 0.17 0.26 0.13 0.10

0.2

0.13 0.10

0.3

0.21 0.14

0.4

0.33

60

Leisure

Working

Eating

Better

70

80.8 82.5 80.2 81.0 80.1 80.0

80

82.1 82.4 80.5 82.2 82.2 80.3

90

84.2 86.2 82.4 83.7 83.0 83.5

89.9 91.9 87.8 89.8 89.0 88.8

0.0

Cooking HouseworkRecreation Weighted

-0.8

Uniform

-1.3

-1.4

EventHOI

75 50 25 Qwen3-VL Qwen3.6 Gemini-3

Local. Acc (%)

Recog. Acc (%)

Figure 12: Effective FPS, uplink bandwidth, and accuracy for baselines and EventHOI ablations on the Nymeria dataset [3]. 75

-1.6

-2.0

50 25 Gemini-2.5 GPT-5.2

Figure 13: EventHOI recognition and localization accuracy. reliably support HOI-aware encoding. EventHOI is also comparable to, and slightly better than, Hands23, whose CV-based detections are not always robust in diverse egocentric scenes. Meanwhile, EventHOI achieves HOI recognition accuracy comparable to both oracle baselines. Overall, EventHOI matches the HOI recognition accuracy of CV-based oracle baselines while requiring lower uplink bandwidth, without relying on continuous RGB recording or heavyweight on-device CV inference (§8). 6.2.5 Performance across Different VLMs and Tasks. We further evaluate whether EventHOI generalizes across foundation models and downstream tasks. For Task 1, HOI Recognition, we compare the VLM-generated narrations produced by open-source Qwen3VL-8B [47], Qwen3.6-27B [48], and proprietary Gemini-3-Flash [77] in different model sizes. Compared with the uniform sampling baseline, EventHOI incurs only negligible accuracy drops of 0.8%, 1.3%, and 1.4%, respectively (Fig. 13). For Task 2, HOI Localization, we evaluate temporal HOI localization using two LLMs. Compared with logs generated from the uniform sampling baseline, temporal localization accuracy using EventHOI-produced logs drops by merely 1.6% for Gemini-2.5-Pro [72] and 2.0% for GPT-5.2 [73]. These results highlight two key findings. First, EventHOI preserves the salient interaction evidence needed not only for HOI recognition, but also for downstream temporal reasoning in episodic memory retrieval. Second, the benefits of EventHOI’s event-driven selective sampling are largely model-agnostic, yielding a consistent accuracy-efficiency tradeoff across different models. 6.2.6 Ablation on Components. We conduct two ablation studies, as described in §6.1.3. Without egomotion cancellation (Vanilla EVS, §4.3), EventHOI cannot reliably distinguish HOI-related motion from background egomotion, causing the effective frame rate to remain close to 1 FPS. In contrast, the HOI motion detector (§4.4) and the HOI-aware video encoder (§4.4) further reduce the uplink

(a) HOI-to-Noise Threshold

78 77

C ≤ 5%

0% 1% 3% 5%

76 EventHOI

75

10%

0 10 20 30 40 50 60 70 80 Frame Rate Reduction (%)

Recognition Acc (%)

0.2

Recognition Acc (%)

0.50 0.50

Better

1.00

1.00 0.96

EventHOI

0.67

1.00

1.00 0.99

0.72 0.72

0.79 0.79

0.78

1.00 0.99

1.00

0.75 0.75

0.74

1.00

1.00 0.98

0.79

EventHOI Detector Only

0.51 0.51

0.68

1.00

1.00 0.98

1.00 0.90

0.69

1.00

1.00 0.97

0.4

Vanilla EVS

0.45 0.45

0.6

Hands23

0.40 0.40

0.8

YOLO

0.61

1.0

76.7 78.1 78.3 77.6 76.2 76.1

Accuracy (%)

Uplink BW (Mbps)

Effective FPS

Uniform

1.00

Zhang et al.

(b) Minimal Frame Size

77

S ≥ 75%

76 75 20

100%

EventHOI

90% 75% 60%

30 40 50 60 70 Bandwidth Reduction (%)

Figure 14: Ablation studies on HOI-to-Noise threshold values for detector and minimal frame size for encoder. bandwidth to below 0.2 Mbps across different activity scenarios with a negligible 0.1% accuracy loss on average. Overall, these results show that each design component of EventHOI is important to achieving the system’s accuracy-efficiency tradeoff. 6.2.7 Ablation on HOI-to-Noise Threshold. We first determine the HOI-to-noise threshold 𝐶 of the HOI motion detector (§4.4), since it critically balances preserving HOIs and filtering distractors. Specifically, we evaluate a 10-hour subset of the Nymeria dataset [3] using the HOI recognition task and the metrics described in §6.1, with a default sampling rate of 1 FPS. We vary 𝐶 from 0% to 10%, and measure (i) the achieved frame-rate reduction and (ii) the overall recognition accuracy in Fig. 14(a). When 𝐶 = 5%, EventHOI achieves about 50% frame reduction while maintaining recognition accuracy within 1% compared to uniform 1 FPS sampling. However, for 𝐶 > 5%, accuracy drops sharply, indicating over-suppression of true HOI motion. Thus, we set 𝐶 = 5% by default. 6.2.8 Ablation on Minimum Frame Size. We study the minimum retained frame size 𝑆 of the HOI-aware video encoder (§4.4) on 720p RGB frames on the same 10-hour subset. Specifically, we fix the HOI-to-noise threshold at the default value 𝐶 = 5% and vary 𝑆 from 60% to 100%, where larger 𝑆 preserves more context but yields lower compression. As shown in Fig. 14(b), EventHOI maintains stable recognition accuracy as 𝑆 decreases, until performance begins to drop at 𝑆 = 60%. Based on this trade-off, we set 𝑆 = 75% by default, which keeps HOI recognition accuracy within 0.3% of using no HOIaware encoder while still achieving substantial spatial compression. 6.2.9 Simulation-to-Real Gaps. Since the benchmark relies on v2esimulated event frames, we validate whether the simulation-to-real gap affects the above results. We evaluate over 2,000 synchronized RGB and real event frame segments from our real-world dataset benchmark (§7). We compare optical flows computed from real event frames against those from v2e [70] events simulated on the paired RGB frames. We then fit the measured angular error into a distribution, inject sampled errors into the simulated flows of the benchmark, and rerun the full EventHOI pipeline (setup details in Appx. A). The results show that perturbed flows introduce negligible degradation to EventHOI: the selected frames maintain a high temporal overlap of 0.95 IoU with those selected from unperturbed flows, system consumption increases only slightly (+2.5% effective frame rate and +5.1% uplink bandwidth), and HOI recognition accuracy changes by only 0.2%. Overall, these results close the simulation-to-real gap and support the correctness of our large-scale simulation benchmark.

7 Real-World Testbed Evaluation 7.1 Testbed Setup 7.1.1 Data Collection. We collect a 20-hour real-world dataset using the hardware described in §5 under IRB approval. The dataset

EventHOI : Event-Driven Resource-Efficient Hand-Object Interaction Logging on Smartglasses

(a) Keyboard

1.00 0.42

0.36

Better

1.00

1.00

0.53

0.55

1.00

1.00

0.36

0.36

0.36

0.53

0.31

RGB

(b) Per scenario

0.64

1.00

1.00

1.00

1.00

1.00

EventHOI

0.59

1.00

0.50

1.00

1.00

0.42

1.00

1.00

1.00

0.41

0.4

0.57

0.6

0.54

0.8 0.41

Effective FPS

1.0

1.00

Uniform (a) Per participant

Event Flows

periodic keys

Egomotion Field

Residual Flow

aliasing

P2

P3

P5

P6

P7

P8

Better

87.6 86.4

78.4 79.8

81.5 78.8

81.3 80.0

87.4 82.5

93.5 92.8

84.0 83.8

100.0 100.0

94.0 90.0

78.4 79.8

93.5 92.8

89.1 87.4

82.0 82.0

89.5 86.9

P4

P9 P10

re ing ing ing rk ion ed isu rk Eat ok ewo eat ight Le Wo Co ous ecr We R H

Figure 15: Effective FPS, uplink bandwidth, and accuracy on the real-world testbed by participants (P1–10) and scenarios. includes 10 participants (9 male and 1 female) in household, office, and open workspace (details in Appx. B). Participants are instructed to perform their daily activities naturally within one of the six scenarios with distributions in Fig. 9(c). We follow the weighted protocol of daily time-use in §6.1. We use synchronized RGB, eventcamera, and IMU streams collected with the acquisition settings described in §5. Representative HOIs collected in our dataset are shown in Fig. 21 in Appx. B. 7.1.2 Benchmark Setup. We use Gemini-3-Flash [77] as a semiautomatic annotation assistant to generate HOI narrations for 5second video windows. Human annotators then verify and correct the generated narrations. This process yields 1,950 HOI narrations for the evaluation of the HOI recognition task. We follow the same evaluation protocol as described in §6.1.

7.2

(b) Tiles

Better

0.13

0.11

0.38

0.40 0.18

0.37 0.19

0.10

0.23

0.40

0.43

0.41

0.33 0.11

0.07

0.18

0.31

0.40 0.19

0.39 0.16

0.39 0.09

0.13

0.38

0.41 0.12

0.34 0.19

0.15

0.36

0.42

0.40 0.14

P1

84.0 84.0

100 90 80 70 60

repetitive tiles aliasing

Figure 16: Failure in flow aliasing on repetitive textures. 94.4 94.4

0.4 0.3 0.2 0.1 0.0

89.0 88.0

Accuracy (%)

Uplink BW (Mbps)

0.2

Testbed Evaluation Results

7.2.1 Real-World Testbed Results. We present per-subject and perscenario testbed results for HOI recognition accuracy, effective frame rate, and uplink bandwidth in Fig. 15. We observe EventHOI reduces the effective RGB frame rate by 2.4× (from 1 to 0.42 FPS) and BLE uplink bandwidth by 2.9× (from 0.38 to 0.13 Mbps) on average compared to the 1-FPS uniform sampling baseline. Despite these reductions, EventHOI incurs only a negligible 1.2% degradation in weighted recognition accuracy. Compared with the Nymeria benchmark [3], which focuses on densely recording HOIs, we observe the frame rate savings in real-world settings come from leisure, working, and recreation scenarios. For common leisure activities such as watching TV, only 14.4% of the windows contain HOIs (e.g., using a remote control). This allows EventHOI to skip 80.1% of RGB frame sampling while still capturing these short HOIs when they occur. Similarly, when participants use computers for working or recreation, we observe that over 59.1% of the windows contain no HOIs in the camera view, as experienced users tend to look at the screen rather than their hands while typing. This allows EventHOI to reduce RGB frame sampling by 73.4%. We present examples from different scenarios in Fig. 18. Overall, our real-world testbed results demonstrate the substantially greater potential of EventHOI to reduce RGB sampling requirements in practical settings since our testbed follows more natural density of daily HOIs.

7.2.2 False Positives from Background Motion. In practice, we find that false-positive HOI detections primarily arise from background motion unrelated to either HOIs or egomotion, such as bystander movements and dynamic content displayed on TVs, smartphones, and monitors. For example, dynamic TV content causes EventHOI to unnecessarily capture 3.8% more windows in our testbed, while motion displayed on smartphones results in 1.2% more captured windows. Furthermore, movements from bystanders and pets result in an additional 0.3% of captured windows. This translates to less than 0.02 Mbps negligible uplink bandwidth increase and 20mW power increase on average (§8). Although the HOI-to-Noise threshold effectively suppresses much of the background motion, these dynamic visual sources can still produce strong optical flow and be misclassified as HOI-related motion. This limitation motivates incorporating more informative visual modalities, such as monochrome or RGB frames, to better distinguish true HOI saliency from background motion during HOI detection. 7.2.3 False Positives from Background Textures. We observe another failure mode caused by highly textured or repetitive backgrounds, including curtains, blinds, keyboards, bed sheets, and tiled surfaces, which lead to 1.7% more captured windows in our testbed. This is equivalent to less than 0.01 Mbps negligible uplink bandwidth increase and 10mW power increase on average (§8). This behavior stems from a known limitation of LK-based optical flow: repetitive local patterns can create ambiguous correspondences, resulting in noisy or incorrect motion estimates [82]. As shown in Fig. 16, the periodic keys of a keyboard and the repetitive tiles of a backsplash produce aliased flow vectors pointing in opposite directions. Since egomotion induces a coherent motion field, egomotion cancellation cannot remove these opposite flows, and the residual is falsely detected as HOI saliency. This limitation motivates future designs that combine event optical flow with lightweight appearance or semantic cues to better suppress false triggers from repetitive background textures. Despite these limitations, EventHOI consistently achieves substantial reductions in RGB frame sampling and uplink bandwidth across different real-world scenarios. The real-world evaluation results demonstrate EventHOI’s practicality in real-world settings.

8

System Evaluation

We evaluate two key aspects of EventHOI ’s on-device performance: (1) power consumption, which measures the computational efficiency of EventHOI, and (2) on-device latency, which evaluates whether the detection pipeline can operate in real time while meeting the RGB sampling duty cycle requirements. We assume the

Zhang et al.

Uniform

Power (W)

0.8 0.70 0.70

35.0x

0.6

0.6

0.4

0.4

0.2 0.0

Power (W)

2.0

0.5

1.26

2.6x

0.48

1.86

1.7x

1.5 1.32

0.0 0.0 Nymeria Weighted

2.0 1.5 1.0

1.25

1.08

2.0

1.0

2.0

OS

Nymeria Total

0.5

1.27

0.12

0.0 Always-on Backend

Uplink

1.5

10.4x

0.70

0.5

0.10

0.00

0.5

1.8x

0.015 0.012

0.0 0.00 Detection Algorithm

1.0

0.72

5.5x 0.02

EventHOI 0.021

0.01

2.0

1.5 1.0

0.55

0.2 0.02

Always-on Sensor

YOLO

0.8

3.0x

1.87

1.0

0.72 0.43

1.8x

1.5 1.32 1.03

0.5

OS 0.0 0.0 Real-World Weighted Real-World Total

Figure 17: Power consumption breakdown. cloud backend VLMs are sufficiently powerful to process uploaded RGB frames in a timely manner.

8.1

On-Device Power Consumption

We compare the on-device power consumption of EventHOI against uniform sampling and YOLO. For a fair comparison, all methods are implemented on the on-device prototype mentioned in §5. EventHOI operates at 2 FPS for translational body-motion detection, at 10 FPS for rotational head motion tracking, while triggering the RGB camera at 1 FPS. In contrast, all baselines run at 1 FPS, since they rely on appearance-based detection and do not require finer-grained temporal sensing. Fig. 17 shows the average power consumption of each baseline. EventHOI reduces the always-on sensing power by 35.0× compared to Uniform and YOLO. EventHOI requires only 0.02 W for continuous event-camera and IMU sensing, whereas Uniform and YOLO both require continuous RGB capture, which consumes 0.7 W. EventHOI also reduces the detection-stage power by 5.5× compared with YOLO. Although EventHOI introduces additional compute overhead for the HOI motion detector (§4.4) and the HOI-aware video encoder (§4.4), this overhead is only about 0.1 W, which remains small relative to the cost of continuous RGB sensing (0.7 W) and YOLO-based hand detection (0.55 W). EventHOI reduces total uplink wireless communication power consumption by 1.3–1.8× compared with Uniform and YOLO. EventHOI requires substantially lower uplink bandwidth and therefore consumes only 0.012 W for BLE. Overall, during non-HOI periods, EventHOI reduces always-on backend power by 5.8× and 10.4× compared with Uniform and YOLO, respectively. When mapped to the daily usage statistics in Fig. 9, this translates to 33% and 41% energy savings on Nymeria and the realworld testbed, respectively, excluding the OS idle power consumption of Raspberry Pi OS.

8.2

On-Device Latency

HOI Detection Algorithm Latency. Our results show that YOLO requires 6.93 s per 256×256 frame and 1.92 s per 128×128 frame on the RPi Zero, failing to localize hands reliably at lower resolutions. Such high latency makes YOLO impractical even at 1 FPS. In contrast, EventHOI processes each event frame in 15–20 ms, while the HOI motion detector adds about 50 ms per 0.5 s window. Even when event frames are processed at 10 FPS, EventHOI incurs only a 25–30% duty cycle when driving 1-FPS RGB sampling. Overall, EventHOI can decide 1-FPS RGB capturing within 100 ms. End-to-end Latency. EventHOI end-to-end on-device pipeline includes the HOI motion detector, RGB camera wake-up, RGB frame capture, and camera’s return to sleep. We measure two metrics: (1)

detection-to-capture latency, from the latest event/IMU input to the RGB trigger; and (2) RGB duty-cycle latency, from waking the camera to putting it back to sleep. The HOI detection algorithm takes less than 100 ms, since event frames are processed in a streaming pipeline. Waking the ESP32-S3 from ultra-low-power sleep takes up to 300 ms. The default ISP pipeline on the ESP32-S3 with the OV5640 camera performs automatic calibrations (e.g., white balance and exposure), which can take up to 1 s to converge and is too slow to meet our 1 FPS duty-cycle constraint. To address this, we implement a checkpoint-and-restore mechanism that stores the most recent camera calibration parameters in non-volatile memory and restores them across power cycles, assuming the user remains in similar lighting conditions across adjacent frames. This reduces parameter convergence to 150–200 ms. Finally, checkpointing parameters and shutting down the camera incur an additional ∼50 ms. Overall, the worst-case detection-to-capture reaction latency is about 500 ms, which is acceptable because EventHOI aggregates saliency maps over a 5 s window and is therefore tolerant to ∼0.5 s temporal staleness. The RGB duty-cycle latency is 500–600 ms, satisfying the < 1 s requirement.

9

Conclusion

We presented EventHOI, an event-driven system for resource-efficient egocentric HOI logging on smartglasses. It uses low-power event sensing and IMU-assisted egomotion cancellation to selectively capture HOI-relevant RGB content without continuous RGB recording or heavyweight on-device computation. More broadly, EventHOI points to a promising design paradigm for resource-constrained systems: using low-power sensors to guide and sparsify the operation of high-power sensors to enable resource-efficient embedded sensing systems.

Appendix A

Simulation-to-Real Evaluation

This appendix details the setup of the simulation-to-real validation summarized in §6.2.9. Data Collection. We evaluate the 2,000 synchronized RGB and event frame segments from our real-world benchmark recording (§7). The RGB and event camera hardware setup is discussed in §5 (1) Data Collection Prototype. We use the RGB frames to simulate events via v2e [70], following the same simulation procedure as the benchmark in §6.1. Flow Error Measurement. For both real and simulated event frames, we compute optical flow at 10 FPS and 320×320 resolution using the LK method (§4.3), and average the resulting flow over 8×8 grids. We remove outliers by discarding the largest anomalous flows (LK failures) and the smallest 10% flows (noise). We then compute the angular error between each pair of corresponding simulated and real grid flows, with examples shown in Fig. 19. Error Injection on the Benchmark. We fit the resulting angular errors into a circular normal distribution [83]. We then sample from this distribution and inject the errors into the optical flows produced by the RGB-v2e pipeline on each benchmark frame. The perturbed flows are processed through the full EventHOI pipeline,

EventHOI : Event-Driven Resource-Efficient Hand-Object Interaction Logging on Smartglasses

RGB FPS

(a) Cooking Scenario Example

RGB FPS

1.0 0.1 00:14:00

1.0 0.1

00:14:20

00:13:00

00:14:40

00:13:20

HOI Capture HOI Recognition

00:15:00

00:15:20 Time

00:15:40

00:16:00

00:16:20

00:13:40

00:14:00 00:14:20 00:14:40 00:15:00 Time (b) Leisure Scenario Example True Positive Captured False Positive Captured Missed Captured Correct Classified Misclassified

00:15:20

Figure 18: EventHOI real-world examples. The plot shows the RGB sampling rate and the icons indicate different HOIs. Table 3: Recording hours and participants (P1–10) of the realworld testbed per location and scenario. Location

(b) Event Frame

Figure 19: Optical flows (red arrows) from (a) v2e-simulated and (b) real event frames while the user holds a water bottle. including egomotion cancellation, HOI motion detection, and HOIaware video encoding, and we generate HOI narrations using a VLM. Evaluation Metrics. We compare the outputs of EventHOI on perturbed and unperturbed flows using the following metrics: (1) Temporal IoU for the overlaps of selected frames, (2) Effective FPS, (3) Uplink Bandwidth, and (4) HOI Recognition Accuracy. Results. The simulation-to-real experiments reveal a median absolute angular error of 10–11 degrees with examples in Fig. 19(a) and (b). These errors likely arise from (1) stereo calibration and rectification inaccuracy, (2) v2e simulation mismatch, and (3) event sensor noise that cannot be fully removed by simple filtering. Despite these error sources, the perturbed flows introduce negligible degradation to EventHOI: the selected frames maintain a high temporal overlap of 0.95 IoU, system consumption increases only slightly by +2.5% effective frame rate and +5.1% uplink bandwidth, and HOI recognition accuracy changes by only −0.2%. Overall, these results close the simulation-to-real gap and support the correctness of our large-scale simulation benchmark.

B

Real-World Testbed Evaluation

This appendix details the setup of the real-world testbed in §7. Testbed Setup. We perform testbed data collection in 2 homes, 1 office, and 2 open workspaces. Each home is in the standard apartment style including kitchen, living room, dining room, and bedrooms etc. The home settings allow user to perform any of the six scenarios, while the office and open workspaces are dominated by working, leisure, and recreation activities. Each participant is instructed to perform their daily routine normally rather than following pre-defined scripts. This allows us to collect and evaluate

Eating

Cooking Housework Recreation Total

2.5 2.6 0.8 1.6 1.6 2.2 Home1 (P1, P2, P6) (P1, P2) (P1, P2, P6) (P1, P6) (P1, P2, P6) (P1, P2, P6) 3.4 0.3 0.8 Home2 – – – (P4) (P4) (P4) 0.3 0.5 0.5 Workspace1 – – – (P5) (P3) (P1) 0.3 0.7 0.8 Workspace2 – – – (P9) (P3, P10) (P8, P9) 0.4 0.7 Office – – – – (P7) (P10) Total

3.5

Leisure

7.8

Working 3.1

3 Hours

(a) Simulation

Leisure Working

1.6

Eating

2.4

Cooking

3.0

1

2.9

Housework

4.5 1.3 1.7 1.1 19.8

Recreation

2.7

2

1.6 1.2

0.8

1.6

11.3

0.8

1.4

1.4

1.6

1.0

1.3

0.1

0 12:00

14:00

16:00 18:00 Time of Day

20:00

22:00

24:00

Figure 20: Recording hours of the real-world testbed across the time of day.

(a) Actions

(b) Objects

Figure 21: Examples of HOIs in the real-world testbed. HOIs at a more natural density across the daily life. We show the distribution across participants, scenarios, and locations in Tab. 3. We show the distribution of activities on time horizon in Fig. 20 and example actions and objects in Fig. 21.

Zhang et al.

References [1] Jacqueline Mogle, Jennifer R. Turner, Laura A. Rabin, Martin J. Sliwinski, Ruixue Zhaoyang, and Nikki L. Hill. Measuring memory lapses and their impact on daily life: Results from two daily diary studies. Assessment, 30(5):1454–1466, 2023. [2] G Hussein Rassool. Failure to take prescribed medicine for chronic diseases is a massive world-wide problem. Journal of Advanced Nursing, 45(4):447, 2004. [3] Lingni Ma et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In Computer Vision – ECCV 2024, pages 445–465, Cham, 2025. Springer Nature Switzerland. [4] Kristen Grauman et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18995–19012, June 2022. [5] Toby Perrett et al. Hd-epic: A highly-detailed egocentric video dataset. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23901–23913, 2025. [6] Meta. How to change video settings on ai glasses, 2026. https://www.meta.com/ help/ai-glasses/1373257549211327. Accessed: 2026-10-31. [7] Project Aria. Project aria hardware specifications, 2026. https://facebookresearch. github.io/projectaria_tools/docs/tech_spec/hardware_spec. Accessed: 2026-1031. [8] Akshay Paruchuri et al. Egotrigger: Toward audio-driven image capture for human memory enhancement in all-day energy-efficient smart glasses. IEEE Transactions on Visualization and Computer Graphics, 31(11):9720–9729, 2025. [9] Soroush Shahi, Farzad Shahabi, Rama Naboulsi, Glenn Fernandes, Aggelos K Katsaggelos, and Nabil Alshurafa. Thor: Thermal-guided hand-object reasoning via adaptive vision sampling. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 9(4), December 2025. [10] Prarthana Bhattacharyya et al. Helios: An extremely low power event-based gesture recognition for always-on smart eyewear. In Computer Vision – ECCV 2024 Workshops, pages 168–184. Springer Nature Switzerland, 2025. [11] Saman Naderiparizi, Pengyu Zhang, Matthai Philipose, Bodhi Priyantha, Jie Liu, and Deepak Ganesan. Glimpse: A programmable early-discard camera architecture for continuous mobile vision. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys), pages 292–305. Association for Computing Machinery, 2017. [12] Cypress Semiconductor. BLE throughput rate benchmark test for cypress BT SoCs. Infineon Developer Community. https://community.infineon.com/ t5/Blogs/BLE-Throughput-Rate-Benchmark-Test-for-Cypress-BT-SoCs/bap/247143, June 2020. Published June 9, 2020. Accessed: 2026-11-02. [13] Ali Hojjat, Janek Haberer, Tayyaba Zainab, and Olaf Landsiedel. Limitnet: Progressive, content-aware image offloading for extremely weak devices & networks. In Proceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services, MOBISYS ’24, page 519–533, New York, NY, USA, 2024. Association for Computing Machinery. [14] Xuedou Xiao, Yingying Zuo, Mingxuan Yan, Wei Wang, Jianhua He, and Qian Zhang. Task-oriented video compressive streaming for real-time semantic segmentation. IEEE Transactions on Mobile Computing, 23(12):14396–14413, 2024. [15] Xingyu Chen, Zihao Feng, Wuqiong Zhao, Jianrong Ding, Ke Sun, and Xinyu Zhang. From bits to tokens: Knowledge-Driven generative communication of multimodal data. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26), pages 191–207, Renton, WA, May 2026. USENIX Association. [16] Boshen Xu, Ziheng Wang, Yang Du, Zhinan Song, Sipeng Zheng, and Qin Jin. Do egocentric video-language models truly understand hand-object interactions? In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 100749–100776, 2025. [17] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F. Fouhey. Understanding human hands in contact at internet scale. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9866–9875, 2020. [18] Eastman Z Y Wu, Yali Li, Yuan Wang, and Shengjin Wang. Exploring pose-aware human-object interaction via hybrid learning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17815–17825, 2024. [19] Baoqi Pei, Yifei Huang, Jilan Xu, Yuping He, Guo Chen, Fei Wu, Jiangmiao Pang, and Yu Qiao. Egothinker: Unveiling egocentric reasoning with spatio-temporal cot. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 44140–44168. Curran Associates, Inc., 2025. [20] Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3272–3282, 2022. [21] Yixin Chen, Sai Kumar Dwivedi, Michael J. Black, and Dimitrios Tzionas. Detecting human-object contact in images. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17100–17110, 2023. [22] Gorjan Radevski, Dusan Grujicic, Matthew Blaschko, Marie-Francine Moens, and Tinne Tuytelaars. Multimodal distillation for egocentric action recognition. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages

5190–5201, 2023. [23] Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Helping hands: An object-aware ego-centric video recognition model. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 13855–13866, 2023. [24] Chen Bao, Jiarui Xu, Xiaolong Wang, Abhinav Gupta, and Homanga Bharadhwaj. Handsonvlm: Vision-language models for hand-object interaction prediction, 2024. [25] Bandhav Veluri, Collin Pernu, Ali Saffari, Joshua Smith, Michael Taylor, and Shyamnath Gollakota. Neuricam: Key-frame video super-resolution and colorization for iot cameras. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, ACM MobiCom ’23, New York, NY, USA, 2023. Association for Computing Machinery. [26] Meta. Ray-Ban Meta Wayfarer (Gen 2), 2025. https://www.meta.com/ai-glasses/ ray-ban-meta-wayfarer-gen-2/. Accessed: 2026-10-31. [27] Jakob Engel et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561, 2023. [28] Elias Mueggler, Christian Forster, Nathan Baumli, Guillermo Gallego, and Davide Scaramuzza. Lifetime estimation of events from dynamic vision sensors. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 4874–4881, 2015. [29] Christian Brandli, Lorenz Muller, and Tobi Delbruck. Real-time, high-speed video decompression using a frame- and event-based davis sensor. In 2014 IEEE International Symposium on Circuits and Systems (ISCAS), pages 686–689, 2014. [30] Guang Chen, Hu Cao, Jorg Conradt, Huajin Tang, Florian Rohrbein, and Alois Knoll. Event-based neuromorphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Processing Magazine, 37(4):34–49, 2020. [31] Zhe Jiang, Yu Zhang, Dongqing Zou, Jimmy Ren, Jiancheng Lv, and Yebin Liu. Learning event-based motion deblurring. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3317–3326, 2020. [32] Cedric Scheerlinck, Henri Rebecq, Daniel Gehrig, Nick Barnes, Robert Mahony, and Davide Scaramuzza. Fast image reconstruction with an event camera. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020. [33] Hao Cao, Jingao Xu, Danyang Li, Zheng Yang, and Yunhao Liu. Eventboost: Event-based acceleration platform for real-time drone localization and tracking. In IEEE INFOCOM 2024 - IEEE Conference on Computer Communications, pages 1851–1859, 2024. [34] Lingdong Kong, Dongyue Lu, Xiang Xu, Lai Xing Ng, Wei Tsang Ooi, and Benoit R. Cottereau. Eventfly: Event camera perception from ground to the sky. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1472–1484, 2025. [35] Julian Moosmann, Pietro Bonazzi, Yawei Li, Sizhen Bian, Philipp Mayer, Luca Benini, and Michele Magno. Ultra-efficient on-device object detection on aiintegrated smart glasses with tinyissimoyolo. In Computer Vision – ECCV 2024 Workshops, pages 262–280, Cham, 2025. Springer Nature Switzerland. [36] Carl Brander, Giovanni Cioffi, Nico Messikommer, and Davide Scaramuzza. Reading in the dark with foveated event vision. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 5035–5043, 2025. [37] Ali Hojjat, Janek Haberer, and Olaf Landsiedel. Mcucoder: Adaptive bitrate learned video compression for iot devices. In DAGM German Conference on Pattern Recognition, pages 123–138. Springer, 2025. [38] Yiting Wang, Pak Hung Chan, and Valentina Donzella. Semantic-aware video compression for automotive cameras. IEEE Transactions on Intelligent Vehicles, 8(6):3712–3722, 2023. [39] FLIR Systems, Inc. FLIR Lepton 2.5: LWIR Micro Thermal Camera Module, May 2019. https://www.mouser.com/datasheet/2/227/lepton_2pt5_datasheet_041952580098.pdf. Datasheet, document no. 19-0372-OEM-COR-Lepton 2.5. Accessed: 2026-08-14. [40] OmniVision Technologies, Inc., Santa Clara, CA. OG02B1B 2-Megapixel Product Brief, v1.5 edition, sep 2024. https://www.ovt.com/wp-content/uploads/2024/09/ OG02B1B-PB-v1.5-WEB.pdf. [41] OmniVision Technologies, Inc. OV5640 Datasheet Product Specification, 2011. https://cdn.sparkfun.com/datasheets/Sensors/LightImaging/OV5640_ datasheet.pdf. Seeed Studio XIAO Series System-on-Modules (SoM), n.d. [42] Seeed Studio. https://files.seeedstudio.com/wiki/XIAO/Seeed-Studio-XIAO-Series-SOMDatasheet.pdf. Accessed: 2026-03-09. [43] BasedHardware. omi, 2025. GitHub repository.https://github.com/ BasedHardware/omi. Accessed: 2026-01-17. [44] Nordic Semiconductor. Power Profiler Kit II, 2025. https://www.nordicsemi.com/ Products/Development-hardware/Power-Profiler-Kit-2. Accessed: 2026-02-24. [45] Rahima Khanam and Muhammad Hussain. Yolov11: An overview of the key architectural enhancements, 2024. [46] Tianyi Cheng, Dandan Shan, Ayda Hassen, Richard Higgins, and David Fouhey. Towards a richer 2d understanding of hands at scale. In Advances in Neural Information Processing Systems, volume 36, pages 30453–30465. Curran Associates,

EventHOI : Event-Driven Resource-Efficient Hand-Object Interaction Logging on Smartglasses

Inc., 2023. [47] An Yang et al. Qwen3 technical report, 2025. [48] Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. https: //qwen.ai/blog?id=qwen3.5. [49] Gemma Team et al. Gemma 4 technical report, 2026. [50] Ana I. Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso García, and Davide Scaramuzza. Event-based vision meets deep learning on steering prediction for self-driving cars. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5419–5427, 2018. [51] Prophesee. 1/5” Format 320 x 320-Pixel Event-Based Metavision® Sensor Compact Optical Module with Embedded Features. Product brief, Prophesee, January 2025. https://www.prophesee.ai/wp-content/uploads/2025/01/GENX320Product-Brief-2025-CM2-OK.pdf. Accessed: 2026-02-16. [52] STMicroelectronics. VD55G1 Datasheet: Tiny, clever, low power, 0.56-megapixel, monochrome global shutter image sensor Datasheet, July 2024. DS13604 – Rev 3. https://www.st.com/resource/en/datasheet/vd55g1.pdf. [53] Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015. [54] Gaspare Santaera, Emanuele Luberto, Alessandro Serio, Marco Gabiccini, and Antonio Bicchi. Low-cost, fast and accurate reconstruction of robotic and human postures via imu measurements. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 2728–2735, 2015. [55] Christian Forster, Luca Carlone, Frank Dellaert, and Davide Scaramuzza. Onmanifold preintegration for real-time visual–inertial odometry. IEEE Transactions on Robotics, 33(1):1–21, 2017. [56] Zhelong Wang, Hongyu Zhao, Sen Qiu, and Qin Gao. Stance-phase detection for zupt-aided foot-mounted pedestrian navigation system. IEEE/ASME Transactions on Mechatronics, 20(6):3170–3181, 2015. [57] Abdelmoumen Norrdine, Zakaria Kasmi, and Jörg Blankenbach. Step detection for zupt-aided inertial pedestrian navigation system using foot-mounted permanent magnet. IEEE Sensors Journal, 16(17):6766–6773, 2016. [58] Carlos Campos, Richard Elvira, Juan J. Gómez Rodríguez, José M. M. Montiel, and Juan D. Tardós. Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):1874– 1890, 2021. [59] Yin Li, Alireza Fathi, and James M. Rehg. Learning to predict gaze in egocentric video. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2013. [60] Jean-Yves Bouguet et al. Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm. Intel corporation, 5(1-10):4, 2001. [61] Berthold K.P. Horn and Brian G. Schunck. Determining optical flow. Artificial Intelligence, 17(1):185–203, 1981. [62] Gunnar Farnebäck. Two-frame motion estimation based on polynomial expansion. In Josef Bigun and Tomas Gustavsson, editors, Image Analysis, pages 363–370, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg. [63] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8934–8943, 2018. [64] Zachary Teed et al. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision – ECCV 2020, pages 402–419, Cham, 2020. Springer International Publishing. [65] Arducam. IMX708 12MP USB3.0 CameraModule, August 2023. https://blog.arducam.com/downloads/datasheet/B0474_12MP_IMX708_ USB3.0_Camera_Datasheet.pdf. Accessed: 2026-03-27. [66] G. Bradski. The OpenCV Library. Dr. Dobb’s Journal of Software Tools, 2000. [67] Raspberry Pi Ltd. Raspberry Pi 4 Model B, release 1.1 edition, March 2024. https://pip-assets.raspberrypi.com/categories/545-raspberry-pi-4-modelb/documents/RP-008341-DS-1-raspberry-pi-4-datasheet.pdf. Accessed: 202603-27. [68] STMicroelectronics. iNEMO 6DoF inertial measurement unit (IMU), for entry level / mid-tier smart phones and Portable PC platforms, 2026. https://www.st.com/en/ mems-and-sensors/lsm6ds3tr-c.html. Accessed: 2026-08-31. [69] Raspberry Pi Ltd. Raspberry Pi Zero 2 W. Raspberry Pi Ltd, April 2024. Product brief. https://pip-assets.raspberrypi.com/categories/584-raspberry-pi-zero2-w/documents/RP-008359-DS-1-raspberry-pi-zero-2-w-product-brief.pdf. Accessed: 2026-03-09. [70] Yuhuang Hu, Shih-Chii Liu, and Tobi Delbruck. v2e: From video frames to realistic dvs events. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1312–1321, 2021. [71] U.S. Bureau of Labor Statistics. American time use survey—2024 results. Technical report, U.S. Department of Labor, June 2025. https://www.bls.gov/news.release/ pdf/atus.pdf. Accessed: 2026-02-18. [72] Gheorghe Comanici et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. [73] Aaditya Singh et al. Openai gpt-5 system card, 2026.

[74] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 46212– 46244. Curran Associates, Inc., 2023. [75] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9772–9781, 2021. [76] Chaoyou Fu et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24108–24118, 2025. [77] Google. Gemini 3 pro model card, December 2025. https://storage.googleapis. com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf. Accessed: 2026-03-12. [78] Jianlyu Chen et al. M3-embedding: Multi-linguality, multi-functionality, multigranularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335. Association for Computational Linguistics, 2024. [79] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China, November 2019. Association for Computational Linguistics. [80] Sheng Zhang, Kevin Duh, and Benjamin Van Durme. Fine-grained entity typing through increased discourse context and adaptive classification thresholds. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 173–179, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. [81] Naman Bansal, Mousumi Akter, and Shubhra Kanti Karmaker Santu. SEM-F1: an automatic way for semantic evaluation of multi-narrative overlap summaries at scale. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 780–792, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. [82] Yiran Zhong, Pan Ji, Jianyuan Wang, Yuchao Dai, and Hongdong Li. Unsupervised deep epipolar flow for stationary or dynamic scenes. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12087–12096, 2019. [83] Nicholas I Fisher. Statistical analysis of circular data. cambridge university press, 1995.

Related documents

• MonoEgo: Monocular Metric Egocentric Demonstration Capture with Passive Wrist Constellations and Sparse Workstation Anchors#1128779
• OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video#1247782
• Reconstructing Humans and Objects in Interaction using Large Reconstruction Models#1215456
• A physical framework to harmonize human interaction analysis across disciplines#308246
• Cross-Embodiment Affordance Transfer for Robotic Manipulation#1343710