SemanticXR: Low Power and Real-time Queryable Semantic Mapping with an Object-Level Device-Cloud Architecture Rahul Singh*
Devdeep Ray†
Connor Smith‡
Sarita Adve§
University of Illinois Urbana-Champaign
NVIDIA
NVIDIA
University of Illinois Urbana-Champaign
Application
Spatial object search app
System Architecture 2D Foundation models
3D Semantic mapping
(extract semantics)
(create point cloud)
Object-centric parallelize
Depth-mapping co-design
Results Update/Query handling
Baseline
arXiv:2606.12849v1 [cs.DC] 11 Jun 2026
Object-centric downsampling
< 2.5 Mbps
Incremental updates Update prioritization
User Query
Network Driver 2% device power increase
They’re on the cabinet App Service
upstream BW
(§3.2)
Server
Network Driver
Ours
Semantic map
(§3.1)
Where did I put my keys?
2.2× faster
(§3.3)
(§3.1)
Mapping Latency
Normal operation Highlight the keys
Network Driver
Depth downsampling
H.264 enc.
RGB
Network Driver
Object-level sparse map
(§3.3)
Pose
Depth
No
(§3.2)
XR Device
Object-level innovations
< 100ms local query latency Yes
Network check? User Query
XR App
(Configurable quality params §3.4)
XR Apps
even under network drops
Flexible Tuning Latency Power Quality Network BW
Figure 1: SemanticXR overview. SemanticXR enables real-time, open-vocabulary semantic mapping for low-power XR through a device-cloud architecture organized around objects as first-class units of communication, execution, and memory footprint. Object-level innovations (green) speed-up server-side mapping (Sec. 3.1), reduce upstream bandwidth (Sec. 3.3); on the device, a sparse local map with incremental updates and update prioritization enables network-robust querying (Sec. 3.2). Organizing every operation at object granularity allows both applications and the system to trade off resource usage against semantic quality, suiting application requirements and operating conditions respectively, without modifying the perception and mapping pipeline (Sec. 3.4). A BSTRACT Semantic mapping is a core service that enables grounded interactions in emerging Extended Reality (XR) applications such as AI assistants and spatial object search. Deploying this capability on mobile XR devices requires a system that is open-vocabulary, realtime, and low-power. Existing approaches are compute-intensive and assume server-class resources. Cloud offloading offers a practical path, but no existing system splits semantic mapping between the device and the cloud, and current approaches do not address how to manage communication, execution, and memory footprint across the device-cloud boundary. We present SemanticXR, the first device-cloud system for realtime, open-vocabulary semantic mapping and querying under XR power, bandwidth, and memory constraints. Our key insight is to elevate semantically identifiable objects to first-class units of system design, governing how the system communicates, executes, and manages memory across the device and the server. On the server, object-level parallelism and geometry downsampling improve mapping latency, while object-level depth-mapping co-design reduces
upstream bandwidth. On the device, an object-level sparse local map with incremental updates and update prioritization enables network-robust querying with bounded memory and downstream bandwidth. Object-level configurable resource usage vs. quality trade-offs allow both applications and the system to adapt semantic mapping behavior to application requirements and operating conditions respectively. Evaluation against a device-cloud baseline using the same perception models shows that object-level system organization improves server-side mapping latency by 2.2× at equivalent semantic quality. Object-level depth-mapping co-design maintains upstream bandwidth under 2.5 Mbps. On the device, SemanticXR sustains sub-100 ms query latency for up to 10,000 objects even under network drops, supports tens of thousands of objects within 500 MB memory footprint, and scales downstream bandwidth with map changes rather than total scene size. The system adds only 2% device power during normal operation. 1
I NTRODUCTION
Extended Reality (XR) has the potential to transform application domains such as education, healthcare, accessibility, and industrial work. To support emerging capabilities such as spatial object search, AI assistants, and context-aware scene understanding, XR devices must go beyond reconstructing 3D geometry to associating semantic meaning with the physical environment. This requires a semantic mapping service that incrementally builds and retains a queryable 3D map linking geometry with meaning. For example,
* e-mail: [email protected] † e-mail: [email protected] ‡ e-mail: [email protected] § e-mail: [email protected]
1
allelism and geometry downsampling manage server-side computation at object granularity, improving real-time mapping latency over frame- or scene-level execution (Sec. 3.1). Objectlevel depth-mapping co-design downsamples depth before transmitting to the server and mitigates quality loss through per-object mapping decisions, providing a lightweight alternative to compression techniques [36] for reducing upstream bandwidth with negligible device-side overhead (Sec. 3.3). On the device, an objectlevel sparse local map bounds device memory and downstream bandwidth while enabling queries under network drops (Sec. 3.2). Object-level incremental updates keep downstream bandwidth proportional to map changes rather than total scene size (Sec. 3.2). Object-level update prioritization further reduces device memory usage and downstream bandwidth by sending and storing only relevant object updates to the device. Object-level configurable resource usage vs. quality trade-offs unify these innovations, allowing both applications and the system to adapt semantic mapping behavior to application requirements and operating conditions respectively, without modifying the perception and mapping pipeline (Sec. 3.4). Together, these innovations lead to the following contributions: 1. We introduce SemanticXR, the first system to enable real-time, open-vocabulary semantic mapping and querying within the power, bandwidth, and memory constraints of mobile XR devices. 2. We identify objects as the core system abstraction for devicecloud semantic mapping, elevating them to first-class units of communication, execution, and memory footprint management. This abstraction enables the innovations that collectively address all the gaps identified in Tab. 1. 3. We enable per-object configurable resource usage vs. quality trade-offs, allowing diverse applications and the system to adapt semantic mapping to application requirements and operating conditions respectively, without modifying the perception and mapping pipeline. Since no existing system implements device-cloud semantic mapping, we construct a device-cloud baseline that uses the same perception models and mapping algorithm as SemanticXR but does not organize system operations at object granularity. This controlled comparison ensures that observed differences are attributable to system design rather than algorithmic or model choice. As discussed, approaches that fuse semantics into monolithic representations remain architecturally incompatible with device-cloud deployment (Sec. 8). Among compatible approaches, this devicecloud baseline is the only one to achieve real-time mapping latency without sacrificing semantic quality: the only other real-time approach has worse quality, and every approach with comparable quality runs offline, taking seconds to minutes per frame (Sec. 5.1). Over this device-cloud baseline, SemanticXR improves real-time mapping latency by 2.2× at equivalent semantic quality, maintains upstream bandwidth under 2.5 Mbps, and enables sub-100 ms query latencies even under network drops while supporting tens of thousands of objects within 500 MB. Downstream bandwidth scales with map changes rather than total scene size, and the system adds only 2% device power over idle during normal operation.
when a user asks ”Where are my keys?”, the system can guide them to the keys’ location, even if the keys are not currently in view, and highlight the keys once they come into view. Deploying semantic mapping as a service on XR devices imposes several concurrent requirements. From the algorithm side, because XR devices operate in diverse and previously unseen environments, the semantic layer must be open-vocabulary, supporting recognition beyond fixed categories. From the system side, the system must operate in real time and within strict power budgets to support interactive applications on battery-constrained, all-day wearable devices. The resulting system must deliver high semantic quality, but quality demands vary across applications, creating opportunities to trade off resource usage for semantic coverage and geometric detail. Recent algorithmic work in robotics has advanced openvocabulary semantic mapping by lifting outputs from 2D foundation models into persistent 3D representations [17, 12, 31, 52, 21, 39, 30, 47, 4]. Some of these approaches achieve real-time performance, but they assume server-class GPUs, with compute and power resources well beyond those available on mobile XR devices. No existing system delivers open-vocabulary, real-time semantic mapping within the power constraints of mobile XR devices. One on-device alternative is specialized hardware acceleration, but rapidly evolving foundation model architectures limit the effective deployment lifetime of such accelerators. Cloud offloading offers a more practical path: it leverages powerful server-side GPUs potentially without increasing device power, avoids dependence on custom hardware, and frees on-device resources for other latencysensitive XR tasks [9, 34, 18, 8, 10, 19, 5, 24, 23, 56, 27, 6, 43, 55]. However, how to partition semantic mapping across the device and the cloud under XR constraints remains an open problem. Tab. 1 explores the design space and system requirements. A natural device-cloud split is to perform both mapping and querying on the server, but this leaves the device unable to answer queries during network drops, which are common in mobile XR. An alternative is to perform mapping on the server and maintain a copy of the semantic map on the device for local querying. This restores query availability during network drops, but introduces new costs: updating the device map requires transferring the full scene to the device, causing downstream bandwidth and device memory footprint to grow with scene size. On the server side, existing approaches incur high per-frame mapping latency despite access to powerful GPUs (Sec. 5.1). No single architecture satisfies all system requirements for deploying semantic mapping under XR constraints. The underlying limitation is how existing approaches organize computation. Many fuse semantics into monolithic scene-level representations such as global volumetric maps [52], tying every cost to the total scene size and making them fundamentally incompatible with device-cloud deployment. Others detect and operate on fine-grained entities such as objects [12, 17, 31, 47, 46], but target algorithmic quality rather than system organization. In both cases, existing work specifies how to construct semantic maps, but not how to manage their communication, execution, and memory footprint across a device-cloud boundary. We present SemanticXR, the first end-to-end device-cloud system that enables real-time, open-vocabulary semantic mapping and querying within the power, bandwidth, and memory constraints of mobile XR. Our key insight is to elevate semantically identifiable objects to first-class units of device-cloud system design, governing communication, execution, and memory footprint across the device and the server. This object-level system organization addresses the gaps identified in Tab. 1, and is not tied to a specific foundation model, generalizing across pipelines that produce per-object mapping representations (Sec. 7.1). Tab. 1 summarizes SemanticXR’s object-level system innovations and the system requirements they address. Object-level par-
2 BACKGROUND 2.1 Geometric Mapping and Semantic Mapping Geometric mapping reconstructs a 3D representation of the environment, typically as a mesh, point cloud, or volumetric model, using inputs such as device pose, depth, and RGB frames. These maps support core XR functionalities such as collision detection, occlusion handling, and spatial audio. However, geometric mapping alone cannot assign semantic meaning to the scene; for example, it cannot distinguish or track objects or determine their semantic attributes.
2
Location System Architecture
Mapping
Query
All on-device
Device
Device
Straightforward device-cloud†
Cloud Cloud
Cloud Device
System Requirements Device Power
Real-Time Mapping
Upstream BW
Query under Network Drops
Downstream BW
Device Memory
×
×
N/A
N/A
N/A
×
✓ ✓
∼ ∼
✓ ✓
× ✓
✓ ×
N/A ×
✓
✓
✓
✓ ✓ ✓ ✓
✓ ✓ ✓ ✓
✓ ✓ ✓
Ours: SemanticXR — co-designed device-cloud semantic mapping Key insight: objects as the core system abstraction for communication, execution, and memory footprint Cloud
Cloud + Device
SemanticXR object-level innovations Object-level parallelism Object-level geometry downsampling Object-level depth-mapping co-design Object-level incremental updates Object-level sparse local map Object-level update prioritization Object-level configurable resource usage vs. quality
✓
✓
✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓
✓
✓
✓
✓
Table 1: System constraints and requirements for deploying semantic mapping under XR constraints. No single current architecture satisfies all system requirements. SemanticXR organizes the system around objects as the core abstraction for communication, execution, and memory footprint, enabling object-level innovations (shaded) that collectively address every gap. ∼ = partially meets the requirement. † No existing system implements device-cloud semantic mapping. Many emerging XR applications require understanding not only where surfaces exist, but also what they represent and how they relate to one another. Supporting such applications requires semantic mapping, which augments geometric reconstructions with persistent, queryable, and spatially grounded semantic attributes. Unlike per-frame semantic perception, semantic mapping maintains a persistent map with spatio-temporal consistency, recognizing previously observed objects, associating new observations with existing entities, and updating their attributes in place rather than creating a new map entry for each observation. Recent advances in openvocabulary recognition enable systems to infer a broad and evolving set of object categories. Thus, realizing semantic mapping in XR requires not only accurate semantic inference, but also a system that manages a persistent semantic map under power, bandwidth, and memory constraints.
damentally incompatible with device-cloud deployment. We therefore build on the object-based family of pipelines, which produce discrete per-object representations. Our contributions are not tied to a specific foundation model but apply to any pipeline that produces per-object representations (Sec. 7.1). 2.3.1 Semantic Mapping Flow Fig. 2 illustrates a representative semantic mapping pipeline. At a high level, the pipeline consists of two stages. First, per-frame semantic information is extracted using open-vocabulary models, producing per-object predictions. Second, these predictions are lifted into 3D using depth and camera pose and incrementally associated with existing objects in the map based on spatial and semantic similarity. Transient observations are pruned over time to mitigate noise.
2.2
Foundation Models and Open-Vocabulary Semantic Mapping Foundation models have substantially advanced 2D scene understanding by enabling open-vocabulary recognition and generalization to previously unseen categories. Models for grounded object detection, segmentation, captioning, and vision-language embedding extraction [28, 57, 22, 41, 40, 16, 7, 42] provide strong building blocks for semantic mapping in XR. However, these models produce view-centric 2D predictions and do not natively maintain open-vocabulary semantics over persistent 3D scenes. Recent work addresses this by lifting outputs from 2D foundation models into persistent 3D representations, as described in Sec. 2.3.
RGB Depth
Pose
2D Foundation Models
3D Semantic Mapping
Captioning Segmentation Detection CLIP
Lift to 3D Merge into map
Scene Semantics
Natural Language Query CLIP conversion
App Object Point Cloud
Figure 2: Representative semantic mapping pipeline using 2D foundation models. Per-frame semantic predictions are lifted into 3D using depth and pose, aggregated into a semantic map, and exposed through a query interface. 2.3.2 Querying the Semantic Map Once a semantic map is constructed, users can issue textual queries to retrieve relevant objects from the scene (Fig. 2). The system matches a semantic embedding of the query text against per-object descriptors (e.g., using CLIP embeddings and cosine similarity) and returns the best-matching objects along with their 3D representations. Because queries operate over the persistent semantic map, their cost depends on how that map is maintained and, in devicecloud settings, how much of it must be retained on the device.
2.3 Semantic Mapping using 2D Foundation Models Semantic mapping using 2D foundation models has been explored in several prior works [17, 12, 52, 30, 50, 4, 47, 53]. These approaches differ in how they structure semantic information. Some embed 2D region semantics into a monolithic 3D representation [52, 59, 58], tying all costs to total scene size. Others organize semantics around identifiable objects [17, 12], producing discrete per-object representations. Both families target mapping quality rather than system organization: neither addresses how the semantic map is managed across a device-cloud boundary. All the above approaches rely on computationally intensive foundation models, requiring server-class GPUs well beyond the power budget of mobile XR devices. As discussed in Sec. 1, monolithic representations that tie every cost to total scene size are fun-
3 S EMANTIC XR SemanticXR builds on the object-based semantic mapping pipelines described in Sec. 2.3 and illustrated in Fig. 2. As discussed, these approaches are largely algorithmic: they target mapping quality on powerful GPUs and are oblivious to the power, bandwidth, and memory constraints of XR devices. As summarized
3
3.2
Query Under Network Drops: Downstream Bandwidth and Device Memory Footprint Interactive semantic queries in XR must remain responsive even during network drops. SemanticXR supports two query modes: Server Querying (SemanticXR-SQ), where queries are evaluated against the full server-side semantic map, and Local Querying (SemanticXR-LQ), where queries are executed on the device using a local semantic map. Enabling SemanticXR-LQ requires maintaining a semantic map on the device, but a full copy of the server map is impractical: device memory would grow with scene size, and keeping the local map updated would require transferring the full map, causing downstream bandwidth to grow with scene size as well. SemanticXR addresses both constraints through three objectlevel innovations. Object-level sparse local map. SemanticXR maintains a local semantic map on the device organized as a collection of per-object entries. Each entry stores a semantic embedding for query matching and a point cloud further downsampled from the server-side representation (Sec. 3.1) to fit device memory constraints; downsampling thus reduces a retrieved object’s geometric detail, not which objects a query retrieves, so query accuracy is unaffected and the retained geometry remains sufficient for spatial localization. Because each object’s geometry is capped at a configurable point budget rather than stored as a dense scene-wide reconstruction, per-object memory is fixed and total device memory grows only with the number of retained objects, not with scene complexity. An application visible system parameter controls the point budget per object, enabling applications and the system to adjust the memory–quality trade-off (Tab. 2). The number of retained objects is further bounded by update prioritization, described below. Object-level incremental updates. Updates to the local map are transmitted as object-level incremental updates. Rather than transferring the full semantic map, the server sends only newly created or modified objects (observed from a different angle). As a result, downstream bandwidth is proportional to the number of changed objects rather than the total scene size. Updates are issued periodically and only after an object has been consistently observed across multiple frames, filtering out transient detections before they propagate to the device. A system parameter controls update frequency, enabling applications and the system to balance map freshness against downstream bandwidth (Tab. 2). Object-level update prioritization. Object-level update prioritization determines which objects are maintained on the device based on semantic relevance, spatial proximity, and application-declared priority classes (for example, task-relevant categories, nearby objects, or distant landmarks), limiting the local map to the most relevant content and reducing device memory and downstream bandwidth. When the local map reaches its memory budget, admitting a higher-priority update evicts the lowest-priority retained objects, keeping the map within budget. The object-level abstraction makes richer replacement policies straightforward to add in the future. Query mode switching. During network drops, pending updates are buffered on the server and applied upon reconnection. Network quality is monitored using latency and transmission error signals from the RGB-D stream. When network latency exceeds a configurable threshold, the system switches from SemanticXR-SQ to SemanticXR-LQ. SemanticXR-LQ may therefore operate on a slightly stale state, but staleness is bounded by the most recent successful update.
in Tab. 1, no existing approach addresses how to manage communication, execution, and memory footprint across a device-cloud boundary. SemanticXR is a device-cloud system that bridges this gap, providing real-time, open-vocabulary semantic mapping and querying for low-power XR devices. The key insight of SemanticXR is to elevate semantically identifiable objects to first-class units of device-cloud system design, governing how semantic information is communicated, executed, and stored across the device and the server. A map object is identified by a stable object ID and consists of a semantic embedding, a class label, and a 3D point cloud. This abstraction generalizes across pipelines that produce per-object representations. As illustrated in Fig. 1, the XR device streams synchronized RGB and depth frames of the user’s physical space along with the associated device pose to the server, where the semantic mapping pipeline (Sec. 2.3) detects objects, extracts semantic embeddings, and incrementally associates observations with existing objects and adds any freshly observed objects to the map. When the network is available, queries (Sec. 2.3.2) are evaluated against the full serverside map. During a network outage, the device falls back to a local semantic map maintained through object-level updates from the server. This object-level system organization enables the innovations summarized in Tab. 1 and detailed in the following subsections: object-level parallelism and geometry downsampling for improved server-side mapping latency (Sec. 3.1), an object-level sparse local map with incremental updates and update prioritization for network-robust querying with bounded device memory and downstream bandwidth (Sec. 3.2), depth-mapping co-design for lightweight upstream bandwidth reduction (Sec. 3.3), and perobject configurable resource usage vs. quality trade-offs that address communication, execution, and memory footprint (Sec. 3.4). 3.1
Mapping Latency
Open-vocabulary semantic mapping composes multiple foundation models followed by incremental 3D association, resulting in substantial server-side computational load. When execution is organized at frame or scene granularity, all computation is treated uniformly regardless of per-object variation in size and complexity, limiting opportunities for parallelism and missing potential efficiency gains available from this variability. Object-level parallelism. SemanticXR structures server-side execution at object granularity. After object proposals are generated for a frame, subsequent processing is performed independently for each detected object (Fig. 1). By moving away from a frame-level execution, segmentation and vision-language feature extraction are parallelized across objects within a frame, improving GPU utilization and reducing per-frame processing latency. Object-level geometry downsampling. Projecting objects into 3D produces highly variable per-object point cloud sizes, depending on the size of the object, leading to disproportionate computational cost. Per-object geometry is needed to place each object in 3D, both for associating and merging observations by spatial proximity and for returning an object’s location in response to queries. Because object association and merging depend on spatial proximity and semantic similarity but not on high-fidelity geometric detail, SemanticXR caps the number of points per object through geometry downsampling, bounding per-object computation and improving mapping latency without degrading semantic quality. A system parameter controls this bound, enabling applications to adjust the trade-off between geometric detail and mapping latency (Tab. 2).
3.3 Upstream Bandwidth Offloading semantic mapping requires the XR device to transmit synchronized RGB and depth frames of the user’s physical space, along with the associated device pose, to the server. While RGB can be efficiently compressed using hardware video encoders available on mobile XR devices, depth is typically produced as high-precision frames that cannot leverage the same hardware di-
Together, object-level parallelism and geometry downsampling improve server-side mapping latency. We evaluate the resulting improvements in Sec. 5.1.
4
full scene updates to support local querying, rather than incremental and sparse object-level transfers. Both SemanticXR and the device-cloud baseline transmit downsampled depth; the effect of depth-mapping co-design on upstream bandwidth is studied independently in Sec. 5.5. Any observed differences are therefore attributable solely to system organization. As shown in Sec. 5.1, this device-cloud baseline already matches or exceeds prior open-vocabulary mapping approaches in semantic quality while achieving lower mapping latency on the evaluated (Replica) dataset [45].
rectly [36]. While prior work has explored depth compression [36], SemanticXR instead asks a co-design question: how much can depth be approximated while preserving semantic quality? This leads to a lightweight alternative – downsampling the depth frames prior to transmission and mitigating quality loss through per-object mapping decisions. Object-level depth-mapping co-design. While depth downsampling substantially lowers transmission cost, it can degrade geometric detail if applied uniformly. Because the semantic map is organized around discrete objects, mapping decisions are made per object rather than per frame. Objects that occupy sufficient image area retain reliable depth even after downsampling and are incorporated immediately, while smaller or distant objects are deferred until additional observations improve depth reliability. A system parameter controls the minimum object area required for incorporating an observation, enabling applications to adjust the trade-off between upstream bandwidth and semantic quality (Tab. 2). We evaluate this trade-off in Sec. 5.5.
4.3 Evaluation System Setup Our evaluation requires fine-grained power instrumentation and controlled low-power configurations to characterize system behavior under XR constraints. Commercial XR headsets do not expose these capabilities, limiting their suitability for evaluating powersensitive system design. Therefore, for the bulk of our work, following prior XR systems work [15, 18, 9], we use an NVIDIA Jetson Orin configured in low-power mode as a proxy XR device. The platform provides embedded GPU compute, configurable power envelopes, and detailed system instrumentation. The configuration in Tab. 3 emulates a modern mobile XR headset by limiting GPU TPC count, operating frequencies, and power envelope to levels representative of all-day wearable devices. Semantic mapping and query execution run on a dedicated server equipped with an AMD Ryzen Threadripper 7960X CPU and an NVIDIA RTX 6000 Ada GPU. This configuration represents a contemporary cloud-class server and is held fixed across all experiments. To study the impact of network conditions on system behavior, we evaluate SemanticXR under three network configurations: (1) a low-latency network with an average round-trip time (RTT) of approximately 20 ms, (2) degraded connectivity with ∼66 ms RTT, and (3) complete network outage to evaluate local querying (Sec. 3.2). These settings are used consistently across experiments unless otherwise noted.
3.4
Object-Level Configurable Resource Usage vs. Quality XR applications impose diverse requirements on semantic maps; e.g., some require broad scene coverage while others prioritize specific object categories or regions. Because SemanticXR organizes all system operations at object granularity, applications can trade off resource usage, including compute, memory, and bandwidth, against semantic quality per object or per category, without modifying the underlying mapping pipeline. Tab. 2 summarizes the exposed parameters and their system-level effects. This configurability cuts across all system operations: applications can adjust geometric detail vs. mapping latency via geometry downsampling (Sec. 3.1), control local map update frequency vs. downstream bandwidth (Sec. 3.2), and reduce upstream bandwidth via depthmapping co-design while preserving semantic quality (Sec. 3.3). 4 E VALUATION M ETHODOLOGY 4.1 SemanticXR Implementation Details We evaluate SemanticXR using the object-based semantic mapping pipeline described in Sec. 2.3. Per-object semantic observations are generated using off-the-shelf zero-shot models for captioning (RAM [57]), object detection (Grounding DINO [28]), instance segmentation (MobileSAM [54]), and vision-language embedding (MobileCLIP [48]). The device transmits RGB, depth, and pose to the server (Sec. 3.3), where all semantic mapping is performed. During network drops, the device executes queries against its local semantic map (Sec. 3.2). We use a fixed configuration (Tab. 2) except when studying a specific tunable parameter (e.g., depth downsampling in Sec. 5.5).
4.4 Dataset and Evaluation Scope We use the Replica dataset for evaluation as adopted by other prior semantic mapping works [12, 31]. Replica consists of diverse indoor room and office scenes and provides synchronized RGB, depth, and camera pose sequences with ground-truth semantic labels, enabling controlled and reproducible evaluation of objectlevel semantic queries. Our evaluation focuses on indoor scenes. Outdoor environments typically require different perception models, but since SemanticXR does not assume specific model architectures, the same system design applies when paired with suitable outdoor models. We leave empirical validation of outdoor deployment to future work. To demonstrate feasibility on commodity hardware with real sensors, we additionally deploy SemanticXR on an iPad-based prototype. This deployment is presented as an end-to-end system demonstration rather than a performance evaluation and is discussed separately in Sec. 6.
4.2 Device-Cloud Baseline and Evaluation Goals Our evaluation characterizes the effect of object-level system organization on device-cloud deployment under XR constraints. Since no existing system implements device-cloud semantic mapping, we construct a controlled and competitive device-cloud baseline to isolate this effect. Splitting semantic mapping requires managing communication, execution, and memory footprint across the device-cloud boundary. Monolithic approaches that fuse semantics into global representations [52] tie every cost to total scene size, incurring prohibitive overhead. Other pipelines [12, 17, 31, 47, 46] produce discrete per-object map representations, which are a prerequisite for the object-level system organization introduced by SemanticXR. Our device-cloud baseline follows this family (Fig. 2) and uses identical perception models and mapping pipeline as SemanticXR (Sec. 4.1) but does not organize system operations at object granularity. The server processes frames without object-level parallelism or geometry downsampling. The device receives periodic
4.5 Evaluation Metrics Our evaluation quantifies how object-level system organization influences device-cloud execution efficiency, and characterizes SemanticXR’s robustness to network drops. To isolate the impact of our system design, specifically whether semantic state is managed at scene or object granularity, all experiments hold the semantic mapping pipeline constant (Sec. 4.1). We evaluate SemanticXR along dimensions that correspond to the system requirements identified in Tab. 1: device power consumption, server-side mapping latency, upstream bandwidth, query latency under network outage, downstream bandwidth, device memory, and the impact of object-level system organization on semantic quality. These metrics collectively characterize how
5
Configurable Trade-off
Exposed System Knobs
Effect and Trade-off
Default
Query latency vs. device power
net latency switch threshold Switches between SemanticXR-SQ and SemanticXR-LQ; trades device power for latency.
Object class mapping policy
skip mapping set, max object points server
Prioritizes selected object classes for detailed mapping; others are skipped or sparsified to reduce compute and downstream bandwidth.
Empty, 2000 pts
Local map geometric detail vs. memory
max object points client, max object points server
Enables denser local geometry for selected classes; trades memory footprint for geometric detail.
200 pts, 2000 pts
Local map freshness vs. downstream bandwidth
local map update frequency
Controls the frequency of incremental local map updates, trades map freshness for downstream bandwidth.
Every 2 frames
Upstream bandwidth budget
min mapping bbox area, depth downsampling ratio
Filters small objects from mapping based on projected size, enabling depth downsampling and reduced upstream load under bandwidth constraints.
2000 px and 5×downsampling
100 ms
Table 2: Application configurable trade-offs and corresponding system-level tuning knobs in SemanticXR. Defaults correspond to the configuration used in SemanticXR. System Parameters CPU Frequency (MHz) GPU Frequency (MHz) CPU count GPU TPC count Idle Power Power Cap
Low Power 1728 1020 8 3 8.6W 20W
MAXN 2202 1301 12 8 10W 60W
4.5.3
Device Memory and Query Latency under Network Outage
To evaluate how the local semantic map scales with scene complexity, we measure device memory and query latency as a function of the number of objects stored on the device. Synthetic semantic maps are constructed by incrementally inserting object point clouds with associated vision-language embeddings. We evaluate map sizes ranging from 80 objects (comparable to room-scale Replica scenes) to 1,000, 5,000, and 10,000 objects, representing increasingly large indoor environments. We additionally report results for 25,000 and 50,000 objects to characterize extreme cases. Results are discussed in Sec. 5.3.
Table 3: Jetson Power Configurations: Low power mode emulates a modern XR headset. MAXN represents the maximum power setting supported by Jetson Orin. Low power mode caps power at 20W while MAXN can go up to 60W. Idle power in both cases is high. SemanticXR’s object-level system organization addresses the requirements of device-cloud semantic mapping (Sec. 5).
4.5.4
Network Bandwidth Usage
We measure upstream bandwidth as the average data rate required to stream RGB, depth, and pose from the XR device to the server, accounting for RGB compression and configurable depth resolution. Downstream bandwidth measures the volume of semantic map data transferred during map synchronization. The resulting tradeoffs are analyzed in Secs. 5.4 and 5.5.
4.5.1 Latency: Semantic Mapping and Queries Semantic mapping latency measures the time required by the server to process a single RGB, depth, and pose frame and update the semantic map. This includes semantic perception using foundation models as well as subsequent object-level map fusion. To report throughput (Frames Per Second), we follow standard practice from the literature [47, 37, 38]: semantic maps are incrementally refined and do not require updates for every input frame, so we sample keyframes at a fixed interval and report throughput as total input frames divided by total keyframe processing time. We use a keyframe interval of 5; prior work commonly uses intervals of 10 or higher [47, 38]. Query latency measures the time required to process a text query and retrieve matching objects from the semantic map (Sec. 2.3.2). This includes text embedding extraction and similarity computation against stored object embeddings. For server-side queries, we additionally include the network round-trip time required to transmit the query and return the resulting object geometry.
4.5.5
XR Device Power Consumption
We measure power consumption on the XR client device using tegrastats [35], sampled at 1 ms intervals. Tegrastats reports power across major subsystems, including CPU, GPU, SoC, DRAM, and system I/O. All reported power measurements include the device’s idle power, which is approximately 8.6 W on the Jetson Orin in low-power mode. We evaluate XR device power consumption under three configurations that characterize SemanticXR’s operating regimes: 1. On-device semantic mapping and querying: Measures total power when the semantic mapping pipeline executes entirely on the device in maximum power mode. This establishes the power cost that SemanticXR’s cloud offloading avoids. 2. SemanticXR under network outage and heavy query rate: Measures device power when semantic mapping is offloaded to the server and the device executes local queries at one query every three seconds, emulating a plausible but heavy query rate. 3. SemanticXR under continuous local querying: Although users do not issue queries continuously, some applications may generate short bursts of quick queries. We therefore measure power consumption under continuous local querying to characterize worst-case peak power on the XR device, representing (unrealistic) sustained querying during network outage.
4.5.2 Semantic Quality We evaluate semantic quality by measuring the accuracy of object retrieval from the semantic map given natural-language text queries. We follow the evaluation methodology of [17] on the Replica dataset [45], which provides ground-truth semantic labels and object annotations. Ground-truth labels are used to generate text queries that are issued against the semantic map constructed by SemanticXR. Retrieved object point clouds are compared against ground-truth using mean class recall (mAcc) and frequency-weighted mean Intersection-over-Union (F-mIoU). These metrics are standard in prior work [17, 12, 31].
6
5 R ESULTS This section evaluates the object-level system innovations introduced by SemanticXR (Tab. 1) along the evaluation dimensions defined in Sec. 4.5. All experiments compare SemanticXR against the device-cloud baseline described in Sec. 4.2; observed differences are attributable to system organization rather than algorithmic or model choice. We first establish that the device-cloud baseline is competitive with prior work in mapping latency and semantic quality. We then evaluate how object-level system organization affects server-side mapping latency (Sec. 5.1), query latency under varying network conditions (Sec. 5.2), device memory and query latency for large local maps (Sec. 5.3), downstream bandwidth (Sec. 5.4), upstream bandwidth and its interaction with semantic quality (Sec. 5.5), and device power consumption (Sec. 5.6).
ConceptFusion [17] ConceptGraphs [12] OpenMask3D [46] Clio (batch)† [31] Clio (online)‡ [31] Our device-cloud baseline SemanticXR
mAcc
F-mIOU
Latency (s/frame) ↓
24.16 38.72 39.54 37.95 N/R (bad)‡
31.31 35.82 49.26 36.98 N/R (bad)‡
Offline Offline Offline Offline ∼0.30*
37.0 37.6
47.23 48.29
0.57 0.26
Processing Latency (ms)
600 500 400 300 200
100 0
B
B
P SD
Baseline
P
B
P SD
B
P SD
B
P SD
B
Object-level Parallelization
P SD
B
P SD
B
P SD
B
P SD
SD Object-level Geometry Downsampling
Figure 3: Server-side semantic mapping latency across 8 scenes from the Replica dataset. For each scene, three bars show cumulative configurations: device-cloud baseline (B), B + object-level parallelism (P), and B + P + object-level geometry downsampling (SD). Each bar is decomposed by pipeline stage.
5.1 Semantic Mapping Latency and Quality Tab. 4 compares semantic quality and per-frame mapping latency across prior open-vocabulary mapping approaches, our devicecloud baseline, and SemanticXR. Most prior approaches are offline and require seconds to minutes per frame. Our device-cloud baseline exceeds all prior approaches in semantic quality except OpenMask3D, while being the only system that operates in real time. Clio-online is the only other real-time approach, but the authors report that quality degrades significantly without offline postprocessing, and do not evaluate quality in online mode on Replica. Method
Mapping CLIP Segmentation Detection Caption
700
Room 0 Room 1 Room 2 Office 0 Office 1 Office 2 Office 3 Office 4 Server Query Processing
Network Latency Fast
Local Query Latency
Network Latency Slow
Figure 4: Average query latency for server-side queries (SemanticXR-SQ), decomposed into server processing and network latency under two network conditions, and for local queries (SemanticXR-LQ) across Replica scenes. Minor variations across scenes arise from differences in map size and object density.
Table 4: Open-set 3D semantic segmentation accuracy and perframe mapping latency on eight Replica [45] scenes. Our devicecloud baseline is defined in Sec. 4.2. † Clio-batch reports quality after offline post-processing, which the authors note is required for adequate quality on Replica. ‡ N/R–Not reported. Clio-online does not report quality metrics; the authors note that quality degrades significantly even compared to Clio-batch without post-processing. * Reported on different hardware; we do not explore further since quality is not acceptable. Fig. 3 shows how object-level parallelism and geometry downsampling reduce per-frame mapping latency over this already competitive device-cloud baseline. Object-level parallelism enables concurrent execution of segmentation and vision-language feature extraction across objects, while geometry downsampling bounds per-object point cloud size. Together, average per-frame mapping latency drops from approximately 570 ms to approximately 260 ms, a 2.2× improvement. Under the throughput methodology described in Sec. 4.5, SemanticXR achieves approximately 20 FPS compared to approximately 8.75 FPS for the device-cloud baseline, at equivalent semantic quality (Tab. 4).
compute capability for text embedding, while reducing XR device power consumption (Sec. 5.6). Under higher-latency network conditions, network delay dominates server-side latency and introduces significant variability, causing SemanticXR-SQ to approach or slightly exceed SemanticXR-LQ latency. In contrast, SemanticXR-LQ remains stable across network conditions because it operates entirely on the device using the objectlevel sparse local map. This stability comes at the cost of higher device power consumption than SemanticXR-SQ. We next examine how local query performance scales as the local semantic map grows. 5.3
Device Memory and Query Latency for Large Local Maps We evaluate device memory footprint and local query latency as the number of objects in the local semantic map increases. Fig. 5 shows both metrics for progressively larger synthetic maps ranging from 80 objects (comparable to room-scale Replica scenes) to 50,000 objects. Because SemanticXR’s object-level sparse local map caps per-object geometry at a configurable point budget rather than storing a dense scene-wide reconstruction (Sec. 3.2), per-object memory is bounded and total device memory grows with the number of retained objects rather than scene complexity. SemanticXR supports local maps containing up to 50,000 objects within a 500 MB memory footprint; associated resource-quality trade-offs are configurable via the parameters in Tab. 2. Local query latency consists of two components: text embedding extraction, which is independent of map size, and similarity computation against stored per-object embeddings, which grows with the number of objects. For maps containing up to 10,000 objects, end-to-end local query latency remains below 100 ms, en-
5.2 Query Latencies: Local and Server Fig. 4 compares average query latency for server-side querying (SemanticXR-SQ) and local querying (SemanticXR-LQ) under two network conditions: a low-latency network and a higher-latency network, evaluated across multiple Replica scenes. Server-side query latency includes server computation and network round-trip time, while local query latency includes only on-device text embedding and similarity computation against the object-level sparse local map (Sec. 3.2). Under low-latency network conditions, SemanticXR-SQ achieves lower end-to-end latency due to the server’s higher
7
250
500
200
400
150
300
100
200
50
100
0
80
1000
5000
10000
25000
50000
0
by 5× in each spatial dimension (25× overall) decreases upstream bandwidth by approximately 90% while causing only a small drop in quality (F-mIoU). Further reductions yield diminishing bandwidth savings while increasingly deferring object integration, as fewer objects meet the minimum area threshold per frame.
Memory Overhead (MB)
Local Query Latency (ms)
abling network-independent querying at interactive latency. Beyond 10,000 objects, query latency increases but remains practical for large indoor environments.
Depth Down-sampling No Down-sampling 2× row, 2× column (4×) 3× row, 3× column (9×) 4× row, 4× column (16×) 5× row, 5× column (25×)
Quality (F-mIoU) 48.29 45.5 44.26 45.73 45.81
Table 5: Upstream bandwidth and semantic query quality (F-mIoU) under different depth resolutions. A 5× reduction in each spatial dimension reduces bandwidth by approximately 90% with small impact on semantic query accuracy and is used as the default configuration.
Number of Synthetic Objects CLIP Embedding Latency Postprocessing Memory Overhead (MB)
Figure 5: Local query latency and memory footprint as a function of number of synthetic objects. Here postprocessing denotes perobject similarity computation.
5.6 XR Device Power Consumption Fig. 7 reports XR device power consumption under the three configurations defined in Sec. 4.5. Running the full semantic mapping pipeline on-device requires approximately 50 W and takes several seconds for mapping a single frame, confirming that on-device mapping is impractical within XR power budgets and motivating SemanticXR’s cloud offloading. Under normal operation with server-side mapping and SemanticXR-SQ, the device transmits RGB, depth, and pose and receives query results; semantic compute remains on the server. The resulting device power is approximately 8.7 W, an increase of only ∼2% over the 8.6 W idle baseline (Tab. 3). Under network outage, the device falls back to SemanticXR-LQ on the object-level sparse local map (Sec. 3.2). Executing local queries at a heavy rate of one query every three seconds increases average power to 9.8 W, an incremental cost of approximately 1.2 W over idle. Because SemanticXR-LQ evaluates similarity against per-object embeddings rather than a dense scene representation, the per-query compute cost remains modest. To characterize worst-case behavior, we additionally measure power under continuous query execution at the maximum achievable rate of 14.7 queries per second. Peak power reaches 13.23 W, an increase of approximately 4.6 W over idle, remaining within the power envelope of contemporary XR devices.
5.4 Downstream Bandwidth The device-cloud baseline transmits the full semantic map to the device on every update, causing downstream bandwidth to grow with the total number of objects in the scene regardless of how many have changed. In contrast, SemanticXR transmits object-level incremental updates (Sec. 3.2), sending only newly created or modified objects. Fig. 6 shows this effect. Downstream bandwidth for the devicecloud baseline grows proportionally with the number of mapped objects, as each update transfers the full scene. SemanticXR instead transfers only the changed object set, so per-update downstream bandwidth is proportional to the number of changed objects rather than total scene size.1 As exploration converges and fewer new objects are discovered, SemanticXR’s per-update cost decreases further, whereas the device-cloud baseline’s cost remains at its plateau. 2 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0
Max Power Mapping Latency: several seconds
1
11
21
31
41
51
61
Avg. Power Consumption (W)
Update Sizes (Mb)
Upstream BW (Mbps) 26.4 7.72 4.26 3.06 2.5
71
Update IDs Incremental Updates
Full Scene Updates
Figure 6: Per-update downstream transfer size as a function of update index for a Replica scene. The device-cloud baseline transfers the full scene on each update, so transfer size grows with total mapped objects until the scene is fully explored (plateau). SemanticXR’s object-level incremental updates transfer only changed objects, so per-update cost remains small and decreases as exploration converges. 5.5 Upstream Bandwidth As discussed in Sec. 3.3, SemanticXR explores a co-design question: how aggressively can depth be downsampled before transmission while preserving semantic quality? Rather than applying compression [36], SemanticXR downsamples the depth frame and mitigates the resulting geometric loss through per-object mapping decisions that defer objects with unreliable depth until additional observations are available. Tab. 5 reports the resulting upstream bandwidth and semantic quality when varying depth downsampling ratio and minimum object area (Tab. 2). Reducing depth resolution
50 45 40 35 30 25 20 15 10 5 0
Local Mapping and Query
Idle power
Server Mapping + Local Query (every 3 seconds)
Continuous Local Query
Figure 7: XR device power consumption on Jetson under different execution modes: full local semantic mapping, idle operation, local queries at one query every three seconds, and continuous local queries illustrating peak power under bursty demand. 5.7 Summary Across all evaluation dimensions (Tab. 1 and Sec. 4.5), object-level system organization improves server-side mapping latency by 2.2× at equivalent quality, enables sub-100 ms local queries for up to 10,000 objects within 500 MB, decouples downstream bandwidth from scene size, reduces upstream bandwidth by ∼90% with minor quality loss, and adds only ∼2% device power under normal operation. 6 D EPLOYMENT C ASE S TUDY: I PAD - BASED P ROTOTYPE To validate SemanticXR with real sensors, we implemented a prototype deployment on a consumer mobile XR device to demonstrate
1 Both systems apply geometry downsampling in this experiment; with-
out it, the device-cloud baseline’s per-update transfer size would be larger.
8
3D RGB point cloud on the server
3D semantic map on the server
7.2 Query Scope SemanticXR is evaluated on object-grounding queries, which retrieve the scene objects that best match a natural-language description. The object-centric organization extends to other query classes through targeted additions rather than a redesign: multi-object and relational queries via an external reasoning agent operating over the object map, affordance queries via additional per-object attributes, and free-space or global-layout queries via complementary representations such as occupancy.
(a) Semantic mapping: Left: Reconstructed color point cloud on the server. Right: 3D semantic map represented as object-based point cloud.
7.3 Relationship to Model-Level Optimization Model-level optimization, such as quantization, pruning, and distillation, is orthogonal to SemanticXR’s object-level system organization. Because the foundation models run on the server, such optimization would improve server-side mapping efficiency and concurrent-client capacity rather than device power, which offloading already holds near idle (about 2%, Sec. 5.6). The two are complementary: model optimization lowers the cost of each model, while object-level organization governs how that cost is communicated and stored across the device-cloud boundary.
iPad client query and display
Point cloud overlayed on the keyboard
7.4 Future Work Future directions include server-side scalability to multiple concurrent devices, support for dynamic environments with moving objects and multi-user collaborative mapping, and extending the object-level interface to richer geometric representations such as per-object meshes with non-uniform detail. SemanticXR’s configurable knobs (Sec. 3.4) also motivate autotuning and dynamic control, i.e., automatically adapting resource-quality parameters to application and network conditions instead of manual configuration. Agentic integration is another direction: humans could issue complex spatial tasks that an agent resolves by issuing spatial queries against SemanticXR’s map and acting on the returned objects. Privacy-aware design for device-cloud XR systems remains an important open problem. Evaluation on commercial XR headsets and outdoor environments would further validate deployment generality.
User Query: Keyboard
(b) iPad client view: The user presses the mic button to submit a voice query for a keyboard. The iPad receives the keyboard point cloud and overlays it on the physical keyboard.
Figure 8: SemanticXR deployed end-to-end with an iPad client: (a) overview of semantic 3D scene mapping; (b) query results on the iPad client. end-to-end feasibility. We deploy the client on an Apple iPad Pro [1], which provides synchronized RGB, depth, and pose streams via ARKit [2]. Current commercial XR headsets such as the Apple Vision Pro do not expose raw sensor streams required for semantic mapping, making the iPad a practical proxy for a mobile XR sensing stack. We deploy a custom ARKit client that captures RGB, depth, and pose at 30 FPS and streams a subset of frames to the server to match semantic mapping throughput. RGB frames (720 × 1280) are compressed using H.264 at 5 Mbps, and transmitted with low resolution depth maps (144 × 256) following the depth-mapping design in Sec. 3.3. The system uses SemanticXR’s tunable interface to adjust frame transmission and mapping parameters under varying scene conditions. For spatial queries, the client performs on-device speech-to-text and forwards textual queries to the server via gRPC. The server returns object-level point clouds that the client renders as worldaligned overlays in ARKit. Although the iPad does not expose fine-grained power telemetry, this deployment demonstrates end-to-end operation with real sensors, mobile hardware, and interactive user queries. Fig. 8 shows the deployed system: the server incrementally constructs the semantic map (Fig. 8a), and the iPad client overlays queried objects in the user’s view (Fig. 8b).
8 R ELATED W ORK Prior work has studied semantic mapping primarily as an algorithmic perception problem, while cloud offloading in XR has largely focused on rendering or isolated perception tasks. In contrast, deploying semantic mapping as a low-power, long-running XR service requires managing communication, execution, and memory footprint across a device-cloud boundary, a problem that existing approaches do not address. 8.1 Semantic Mapping in Robotics and XR A large body of work has explored open-vocabulary semantic mapping by embedding image- or text-derived semantics into 3D representations. Representative systems include Clio [31], SpatialLM [32], OpenFusion [52], ConceptFusion [17], ConceptGraph [12], and One Map to Find Them All [4]. These approaches differ in perception models, representations, and update strategies, and collectively demonstrate the feasibility of constructing semantic maps from RGB-D observations. Many support online RGB-D processing [31, 52], while others operate offline over preprocessed point clouds. However, these systems target algorithmic quality under server-class hardware assumptions, whether through taskdriven scene representations [31] or monolithic scene-level representations [52, 50], rather than system organization for devicecloud deployment under XR constraints. Unlike these open-vocabulary systems, a related line of robotics work builds real-time hierarchical 3D scene graphs from closedvocabulary perception, including Hydra [13, 14], and SceneGraphFusion [49]. Because this perception is lightweight, these systems run on a single machine and do not address the device-
7 D ISCUSSION 7.1 Applicability Across Semantic Mapping Pipelines SemanticXR’s system innovations (Tab. 1) are not tied to a specific perception model; they require only that the backend produce persistent, discrete object-level semantic state that can be incrementally maintained. Backends that already maintain per-object representations (e.g., [12, 17, 31, 47]) can adopt SemanticXR’s abstractions directly by promoting objects to independently managed system state. Pipelines that aggregate object inference into monolithic scene representations [52, 50] must first expose persistent object entities, requiring more substantial restructuring.
9
vice and the server. This object-level system organization enables a family of innovations (Tab. 1) that collectively address the requirements of XR device-cloud semantic mapping: object-level parallelism and geometry downsampling improve server-side mapping latency by 2.2× at equivalent semantic quality, object-level depthmapping co-design reduces upstream bandwidth by ∼90% with a small quality loss, and an object-level sparse local map with incremental updates and update prioritization enables sub-100 ms query latency for up to 10,000 objects within 500 MB, with downstream bandwidth proportional to map changes rather than total scene size. The system adds only ∼2% device power during normal operation. Because SemanticXR operates on object-level semantic state rather than model-specific internals, its system innovations generalize across any semantic mapping pipeline that produces persistent per-object representations.
cloud split that foundation-model perception forces on mobile XR. Clio [31] extends this scene-graph line toward open-vocabulary, task-driven mapping and is the closest comparison among these systems (Tab. 4). Separately, 3D instance segmentation methods [46, 51, 47, 30] focus on geometric shape decomposition rather than persistent, queryable semantic state and do not target devicecloud deployment. In contrast, SemanticXR treats existing semantic mapping pipelines as backends and elevates objects to first-class units of device-cloud system design, governing communication, execution, and memory footprint across the device-cloud boundary (Tab. 1). 8.2 VLM-based Assistants and Scene Understanding Recent XR systems have explored natural language interaction using VLM-based assistants such as Google’s Project Astra [11] and XaiR [44]. These systems reason over visual observations together with short-term textual or embedding-based memory [11], or attach per-frame features to pre-built geometry without distinguishing individual objects [44]. Neither builds a persistent, queryable 3D semantic map with explicit object identities. SemanticXR addresses an orthogonal problem: how to build and manage such a map across a device-cloud boundary under the power, bandwidth, and memory constraints of mobile XR.
R EFERENCES [1] Apple Inc. Apple iPad Pro, 2020. Tablet device with LiDAR sensor. 9 [2] Apple Inc. Apple ARKit, 2023. Apple’s AR Developer tool. 9 [3] A. Behroozi, Y. Chen, V. Fruchter, L. Subramanian, S. Srikanth, and S. Mahlke. Slimslam: An adaptive runtime for visual-inertial simultaneous localization and mapping. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS ’24, p. 900–915. Association for Computing Machinery, New York, NY, USA, 2024. doi: 10.1145/3620666.3651361 10 [4] F. L. Busch, T. Homberger, J. Ortega-Peimbert, Q. Yang, and O. Andersson. One map to find them all: Real-time open-vocabulary mapping for zero-shot multi-object navigation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 14835–14842, 2025. doi: 10.1109/ICRA55743.2025.11128393 2, 3, 9 [5] G. Chen, F. H’acha, L. V’aša, and M. Dasari. Tvmc: Timevarying mesh compression using volume-tracked reference meshes. In Proceedings of the 16th ACM Multimedia Systems Conference, MMSys ’25, p. 79–89. Association for Computing Machinery, New York, NY, USA, 2025. doi: 10.1145/3712676.3714440 2 [6] K. Chen, T. Li, H.-S. Kim, D. E. Culler, and R. H. Katz. Marvel: Enabling mobile augmented reality with low energy and low latency. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, SenSys ’18, p. 292–304. Association for Computing Machinery, New York, NY, USA, 2018. doi: 10.1145/3274783. 3274834 2, 10 [7] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818–2829, 2023. 3 [8] A. Dhakal, X. Ran, Y. Wang, J. Chen, and K. K. Ramakrishnan. Slamshare: visual simultaneous localization and mapping for real-time multi-user augmented reality. In Proceedings of the 18th International Conference on Emerging Networking EXperiments and Technologies, CoNEXT ’22, p. 293–306. Association for Computing Machinery, New York, NY, USA, 2022. doi: 10.1145/3555050.3569142 2, 10 [9] S. Gao, J. Liu, Q. Jiang, F. Sinclair, W. Sentosa, B. Godfrey, and S. Adve. Xrgo: Design and evaluation of rendering offload for lowpower extended reality devices. In Proceedings of the 16th ACM Multimedia Systems Conference, MMSys ’25, p. 124–135. Association for Computing Machinery, New York, NY, USA, 2025. doi: 10. 1145/3712676.3714444 2, 5, 10 [10] Google. Immersive stream for xr overview. Streaming for extended reality. 2 [11] Google. Project astra, 2024. Google’s universal AR AI agent. 10 [12] Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, C. Gan, C. M. de Melo, J. B. Tenenbaum, A. Torralba, F. Shkurti, and L. Paull. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5021–5028, 2024. doi: 10.1109/ICRA57147
8.3 Cloud Offloading in XR Several prior works have explored cloud offloading for XR to address the compute and power limitations of mobile devices, including offloading components such as head pose estimation, visual tracking, and rendering [18, 34, 9, 29, 25, 3, 8, 36]. These systems primarily focus on latency-sensitive rendering pipelines or geometric reconstruction, and do not consider the challenges associated with maintaining and querying a persistent semantic map. In addition, offloading AI workloads such as object detection and segmentation has been widely studied [24, 23, 56, 27, 6, 43, 55]. While some of these systems operate at object granularity, they typically target task-specific domains (e.g., object detection over a limited and predefined set of categories) and do not address the requirements of open-vocabulary semantic mapping. Techniques such as keyframe sampling reduce bandwidth or server compute, but do not address the system-level challenges introduced by offloading semantic mapping in XR, including maintaining persistent semantic state and robustness to network variability. 8.4 Semantic Maps on Neural Representations Recent work has explored semantic scene representations based on neural fields, such as NeRFs [33] augmented with language grounding [21], and Gaussian-based representations [20] extended for semantic queries [39, 26]. While effective for dense reconstruction and semantic querying, these representations typically require optimization over accumulated observations and are not yet designed for incremental, low-latency updates under the power, bandwidth, and memory constraints of mobile XR systems. Taken together, prior work provides strong semantic mapping backends and cloud-XR mechanisms, but does not address how to manage communication, execution, and memory footprint for semantic mapping across a device-cloud boundary under the power, bandwidth, and memory constraints of mobile XR. SemanticXR addresses this gap through object-level system organization that is compatible with a broad class of existing semantic mapping pipelines. 9 C ONCLUSION We presented SemanticXR, the first device-cloud system for realtime, open-vocabulary semantic mapping and querying within the power, bandwidth, and memory constraints of mobile XR. Our key insight is that semantically identifiable objects can serve as firstclass units of device-cloud system design, governing how the system communicates, executes, and manages memory across the de-
10
.2024.10610243 2, 3, 5, 6, 7, 9 [13] N. Hughes, Y. Chang, and L. Carlone. Hydra: A real-time spatial perception system for 3D scene graph construction and optimization. 2022. 9 [14] N. Hughes, Y. Chang, S. Hu, R. Talak, R. Abdulhai, J. Strader, and L. Carlone. Foundations of spatial perception for robotics: Hierarchical representations and real-time systems. The International Journal of Robotics Research, 2024. doi: 10.1177/02783649241229725 9 [15] M. Huzaifa, R. Desai, S. Grayson, X. Jiang, Y. Jing, J. Lee, F. Lu, Y. Pang, J. Ravichandran, F. Sinclair, B. Tian, H. Yuan, J. Zhang, and S. V. Adve. Illixr: An open testbed to enable extended reality systems research. IEEE Micro, 42(4):97–106, 2022. doi: 10.1109/MM.2022. 3161018 5 [16] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. If you use this software, please cite it as below. doi: 10.5281/zenodo.5143773 3 [17] K. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, S. Li, G. Iyer, S. Saryazdi, N. Keetha, A. Tewari, J. Tenenbaum, C. de Melo, M. Krishna, L. Paull, F. Shkurti, and A. Torralba. Conceptfusion: Open-set multimodal 3d mapping. Robotics: Science and Systems (RSS), 2023. 2, 3, 5, 6, 7, 9 [18] Q. Jiang, Y. Pang, W. Sentosa, S. Gao, M. Huzaifa, J. Zhang, J. PerezRamirez, D. Das, D. Gonzalez-Aguirre, B. Godfrey, and S. Adve. Remotevio: Offloading head tracking in an end-to-end xr system. In Proceedings of the 16th ACM Multimedia Systems Conference, MMSys ’25, p. 101–112. Association for Computing Machinery, New York, NY, USA, 2025. doi: 10.1145/3712676.3714442 2, 5, 10 [19] T. Jin, M. Dasa, C. Smith, K. Apicharttrisorn, S. Seshan, and A. Rowe. Meshreduce: Scalable and bandwidth efficient 3d scene capture. In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pp. 20–30, 2024. doi: 10.1109/VR58804.2024.00026 2 [20] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. 10 [21] J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. In International Conference on Computer Vision (ICCV), 2023. 2, 10 [22] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023. 3 [23] Z. J. Kong, Q. Xu, and Y. C. Hu. Arise: High-capacity ar offloading inference serving via proactive scheduling. In Proceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services, MOBISYS ’24, p. 451–464. Association for Computing Machinery, New York, NY, USA, 2024. doi: 10.1145/3643832. 3661894 2, 10 [24] Z. J. Kong, Q. Xu, J. Meng, and Y. C. Hu. Accumo: Accuracycentric multitask offloading in edge-assisted mobile augmented reality. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, ACM MobiCom ’23. Association for Computing Machinery, New York, NY, USA, 2023. doi: 10. 1145/3570361.3592531 2, 10 [25] Z. Lai, Y. C. Hu, Y. Cui, L. Sun, and N. Dai. Furion: Engineering high-quality immersive virtual reality on today’s mobile devices. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking, MobiCom ’17, p. 409–421. Association for Computing Machinery, New York, NY, USA, 2017. doi: 10.1145/ 3117811.3117815 10 [26] M. Li, S. Liu, H. Zhou, G. Zhu, N. Cheng, T. Deng, and H. Wang. Sgsslam: Semantic gaussian splatting for neural dense slam. p. 163–179. Springer-Verlag, Berlin, Heidelberg, 2024. doi: 10.1007/978-3-031 -72751-1 10 10 [27] L. Liu, H. Li, and M. Gruteser. Edge assisted real-time object detection for mobile augmented reality. In The 25th Annual International Conference on Mobile Computing and Networking, MobiCom ’19. Association for Computing Machinery, New York, NY, USA, 2019. doi: 10.1145/3300061.3300116 2, 10 [28] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with
grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 3, 5 [29] E. Lu, S. Bharadwaj, M. Dasari, C. Smith, S. Seshan, and A. Rowe. Renderfusion: Balancing local and remote rendering for interactive 3d scenes. In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 312–321, 2023. doi: 10.1109/ ISMAR59233.2023.00046 10 [30] S. Lu, H. Chang, E. P. Jing, A. Boularias, and K. Bekris. Ovir-3d: Open-vocabulary 3d instance retrieval without training on 3d data. In 7th Annual Conference on Robot Learning, 2023. 2, 3, 10 [31] D. Maggio, Y. Chang, N. Hughes, M. Trang, D. Griffith, C. Dougherty, E. Cristofalo, L. Schmid, and L. Carlone. Clio: Real-time task-driven open-set 3d scene graphs. IEEE Robotics and Automation Letters, 9(10):8921–8928, 2024. doi: 10.1109/LRA.2024.3451395 2, 5, 6, 7, 9, 10 [32] Y. Mao, J. Zhong, C. Fang, J. Zheng, R. Tang, H. Zhu, P. Tan, and Z. Zhou. Spatiallm: Training large language models for structured indoor modeling, 2025. 9 [33] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: representing scenes as neural radiance fields for view synthesis. Commun. ACM, 65(1):99–106, Dec. 2021. doi: 10. 1145/3503250 10 [34] NVIDIA. Cloudxr, 2023. Streaming for extended reality. 2, 10 [35] NVIDIA. tegrastats Utility. https://docs.nvidia.com/ drive/drive_os_5.1.6.1L/nvvib_docs/index.html#page/ DRIVE_OS_Linux_SDK_Development_Guide/Utilities/util_ tegrastats.html, 2024. 6 [36] Y. Pang, S. Kondguli, S. Wang, and S. Adve. Ada: A distributed, power-aware, real-time scene provider for xr. IEEE Transactions on Visualization and Computer Graphics, 31(11):9677–9687, 2025. doi: 10.1109/TVCG.2025.3616835 2, 5, 8, 10 [37] Z. Peng, T. Shao, L. Yong, J. Zhou, Y. Yang, J. Wang, and K. Zhou. Rtg-slam: Real-time 3d reconstruction at scale using gaussian splatting. 2024. 6 [38] Z. Peng, K. Zhou, and T. Shao. Gaussian-plus-sdf slam: High-fidelity 3d reconstruction at 150+ fps. Computational Visual Media, 2025. 6 [39] M. Qin, W. Li, J. Zhou, H. Wang, and H. Pfister. Langsplat: 3d language gaussian splatting. arXiv preprint arXiv:2312.16084, 2023. 2, 10 [40] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021. 3 [41] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. Girshick, P. Dollár, and C. Feichtenhofer. Sam 2: Segment anything in images and videos, 2024. 3 [42] C. Schuhmann, R. Beaumont, R. Vencu, C. W. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. R. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. 3 [43] S. Shi, J. Cui, Z. Jiang, Z. Yan, G. Xing, J. Niu, and Z. Ouyang. Vips: real-time perception fusion for infrastructure-assisted autonomous driving. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking, MobiCom ’22, p. 133–146. Association for Computing Machinery, New York, NY, USA, 2022. doi: 10.1145/3495243.3560539 2, 10 [44] S. Srinidhi, E. Lu, and A. Rowe. Xair: An xr platform that integrates large language models with the physical world. In 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 759–767, 2024. doi: 10.1109/ISMAR62088.2024. 00091 10 [45] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge, Y. Yan, X. Pan, J. Yon, Y. Zou, K. Leon, N. Carter, J. Briales, T. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. D. Nardi, M. Goesele, S. Lovegrove, and R. Newcombe.
11
The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797, 2019. 5, 6, 7 [46] A. Takmaz, E. Fedele, R. W. Sumner, M. Pollefeys, F. Tombari, and F. Engelmann. OpenMask3D: Open-Vocabulary 3D Instance Segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 2, 5, 7, 10 [47] Y. Tang, J. Zhang, Y. Lan, Y. Guo, D. Dong, C. Zhu, and K. Xu. Onlineanyseg: Online zero-shot 3d segmentation by visual foundation model guided 2d mask merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3676–3685, June 2025. 2, 3, 5, 6, 9, 10 [48] P. K. A. Vasu*, H. P. Ansari*, F. Faghri*, R. Vemulapalli, and O. Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. In CVPR, 2024. 5 [49] S.-C. Wu, J. Wald, K. Tateno, N. Navab, and F. Tombari. SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 9 [50] C. Xu, R. Kumaran, N. Stier, K. Yu, and T. Höllerer. Multimodal 3d fusion and in-situ learning for spatially aware ai. In 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2024. doi: 10.1109/ISMAR62088.2024.00063 3, 9 [51] X. Xu, H. Chen, L. Zhao, Z. Wang, J. Zhou, and J. Lu. Embodiedsam: Online segment any 3d thing in real time. arXiv preprint arXiv:2408.11811, 2024. 10 [52] K. Yamazaki, T. Hanyu, K. Vo, T. Pham, M. Tran, G. Doretto, A. Nguyen, and N. Le. Open-fusion: Real-time open-vocabulary 3d mapping and queryable scene representation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 9411–9417, 2024. doi: 10.1109/ICRA57147.2024.10610193 2, 3, 5, 9 [53] Y. Yang, H. Yang, J. Zhou, P. Chen, H. Zhang, Y. Du, and C. Gan. 3d-mem: 3d scene memory for embodied exploration and reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17294–17303, June 2025. 3 [54] C. Zhang, D. Han, Y. Qiao, J. U. Kim, S.-H. Bae, S. Lee, and C. S. Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289, 2023. 5 [55] Q. Zhang, X. Zhang, R. Zhu, F. Bai, M. Naserian, and Z. M. Mao. Robust real-time multi-vehicle collaboration on asynchronous sensors. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, ACM MobiCom ’23. Association for Computing Machinery, New York, NY, USA, 2023. doi: 10. 1145/3570361.3613271 2, 10 [56] W. Zhang, Z. He, L. Liu, Z. Jia, Y. Liu, M. Gruteser, D. Raychaudhuri, and Y. Zhang. Elf: accelerate high-resolution mobile deep vision with content-aware parallel offloading. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, MobiCom ’21, p. 201–214. Association for Computing Machinery, New York, NY, USA, 2021. doi: 10.1145/3447993.3448628 2, 10 [57] Y. Zhang, X. Huang, J. Ma, Z. Li, Z. Luo, Y. Xie, Y. Qin, T. Luo, Y. Li, S. Liu, et al. Recognize anything: A strong image tagging model. arXiv preprint arXiv:2306.03514, 2023. 3, 5 [58] Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, et al. Regionclip: Region-based languageimage pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16793–16803, 2022. 3 [59] X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Wang, L. Wang, J. Gao, and Y. J. Lee. Segment everything everywhere all at once. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23. Curran Associates Inc., Red Hook, NY, USA, 2024. 3
12