OneCanvas: 3D Scene Understanding via Panoramic Reprojection
Bartłomiej Baranowski1
arXiv:2606.19253v1 [cs.CV] 17 Jun 2026
1
Dave Zhenyu Chen2
Technical University of Munich
2
Matthias Nießner1 Huawei
Project page: https://baranowskibrt.github.io/onecanvas/ 1
Feature Backprojection
2
Aggregation in a Common Reference Frame
3
Resulting panorama
Figure 1: OneCanvas (1) backprojects per-frame patch features to 3D, (2) aggregates them in a common reference frame centered on a chosen viewpoint, and (3) places each patch at its continuous angular position on a panoramic canvas consumed by the VLM as one image.
Abstract Existing approaches to 3D scene understanding in Vision-Language Models (VLMs) either rely on complex, model-specific geometry encoders or large training budgets in pursuit of spatial reasoning. Instead, OneCanvas aggregates patch features from all views onto a single equirectangular panoramic canvas. Namely, each patch is unprojected to a 3D world coordinate using its depth and camera pose, then placed on the canvas at the continuous longitude and latitude of that point as seen from the canvas origin, with no rasterization or aggregation across overlapping views. A 3D position embedding of the patch’s metric coordinates is added to its feature, restoring the depth lost when collapsing the world position to an angular canvas coordinate. Patches from all frames thus share one spatial coordinate system with no fusion or major architectural modifications of the backbone. The pretrained VLM consumes this representation as if it were an ordinary image. Because the canvas can be centered on any pose of interest, the same representation directly supports situated reasoning from a specific viewpoint, a common requirement in robotics and embodied AI. Thanks to this representation, we can also introduce a spatial pretraining curriculum: by procedurally placing patch features of objects, drawn from real images, at chosen 3D world positions on an otherwise empty canvas, we generate on-the-fly supervision spanning a broad range of spatial reasoning tasks, with answer distributions controlled to reduce spatial reasoning shortcuts. OneCanvas achieves state-of-the-art accuracy on SQA3D and VSI-Bench, and generalizes to out-of-distribution data on SPBench, using an order of magnitude less training compute than the strongest competing methods.
(b) SPBench Zero-Shot
(a) SQA3D Per Question Type
(c) Accuracy vs. Compute
MV NQ
Others
Is
SI Avg.
MV MCQ
Which
SI How MCQ
MV Avg.
Can
VSI-Bench avg. (%)
What
104 105 10 103 Training compute (A100-equiv. GPU-hours)
SI NQ
OneCanvas (Ours) VLM-3R
Ross3D ViCA
SR-3D
SpaceMind
VILASR-7B
72 70 68 66 64 62 60 58 2
SpaceR-7B
SenseNova-SI
Video-3D LLM
Spatial-MLLM-4B
Figure 2: OneCanvas benchmark comparison. (a) Per-question-type accuracy on SQA3D and (b) zero-shot accuracy on SPBench (multi-view and single-image splits), shown as radar charts against the strongest competing methods. (c) Training compute (A100-equivalent GPU-hours, log scale) versus VSI-Bench accuracy.
1
Introduction
Vision-language models have transformed how machines interpret images, video, and natural language jointly, but a great deal of practically valuable understanding is intrinsically three-dimensional. Robots that act in human environments, augmented-reality assistants that ground instructions in a user’s physical surroundings, and autonomous agents that plan trajectories through unfamiliar spaces all need to answer questions about where things are, how far apart they are, which direction one must turn to face them, and what is or is not visible from a given vantage point. Building VLMs that answer such questions reliably from ordinary video of a scene is the goal of 3D scene understanding. Most existing 3D-aware VLMs are built on top of pretrained 2D vision-language backbones with no native machinery for spatial reasoning. Adapting them to 3D therefore demands either inducing geometry implicitly from raw video, which is data-inefficient and tends to confound geometry with scene-specific texture priors, or augmenting the backbone with a dedicated geometry encoder. Even with these adaptations in place, the supplied geometry is often not actually used. Recent audits on situated 3D question answering find that text-only baselines match or approach state-of-the-art geometry-aware models on widely cited benchmarks Ma et al. [2026], indicating that the models fall back on statistical regularities of the question and scene class rather than reasoning over the geometric input they are given. Existing 3D VLMs have nonetheless made substantial progress on 3D reasoning. One family augments the VLM with a dedicated geometric module, including point-cloud tokenisers Chen et al. [2024], Huang et al. [2024b], Wang et al. [2023], Zhu et al. [2024], depth-conditioned video position encodings Zheng et al. [2025b], Zhu et al. [2025], and fused geometry encoders such as VLM-3R Fan et al. [2026] and SpaceMind Zhao et al. [2025] that read 3D structure from monocular video and inject it into the visual backbone. A second family scales spatial supervision: SenseNova-SI Cai et al. [2026] and Cambrian-S Yang et al. [2026] curate large collections of spatial QA pairs to teach a geometrically vanilla VLM to reason about distance and direction by sheer exposure. Both directions deliver concrete improvements on every major spatial benchmark, and together they show that 3D reasoning is reachable from a pretrained VLM if one is willing to invest in either architecture or data. Both directions also pay a heavy bill. Architectural fusion adds modules that must be jointly trained at scale to align a new geometry encoder with the language backbone, while data scaling demands curating and sorting millions of high-quality spatial question-answer pairs. Neither directly closes the gap flagged by the audits above: methods in both families still see the scene only piecewise across frames and have no clear handle on which signals in the input actually encode the spatial relationships they are being asked to learn. Instead, we propose OneCanvas (Figure 1): each patch from each view is lifted to a 3D world coordinate using its depth and camera pose, then placed on a single equirectangular canvas at the longitude and latitude of that point as seen from a chosen canvas origin. These spherical coordinates fill the spatial axes of the patch’s MRoPE position, while the 2
source-frame index occupies the temporal axis. All patches from all views thus share one spatial coordinate system, and the pretrained VLM consumes the result as if it were an ordinary single-image input, with no major architectural modifications. This representation also enables a spatial pretraining curriculum that would be difficult to assemble in conventional 2D or video pipelines. By procedurally placing patch features of objects, drawn from real images, at chosen 3D world positions on an otherwise empty canvas, we generate on-the-fly supervision for tasks ranging from metric distances to directions, counting, observability, and navigation. Answer distributions are controlled to suppress the scene-statistic shortcuts recent audits have flagged Ma et al. [2026]. OneCanvas reaches state of the art on SQA3D Ma et al. [2023] (65.3 EM@1, 2.3 points above the previous best), VSI-Bench Yang et al. [2025a] (70.1 average), and SPBench Li et al. [2026] (72.1 zero-shot overall, 4.8 points above the next best method), while using an order of magnitude less training compute than the strongest competing methods (Figure 2, with the full normalization methodology in Section B). Our contributions are: • Panoramic feature reprojection. A mechanism that aggregates patch features from all views onto a single panoramic canvas, matching the input format the VLM was pretrained on, allowing for flexible choice of canvas origin and orientation to suit different tasks. • Spatial pretraining curriculum enabled by the representation. On-the-fly geometric supervision built by procedurally placing patch features of objects, drawn from real images, at chosen 3D positions on the canvas, with answer distributions controlled to reduce the dataset-statistic shortcuts.
2
Related Work
3D-aware VLMs via architectural extensions. Most existing 3D VLMs equip a pretrained language backbone with auxiliary modules that supply geometric information Ma et al. [2024a], Chen et al. [2024], Huang et al. [2024b], Wang et al. [2023], Zhu et al. [2024], Li et al. [2024], Zheng et al. [2025b], Fan et al. [2026], Zhao et al. [2025], Wang et al. [2025a], Cheng et al. [2025], Qi et al. [2026]. Early point-cloud-based 3D VLMs Hong et al. [2023], Chen et al. [2024], Huang et al. [2024b], Wang et al. [2023], Zhu et al. [2024], Li et al. [2024] pair a language backbone with a dedicated point cloud tokeniser and train end-to-end on 3D QA data, establishing the feasibility of LLM-based 3D scene reasoning. RGB-D-based 3D VLMs Zheng et al. [2025b], Zhu et al. [2025] instead treat the input as a video sequence and inject 3D position encodings computed from depth maps into the video representation, bridging the video and 3D domains without an explicit point-cloud encoder. HiSpatial Liang et al. [2026] extends this line by feeding metric-scale point maps as auxiliary inputs alongside RGB-D and supervising the VLM through a hierarchical curriculum from geometric perception to abstract spatial reasoning. With the emergence of geometry foundation models Wang et al. [2024a], Leroy et al. [2024], Wang et al. [2025b], Lin et al. [2026], more recent 3D VLMs extract 3D tokens directly from monocular video and fuse them with visual features before the LLM Fan et al. [2026], Zhao et al. [2025], Wang et al. [2025a], Cheng et al. [2025], Qi et al. [2026], Zheng et al. [2025a]. VLM-3R Fan et al. [2026] fuses implicit 3D tokens from a geometry encoder with visual features. SpaceMind Zhao et al. [2025] adds a Camera-Guided Modality Fusion module that combines a spatial encoder with a visual encoder. Ross3D Wang et al. [2025a] introduces a reconstructive visual instruction-tuning objective that supplies 3D-aware supervision during fine-tuning of a video VLM. SR-3D Cheng et al. [2025] is a region-promptable VLM that unifies single-view 2D and multi-view 3D inputs in a shared visual token space. GPT4Scene Qi et al. [2026] fine-tunes Qwen2VL on video sequences of 3D scenes. SD-VLM Chen et al. [2025] introduces a sinusoidal depth positional encoding that adds depth information along the camera’s z-axis. Loc3R-VLM Qu et al. [2026] pairs camera priors from a pretrained geometry foundation model with auxiliary BEV-layout and situation-modeling objectives. A parallel line uses reinforcement learning with spatial reward signals Ouyang et al. [2025], Wu et al. [2025b], Batra et al. [2025], Chen et al. [2026], which is orthogonal to representation choice and could in principle be combined with our canvas. All of these methods introduce new architectural components, auxiliary objectives, or dedicated 3D encoders on top of the base VLM. In contrast, our approach introduces no new architectural components, no auxiliary loss terms, and no dedicated 3D encoder. 3D understanding emerges from how the input is prepared. 3
1
Multi-view RGB-D
2
Feature extraction
3
Lifted 3D feature patches
4
Shared equirectangular feature space
Feature extractor
5
Frozen model Fine-tuning (LoRA) Source cameras Panoramic orientation
Question answering
Question Where do I turn to look at the TV?
VLM
Answer right
Figure 3: Method overview. Multi-view images are encoded by Qwen3-VL’s frozen vision encoder, and lifted patches land at continuous positions on a shared equirectangular panorama. The VLM consumes this representation through its native attention layers. Unified scene representations. Panoramic image representations offer a 360◦ field of view that naturally preserves long-range spatial relationships, a property exploited for scene understanding Zheng et al. [2025c] and recently for 3D visual grounding. PanoGrounder Jung et al. [2025] renders multiple equirectangular panoramas from virtual cameras placed around a pre-reconstructed mesh or 3D Gaussian Splatting scene and processes them with a large VLM for grounding. PanoEnv Lin and Zheng [2026] studies VLM spatial intelligence on equirectangular images from synthetic environments, showing that geometric distortion in ERP images remains a challenge for off-the-shelf VLMs. The broader idea of unprojecting 2D features into 3D and reprojecting them onto a canonical 2D view is well-established in autonomous driving through BEV methods Philion and Fidler [2020], Ma et al. [2024b]. Lift3D T et al. [2024] lifts per-view features into a neural field and renders target views, and LiftProj Jia et al. [2025] uses a similar lift-then-project paradigm for panorama stitching. Our work brings this principle to VLM-based 3D scene understanding: per-view features are lifted with metric depth and camera poses, projected once onto a panoramic canvas, and consumed directly by the VLM, with the canvas origin chosen freely to support both scene-centered and pose-centered situated reasoning. Because we project features rather than raw pixels, the distortion that PanoEnv documents is absorbed by the projection geometry rather than forced onto the VLM’s perception.
3
Method
Given a set of K posed RGB images {Ik }K k=1 with metric depth maps {Dk } and pinhole camera intrinsics Ck = (fx , fy , cx , cy ) (per-frame focal lengths and principal point), our goal is to construct a single panoramic representation that a pretrained VLM can ingest directly. The representation is built in two steps (Figure 3): (1) extract per-patch features from each view and lift them into 3D world coordinates (Section 3.1), and (2) reproject the lifted patches onto a shared equirectangular canvas, add a 3D position embedding of each patch’s metric coordinates, and assemble the VLM’s input sequence with its native 3D-RoPE position encoding (Section 3.2). Training proceeds in two stages (Figure 4): a stage-1 spatial pretraining phase (Section 3.3) followed by stage-2 adaptation on downstream target data (Section 3.4). 3.1
Feature Extraction and 3D Lifting
Each image Ik is passed through Qwen3-VL’s frozen vision encoder Bai et al. [2025] to obtain a patch-level feature map Fk ∈ RHf ×Wf ×D at the encoder’s grid resolution Hf × Wf and per-patch feature dimension D. Each patch (u, v) is lifted to world coordinates using its depth z = Dk (u, v), intrinsics scaled to the feature-map resolution, and the camera-to-world pose Tk ∈ SE(3): ! (u − cx ) z/fx (v − cy ) z/fy , pworld (1) u,v = Tk z where Tk ∈ SE(3) acts as Tk x = Rk x + tk , and (u, v, z) follow the OpenCV camera convention (x right, y down, z forward). After lifting all K views we obtain a set P = {(pi , fi , ti )} of lifted patches, 4
with world coordinates pi ∈ R3 , features fi ∈ RD , and a per-patch temporal index ti ∈ {1, . . . , K} recording the position of the source frame in the input video. We carry ti through to panoramic placement (Section 3.2), where it becomes the patch’s coordinate on the language model’s temporal MRoPE axis. 3.2
Panoramic Canvas and Position Encoding
We place each lifted patch on an equirectangular canvas defined by its origin c ∈ R3 and orientation R ∈ SO(3). The pair (c, R) is a free design choice and can be set to whatever frame is most natural for the task: an agent pose for situated reasoning, a viewpoint that maximizes scene coverage, or any other reference. Specific conventions used at training and inference are given in Section 4.1. For each lifted patch, we compute the local offset qi = R⊤ (pi − c) = (qx , qy , qz ) in the canvas frame and continuous spherical coordinates: longitude θi = atan2(qx , qz ) and latitude p ϕi = arctan(−qy / qx2 + qz2 ). The image y-axis points downward in the camera frame, so qy measures the downward direction and the −qy in the latitude formula gives positive latitude upward. Continuous positions, not a pixel grid. A natural alternative is to rasterize the lifted patches onto a fixed pixel grid and reduce cell contents through a hand-designed rule (averaging, depth-based selection, last-write). This collapses patches that project to the same or nearby angular position (for instance occluded surfaces along a common viewing ray or co-located observations from overlapping frames), destroying the distinction between surfaces and replacing a pretrained-attention decision with a fixed reduction. We instead keep each lifted patch as its own input token at its continuous (ϕ, θ) position. Co-located patches remain distinct tokens, disambiguated by the 3D position embedding described below, and the model resolves overlap through attention rather than a pre-baked rule. Position encoding. The VLM uses its stock 3D Rotary Position Embedding (3D-RoPE) Bai et al. [2025] unchanged. Only the content of the position IDs differs from the standard image/video setting. We set W to the longitude θi ∈ [−π, π], H to the latitude ϕi ∈ [−π/2, π/2], and T to the source frame index ti ∈ {1, . . . , K}, with all three linearly rescaled to [0, 100] to stay within the effective positional distribution the model was pretrained on. T retains its standard “temporal / frame-ordering” semantics from video. 3D position embedding. A 2D VLM picks up only a class-conditioned scale prior from pixels, not a metric reading of the actual layout. We supply explicit per-patch metric position by encoding the canvas-frame offset qi = (qx , qy , qz ) as the concatenation of (i) per-axis sinusoids {sin(ωk qa ), cos(ωk qa ), qa } for a ∈ {x, y, z} at 16 log-spaced p frequencies ωk ∈ [0.1, 100] rad/m, (ii) the same form on the radial primitives ∥qi ∥ and rxz = qx2 + qz2 at 8 log-spaced frequencies νk ∈ [0.1, 10] rad/m, and (iii) the unit-ray direction qi /∥qi ∥ as raw passthrough. These radial primitives directly expose quantities relevant to pairwise distance and horizontal-radial reads such as floor area, rather than requiring the model to synthesise them from per-axis components. The 136-channel encoding is projected by a 2-layer MLP and added to the patch feature through a learned scalar gate. Injecting metric position in feature space rather than as additional RoPE dimensions keeps 3D-RoPE’s pretrained angular and temporal semantics intact, and lets the VLM attend to metric position through the same content-attention pathway it uses for any other feature. 3.3
Spatial Pretraining Curriculum
Spatial reasoning is difficult to acquire from downstream QA alone. The supervision is indirect, target-task distributions are narrow, and scene-appearance cues dominate the gradient before any geometric reading forms, so a model fits the per-template answer marginal long before it learns to read the input. Once that prior is absorbed it is hard to displace, since switching to a geometric readout requires a transient drop in accuracy that the QA gradient is not strong enough to reward. We therefore want supervision in which the answer follows causally from the input geometry, with no language-prior or scene-memorisation shortcut. The panoramic canvas is the natural input format for it: arbitrary 3D content can be placed at any world position with metric coordinates carried in the per-patch embedding, so the geometric answer can be read directly from the placements through the same path the model uses on a real scene. 5
1 Spatial pretraining
Distance
Direction
Tasks generated on-the-fly
Visibility
No scene clutter
Navigation
2 QA Fine-tuning
Noisy real-world scenes
Counting
Q: What is the size of this room (in square meters)? A: 18.4
Figure 4: Two-stage training. Stage 1 trains LoRA Hu et al. [2022] adapters and the 3D position embedding on the spatial pretraining curriculum, where objects placed on an otherwise empty canvas carry all the geometric signal. Stage 2 merges the stage-1 adapter back into the base language model and trains a fresh, smaller adapter on real-scene downstream QA. Token embeddings stay frozen across both stages. Curriculum objects. Placing recognisable real objects on the canvas can weaken the supervision we are trying to provide, since doors tend to be the same height and bathrooms a fairly consistent size, so a model that has seen enough scenes can often answer geometric questions from class identity alone. In our default formulation we populate a box-sized area of the canvas with a single feature patch sampled from a real scene, and refer to that placement in the prompt by inserting the same patch where a class name would appear. This inline copy keeps the MRoPE position of its slot in the prompt text, like any other token, rather than the longitude/latitude canvas position carried by its counterpart on the canvas. The model resolves the reference through the same visual-matching pathway it uses during vision-language pretraining. With appearance decoupled from size and location, geometry on the canvas becomes the principal signal for solving the task. The patch is drawn from a precomputed pool of activations harvested from held-out scenes of ScanNet Dai et al. [2017], ScanNet++ Yeshwanth et al. [2023], and ARKitScenes Baruch et al. [2021], so the vision encoder treats it as ordinary visual content. An alternative class-labelled scheme based on EmbodiedScan Wang et al. [2024b] object features, and a head-to-head VSI-Bench comparison, are reported in Section A. Task types. To learn geometry-based spatial understanding we construct a compact set of tasks generated on-the-fly across six families: • Metric measurement. Surface-to-surface distance between two placed objects, relativedistance comparisons (which of several candidates is closer to a reference, matching VSIBench’s relative-distance task), and floor-area readout as a room-size signal. • Egocentric direction. “Left / right / front / back” queries from a chosen viewpoint, at three difficulty levels, plus an o’clock-direction variant and a camera-perspective variant matching downstream SI-style queries. • Multi-turn navigation. N -turn path queries between two placed targets, N ∈ {1, 2, 3, 4}. • Observability. When a placement first becomes visible along the frame sequence (appearance-order), and whether a target is visible from a given pose. • Counting. Counts of placed objects under divisibility conditions (parity, divisibility by three) or location conditions (placements on the agent’s left, right, front, or back). • Multi-target bounding-box readout. Two to four placed objects, with the prompt asking for each one’s 3D bounding box in canonical order. 3.4
Two-Stage Training
Stage 1: spatial pretraining. We train on the spatial pretraining curriculum alone, teaching the geometric reasoning skills downstream 3D-QA depends on under supervision designed to suppress the scene-prior shortcuts available in real-scene QA. Because each curriculum sample carries far fewer tokens than a full scene panorama, stage 1 is cheap to run and we can afford a long training schedule. Stage 2: target adaptation. We merge the stage-1 LoRA weights back into the base language model and train a fresh, lower-rank LoRA adapter (r=64) on top, on a weighted mixture of downstream 6
spatial QA (SQA3D, VSI-Bench-style data, and ViCA-family corpora). Merging freezes the stage-1 update into the base weights, and the lower rank of the new adapter bounds how far stage-2 can drift from it, anchoring the stage-1 geometric reading against real-scene answer-prior shortcuts while still allowing answer-format adaptation.
4
Experiments
4.1
Implementation Details
Panoramic projection. We sample K=32 frames per scene at evenly spaced timestamps along the video. The scene center is set to the annotated agent position for SQA3D and to the queried camera pose for SPBench single-image queries, and to the centroid of all camera positions otherwise. Training is run at 320×240 input resolution across all data sources. At evaluation we use 320×240 for SQA3D, where raising it gave no aggregate gain, and 640×480 for VSI-Bench and SPBench, where explicit numeric-measurement queries benefit from higher input resolution. For ScanNet scenes we crop 3% from each image edge (with intrinsics adjusted) to avoid spurious lifted points outside the scene. We use ground-truth depth and ground-truth camera poses whenever the dataset provides them, and fall back to estimates from a feed-forward reconstruction model on the small subset of scenes where the released ground-truth pose track is corrupted. During training only, we randomize the panorama’s location and orientation per sample by drawing the scene center from the available camera positions and rotating the forward axis by a uniform random yaw. Training. We build on Qwen3-VL-8B Bai et al. [2025], a vision-language model with a ViT-based vision encoder and a 3D RoPE position encoding scheme with temporal, height, and width dimensions. Training proceeds in two stages on 8 A6000 GPUs, with LoRA adapters targeting the attention and MLP projections of the language model. Stage 1 trains on the spatial pretraining curriculum with LoRA rank r=256 (α=512, dropout 0.05), effective batch 16, learning rate 2×10−5 with cosine schedule and 3% warmup. During stage 1 we cap the placed-object patch count at 200 per sample, distributed uniformly across the placed boxes with a floor of 8 patches per box to retain shape, which keeps stage 1 fast while leaving the geometry signal intact. The stage-2 LoRA uses r=64, α=128, dropout 0.05, and trains for 10k steps with effective batch 32 at the same learning rate, schedule, and warmup. Stage-2 training data is a weighted mixture of VLM-3R-VSIBench Fan et al. [2026], SQA3D Ma et al. [2023], and ViCA Feng [2025], mixed at approximately 70/15/15 by sampling weight. 4.2
Evaluations
We evaluate on three benchmarks: SQA3D Ma et al. [2023] (situated question answering), VSIBench Yang et al. [2025a] (eight spatial subtasks split between numerical and multiple-choice), and SPBench Li et al. [2026] (single-image and multi-view spatial reasoning, evaluated zero-shot). Spatial question answering. Table 1 reports results on SQA3D, comparing our method against point-cloud-based 3D VLMs and video-input approaches. We achieve the best overall accuracy. We obtain the best results on the question types that most directly test situated reasoning (Which, Can, Others), with the largest margin on Which. The large Which margin tracks the canvas-origin ablation (Table 5): agent-pose centering drives most of it, and Which is the most viewpoint-dependent question type. The remaining gap concentrates on Is (yes/no questions, where textual priors carry most of the signal) and How (predominantly counting, a capability the base 2D VLM is already heavily pretrained on). Table 2 presents zero-shot results on SPBench, where every method is evaluated without SPBench training data. We top the leaderboard, with the largest gain concentrated on multi-view MCQ, consistent with the intuition that a single panoramic canvas gives multi-view questions a unified frame of reference rather than forcing the VLM to stitch several independent views. We also top the single-image split on average accuracy. Table 3 shows results on VSI-Bench, where we again come out on top, with the largest margin on route planning, the subtask that most directly exercises multi-step reasoning over a unified panoramic canvas. We also lead on room size and finish second on absolute distance, relative distance, and relative direction. 7
Table 1: Evaluation on SQA3D Ma et al. [2023] test split. Comparison of methods across question types. EM@1 and EM@R1 are the primary metrics. Best results in bold, second best underlined. Dashes indicate scores not reported by the method. Per Question Type
Method
EM@1 EM@R1
What
Is
How Can Which Others
PQ3D Zhu et al. [2024] 3D-VisTA Zhu et al. [2023] LEO Huang et al. [2024b] SIG3D Man et al. [2024] Scene-LLM Fu et al. [2025] ChatScene Huang et al. [2024a] Video-3D LLM Zheng et al. [2025b] GPT4Scene-HDM Qi et al. [2026] VLM-3R Fan et al. [2026] SpaceMind Zhao et al. [2025] SR-3D Cheng et al. [2025] Loc3R-VLM Qu et al. [2026] Ross3D Wang et al. [2025a]
37.1 34.8 39.0 35.6 40.9 45.4 51.1 55.9 – 54.1 55.0 – 56.0
61.3 63.3 63.9 67.2 69.1 67.0 72.4 69.9 – 74.8 76.4 – 79.8
44.5 45.4 44.9 48.5 45.0 52.0 55.5 50.8 – 61.7 59.8 – 60.6
60.9 69.8 66.2 71.4 70.8 69.5 69.8 68.7 – 71.0 71.6 – 70.4
47.0 47.2 47.7 49.1 47.2 49.9 51.3 53.3 – 51.9 54.7 – 55.3
45.1 48.1 51.1 45.8 52.3 55.0 56.0 60.4 – 53.6 61.1 – 60.1
47.1 48.5 50.0 52.6 54.2 54.6 58.6 59.4 60.7 61.1 62.2 62.8 63.0
49.3 50.9 52.4 54.4 56.2 57.5 60.8 62.4 63.4 63.8 – 65.0 65.7
OneCanvas (Ours)
62.1
76.2 61.1 75.4
74.4
70.5
65.3
68.4
Table 2: Evaluation on SPBench Li et al. [2026]. All models are evaluated without using SPBench training data. Best results per section in bold, second best underlined.
Method
SPBench-MV
Overall
SPBench-SI
NQ
MCQ Avg.
NQ
MCQ Avg.
Video-R1 Feng et al. [2025] SpaceR-7B Ouyang et al. [2025] VILASR-7B Wu et al. [2025b] Spatial-MLLM-4B Wu et al. [2025a] SpaceMind Zhao et al. [2025]
43.8 53.5 54.0 52.5 67.3
32.5 63.2 56.2 63.7 76.2
53.0 53.7 59.6 58.9 70.5
42.8 58.5 57.9 61.3 73.8
27.7 35.7 36.6 38.1 66.3
62.0 61.5 63.7 49.3 53.2
44.9 48.6 50.2 43.7 59.7
OneCanvas (Ours)
72.1
79.0
91.8
81.5 62.8
62.7
62.8
Table 3: Evaluation on VSI-Bench Yang et al. [2025a]. Comparison across numerical and multiplechoice spatial reasoning subtasks. Best results per section in bold, second best underlined. Method
Numerical Question
Avg.
Multiple-Choice Question
Obj.Cnt. Abs.Dist. Obj.Size Room Size Rel.Dist. Rel.Dir. Route App.Order Spacer Ouyang et al. [2025] ViLaSR Wu et al. [2025b] Spatial-MLLM Wu et al. [2025a] ViCA Feng [2025] VLM-3R Fan et al. [2026] VST Yang et al. [2025b] Cambrian-S Yang et al. [2026] SenseNova-SI Cai et al. [2026] SpaceMind Zhao et al. [2025] OneCanvas (Ours)
4.3
45.5 45.4 48.4 60.6 60.9 61.2 67.5 68.8 69.6 70.1
57.8 63.5 65.3 68.8 70.2 71.6 73.2 72.0 73.3 68.4
28.2 34.4 34.8 57.0 49.4 43.8 50.5 53.5 61.4 59.5
59.9 60.6 63.1 79.2 69.2 75.5 74.9 76.8 77.3 75.4
47.1 30.9 45.1 75.1 67.1 69.2 72.2 72.8 74.2 76.5
40.1 48.9 41.3 58.5 65.4 60.0 71.1 69.6 67.2 71.0
45.4 45.2 46.2 42.6 80.5 55.6 76.2 80.8 88.4 84.8
33.5 30.4 33.5 34.5 45.4 44.3 41.8 48.5 44.3 59.8
52.1 49.2 46.3 68.8 40.1 69.2 80.1 76.4 70.6 65.7
Ablation Studies
Component ablation. In Table 4 we ablate the key components of our method on VSI-Bench. To match the total LoRA capacity of the two-stage full model (stage 1 r=256 merged into the base, stage 2 r=64), the single-stage ablation variants are trained with r=256 (α=512). Without the spatial pretraining stage we observe a large drop on the route task, confirming the value of the curriculum for this capability. We also perform better on the tasks that require estimating metric quantities, with 8
Table 4: Ablation study on VSI-Bench. Each row lists the components present in that variant. The full model uses panoramic reprojection, 3D position encoding, and stage-1 spatial pretraining, while the bottom row is the base VLM with multi-view input and none of these components. All variants are LoRA-finetuned with matched training settings. Best results per column in bold, second best underlined. Configuration
Avg.
Full model (ours) Full w/o 3D PE Panorama + 3D PE Panorama only Base VLM (multi-view)
70.1 69.0 66.6 63.7 63.5
Numerical Question
Multiple-Choice Question
Obj.Cnt. Abs.Dist. Obj.Size Room Size Rel.Dist. Rel.Dir. Route App.Order 59.5 55.9 53.7 52.1 50.3
68.4 68.5 67.6 67.3 71.3
76.5 69.8 71.5 56.2 66.0
75.4 75.3 75.6 75.8 74.8
71.0 68.9 70.4 69.3 67.9
84.8 86.2 85.9 83.6 72.2
59.8 63.4 46.4 44.3 41.8
65.7 63.8 61.3 60.4 63.4
Table 5: Canvas origin ablation on SQA3D. Same checkpoint evaluated under four canvas origin strategies. Agent pose tracks the agent’s position and heading. The other three fix forward to world +X with origins at the scene centroid, a random camera, or outside the scene. Best per column in bold, second best underlined. Per Question Type
Canvas origin Agent pose Scene center Random camera Outside scene
What
Is
How
Can
Which
Others
62.1 60.3 61.1 59.5
76.2 76.1 76.2 75.3
61.1 60.4 61.9 60.6
75.4 68.9 68.9 68.3
74.4 58.7 57.5 53.0
70.5 62.4 62.6 59.1
EM@1
EM@R1
65.3 61.2 61.4 59.8
68.4 64.3 64.6 62.7
the exception of object size, which relies much more heavily on text priors. Removing the positional encoding produces a large additional drop on the room size task, while the other metric questions stay relatively stable as the model can still infer sizes and distances from the scene’s appearance. Removing only the 3D position embedding from the full model isolates the same effect with the curriculum intact: the drop is concentrated on the metric subtasks, while angular and categorical subtasks are unaffected, consistent with the 3D position embedding acting as a metric-scale auxiliary channel on top of the panoramic canvas. We also see how much the unified panoramic representation contributes compared to passing the images straight: tasks that require relating objects across different parts of the scene perform notably better with our method, with the largest improvement on relative direction. The exception is object counting, where the base VLM retains a small lead, which we suspect reflects small reprojection inaccuracies that can fragment or merge instances on the canvas. Canvas origin. We evaluate the same checkpoint on SQA3D under four canvas origin strategies (Table 5). Using the agent’s pose increases the results noticeably on most tasks, except on the Is and How categories, which are generally viewpoint agnostic. Other camera placement strategies generally do not greatly influence the results, except for the outside-scene origin where the results drop consistently across the tasks.
5
Conclusion
OneCanvas reframes 3D scene understanding as 2D spatial reasoning on a panoramic canvas. Multiview RGB-D is reprojected onto a single equirectangular panorama with each patch carrying its metric 3D position through an additive 3D position embedding, and the pretrained VLM reads the result through its native attention. A spatial pretraining curriculum of objects placed on an otherwise empty canvas and referenced from the prompt warms up geometric reading before downstream fine-tuning. The combination sets a new state of the art on VSI-Bench and SQA3D, and tops the zero-shot SPBench leaderboard. Limitations. OneCanvas requires depth and camera poses, which pure RGB methods avoid, and pose-estimation failures can still degrade the canvas, though feed-forward metric reconstruction is narrowing this gap. The single global panorama trades fine spatial precision for compactness and 9
may limit very large or outdoor scenes. The pretraining curriculum is hand-authored, so adding a qualitatively new spatial skill requires writing a new task generator rather than collecting more data. Finally, the method assumes a backbone with at least a 2D position encoding to carry each patch’s longitude and latitude, which holds for the multi-axis RoPE in Qwen3-VL and similar video-VLMs but not for 1D-position models. All experiments use Qwen3-VL-8B, and validating on additional backbones is left to future work.
Acknowledgments and Disclosure of Funding This project was funded by the ERC Consolidator Grant Gen3D (101171131). We also thank Angela Dai for the video voice-over.
References Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitScenes: A diverse realworld dataset for 3d indoor scene understanding using mobile RGB-D data. In Advances in Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021. Hunar Batra, Haoqin Tu, Hardy Chen, Yuanze Lin, Cihang Xie, and Ronald Clark. Spatialthinker: Reinforcing 3d reasoning in multimodal llms via spatial rewards. arXiv preprint arXiv:2511.07403, 2025. Zhongang Cai, Ruisi Wang, Chenyang Gu, Fanyi Pu, Junxiang Xu, Yubo Wang, Wanqi Yin, Zhitao Yang, Chen Wei, Qingping Sun, et al. Scaling spatial intelligence with multimodal foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. Jieneng Chen, Wenxin Ma, Ruisheng Yuan, Yunzhi Zhang, Jiajun Wu, and Alan Yuille. Thinking with spatial code for physical-world video reasoning. arXiv preprint arXiv:2603.05591, 2026. Pingyi Chen, Yujing Lou, Shen Cao, Jinhui Guo, Lubin Fan, Yue Wu, Lin Yang, Lizhuang Ma, and Jieping Ye. Sd-vlm: Spatial measuring and understanding with depth-encoded vision-language models. In Advances in Neural Information Processing Systems (NeurIPS), 2025. Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26428–26438, 2024. An-Chieh Cheng, Yang Fu, Yukang Chen, Zhijian Liu, Xiaolong Li, Subhashree Radhakrishnan, Song Han, Yao Lu, Jan Kautz, Pavlo Molchanov, et al. 3d aware region prompted vision language model. arXiv preprint arXiv:2509.13317, 2025. Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017. Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Shijie Zhou, Dilin Wang, et al. Vlm-3r: Vision-language models augmented with instructionaligned 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 31054–31065, 2026. Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776, 2025. Qi Feng. Visuospatial cognitive assistant. arXiv preprint arXiv:2505.12312, 2025. 10
Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025. Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494, 2023. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances in Neural Information Processing Systems, 37:113991– 114017, 2024a. Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024b. Yuan Jia, Ruimin Wu, Rui Song, Jiaojiao Li, and Bin Song. Liftproj: Space lifting and projectionbased panorama stitching. arXiv preprint arXiv:2512.24276, 2025. Seongmin Jung, Seongho Choi, Gunwoo Jeon, Minsu Cho, and Jongwoo Lim. Panogrounder: Bridging 2d and 3d with panoramic scene representations for vlm-based 3d visual grounding. arXiv preprint arXiv:2512.20907, 2025. Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European conference on computer vision, pages 71–91. Springer, 2024. Hongxing Li, Dingming Li, Zixuan Wang, Yuchen Yan, Hang Wu, Wenqi Zhang, Yongliang Shen, Weiming Lu, Jun Xiao, and Yueting Zhuang. Spatialladder: Progressive training for spatial reasoning in vision-language models. In International Conference on Learning Representations (ICLR), 2026. Xiang Li, Jian Ding, Zhaoyang Chen, and Mohamed Elhoseiny. Uni3dl: A unified model for 3d vision-language understanding. In European Conference on Computer Vision, pages 74–92. Springer, 2024. Huizhi Liang, Yichao Shen, Yu Deng, Sicheng Xu, Zhiyuan Feng, Tong Zhang, Yaobo Liang, and Jiaolong Yang. Hispatial: Taming hierarchical 3d spatial understanding in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views. In International Conference on Learning Representations (ICLR), 2026. Zekai Lin and Xu Zheng. Panoenv: Exploring 3d spatial intelligence in panoramic environments with reinforcement learning. arXiv preprint arXiv:2602.21992, 2026. Xianzheng Ma, Brandon Smart, Yash Bhalgat, Shuai Chen, Xinghui Li, Jian Ding, Jindong Gu, Dave Zhenyu Chen, Songyou Peng, Jia-Wang Bian, et al. When llms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models. arXiv preprint arXiv:2405.10255, 2024a. Xianzheng Ma, Tao Sun, Shuai Chen, Yash Bhalgat, Jindong Gu, Angel X Chang, Iro Armeni, Iro Laina, Songyou Peng, and Victor Adrian Prisacariu. Do 3d large language models really understand 3d spatial relationships? In The Fourteenth International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2603.23523. 11
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. In International Conference on Learning Representations (ICLR), 2023. Yuexin Ma, Tai Wang, Xuyang Bai, Huitong Yang, Yuenan Hou, Yaming Wang, Yu Qiao, Ruigang Yang, and Xinge Zhu. Vision-centric bev perception: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):10978–10997, 2024b. Yunze Man, Liang-Yan Gui, and Yu-Xiong Wang. Situational awareness matters in 3d vision language reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13678–13688, 2024. Kun Ouyang, Yuanxin Liu, Haoning Wu, Yi Liu, Hao Zhou, Jie Zhou, Fandong Meng, and Xu Sun. Spacer: Reinforcing mllms in video spatial reasoning. arXiv preprint arXiv:2504.01805, 2025. Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European conference on computer vision, pages 194–210. Springer, 2020. Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models. In International Conference on Learning Representations (ICLR), 2026. Kevin Qu, Haozhe Qi, Mihai Dusmanu, Mahdi Rad, Rui Wang, and Marc Pollefeys. Loc3r-vlm: Language-based localization and 3d reasoning with vision-language models, 2026. URL https: //arxiv.org/abs/2603.18002. Mukund Varma T, Peihao Wang, Zhiwen Fan, Zhangyang Wang, Hao Su, and Ravi Ramamoorthi. Lift3d: Zero-shot lifting of any 2d vision model to 3d. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21367–21377, 2024. Haochen Wang, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiangyu Zhang, and Zhaoxiang Zhang. Ross3d: Reconstructive visual instruction tuning with 3d-awareness. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9275–9286, 2025a. Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5294–5306, 2025b. Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20697–20709, 2024a. Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, Xihui Liu, Cewu Lu, Dahua Lin, and Jiangmiao Pang. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b. Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08769, 2023. Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan. Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence. In Advances in Neural Information Processing Systems (NeurIPS), 2025a. Junfei Wu, Jian Guan, Kaituo Feng, Qiang Liu, Shu Wu, Liang Wang, Wei Wu, and Tieniu Tan. Reinforcing spatial reasoning in vision-language models with interwoven thinking and visual drawing. In Advances in Neural Information Processing Systems (NeurIPS), 2025b. Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025a. 12
Rui Yang, Ziyu Zhu, Yanwei Li, Jingjia Huang, Shen Yan, Siyuan Zhou, Zhe Liu, Xiangtai Li, Shuangye Li, Wenqian Wang, et al. Visual spatial tuning. arXiv preprint arXiv:2511.05491, 2025b. Shusheng Yang, Jihan Yang, Pinzhi Huang, Ellis L Brown II, Zihao Yang, Yue Yu, Shengbang Tong, Zihan Zheng, Yifan Xu, Muhan Wang, et al. Cambrian-s: Towards spatial supersensing in video. In The Fourteenth International Conference on Learning Representations, 2026. Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A highfidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. Ruosen Zhao, Zhikang Zhang, Jialei Xu, Jiahao Chang, Dong Chen, Lingyun Li, Weijian Sun, and Zizhuang Wei. Spacemind: Camera-guided modality fusion for spatial reasoning in vision-language models. arXiv preprint arXiv:2511.23075, 2025. Duo Zheng, Shijia Huang, Yanyang Li, and Liwei Wang. Learning from videos for 3d world: Enhancing mllms with 3d vision geometry priors. In Advances in Neural Information Processing Systems (NeurIPS), 2025a. Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d llm: Learning position-aware video representation for 3d scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8995–9006, 2025b. Xu Zheng, Chenfei Liao, Ziqiao Weng, Kaiyu Lei, Zihao Dongfang, Haocong He, Yuanhuiyi Lyu, Lutao Jiang, Lu Qi, Li Chen, et al. Panorama: The rise of omnidirectional vision in the embodied ai era. arXiv preprint arXiv:2509.12989, 2025c. Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pretrained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911–2921, 2023. Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision-language understanding via promptable queries. In European Conference on Computer Vision, pages 188–206. Springer, 2024.
A
Object Referencing: Alternative Class-Labelled Design
The main paper places synthetic axis-aligned boxes on the canvas whose per-patch features are drawn from a shared random pool of activations harvested from held-out scenes, and references each placement in the prompt via an inline copy of the same patch feature. The motivation is to disconnect object size and appearance from class identity, so geometry on the canvas becomes the principal signal for solving the curriculum (see the curriculum-objects discussion in the main method). A natural alternative is to populate the canvas with real objects harvested in-context from scene instance annotations Wang et al. [2024b], pasted onto the canvas with their original per-patch features, and referenced in the prompt by class name. Features are extracted within the original scene rather than re-encoded from cropped objects in isolation, since the latter would discard the surrounding-scene context each patch was encoded with. This variant adds text-to-object grounding as a side benefit, at the cost of re-introducing the class priors (typical door height, typical bathroom size) that the synthetic-box scheme is designed to suppress, and of depending on ground-truth instance bounding boxes to harvest the objects. Table 6 compares the two curriculum content choices on the full VSI-Bench test split, holding the backbone, canvas construction, MRoPE schedule, task families, family-flat weighting, stage-2 fine-tuning, and evaluation protocol fixed. The only thing that changes between rows is what populates the canvas in stage 1. 13
Table 6: Curriculum content study on VSI-Bench (full test split). Both rows share the same backbone, canvas construction, task families, stage-2 fine-tuning, and evaluation protocol. Only the stage-1 curriculum content differs. Synthetic boxes: axis-aligned boxes with shared-random per-patch features, referenced in the prompt by inline patch. Scene-harvested objects: real objects pasted from in-context scene crops with their original per-patch features, referenced by class name. Curriculum content
Numerical Question
Avg.
Multiple-Choice Question
Obj.Cnt. Abs.Dist. Obj.Size Room Size Rel.Dist. Rel.Dir. Route App.Order Synthetic boxes (main paper) 70.1 Scene-harvested objects 69.4
B
68.4 69.1
59.5 58.8
75.4 75.8
76.5 79.3
71.0 68.5
84.8 83.4
59.8 59.8
65.7 60.8
Training Compute Budget
All OneCanvas experiments use 8 NVIDIA RTX A6000 GPUs (Ampere architecture, 48 GB GDDR6 each) with DeepSpeed ZeRO-2 and Flash Attention 2. Stage 1 (spatial pretraining) takes ≈35 h of wall-clock time, and Stage 2 (QA fine-tuning) takes ≈37.5 h, for a total of ≈72.5 h on 8 GPUs (580 raw A6000-GPU-hours). Table 7 compares this to the reported training cost of competing methods, normalized to A100-equivalent GPU-hours. Table 7: Training compute comparison. Wall-clock training time and total compute for OneCanvas and competing methods, normalized to A100-equivalent GPU-hours using the per-GPU-type factors in Table 8. OneCanvas numbers combine both training stages. Method
GPU type
# GPUs
Wall-clock
Raw GPU-h
A100-equiv. GPU-h
OneCanvas (Ours) VLM-3R Fan et al. [2026] ViCA Feng [2025] SpaceMind Zhao et al. [2025] SenseNova-SI Cai et al. [2026]
A6000 H200 H100 H100 H100†
8 16 8 64 128
≈72.5 h 5h 55 h 25 h 72 h
580 80 440 1,600 9,216
290 240 1,320 4,800 27,648
† GPU type not stated in the paper, assumed H100.
GPU normalization. We convert raw GPU-hours to A100-equivalent GPU-hours by multiplying by the dense BF16 Tensor Core throughput ratio between the source GPU and the A100 SXM4. We use dense BF16 (no structured 2:4 sparsity) because that matches what training actually runs: BF16 numerics, no sparsity kernels. The factors are listed in Table 8 and are simply TFLOPSsrc /TFLOPSA100 rounded to one significant digit. This conversion captures peak compute only. In practice the achievable speedup on H100/H200 is bounded below the 3× ratio by memory bandwidth (HBM ratio 1.67× for H100, 2.4× for H200) and by multi-node scaling efficiency at the 64- and 128-GPU jobs in Table 7, both of which would shrink the competitor wall-clock conversions. We use the peak ratio anyway because it is the most-cited and least-disputed figure of merit, and the order-of-magnitude conclusion against the strongest competitors holds even at the conservative bandwidth-bound conversion. Table 8: GPU normalization factors used in Table 7. Dense BF16 Tensor Core TFLOPS (no 2:4 sparsity) are NVIDIA datasheet values, and HBM bandwidth is shown for reference but is not used in the conversion. The A100-equivalent factor is computed as TFLOPSsrc /TFLOPSA100 = TFLOPSsrc /312, rounded to one significant digit. GPU A6000 A100 SXM4 H100 SXM5 H200 SXM5
Dense BF16
HBM bandwidth
A100-equiv. factor
154.8 TFLOPS 312 TFLOPS 989 TFLOPS 989 TFLOPS
768 GB/s 2,000 GB/s 3,350 GB/s 4,800 GB/s
154.8/312 ≈ 0.5× 1.0× 989/312 ≈ 3.0× 989/312 ≈ 3.0×
The H200 shares the GH100 compute die with the H100 and differs only in memory capacity (141 GB HBM3e vs. 80 GB HBM3) and bandwidth, so it carries the same dense BF16 throughput and the same conversion factor. The 989/312 ratio rounds down from 3.17 to 3.0, which mildly understates competitor compute and therefore biases the comparison against our method. 14
Per-method calculations. reported by each method.
Table 9 applies the factors from Table 8 to the wall-clock training time
Table 9: Per-method A100-equivalent compute. A100-equivalent GPU-hours equal (# GPUs) × (wall-clock hours) × (A100-equiv. factor). # GPUs and wall-clock are taken from each method’s paper, factors come from Table 8. Method OneCanvas (Ours) VLM-3R Fan et al. [2026] ViCA Feng [2025] SpaceMind Zhao et al. [2025] SenseNova-SI Cai et al. [2026]
C
# GPUs
Wall-clock (h)
Factor
A100-equiv. GPU-h
vs. Ours
8 16 8 64 128
72.5 5 55 25 72
0.5× 3× 3× 3× 3×
290 240 1,320 4,800 27,648
1× 0.8× 4.6× 17× 95×
Spatial Pretraining Curriculum: Task Details
This section details the tasks that make up the stage-1 spatial pretraining curriculum. The curriculum organizes its task instances into six families and applies family-flat weighting: each family contributes an equal share of every minibatch, and every task within a family carries the same weight. The six families are Metric Measurement, Egocentric Direction, Multi-Turn Navigation, Observability, Counting, and Multi-Target Bounding-Box Readout. All tasks are trained on a canvas containing only the placed objects, at procedurally sampled centres, dimensions, and yaws. No scene is ever loaded onto the canvas during stage 1, so the model only sees the placed objects. Each placed box is realised as the set of canvas patches that fall within its volume, all carrying a single feature vector sampled from a precomputed pool of real ViT patch features. The pool is built once by running Qwen3-VL’s frozen vision encoder on held-out scenes from ScanNet Dai et al. [2017], ARKitScenes Baruch et al. [2021], and ScanNet++ Yeshwanth et al. [2023], and box features are drawn independently from it at training time. Because the pool is scene-agnostic, a box cannot leak the current scene’s identity, and multi-target tasks cannot shortcut by clustering on tokens from a shared source frame. Because the patches are real ViT outputs, their per-token feature distribution matches what the VLM sees on real scenes, so the stage-1 reading capability transfers to stage 2 where the canvas is populated with real-scene patches. We describe each family below, giving each task a short descriptive name followed by its precise input and output format. C.1
Metric Measurement
Pairwise distance. Two boxes are placed on the canvas with non-overlapping supports and a surface-to-surface separation of at least 5 cm. The prompt references each via an inline patch, and the model is asked to report the shortest distance between the two surfaces in metres, rounded to one decimal. The label is the scalar regression target. Closest-of-four. Five boxes are placed on the canvas (one target, four candidates), with the winnervs.-runner-up surface-distance gap constrained to at least 5 cm. The prompt references the target and the four candidates via inline patches, and the model picks which of the four candidates is closest to the target as a 4-way MCQ with answers {A, B, C, D}. Non-rectangular floor area. A synthetic L- or T-shaped floor polygon is placed in the scene, rotated by a random yaw, and rendered as a thin slab of lifted points. The model reads off the floor area in square metres (one-decimal regression). C.2
Egocentric Direction
The world-frame direction tasks (the two-way, three-way, four-way diagonal, four-way cardinal, and clock-face variants below) place 3 boxes: a “ref” box at the agent position, a “fwd” box defining the 15
facing direction, and a target box to classify. The prompt reads “If you are standing at [ref] and facing [fwd], . . .”, with the bracketed names rendered as inline patches bound to the corresponding canvas patches. The camera-frame direction tasks place 2 boxes (target and pivot), with the canvas itself reoriented onto a real scene camera pose so no explicit facing reference is needed. The answer space varies by task. Two-way direction. 2-way MCQ: {left, right}. The question is “is the target to the left or the right?”. Three-way direction. 3-way MCQ: {left, right, back}. The “back” bin activates when the target lies more than 135◦ from the heading. Four-way diagonal direction. 4-way diagonal MCQ: {front-left, front-right, back-left, back-right}. Sampling enforces an angular margin from the cardinal axes so the diagonal label is well-defined. Four-way cardinal direction. 4-way cardinal MCQ {front, back, left, right} with ±45◦ bin boundaries. Camera-frame direction (easy / medium / hard). Same three answer spaces as the world-frame two-way, three-way, and four-way diagonal variants, but the canvas is recentered and reoriented onto a real scene camera pose drawn from the frame sequence. The question is phrased from the camera’s perspective, “From the camera’s perspective, is X to Y’s . . . ?”. Clock-face bearing. The answer is a clock-face bearing in {1, 2, . . . , 12}, computed as the clockwise angle from the heading (12 = forward, 3 = right, 6 = back, 9 = left). Finer angular resolution than the cardinal or diagonal variants. C.3
Multi-Turn Navigation
The Navigation family carries the N-turn route-planning task at four turn counts (N ∈ {1, 2, 3, 4}), each carrying equal weight in the family. N-turn route planning (N ∈ {1, 2, 3, 4}). A start box, a goal box, and N intermediate waypoint boxes are placed at sampled positions on the canvas. The prompt references each via inline patches. The answer is the sequence of egocentric turn actions that takes the agent from start to goal, with one action per turn. The format is a fill-in MCQ where the question lists the actions as “[please fill in]” slots and the model selects one of 4 letter options. Each path-turn is drawn from {Turn Left, Turn Right}. Additionally, on the first turn we add a “Turn Back” option, so the initial move is drawn from {Turn Left, Turn Right, Turn Back}. C.4
Observability
Appearance order. Four boxes are placed at sampled positions on the canvas. Each box’s per-patch MRoPE T values are overridden to a controlled distribution: the four boxes are assigned consecutive first-appearance T -starts {k, k + 1, k + 2, k + 3} for a random k, and each box’s remaining patch T values are sampled uniformly in [tstart , Tmax ] with one patch pinned to tstart . The prompt references the four via inline patches and asks for the full first-time appearance order as a 4-way MCQ (correct permutation plus three random-permutation distractors). The construction isolates fine-grained T -axis resolution at the one-frame scale, with no scene content available to shortcut the answer. Line-of-sight visibility. A yes/no task: given a viewer box p1 , a target box p2 , and a synthetic axis-aligned occluder box between them, does the line of sight from p1 reach p2 ? Visibility is computed from first principles as the fraction of p2 ’s AABB reachable from p1 by a segment not intersecting the occluder, and the label is “yes” if that fraction is at least 10%. The occluder is always present so box count cannot leak the label. Blocker side, along-segment position, and segment length are drawn from a shared distribution across both labels, with only the perpendicular offset of the occluder centre label-conditional. The task is not temporal. The features are pose-independent, and the task is listed under Observability because it tests the same “can the model resolve whether a region is reachable” capability as appearance order. 16
C.5
Counting and Arithmetic Readout
Object counting. N boxes are placed on the canvas with N drawn uniformly over a wide integer range. All N boxes share their feature draw with a single reference box that the prompt references via an inline patch: “How many [ref] are in the room?”. The model emits the integer count. The shared feature gives the N boxes a common identity that the model can read through content attention, while non-referenced distractor boxes carry their own different feature draws so they cannot be mistaken for the referent. Count parity. Same setup as object counting but the question is “Is the number of [ref] in the room odd or even?”. The answer is “even” or “odd”. The placed count is sampled to make the two labels equiprobable, so the model cannot solve the task by guessing the marginal. Count divisibility-by-three. Same setup but the question is “Is the number of [ref] in the room a multiple of three?”. The answer is “yes” or “no”. The placed count is sampled to make the two labels equiprobable. Phrased as a binary divisibility check rather than a three-way residue classification, which keeps the answer format consistent with the other yes/no tasks. Count-by-side. A hybrid task. Given an origin pose with a forward heading and a reference box, the model counts how many same-feature boxes lie on a specified cardinal side from the agent’s perspective. The side is drawn uniformly from {right, left, front, back} and the target count is drawn from {3, 4, 5, 6} with weights [30, 35, 25, 10]. The answer is the integer. The task combines direction classification with counting. C.6
Multi-Target Bounding-Box Readout
Multi-target bounding-box readout. Two to four boxes are placed on the canvas at sampled centres, dimensions, and yaws. The prompt references each via an inline patch tagged with a letter: “Provide the 3D bounding box of each highlighted region, in the same order: A: [m1], B: [m2], . . .”. The model emits the axis-aligned 3D bounding box (centre and metric extents) for each box in canonical near-to-far order.
17