arXiv:2605.15186v1 [cs.CV] 14 May 2026
VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Field Prediction Kaixin Zhu1∗, Yiwen Tang2 *†, Yifan Yang1 *, Renrui Zhang3 *†, Bohan Zeng1 , Ziyu Guo3 , Ruichuan An1 ,Zhou Liu1 ,Qizhi Chen4 , Delin Qu4 , Jaehong Yoon5 , Wentao Zhang1,6,7‡ 1 Peking University 2 Tencent 3 The Chinese University of Hong Kong 4 Shanghai AI Lab 5 NTU 6 Zhongguancun Academy 7 Beijing Key Lab of Data Intel. & Security (PKU)
Abstract High-quality 3D scene reconstruction has recently advanced toward generalizable feed-forward architectures, enabling the generation of complex environments in a single forward pass. However, despite their strong performance in static scene perception, these models remain limited in responding to dynamic human instructions, which restricts their use in interactive applications. Existing editing methods typically rely on a “2D-lifting” strategy, where individual views are edited independently and then lifted back into 3D space. This indirect pipeline often leads to blurry textures and inconsistent geometry, as 2D editors lack the spatial awareness required to preserve structure across viewpoints. To address these limitations, we propose VGGT-Edit, a feed-forward framework for text-conditioned native 3D scene editing. VGGT-Edit introduces depth-synchronized text injection to align semantic guidance with the backbone’s spatial poses, ensuring stable instruction grounding. This semantic signal is then processed by a residual transformation head, which directly predicts 3D geometric displacements to deform the scene while preserving background stability. To ensure high-fidelity results, we supervise the framework with a multi-term objective function that enforces geometric accuracy and cross-view consistency. We also construct the DeltaScene Dataset, a large-scale dataset generated through an automated pipeline with 3D agreement filtering to ensure ground-truth quality. Experiments show that VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speed.
1
Introduction
High-quality 3D scene reconstruction and understanding are essential for autonomous systems and spatial computing Deitke et al. [2023], Fan et al. [2024], Szymanowicz et al. [2024], Ravi et al. [2025], Tang et al. [2025], Yang et al. [2025b], Zeng et al. [2024]. Recently, the field has shifted from time-consuming per-scene optimization to generalizable feed-forward architectures Team et al. [2026], Zeng et al. [2026]. Models such as VGGT Wang et al. [2025a] and π 3 Wang et al. [2025b] represent this emerging paradigm, enabling complex 3D environments to be reconstructed from sparse input images in a single forward pass. By avoiding expensive iterative optimization for each new scene, these methods provide an efficient geometric foundation for real-time applications. However, fast reconstruction does not naturally imply editable scene understanding Tang et al. [2024], You et al. [2026], Wang et al. [2024]. Existing feed-forward models Yuan et al. [2026], Wang et al. [2025a], Maggio et al. [2025] are mainly designed for static perception and lack mechanisms to ∗ Equal Contribution. † Project Lead ‡ Corresponding Author.
Preprint.
Traditional 3D Feed-forward
Generic 3D Feed-forward Model
…
Input Multiview Image
…
3D Encoder
2D-Lifting Editing
3D Output
3D Decoder
3D Feature Map
Reconstruction Scene
No Editing Capability
SAM2 “move the chair leave the table”
Input Multiview Image
3D Feed-forward model
Edited 3D Output Minutes
Reconstruction Scene
Slow, Inconsistent, 2D-Domain Gap
VGGT-Edit (Our) 3D Displacement Fleld
Input Multiview Image
+
Text Instruction “move the chair leave the table”
VGGT-Edit
Frozen Reconstruction Prior
Edited 3D Output Seconds
Reconstruction Scene
Fast, Consistent, Native 3D Domain
Figure 1: Comparison of 3D Editing Methods.
respond to dynamic human instructions Chen et al. [2024c]. Current 3D editing methods Wu et al. [2024], Chen et al. [2024a], Liu et al. [2025], Liyi et al. [2026] often rely on a “2D-lifting” pipeline, where individual views are edited independently using 2D image editors and then input into the reconstruction model. This indirect process is inherently limited for complex scene transformations: because different views are processed separately, it often breaks multi-view geometric consistency and fails to produce stable 3D structures. This limitation is especially problematic for high-precision applications such as robotic manipulation and interactive simulation, where 3D control is required. To address these challenges, we propose VGGT-Edit, a feed-forward framework for text-conditioned native 3D scene editing. Unlike optimization-based editing methods, VGGT-Edit performs complex scene modifications in a single forward pass. Built on a pre-trained reconstruction backbone, our method introduces a lightweight residual field prediction paradigm. Instead of re-learning the entire scene, VGGT-Edit treats editing as an incremental update to a strong geometric prior and focuses on predicting precise geometric displacements in the 3D field. This design preserves background structure while enabling controllable local modifications. VGGT-Edit is built upon three synergistic technical components. First, we design a multimodal prompt injection module that maps linguistic intent directly into the 3D geometric space. Specifically, depth-synchronized attention aligns instruction embeddings with the backbone’s intrinsic posemodulated features, enabling semantic guidance to be fused at the same feature depth where spatial geometry is represented. In addition, a view-aware weighting mechanism dynamically prioritizes viewpoints with clearer observations, reducing noise and artifacts caused by occlusions and camera boundaries. Second, a dedicated residual transformation head predicts a dense displacement field from the spatially fused features. By adding the predicted residuals directly to the base geometry, the model preserves the structural integrity of unchanged regions while concentrating its capacity on the target edit. Third, we introduce a residual-oriented training objective, including masked scale alignment to address global reconstruction ambiguity, as well as normal and projective consistency losses to enforce fine-grained geometry and multi-view alignment. Together, these components enable VGGT-Edit to perform precise scene transformations, such as moving or resizing objects, while maintaining efficient feed-forward inference. To evaluate the effectiveness of our framework, we conduct extensive experiments on the DeltaScene test set, which consists of 500 high-quality and diverse 3D editing cases. Quantitative results demonstrate that VGGT-Edit significantly outperforms state-of-the-art baselines. Specifically, our model achieves a 30.2 CLIP Radford et al. [2021] Score, representing a 1.3 point improvement over the best existing method, while reducing the C-FID to a record low of 122.4. Most notably, our framework reduces the per-scene editing time to approximately 5 seconds. This performance represents a speedup of about 2 to 120 times compared to current 2D-lifting and optimization-based approaches. These results confirm that VGGT-Edit provides a practical and efficient foundation for interactive 3D scene manipulation. Our contributions are as follows: 2
• VGGT-Edit Framework: We propose a native feed-forward 3D scene editing framework that operates directly in the geometric field, eliminating the multi-view inconsistency and high latency inherent in traditional 2D-lifting approaches. • Synchronized and Weighted Fusion Mechanism: We design a depth-synchronized feature injection strategy together with a view-aware weighting mechanism, enabling stable, controllable, and instruction-driven 3D editing. • DeltaScene Dataset and Automated Pipeline: We develop a scalable data generation pipeline featuring 3D agreement filtering to construct the DeltaScene dataset. This largescale dataset provides approximately 100,000 high-quality training pairs. • Superior Performance: Our method achieves state-of-the-art results in both geometric accuracy and multi-view consistency. Furthermore, VGGT-Edit enables near-instantaneous inference, providing a practical and efficient foundation for interactive applications in spatial computing and robotics.
2
Related Work
2.1
Feed-forward 3D Reconstruction
The field of neural 3D representations has evolved rapidly since the introduction of NeRF Mildenhall et al. [2021], which initially relied on time-consuming per-scene optimization. To enhance efficiency, generalizable methods such as PixelNeRF Yu et al. [2021] and MVSNeRF Chen et al. [2021] introduced feed-forward mechanisms capable of inferring volumetric fields from sparse inputs in a single pass. Recently, the emergence of 3D Gaussian Splatting (3DGS) Kerbl et al. [2023] has shifted the focus toward more efficient rasterization techniques, leading to the development of feedforward Gaussian models like pixelSplat Charatan et al. [2024] and MVSplat Chen et al. [2024b]. Modern architectures have further pushed these boundaries by introducing pose-agnostic learning in PF3plat Hong et al. [2024] and permutation-equivariant geometry priors in π 3 Wang et al. [2025b], while Speed3R Ren et al. [2026] has utilized sparse attention to enable the reconstruction of largescale environments. While these advancements provide a robust geometric foundation for passive perception, they are fundamentally designed for static recovery rather than dynamic interaction. Our work, VGGT-Edit, leverages the powerful geometric priors of π 3 to extend these feed-forward capabilities into the realm of active, instruction-conditioned scene manipulation. 2.2
3D Scene Editing
Traditional 3D editing frameworks, such as Instruct-NeRF2NeRF Haque et al. [2023] and GaussianEditor Chen et al. [2024a], primarily rely on Score Distillation Sampling (SDS) or iterative dataset updating, which often results in extreme computational latency and precludes real-time interaction. To bridge this gap, recent research has shifted toward more efficient pipelines, which can be broadly categorized into optimization-based and feed-forward 2D-lifting approaches. Methods like GaussCtrl Wu et al. [2024] and EditSplat Lee et al. [2025] utilize depth-conditioned diffusion to guide 3D updates, while emerging feed-forward models such as Edit3r Liu et al. [2025] and TRACE Hu et al. [2026] attempt to accelerate the process into a single forward pass. However, these methods remain fundamentally tied to the 2D domain, as their architectures still operate on image-space features or rely heavily on 2D-rendering consistency during both training and inference, leading to spatial ambiguities and compromised geometric integrity in complex scenarios. In contrast, VGGT-Edit introduces a native 3D residual learning paradigm that operates directly within the 3D geometric field by predicting point-level displacements on a fixed prior. By shifting from 2D-dependent modifications to native 3D residual learning, our model effectively handles sophisticated compositional operations, such as moving and deleting multiple objects simultaneously, while ensuring strict geometric stability and near-instantaneous inference.
3
3D Editing Data Pipeline
To train VGGT-Edit, we construct an automated 3D editing data generation pipeline that produces large-scale pairs of original and edited 3D scenes. Given raw multi-view observations, the pipeline 3
3D Editing Data Pipeline 1
Instruction Generation (Qwen3.5-Plus) Input
Object Segmentation (SAM3)
2
Qwen3.5-Plus
…
Object Info
Edit Operations
Input
Output
Object Mask
Object Mask
3D Mask Representation
…
Multi-view Image
Edit Prompt
(Binary Volume)
Category: Sofa Attributes: Leather, Gray Location: Next to Table
Delete
Modify
Move
Add
Challenge
Solution
Object not found or too small
VLM Verification
Challenge
…
Qwen-ImageEditing-max
Edit Prompt
Mask jitter and noise
Final Output 3D Editing Scene Data
…
…
SAM3
Multi-view Image
Output
Multi-view Editing (Qwen-Image-Editing-Max)
Output
Input
Multi-view Image
3
Process
Solution 3D Consistency Filter & Consensus Volume
Point Cloud
Edited Multi-view Images
Per-view Editing
Cross-view Propagation
Consistency Check
Updated Views
Figure 2: The DeltaScene Data Pipeline. Our automated framework leverages LLMs and VLMs to generate diverse 3D editing pairs through a multi-stage consensus filtering process. converts them into high-quality, instruction-aligned, and view-consistent 3D editing pairs through four key stages. The overall design is illustrated in Fig. 2. 3.1
Instruction Generation and Target Selection
The pipeline begins by using Qwen3.5-Plus Yang et al. [2025a] to analyze the multi-view observations of a scene and generate candidate editing instructions. A common failure mode is that the language model may propose targets that are absent, ambiguous, or too small to support reliable 3D editing. To mitigate this issue, we introduce a VLM-based Yang et al. [2025a] verification step. Specifically, the LLM first proposes a set of candidate objects, and the VLM Bai et al. [2025] then verifies their visibility and spatial consistency across multiple views. Only objects that can be clearly identified in most views are retained. This process ensures that the final instruction I is grounded in real, visible, and geometrically valid scene content. 3.2
3D Mask Refinement
After selecting the target object, we use SAM3 Carion et al. [2025] to obtain object masks in each view. However, independently predicted 2D masks often suffer from boundary noise, partial occlusions, and cross-view jitter, which can lead to inconsistent 3D supervision. To improve mask reliability, we apply a 3D consensus filtering strategy. Specifically, we project all 2D masks into 3D space and estimate a consensus volume V̄, representing the region where most views agree on the target object’s location. This consensus volume is then re-projected back to each image plane to obtain a refined mask M̂v . A view-specific mask is considered valid only when it sufficiently overlaps with its consensus projection: 1, if IoU(Mv , M̂v ) > τ, Valid(Mv ) = (1) 0, otherwise. This refinement step reduces noisy supervision and enforces stronger multi-view consistency. 3.3
Sequential Multi-View Editing
A central challenge in data generation is maintaining appearance and geometry consistency of the edited target across viewpoints. Editing each view independently can introduce inconsistent colors, textures, shapes, or spatial layouts, making the resulting data unsuitable for learning native 3D scene editing. To address this issue, we adopt a sequential multi-view editing strategy. Instead of editing all views independently, we edit them in an ordered sequence and condition the current edit on the previously edited viewWu et al. [2025]: Iˆv = Edit(Iv , Mv , I, Cond = Iˆv−1 ). 4
(2)
By propagating visual context across adjacent views, this strategy encourages consistent object appearance and spatial placement throughout the sequence. As a result, the generated editing pairs provide more reliable supervision for learning residual field prediction in 3D space. 3.4
Viewpoint Selection and Quality Control
Not all views provide equally reliable supervision. Some viewpoints may contain severe occlusion, truncation, extreme viewing angles, or weak target visibility. To select high-quality observations, we introduce a Re-projection Fidelity score to evaluate each view. For a given view v, we project its mask into 3D and then re-project it back to the image plane, obtaining a reconstructed mask M̃v . The score is defined as: RF (Mv ) = IoU(Mv , M̃v ) · cos(θv ), (3) where θv denotes the viewing angle. This metric favors views with accurate geometric projection and frontal, unobstructed observations. By filtering out unreliable views, the pipeline provides cleaner supervision and improves the stability of VGGT-Edit training.
4
The DeltaScene Dataset Overview of Our Multi-view 3D Editing Dataset
Scene Type
(a) Scene Category Distribution
(b) Edit Operation Distribution
(c) Edited Object Category Distribution Household Items
Office Supplies
Bedroom / Hotel
Electronic Appliances
Architectural Elements
Living room / Lounge
Large-scale Furniture
MODIFY
DELETE
Bathroom Office
18% 34%
Conference Room Kitchen
MOVE
Bookstore / Library 0
100
150
Total Scene 2000+
200 Count
250
ADD
16% 12%
300
Avg. views 60 frames
Total Samples 45000+
20%
Source Datasets Replica & ScanNet
Figure 3: Overview of the DeltaScene Dataset Details. In this section, we provide a detailed description of the DeltaScene Dataset, which is specifically constructed to address the lack of large-scale, view-consistent data for native 3D scene editing. High-quality data is fundamental to training our residual learning paradigm, as it requires precise geometric alignment between the original and edited scenes, as illustrated in Fig. 3. Data Generation Pipeline. We develop an automated pipeline to generate large-scale, multiview consistent editing pairs. The process begins with a diverse collection of high-quality 3D scene priors from sources including Replica Straub et al. [2019], ScanNet Dai et al. [2017], and ScanNet++ Yeshwanth et al. [2023]. For each scene, we leverage Large Language Models (LLMs) to brainstorm realistic and complex editing instructions. To ensure these edits are spatially grounded, we use Vision-Language Models (VLMs) to identify the target regions within the 3D field. We then apply a multi-view rendering engine to generate the corresponding "before" and "after" image sequences. All editing pairs are refined using 3D consensus filtering and re-projection fidelity scoring, ensuring every edit maintains strict geometric consensus across all viewpoints and providing the necessary ground truth for residual displacement learning. Dataset Statistics and Diversity. DeltaScene consists of approximately 100,000 high-quality editing pairs (including 95,000 training and 500 manual-verified testing samples), covering a wide range of indoor and outdoor environments such as offices, living rooms, and residential spaces. The dataset is designed to be operationally and semantically diverse, incorporating four atomic 3D editing operations: (1) Add, which inserts new style-matched elements; (2) Delete, which removes target objects and recovers the background; (3) Modify, which changes object attributes such as color, material, or texture; and (4) Move, which alters object position or orientation. These operations further support compositional editing, where multiple modifications are applied within the same scene. To evaluate varied geometric properties, we curate a wide selection of household items, 5
office supplies, electronic appliances, and large-scale furniture, alongside architectural elements like windows and doors. This holistic design ensures that VGGT-Edit learns a robust mapping from text instructions to complex 3D changes across any scene context. Quality Control Mechanisms. To guarantee the reliability of our benchmarks, the 500 testing pairs underwent rigorous manual verification and refinement. Each sample was checked for both semantic accuracy—ensuring the visual change strictly follows the text instruction—and geometric stability, confirming that non-edited background regions remain perfectly static. This careful selection process, combined with our re-projection fidelity scoring, ensures that our quantitative evaluations, such as the C-FID and CLIP Score, accurately reflect the model’s performance in real-world scenarios.
5
Model
5.1
Overview
VGGT-Edit
Frozen
View-aware Weighting Module
+ Injection (Add)
Per-view Geometric Descriptors
OpenCLIP Text Encoder
Global Text Instruction
Learnable
Mask Area (𝑎𝑛 )
Global Text Embedding
𝒘𝟏 MLP
Boundary Ratio (𝑏𝑛 )
“move the chair near the window”
+
+
+
Multi-view Inputs (RGB, Mask, Poses)
…
+
L+0
L+8
L+16
L+24
L+32
Decoder Stage 1
Decoder Stage 2
Decoder Stage 3
Decoder Stage 4
𝒘𝟐
𝒘𝑵
Decoder
Decoder Stage 0
Softmax
…
Confidence (𝑐𝑛 )
+
View Weights 𝒘𝒏
Output Tokens
Output Tokens
Pi3 Encoder Original Pi3 Pathway
Training Loss 𝑳𝒆𝒅𝒊𝒕
𝑳𝒑𝒓𝒆𝒔
𝑳𝒏𝒐𝒓𝒎𝒂𝒍
𝑳𝒄𝒂𝒎
𝑳𝚫
Editing Fidelity
Content Preservation
Normal Consistency
Multiview Consistency
Displacement Regularization
Base Geometry (Input) 𝑷𝒃𝒂𝒔𝒆
Base Head Residual Transformation Head Predicted Displacement Field 𝚫𝑷
Edited Geometry (Output) 𝑷𝒆𝒅𝒊𝒕 = 𝑷𝒃𝒂𝒔𝒆 + 𝚫𝐏
+
Figure 4: Overview of the VGGT-Edit Model Architecture. Our model features a synchronized prompt injection mechanism and a residual transformation head, enabling native 3D scene editing through a single forward pass. VGGT-Edit is designed for efficient, instruction-driven native 3D scene editing. Given sparse-view N images {In }N n=1 , camera parameters {θ n }n=1 , and a text instruction I, the model predicts an edited 3D geometry in a single forward pass. Unlike 2D-lifting methods that edit each image independently and then reconstruct the edited scene, VGGT-Edit performs editing directly in the 3D geometric field. This design avoids cross-view conflicts and enables stable, localized scene modifications. As shown in Fig. 4, VGGT-Edit consists of three main architectural components. First, a frozen feedforward reconstruction backbone provides a strong geometric prior. Second, a depth-synchronized text injection module aligns the editing instruction with spatially grounded multi-view features. Third, a residual transformation head predicts a dense residual displacement field, which is added to the base geometry under the guidance of an edit mask. To train this architecture, we further introduce a residual-oriented objective that combines edit reconstruction, non-edit preservation, normal consistency, camera-frame consistency, and residual regularization. This formulation enables VGGT-Edit to preserve unchanged regions and perform localized geometry deformation. 5.2
Feed-forward Geometric Prior
We build VGGT-Edit upon π 3 Wang et al. [2025b], a generalizable feed-forward reconstruction model. Given N sparse-view images and their corresponding camera parameters, π 3 extracts multi-view features using an image encoder Φ and a permutation-equivariant transformer Ψ: F = Ψ (Φ(I1 , . . . , IN ), θ 1 , . . . , θ N ) .
(4)
The backbone further predicts a dense base point map Pbase ∈ RN ×H×W ×3 , which represents the reconstructed 3D geometry of the original scene. Rather than training a scene-specific representation 6
from scratch, we freeze the reconstruction backbone and use it as a generalizable geometric prior. This choice is important for two reasons. First, it preserves the robust spatial structure learned from large-scale reconstruction data. Second, it allows the editing module to focus on modeling the requested change rather than re-learning the entire scene geometry. 5.3
Depth-Synchronized Text Injection
To perform instruction-driven editing, VGGT-Edit must map the semantic intent of a text instruction to the correct spatial region in the 3D field. We therefore introduce a depth-synchronized text injection module, which injects textual guidance into the reconstruction features at layers aligned with the backbone’s pose-modulation stages. Given an instruction I, we obtain a text embedding etext using a pre-trained OpenCLIP Ilharco et al. [2021] encoder. Instead of injecting this embedding only once, we fuse it into the transformer decoder at multiple synchronized layers: L = {l · k + 1}4l=0 ,
k = 8.
(5)
These layers are selected to match the major pose-injection blocks of the reconstruction backbone. As a result, semantic guidance is introduced at the same feature depths where spatial geometry is progressively formed. At each selected layer, we perform text-driven cross-attention between the multi-view features and the instruction embedding. This synchronized design provides continuous semantic guidance throughout the decoding process. Compared with a single early injection, it reduces the risk that textual information fades in deeper layers. Compared with injecting text into every layer, it avoids unnecessary computation and training instability. In practice, this enables the model to produce edits that are both semantically aligned with the instruction and spatially consistent across views. 5.4
View-Aware Importance Weighting
Multi-view observations are not equally informative for editing. In some views, the target object may be clearly visible, while in others it may be occluded, truncated, or close to the image boundary. Treating all views equally can therefore introduce noisy semantic guidance. To address this issue, we introduce a view-aware importance weighting mechanism. For each view n, we construct a geometric descriptor gn = [sn , an , cn ], (6) where sn denotes the visible mask area, an denotes the boundary ratio, and cn denotes the backbone confidence score. These quantities jointly describe the reliability of the target observation in that view. A lightweight MLP predicts a normalized importance weight: exp(MLP(gn )) wn = PN . j=1 exp(MLP(gj ))
(7)
The resulting weight is used to modulate the key and value features derived from the text embedding: √ √ Kn = wn Wk etext , Vn = wn Wv etext . (8) This formulation allows views with more complete observations to contribute more strongly to the editing process, while suppressing unreliable views caused by occlusion or boundary artifacts. 5.5
Residual Field Prediction
After text injection and view-aware weighting, the spatially fused features are passed to a residual transformation head. Unlike the original reconstruction head, which predicts the full scene geometry, our head predicts only the incremental geometric change required by the instruction. Specifically, the head outputs a dense residual displacement field ∆P ∈ RN ×H×W ×3 . Given an edit mask M, the edited point map is obtained by: Pedit = Pbase + ∆P ⊙ M, 7
(9)
where ⊙ denotes element-wise multiplication. This residual formulation is central to VGGT-Edit. Since most regions in a scene remain unchanged after an edit, directly predicting the complete edited geometry is unnecessary and may destabilize training. By predicting only localized residual displacements, VGGT-Edit preserves the background structure inherited from the frozen backbone and concentrates its modeling capacity on the edited region. 5.6
Training Objectives
We train VGGT-Edit with a multi-term objective that supervises edited geometry, preserves unchanged regions, and enforces geometric and projective consistency. Masked Scale Alignment. Feed-forward reconstruction models may exhibit global scale ambiguity relative to ground-truth geometry. Since our goal is to learn accurate relative editing, we compute a per-sample masked least-squares scale factor within the edit region: P n,h,w n,h,w ) · (Pn,h,w ⊙ Mn,h,w ) gt n,h,w (Pedit ⊙ M . (10) s= 2 P n,h,w n,h,w + ϵ P ⊙ M n,h,w edit 2
The aligned prediction is then: P̂ = s · Pedit .
(11)
For compactness, we define a masked ℓ1 loss as: X 1 L1 (A, B; M) = P Mn,h,w An,h,w,c − Bn,h,w,c . 3 M
(12)
n,h,w,c
Edit Reconstruction and Preservation. The edit reconstruction loss supervises the edited region against the target point map: Ledit = L1 (P̂, Pgt ; M). (13) To preserve the unchanged scene structure, we constrain the non-edit region to remain close to the frozen base geometry: Lpres = L1 (Pedit , stopgrad(Pbase ); 1 − M).
(14)
This term discourages unnecessary deformation outside the region and improves background stability. Normal Consistency. To encourage geometrically plausible surfaces, we compute surface normals from the dense point maps using finite differences and penalize angular deviation: 1 X ′ Lnormal = P ′ M (1 − ⟨npred , ngt ⟩) , (15) M where M′ is the mask resized or cropped to match the normal grid. Camera-Frame Consistency. To further enforce multi-view geometric alignment, we supervise the prediction in camera space. We transform the aligned prediction and ground truth using the world-to-camera pose, obtaining camera-space points Xcam and Ycam . We then constrain both perspective rays and log-depth: Lcam = L1 (r(Xcam ), r(Ycam ); M) + L1 (ℓ(Xcam ), ℓ(Ycam ); M),
(16)
where r(x) = (x/z, y/z) denotes the perspective ray and ℓ(x) = log z denotes log-depth. This loss encourages the edited geometry to remain consistent under camera projection. Residual Regularization. sarily large deformations:
Finally, we regularize the residual displacement field to avoid unnecesL∆ =
3
1 P
X M
2
Mn,h,w ∆Pn,h,w 2 .
(17)
n,h,w
The final training objective is: Ltotal = λedit Ledit + λpres Lpres + λnormal Lnormal + λcam Lcam + λ∆ L∆ . 8
(18)
6
Experiments
In this section, we evaluate VGGT-Edit on the proposed DeltaScene Dataset. We conduct experiments to assess its performance in geometric accuracy, multi-view consistency, semantic alignment, and inference efficiency. 6.1
Implementation Details
We implement VGGT-Edit in PyTorch and train it on 8 NVIDIA A100 GPUs for 10k iterations with a batch size of 16. The model is initialized with a frozen π 3 backbone Wang et al. [2025b], while the decoder transformer, text injection module, and residual head are trained using Adam with a learning rate of 1 × 10−4 and cosine decay. The training loss weights are set to λedit = 1.0, λpres = 1.0, λnormal = 0.1, λcam = 0.1, and λ∆ = 0.01. We utilize 95,000 pairs from the DeltaScene Dataset for training and 500 pairs for testing, covering diverse scene categories and editing operations. 6.2
Baselines
We compare VGGT-Edit with representative 3D editing methods across three major paradigms to evaluate its performance. For 2D-lifting approaches, we include GaussCtrl Wu et al. [2024] and Omni-3DEdit Liyi et al. [2026]. It is important to note that GaussCtrl is a per-scene optimization method that maintains multi-view consistency through an iterative refinement process, whereas Omni-3DEdit focuses on unified image-domain editing before 3D lifting. We also compare our model with EditSplat Lee et al. [2025], another optimization-based baseline that achieves high-fidelity results at the cost of substantial per-scene computation time. Finally, we evaluate feed-forward editing methods, including Edit3r Liu et al. [2025] and an adapted version of NoPoSplat Ye et al. [2024]. Since NoPoSplat was originally designed as a generalizable reconstruction model, we extend it into a 2D-lifting baseline by employing Qwen-Image-Editing-Max as a pre-processor to edit input views prior to 3D reconstruction. These comprehensive comparisons highlight the advantages of our native 3D formulation, depth-synchronized text injection, and residual field prediction over both iterative optimization-based pipelines and existing feed-forward architectures. 6.3
Main Results
Evaluation Metrics. We evaluate VGGT-Edit from three aspects: semantic alignment, multi-view consistency, and inference efficiency. For semantic alignment, we report the CLIP Score between rendered views of the edited scene and the input text instruction. To measure cross-view visual and geometric consistency, we compute C-FID and C-KID across rendered viewpoints. These metrics capture view-dependent artifacts such as ghosting, flickering, and inconsistent object structure, which are common in 2D-lifting pipelines. Finally, we report the average inference time per edit to measure efficiency against optimization-based and diffusion-based methods. Quantitative Comparison. Table 1 summarizes the quantitative results on the DeltaScene Dataset. VGGT-Edit achieves the best overall performance across semantic alignment, multi-view consistency, and efficiency. Compared with 2D-lifting methods such as Omni-3DEdit and GaussCtrl, VGGT-Edit obtains substantially lower C-FID and C-KID scores, demonstrating that residual field prediction in the 3D geometric field better preserves cross-view structural consistency than independently editing 2D views. In addition, VGGT-Edit requires only approximately 2 seconds per scene, making it nearly two orders of magnitude faster than optimization-based methods such as EditSplat and clearly more efficient than recent feed-forward baselines such as Edit3r. These results show that VGGT-Edit provides a strong balance between high-fidelity native 3D editing and practical inference efficiency. Qualitative Comparison. As shown in Fig. 5, the qualitative results further demonstrate the effectiveness of VGGT-Edit for native 3D scene editing. Existing 2D-lifting and feed-forward baselines often produce geometric misalignment, blurred boundaries, and view-dependent artifacts. For example, in the ”Erase the middle chair” and ”Add a chair” tasks, methods such as Edit3r and NoPoSplat exhibit ghosting effects and unstable object shapes, as their edits are not directly grounded in the 3D geometric field. In several cases, the edited regions appear as 2D overlays that are weakly attached to the underlying scene geometry. 9
Input Images
Edit Prompts
Edit3r
VGGT-Edit Omni3d-edit (Ours)
NoPoSplat
“Erase the middle chair”
“Add a chair near the door” “Turn a gray leather sofa into a white long-pile sofa” “Move the sofa away from the table”
Figure 5: Qualitative Comparison on Diverse 3D Editing Tasks. We evaluate VGGT-Edit against state-of-the-art baselines across various scene-level operations, including object addition, removal, and transformation. Our method produces sharp, instruction-accurate results while maintaining superior multi-view consistency compared to 2D-lifting approaches. Table 1: Quantitative results on the DeltaScene dataset. CLIP Score measures semantic alignment, while C-FID and C-KID evaluate geometric consistency. Time (s) denotes the end-to-end inference latency, measuring the full process from multi-view input images to the final edited 3D scene. Method GaussCtrl Wu et al. [2024] EditSplat Lee et al. [2025] Omni-3DEdit Liyi et al. [2026] NoPoSplat Ye et al. [2024] Edit3r Liu et al. [2025] VGGT-Edit (Ours)
CLIP Score ↑ 26.4 27.1 28.5 25.8 28.9 30.2
C-FID ↓ 145.2 138.5 128.1 135.4 130.8 122.4
C-KID ↓ 0.192 0.154 0.85 0.112 0.92 0.048
Time (s) ↓ ∼300 ∼600 ∼115 ∼20 ∼10 ∼5
In contrast, VGGT-Edit generates sharp and spatially grounded edits that remain consistent across camera viewpoints. By leveraging a frozen geometric prior and predicting localized residual displacement fields, our method preserves background structure while accurately executing instruction-guided modifications. These visual improvements are consistent with the quantitative results: VGGT-Edit achieves a CLIP Score of 30.2, improving the best competing method by 1.3 points, and obtains the lowest C-FID of 122.4. These results indicate that depth-synchronized text injection and residual field prediction effectively translate semantic instructions into accurate 3D deformations, leading to high visual quality and strong multi-view consistency. 6.4
Ablation Study
Table 2: Ablation study of VGGT-Edit components. Sync-Attn: depth-synchronized attention; ViewW: view-aware weighting; Res-Head: residual transformation head. Sync-Attn ✓ ✓ ✓
View-W ✓ ✓ ✓
Res-Head ✓ ✓ ✓
CLIP Score ↑ 28.1 27.8 29.5 30.2
C-FID ↓ 126.5 127.2 131.4 122.4
C-KID ↓ 0.062 0.068 0.085 0.048
Architecture Effectiveness. We conduct ablation studies to evaluate the contribution of each core component, with quantitative results reported in Table 2. Replacing depth-synchronized text injection 10
with single-layer injection reduces the CLIP Score from 30.2 to 28.1, indicating that multi-stage semantic guidance is important for accurate instruction alignment. Removing view-aware weighting increases C-FID, suggesting that unreliable observations from occluded or boundary views introduce geometric noise. Finally, replacing the residual transformation head with a standard reconstruction head leads to the weakest geometric consistency and background stability. These results demonstrate that depth-synchronized text injection, view-aware weighting, and residual field prediction jointly contribute to accurate, stable, and view-consistent native 3D editing. Operational Efficiency and Robustness. We further analyze the practical utility of VGGT-Edit in terms of efficiency and operational robustness. Unlike 2D-lifting methods whose latency increases with the number of views, VGGT-Edit maintains a constant, low-latency inference of approximately 2 seconds per scene due to its native feed-forward nature. This capability stems from our 3D residual learning paradigm, which empowers the model to map semantic intent to arbitrary point-level displacements while maintaining structural integrity. 6.5
Generalization to Unseen Instructions
Edit Prompt “Rotate the middle chair 90 degrees clockwise."
Output
VGGT-Edit
Input
Figure 6: Generalization to Unseen Instructions. Beyond the primary editing operations used during training, such as addition, removal, relocation, and attribute transformation, we observe that VGGT-Edit demonstrates remarkable generalization to unseen text instructions. This zero-shot capability allows the model to execute complex geometric commands that were not explicitly included in the training set. For instance, as shown in Fig 6, when given an instruction like "rotate the middle chair 90 degrees clockwise." our model can successfully produce the corresponding geometric shift while maintaining the chair’s structural integrity. This flexibility stems from our 3D residual field prediction paradigm. Instead of simply memorizing predefined actions, the model learns a fundamental mapping between semantic intent and raw pointlevel displacements (∆P). By understanding how to translate text guidance into precise spatial deformations, VGGT-Edit can synthesize arbitrary motion fields to satisfy diverse human requests. This confirms that our framework has captured a deep, generalizable understanding of 3D spatial manipulation, making it highly effective for real-world interactive applications. 6.6
Qualitative Analysis
To further illustrate the impact of each component in VGGT-Edit, we provide qualitative comparisons in Figure 7. The visual evidence aligns with our quantitative findings: w/o Depth-Synchronized Attention: Without rhythmic semantic reinforcement, the model exhibits "incomplete editing" or color mismatches, failing to fully manifest the requested changes (e.g., the added object appears faded or incorrectly textured). 11
Reference & Proposed
Target Task “Turn a gray leather sofa into a white long-pile sofa”
Ablation Cases
Edit Mask
(a) Source
(b) Target Task
Incomplete Modify
(d) w/o Sync-Attention
Boundary Noise and Artifacts (e) w/o View-Aware Weighting
(c) Ours (Full Method)
Static Background Distortion (f) w/o Residual Head
Figure 7: Qualitative ablation study. Given the source scene (a) and the editing task (b), our full method (c) produces a coherent result. Removing Sync-Attention (d) leads to incomplete material modification; removing View-Aware Weighting (e) introduces boundary noise and artifacts near occluded regions; removing the Residual Head (f) causes subtle distortion in static background areas. w/o View-Aware Weighting: The absence of this module leads to noticeable geometric artifacts and "floaters," particularly at the edges of occluded regions where the model fails to resolve spatial ambiguities. w/o Residual Transformation Head: Replacing our residual paradigm with a full reconstruction head causes significant background "drifting." Background objects that should remain static exhibit subtle deformations or shifts, confirming the importance of our 3D residual learning for maintaining structural integrity.
7
Conclusion
In this paper, we present VGGT-Edit, a feed-forward framework for instruction-driven native 3D scene editing. By treating editing as a 3D residual field prediction task, our model avoids the multiview inconsistencies common in 2D-lifting pipelines while preserving the structural integrity of the original scene. Through depth-synchronized text injection and view-aware weighting, VGGT-Edit achieves precise semantic-to-spatial alignment and robust feature fusion, enabling localized geometry deformations in a single forward pass. Experimental results on the DeltaScene Dataset show that our method outperforms existing optimization-based and feed-forward baselines in terms of geometric fidelity, multi-view consistency, and inference efficiency. With its ability to handle diverse operations and maintain high-speed performance, VGGT-Edit provides a practical and generalizable solution for real-time, interactive 3D scene editing.
References Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025. David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19457–19467, 2024. Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14124–14133, 2021.
12
Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang, Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21476–21485, 2024a. Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In European conference on computer vision, pages 370–386. Springer, 2024b. Yuedong Chen, Chuanxia Zheng, Haofei Xu, Bohan Zhuang, Andrea Vedaldi, Tat-Jen Cham, and Jianfei Cai. Mvsplat360: Feed-forward 360 scene synthesis from sparse views. Advances in Neural Information Processing Systems, 37:107064–107086, 2024c. Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13142–13153, 2023. Zhiwen Fan, Jian Zhang, Wenyan Cong, Peihao Wang, Renjie Li, Kairun Wen, Shijie Zhou, Achuta Kadambi, Zhangyang Wang, Danfei Xu, et al. Large spatial model: End-to-end unposed images to semantic 3d. Advances in neural information processing systems, 37:40212–40229, 2024. Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 19740–19750, 2023. Sunghwan Hong, Jaewoo Jung, Heeseong Shin, Jisang Han, Jiaolong Yang, Chong Luo, and Seungryong Kim. Pf3plat: Pose-free feed-forward 3d gaussian splatting. arXiv preprint arXiv:2410.22128, 2024. Jiyuan Hu, Zechuan Zhang, Zongxin Yang, and Yi Yang. Trace: High-fidelity 3d scene editing via tangible reconstruction and geometry-aligned contextual video masking. arXiv preprint arXiv:2604.01207, 2026. Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, et al. Openclip. Zenodo, 2021. Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis, et al. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1, 2023. Dong In Lee, Hyeongcheol Park, Jiyoung Seo, Eunbyung Park, Hyunje Park, Ha Dam Baek, Sangheon Shin, Sangmin Kim, and Sangpil Kim. Editsplat: Multi-view fusion and attention-guided optimization for viewconsistent 3d scene editing with 3d gaussian splatting. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 11135–11145, 2025. Jiageng Liu, Weijie Lyu, Xueting Li, Yejie Guo, and Ming-Hsuan Yang. Edit3r: Instant 3d scene editing from sparse unposed images. arXiv preprint arXiv:2512.25071, 2025. Chen Liyi, Wang Pengfei, Zhang Guowen, Ma Zhiyuan, and Zhang Lei. Omni-3dedit: Generalized versatile 3d editing in one-pass. arXiv preprint arXiv:2603.17841, 2026. Dominic Maggio, Hyungtae Lim, and Luca Carlone. Vggt-slam: Dense rgb slam optimized on the sl (4) manifold. arXiv preprint arXiv:2505.12549, 2025. Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99–106, 2021. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. In International Conference on Learning Representations, volume 2025, pages 28085–28128, 2025.
13
Weining Ren, Xiao Tan, and Kai Han. Speed3r: Sparse feed-forward 3d reconstruction models. arXiv preprint arXiv:2603.08055, 2026. Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797, 2019. Stanislaw Szymanowicz, Chrisitian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10208–10217, 2024. Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1–18. Springer, 2024. Yiwen Tang, Zoey Guo, Kaixin Zhu, Ray Zhang, Qizhi Chen, Dongzhi Jiang, Junli Liu, Bohan Zeng, Haoming Song, Delin Qu, et al. Are we ready for rl in text-to-3d generation? a progressive investigation. arXiv preprint arXiv:2512.10949, 2025. DataFlow Team, Bohan Zeng, Daili Hua, Kaixin Zhu, Yifan Dai, Bozhou Li, Yuran Wang, Chengzhuo Tong, Yifan Yang, Mingkun Chang, et al. Openworldlib: A unified codebase and definition of advanced world models. arXiv preprint arXiv:2604.04707, 2026. Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5294–5306, 2025a. Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20697–20709, 2024. Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He. π 3 : Permutation-equivariant visual geometry learning. arXiv preprint arXiv:2507.13347, 2025b. Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508.02324, 2025. Jing Wu, Jia-Wang Bian, Xinghui Li, Guangrun Wang, Ian Reid, Philip Torr, and Victor Adrian Prisacariu. Gaussctrl: Multi-view consistent text-driven 3d gaussian splatting editing. In European conference on computer vision, pages 55–71. Springer, 2024. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. Ling Yang, Kaixin Zhu, Juanxi Tian, Bohan Zeng, Mingbao Lin, Hongjuan Pei, Wentao Zhang, and Shuicheng Yan. Widerange4d: Enabling high-quality 4d reconstruction with wide-range movements and scenes. arXiv preprint arXiv:2503.13435, 2025b. Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Marc Pollefeys, Ming-Hsuan Yang, and Songyou Peng. No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images. arXiv preprint arXiv:2410.24207, 2024. Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12–22, 2023. Junqi You, Chieh Lin, Weijie Lyu, Zhengbo Zhang, and Ming-Hsuan Yang. Instainpaint: Instant 3d-scene inpainting with masked large reconstruction model. Advances in Neural Information Processing Systems, 38: 153574–153594, 2026. Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4578–4587, 2021. Shuai Yuan, Yantai Yang, Xiaotian Yang, Xupeng Zhang, Zhonghao Zhao, Lingming Zhang, and Zhipeng Zhang. Infinitevggt: Visual geometry grounded transformer for endless streams. arXiv preprint arXiv:2601.02281, 2026.
14
Bohan Zeng, Ling Yang, Siyu Li, Jiaming Liu, Zixiang Zhang, Juanxi Tian, Kaixin Zhu, Yongzhen Guo, Fu-Yun Wang, Minkai Xu, et al. Trans4d: Realistic geometry-aware transition for compositional text-to-4d synthesis. arXiv preprint arXiv:2410.07155, 2024. Bohan Zeng, Kaixin Zhu, Daili Hua, Bozhou Li, Chengzhuo Tong, Yuran Wang, Xinyi Huang, Yifan Dai, Zixiang Zhang, Yifan Yang, et al. Research on world models is not merely injecting world knowledge into specific tasks. arXiv preprint arXiv:2602.01630, 2026.
15