MonoVoc: Decoupling Geometry and Semantics for Lightweight Monocular Open-Vocabulary 3D Gaussians Pouya Ardehkhani, Zahra Dehghanian, Morteza Abolghasemi, Hamid R. Rabiee Sharif University of Technology, Tehran, Iran {pouya.ardehkhani02,zahra.dehghanian97, m.abolghasemi77, rabiee}@sharif.edu
arXiv:2607.28300v1 [cs.CV] 30 Jul 2026
Abstract Open vocabulary 3D scene understanding is essential for next-generation interactive systems, empowering users to intuitively query and navigate reconstructed environments using natural language. However, current 3D Gaussian frameworks are often bottlenecked by restrictive multiview capture requirements, costly scenespecific optimization, and the massive memory overhead of storing dense language features. We present a novel, training-free pipeline that fundamentally reimagines this paradigm by explicitly decoupling 3D geometric reconstruction from semantic integration. Given a standard monocular video sequence as input, our method efficiently outputs a compact, highly interpretable, and fully searchable objectlevel semantic Gaussian map. Rather than entangling heavy language embeddings within the mapping loop, we extract geometry independently and ground semantics through a lightweight, modular post-processing framework. Extensive evaluations on the Replica dataset demonstrate that this decoupled architecture preserves strong rendering fidelity and competitive segmentation accuracy. Crucially, by replacing dense per-Gaussian storage with modular, object-level semantic embeddings, our approach delivers an order-of-magnitude reduction in memory usage compared to SOTA baselines. This provides a highly efficient, scalable, and practical solution for open-vocabulary 3D retrieval and question answering directly from everyday monocular video.
1
Introduction
Open-vocabulary 3D scene understanding is important for interactive systems that must interpret real environments through natural language. In augmented reality [9], robotics [30], and inspection tasks [28], users should be able to refer to objects with everyday descriptions, such as “the red toolbox” or “the chair behind the table,” without relying on a fixed category set. Recent 3D Gaussian Splatting (3DGS) [11] methods have made 3D scenes searchable with language by attaching semantic or language features to reconstructed scenes. For example, SceneSplat [19] learns open-vocabulary 3D features, OpenGaussian [31] builds instance-level semantic links, ObjectGS [35] brings object awareness to monocular Gaussian reconstruction, and LangSplatV2 [17] and 4D LangSplat [18] improve query efficiency or support dynamic scenes. This line of work shows the promise of language-aware Gaussian maps, but practical monocular deployment also requires efficient reconstruction. Gaussian SLAM methods have made movingcamera 3D reconstruction increasingly practical [22, 10], while open-vocabulary grounding and language-driven segmentation [14] has enabled visual systems to respond to free-form language queries. Yet, in the monocular setting, this integration remains limited: semantics are often coupled with reconstruction, feature learning, or per-scene optimization, and dense semantic features are stored across many Gaussians. This makes the resulting maps costly, less editable, and harder to Preprint.
interpret as object-level representations. We therefore address a missing link in this landscape: a lightweight monocular 3DGS framework that produces maps that are not only renderable, but also searchable, segmentable, and queryable through compact open-vocabulary object semantics. To address this gap, we propose a training-free open-vocabulary scene understanding and question answering framework built on a SLAM-generated 3DGS map. Given monocular video frames and segmentation masks, we reconstruct a geometry-aware Gaussian map and then add semantics as a post-processing step. First, we reverse the effect of alpha blending through our Semantic Color Deblending (SCD) algorithm to recover semantic color evidence for individual Gaussians. Second, we match the recovered colors to a fixed semantic palette, producing discrete object identities. Third, we assign one embedding to each object, so all Gaussians belonging to the same object share a compact semantic representation. This avoids storing a separate language feature for every Gaussian and enables efficient retrieval and question answering.
2
Related Works
Our work bridges the gap between 3D Gaussian Splatting (3DGS) and open-vocabulary language grounding. In particular, we focus on 3D representations that can be searched, segmented, and queried using free-form language. Existing approaches can be broadly categorized into two paradigms: multiview pipelines and monocular single-video pipelines. While both aim to bridge geometry and semantics, they differ fundamentally in capture constraints, reconstruction complexity, runtime efficiency, and the manner in which semantic information is encoded within the resulting 3D representation. 2.1
Multiview language Gaussians
Early open-vocabulary 3D grounding was shaped by LeRF [12], which learned dense language features on a NeRF radiance field [24] and showed that text queries can be localized in 3D through rendered relevancy maps. However, LeRF depends on well-covered, calibrated multiview captures, requires costly optimization, and stores semantics densely, making it less suitable for long monocular videos. Recent Gaussian-based methods improve efficiency and scalability. LangSplatV2 [17] enables faster rendering of high-dimensional language features, while 4D LangSplat [18] extends language-aware Gaussians to dynamic scenes. Other methods improve semantic structure and boundary quality: OpenGaussian [31] uses 2D masks and 2D–3D links, LangSurf [16] aligns language features with object surfaces, and systems such as SLGaussian [3] and SLAG [27] attach language features to Gaussian maps. Together, these methods achieve strong open-vocabulary grounding, faster querying, and sharper segmentation in multiview or well-controlled reconstruction settings. However, they are less suited to our target setting: a single monocular walkthrough. In this case, the input is less complete, camera coverage is more limited, and optimization-heavy semantic learning or dense feature storage becomes costly and harder to scale. As a result, the performance and practicality of these methods do not directly carry over to ordinary monocular videos. In contrast, our approach adds semantics after reconstruction and produces a lightweight, compact, and interpretable object-level map. 2.2
Monocular open-vocabulary 3DGS understanding
Monocular methods are closer to realistic capture scenarios, where a user records a scene with one moving camera. However, existing approaches often still couple semantic understanding with reconstruction and store semantic information densely inside the Gaussian representation. ObjectGS [35] moves toward object-centric monocular Gaussian reconstruction. It models objects using local anchors that generate associated Gaussians while preserving object identity. This improves object separation and supports more object-level reasoning. However, reconstruction and semantic modeling remain tightly coupled in an optimization-heavy pipeline, and the semantic state is not necessarily lightweight or easy to inspect. SceneSplat [19] takes a different direction by using vision–language pretraining and large-scale training to learn generalizable 3D features on Gaussian representations. This is useful for open2
vocabulary scene understanding, but it introduces high training cost and does not directly provide a simple end-to-end monocular reconstruction-and-query pipeline. Its output is also still mainly feature-based rather than an explicitly readable object-level semantic map. Our work is closest in spirit to ObjectGS [35] and SceneSplat [19], since both address semantic understanding for 3DGS in settings closer to monocular capture. However, their efficiency does not directly match our target setting. ObjectGS couples object semantics with reconstruction and optimization, while SceneSplat predicts dense open-vocabulary features over 3DGS representations and does not include 3D map generation in its reported runtime. These design choices lead to larger maps, higher memory cost, or less direct end-to-end runtime comparison. To the best of our knowledge, we are the first to study a training-free, SLAM-based monocular 3DGS pipeline that attaches compact object-level open-vocabulary semantics after reconstruction, without scene-specific semantic optimization or dense per-Gaussian language storage. By assigning one embedding per object rather than one feature per Gaussian, MonoVoc produces a compact, object-readable semantic map for efficient retrieval and question answering.
3
Proposed Method
Figure 1 summarizes our training-free pipeline for open-vocabulary semantic Gaussian mapping. Given a monocular RGB video, we first reconstruct a geometry-aware 3D Gaussian map with HISLAM2 [34], then keep the geometry fixed and add semantics as post-processing. 2D segmentation masks provide semantic color observations, which are assigned to Gaussians using Semantic Color Deblending (SCD); the recovered colors are then quantized to the segmentation palette to obtain objectlevel identities. Finally, representative masked keyframes are used to extract language embeddings for each object color. The resulting compact database links Gaussians, object identities, and embeddings for open-vocabulary retrieval and question answering. We next detail the pipeline components: alpha compositing, SCD, palette quantization, and language embedding assignment.
Figure 1: Overview of the proposed training-free open-vocabulary semantic Gaussian mapping pipeline. Given a monocular RGB video, we first reconstruct a geometry-aware 3D Gaussian map using HI-SLAM2 [34]. We then keep the geometry fixed and add semantics as a post-processing step. 2D segmentation masks provide semantic color observations, which are assigned to Gaussians through Semantic Color Deblending (SCD). The recovered colors are then snapped to a discrete segmentation palette to obtain object-level identities. For each object, representative masked keyframes are used to extract language embeddings. The final database links Gaussians, semantic object colors, and embeddings, enabling open-vocabulary retrieval and question answering. Additional discussion of overlap in deblending is provided in Appendix B.
3
3.1
Training-free open-vocabulary semantic mapping
Our method is a SLAM-based, training-free pipeline for open-vocabulary 3D Gaussian understanding from monocular videos, as shown in Figure 1. Its key idea is to decouple geometry reconstruction from semantic grounding: we first build a stable Gaussian map from RGB input, then attach object colors and language embeddings. This avoids retraining the SLAM system or optimizing language features inside the mapping loop, while producing a compact semantic map queryable with natural language. The pipeline has four stages. First, we reconstruct the scene with HI-SLAM2 [34], a geometry-aware Gaussian SLAM system for fast monocular RGB reconstruction. HI-SLAM2 combines monocular geometry priors with dense SLAM and represents the map using 3D Gaussian splatting, yielding a renderable, geometrically meaningful scene representation. We use this Gaussian map as fixed geometric support for semantic assignment. Section 3.2 reviews the forward alpha-compositing process used by the map. Second, we recover an initial semantic RGB color for each Gaussian using our Semantic Color Deblending (SCD) algorithm. Given 2D segmentation maps, which can come from any reliable segmentation source such as SAM [13], and the fixed Gaussian geometry, SCD uses the alphacompositing contribution of each Gaussian to collect semantic evidence from all pixels it helps render. The resulting per-Gaussian estimate is P p wi,p yp ŝi = P , (1) p wi,p where wi,p is the contribution of Gaussian i at pixel observation p, and yp is the backgroundsubtracted semantic color. Intuitively, this step reverses the renderer: instead of asking what color a Gaussian produces at a pixel, we ask which semantic color best explains all pixels influenced by that Gaussian. The derivation, exact-recovery condition, and overlap-bias analysis of SCD are provided in Section 3.3. Third, we correct the deblended colors using palette quantization. Because alpha blending can mix colors from overlapping Gaussians, the recovered color ŝi may lie between valid 2D segmentation colors. We therefore extract the discrete object palette P from the segmented images and assign each Gaussian to the closest valid semantic color. To make this comparison perceptually meaningful, we compare colors in CIELAB space [4] using CMC distance [5]: sqi = arg minc∈P dCMC (Lab(ŝi ), Lab(c)) .
(2)
This step snaps noisy or blended Gaussian colors back to the object-level labels observed in the 2D segmentations. Additional explanation of why overlap appears in Gaussian deblending is provided in Appendix B. Finally, we attach language embeddings to object colors. For each semantic RGB color, we select representative keyframes based on its prevalence across segmented frames, providing clean object views for embedding extraction. The embedding backend is modular: masked regions can be captioned with a vision-language model such as Osprey [32] and embedded with a language model, or directly encoded with alignment models such as CLIP [25], DINO [2], or Perception Encoder [1]. This masked-region design relates to open-vocabulary segmentation methods such as OVSeg [20], which classify mask proposals with vision-language models. In our implementation, we use CLIP and Perception Encoder with hard- and soft-mask variants, store each semantic-color embedding in a database, and link it to all Gaussians assigned that color. Our contribution is therefore a modular monocular pipeline for open-vocabulary 3DGS understanding. Geometry comes from HI-SLAM2, object identity comes from deblended and quantized segmentation colors, and language meaning comes from masked keyframe embeddings. This produces a compact and readable semantic Gaussian map without per-scene semantic training, while supporting efficient retrieval, question answering, and command grounding in reconstructed 3D scenes. 3.2
Forward Rendering Equation
We first describe the forward rendering process that our method later inverts. For each pixel p, the projected Gaussians are sorted from front to back and alpha-composited into a final semantic color. 4
Each Gaussian i has a semantic color si (not RGB value rather a learnable parameter), opacity oi , projected mean µi , and screen-space covariance Σi . Its pixel-level opacity is computed as 1 αi (p) = min 0.99, oi exp − (p − µi )⊤ Σ−1 (p − µ ) . (3) i i 2 As shown in Equation 3, the contribution is high near the Gaussian center and decreases smoothly according to the projected covariance. Occlusion from closer Gaussians is captured by the accumulated transmittance: Y Ti (p) = (1 − αj (p)) , wi (p) = αi (p)Ti (p). (4) j<i
Equation 4 defines how much visibility remains before Gaussian i and its effective weight wi at pixel p. The final rendered semantic color is then C(p) =
N X
wi (p)si + TN (p)b,
(5)
i=1
where b is the background color and TN is the remaining transmittance after all Gaussians. Equation 5 makes the key structure explicit: each pixel is a weighted mixture of Gaussian semantic colors. Our inverse formulation uses these same weights to recover the underlying semantic colors from rendered observations. 3.3
Semantic Color Deblending (SCD)
We now derive the Semantic Color Deblending (SCD) algorithm used to assign one semantic color to each Gaussian. Let p index a pixel observation, or a pixel–frame observation in the multi-view setting. For fixed Gaussian geometry, opacity, and depth ordering, the compositing weights are known, and the only unknowns are the Gaussian semantic colors si . The forward semantic rendering equation is Cp =
N X
wi,p si + TN +1,p b,
wi,p = αi,p Ti,p ,
(6)
i=1
where b is the background semantic color. Since wi,p and TN +1,p do not depend on si , the renderer is affine in the semantic colors. After subtracting the known background term, yp = Cp − TN +1,p b, we obtain the linear inverse problem yp =
N X
wi,p si .
(7)
i=1
Theorem 1 (Compositing weights are exact mixing coefficients). Statement. Let Ti,p = Q j<i (1 − αj,p ) and wi,p = αi,p Ti,p . Then wi,p ≥ 0,
N X
wi,p + TN +1,p = 1.
(8)
i=1
Thus each pixel is an exact convex mixture of Gaussian semantic colors and the background. Proof. Since αi,p ∈ [0, 1) and Ti,p is a product of nonnegative terms, wi,p ≥ 0. Also, Ti+1,p = Ti,p (1 − αi,p ), so wi,p = αi,p Ti,p = Ti,p − Ti+1,p . (9) Summing Equation (9) over all Gaussians gives N X
wi,p = T1,p − TN +1,p = 1 − TN +1,p ,
i=1
which proves Equation (8). 5
(10)
Theorem 2 (Weighted-average deblending). Statement. For each Gaussian i, consider the perP P 2 Gaussian weighted least-squares problem ŝi = arg mins∈R3 p wi,p ∥s − yp ∥2 . If Wi = p wi,p > 0, then the unique minimizer is P p wi,p yp . (11) ŝi = P p wi,p Proof. Let Ji (s) =
2 p wi,p ∥s − yp ∥2 . Differentiating with respect to s gives
P
X ∂Ji =2 wi,p ∂s p
! s−2
X
wi,p yp .
(12)
p
Setting Equation (12) to zero gives ! X
wi,p
ŝi =
p
X
wi,p yp .
(13)
p
Because Wi > 0, division by Wi gives Equation (11). The objective is a strictly convex quadratic when Wi > 0, so the minimizer is unique. Theorem 3 (Exact recovery in the non-overlapping case). Statement. Assume each pixel is explained by exactly one Gaussian. That is, for each p there exists an index ι(p) such that wι(p),p = 1,
wj,p = 0
for
j ̸= ι(p).
(14)
Then the estimator in Equation (11) recovers the true semantic color exactly: ŝi = si for every Gaussian with Wi > 0. Proof. Using Equation (7) and the one-hot condition in Equation (14), each observation satisfies yp = sι(p) . For Gaussian i, only pixels with ι(p) = i have nonzero weight. Therefore, P P p:ι(p)=i yp p:ι(p)=i si ŝi = P = P = si . (15) p:ι(p)=i 1 p:ι(p)=i 1 Thus the estimator is exact when Gaussian contributions do not overlap. Theorem 4 (Overlap bias of weighted-average deblending). Statement. Assume the linear model P P in Equation (7). Define Gij = p wi,p wj,p , and Wi = p wi,p . If Wi > 0, then the estimator in Equation (11) satisfies X Gij Gii si + sj . (16) ŝi = Wi Wi j̸=i
Proof. Substituting Equation (7) into Equation (11) gives N X 1 X ŝi = wi,p wj,p sj . Wi p j=1
(17)
Rearranging the sums gives N 1 X X ŝi = wi,p wj,p Wi j=1 p
!
N
sj =
1 X Gij sj . Wi j=1
(18)
Separating the term j = i gives Equation (16). Therefore the bias comes only from overlap with other Gaussians, measured by the off-diagonal terms Gij . Equations (11) and (16) summarize the proposed deblending step: it is the exact minimizer of a per-Gaussian weighted least-squares problem, it is exact for one-hot visibility, and under overlap it has an explicit mixing bias. We include only the main derivations here for clarity; additional details and full proofs are provided in Appendix A. 6
4
Experiment
We evaluate the proposed pipeline in terms of reconstruction quality, semantic color quantization, language-based object retrieval, and map efficiency: including memory, runtime, and number of Gaussians. Further analyses and additional experiments are provided in Appendix C. 4.1
Experiment Setup
We evaluate on Replica [26] using monocular RGB sequences as input and the corresponding semantic segmentations as 2D supervision for our training-free semantic mapping pipeline. For each scene, we reconstruct a 3D Gaussian map with HI-SLAM2 [34] using its fixed Replica configuration; our method uses no additional training, optimizer, or train/test split. We then apply semantic deblending and palette quantization with identical settings across scenes: batch size 32 for extracting unique colors from ground-truth segmentations, Stage 2 deblending with 9000 Gaussians per batch at 616 × 344 segmentation resolution, and Stage 3 palette quantization using CMC distance in Lab space with l = 2.0, c = 1.0, and GPU batch size 100,000. For end-to-end semantic rendering error, we render the colored Gaussian point cloud with point size 0.05 for measuring error in stage 3. All experiments are single-run evaluations on one NVIDIA A100 GPU. 4.2
Reconstruction and Rendered Segmentation Evaluation
In the first stage, we assess whether the reconstructed Gaussian map preserves visual quality and semantic consistency after rendering. We report PSNR [8], SSIM [29], and LPIPS [33] for rendering quality, and mIoU [7] for rendered mask accuracy. As shown in Table 1, our method achieves the best PSNR and LPIPS, indicating sharper and more perceptually faithful renderings. While SceneSplat attains the highest mIoU, our method is close without scene-specific semantic training or dense per-Gaussian language storage. These results show that HI-SLAM2 provides a strong geometric backbone, and that our post-processing semantic assignment maintains competitive segmentation quality in a lighter, more modular pipeline. Table 1: MonoVoc’s Stage 1 reconstruction and rendered segmentation results on Replica. We compare with ObjectGS [35] and SceneSplat [19], the closest monocular 3D Gaussian-based methods to our setting. Reported variation is the standard deviation over multiple scenes. Model ObjectGS [35] SceneSplat [19] MonoVoc (ours) 4.3
PSNR ↑
SSIM ↑
LPIPS ↓
mIoU ↑
40.26 ± 0.32 41.25 ± 0.35 41.91 ± 0.34
0.984 ± 0.0015 0.980 ± 0.0018 0.981 ± 0.0012
0.028 ± 0.0021 0.040 ± 0.0025 0.025 ± 0.0016
88.39 ± 0.78% 91.89 ± 0.84% 91.22 ± 0.72%
Quantization Metric Selection
In Stage 3, the deblended Gaussian colors are mapped back to the nearest valid color in the semantic palette. Because alpha blending and overlapping Gaussians can slightly shift these colors, the distance metric used for palette snapping has a direct impact on the final segmentation quality. We compare six quantization metrics: CAM16 [15], CIE94 [23], CIEDE2000 [21], CMC [5], DIN99o [6], and Euclidean distance in RGB space. Each metric is evaluated using a pixel-level semantic mismatch score over 2000 rendered semantic images. To generate each rendered image, we project the colored 3D point cloud into the camera view using the estimated pose and intrinsics, remove points behind the camera, depth-sort the visible points, and draw them on a 2D canvas as colored circles or squares with a point size of 0.05. The rendered segmentation is then compared with the reference segmentation after nearest-neighbor alignment to avoid interpolation artifacts and preserve semantic colors. Pure-white target pixels are ignored since they represent unlabeled or empty regions, and we report errors both over all pixels and over non-white pixels only. As shown in Table 2, CMC achieves the fewest mismatched pixels and the lowest error rates, reducing the non-white error to 7.48%. This suggests that CMC is the most reliable choice for our palette snapping step, as it handles small perceptual color shifts from deblending while still assigning Gaussians to stable object-level semantic colors, as illustrated in Figure 2. Therefore, we use CMC as the default quantization distance. 7
Table 2: Comparison of color-distance metrics for Stage 3 palette quantization over 2000 rendered semantic images and 10,566,418,000 total pixels. Reported variation is the std over multiple scenes. Mismatched Pixels ↓
Mismatch Ratio ↓
Overall Err. (all) ↓
Overall Err. (non-white) ↓
804,803,899 ± 17M 829,702,339 ± 20M 795,231,886 ± 14M 768,638,180 ± 12M 814,373,731 ± 18M 804,133,147 ± 16M
7.62 ± 0.15% 7.85 ± 0.18% 7.53 ± 0.13% 7.27 ± 0.10% 7.71 ± 0.16% 7.61 ± 0.14%
7.62 ± 0.16% 7.85 ± 0.19% 7.53 ± 0.14% 7.27 ± 0.11% 7.71 ± 0.17% 7.61 ± 0.15%
7.83 ± 0.17% 8.07 ± 0.20% 7.74 ± 0.15% 7.48 ± 0.12% 7.93 ± 0.18% 7.83 ± 0.16%
Method CAM16 [15] CIE94 [23] CIEDE2000 [21] CMC [5] DIN99o [6] RGB
Figure 2: CMC [5]-based palette quantization snaps deblended colors to valid semantic colors. 4.4
Language Embedding Evaluation
To evaluate language grounding in the final object-level map, we use 15 text queries and test whether the correct object is retrieved from the semantic Gaussian database. For each object, masked renderings are encoded with CLIP [25] or Perception Encoder [1], using either hard masking, which keeps only the object, or soft masking, which retains limited context. As shown in Table 3, soft masking improves retrieval across models, suggesting context aids recognition. Perception Encoder with soft masking performs best, achieving 80% Top-1 and 87% Top-3 accuracy. This supports storing one compact embedding per object instead of dense language features on every Gaussian. Table 3: Text-based object retrieval results on MonoVoc using 15 text queries.
4.5
Embedding
Masking Type
Top-1 Accuracy (%) ↑
Top-3 Accuracy (%) ↑
CLIP [25] CLIP [25] Perception Encoder [1] Perception Encoder [1]
Hard mask Soft mask Hard mask Soft mask
66.7 ± 5.0 73.3 ± 5.0 73.3 ± 7.1 80.0 ± 5.0
73.3 ± 3.6 80.0 ± 6.2 80.0 ± 5.0 86.7 ± 3.6
Efficiency Analysis
We also evaluate the efficiency of the generated 3D semantic maps in terms of map size, memory usage, and end-to-end runtime. As shown in Table 4, MonoVoc produces a much smaller representation than the compared methods, using only 140K Gaussians per scene, versus 1M in ObjectGS [35] and 1.5M in SceneSplat [19]. This compact map reduces memory usage to 14 MB per scene, nearly an order of magnitude below both baselines. Runtime is also considered: although SceneSplat reports a shorter 8
time, it excludes 3D map generation, making the comparison less direct. In contrast, MonoVoc includes both SLAM-based reconstruction and semantic post-processing, better reflecting the cost of a full monocular pipeline. Overall, these results show that decoupling semantics from reconstruction and storing compact object-level representations, rather than dense per-Gaussian language features, yields a much lighter semantic map while remaining practical in memory and runtime. Table 4: Efficiency comparison in terms of average map size, average memory usage, and average runtime. MonoVoc uses fewer Gaussians and substantially less memory while maintaining a practical end-to-end runtime. Model ObjectGS [35] SceneSplat [19] MonoVoc (ours) ∗
5
# Gaussians per Scene ↓
Memory Usage per Scene ↓
Runtime ↓
1M 1.5M 140K
90 MB 149 MB 14 MB
186 min ± 34 min 40 min∗ ± 22 min 81 min ± 17 min
Runtime is reported without 3D map generation; SceneSplat does not generate the 3D map.
Discussion and Limitations
Discussion. The results support our main design choice: geometry and semantics need not be learned jointly for monocular open-vocabulary 3D understanding. Using HI-SLAM2 [34] as a fixed geometric backbone, we add semantics after reconstruction without retraining the mapping pipeline, achieving strong rendering quality, competitive rendered segmentation accuracy, and a far more compact semantic map than ObjectGS and SceneSplat. Ablations show that deblending and semantic color correction are important, with CMC in Lab space [4, 5] yielding the lowest semantic mismatch after alpha blending. For language grounding, Perception Encoder with soft masking performs best, though the pipeline is model- and segmenter-agnostic. Overall, compact object-level semantics suffice for efficient open-vocabulary retrieval and question answering without storing dense language features on every Gaussian. Limitations. The method has two main limitations, both in semantic assignment. First, palette quantization can be ambiguous when object colors are close in the semantic palette or when deblending yields intermediate colors. We partly address this with CMC distance in Lab space, which is more perceptually meaningful than RGB distance, but performance could improve with better-separated or adaptive palettes. Second, SCD uses an efficient per-Gaussian approximation rather than solving the full global inverse problem over all overlapping Gaussians. This keeps the method lightweight, but may cause errors near object boundaries or in highly overlapping regions. Quantization reduces many mixed-color errors, while future work could add confidence weighting or sparse global refinement. Broader impact, privacy considerations, and deployment risks are discussed in Appendix D.
6
Conclusion
We presented a training-free pipeline for open-vocabulary 3D scene understanding from monocular videos. Rather than learning dense language features during reconstruction, our method first builds a geometry-aware Gaussian map with HI-SLAM2, then assigns semantics in post-processing. Semantic colors are recovered for individual Gaussians via deblending, refined through perceptual palette quantization, and linked to compact object-level language embeddings. This keeps the map readable and lightweight while supporting text-based object retrieval and question answering. Experiments on Replica show that this modular design is effective, achieving strong reconstruction quality, competitive rendered segmentation accuracy, and substantially lower memory usage than baselines. Overall, our work suggests a simple direction for semantic 3D Gaussian maps: keep geometry, object identity, and language meaning modular. Although the method still depends on segmentation quality, color quantization, and approximate deblending, it provides a compact foundation for monocular open-vocabulary scene understanding. Future work can extend it to realworld videos, diverse environments, stronger object-level reasoning, and more robust handling of overlap, occlusion, and noisy semantic observations. 9
References [1] Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr Dollár, and Christoph Feichtenhofer. Perception encoder: The best visual embeddings are not at the output of the network. arXiv:2504.13181, 2025. [2] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9650–9660, October 2021. [3] Kangjie Chen, BingQuan Dai, Minghan Qin, Dongbin Zhang, Peihao Li, Yingshuang Zou, and Haoqian Wang. Slgaussian: Fast language gaussian splatting in sparse views. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 3047–3056, 2025. [4] CIE. Colorimetry – part 4: CIE 1976 L*a*b* colour space. CIE Standard S 014-4/E:2007, 2007. [5] Frank J. J. Clarke, Roderick McDonald, and Bryan Rigg. Modification to the JPC79 colour– difference formula. Journal of the Society of Dyers and Colourists, 100(4):128–132, 1984. doi: 10.1111/j.1478-4408.1984.tb00969.x. [6] Guihua Cui, Ming Ronnier Luo, Bryan Rigg, Gerhard Roesler, and Klaus Witt. Uniform colour spaces based on the din99 colour-difference formula. Color Research & Application, 27(4): 282–290, 2002. doi: 10.1002/col.10066. [7] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303–338, 2010. doi: 10.1007/s11263-009-0275-4. [8] Alain Hore and Djemel Ziou. Image quality metrics: PSNR vs. SSIM. Proceedings of the 20th International Conference on Pattern Recognition (ICPR), pages 2366–2369, 2010. doi: 10.1109/ICPR.2010.579. [9] Shunya Kato, Shuhei Kurita, Chenhui Chu, and Sadao Kurohashi. ARKitSceneRefer: Textbased localization of small objects in diverse real-world 3d indoor scenes. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 784–799. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-emnlp.56. [10] Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [11] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023. doi: 10.1145/3592433. [12] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pages 19729–19739, 2023. [13] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023. [14] Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Languagedriven semantic segmentation. In International Conference on Learning Representations, 2022. 10
[15] Changjun Li, Zhiqiang Li, Zhifeng Wang, Yang Xu, Ming Ronnier Luo, Guihua Cui, Manuel Melgosa, Michael H. Brill, and Michael R. Pointer. Comprehensive color solutions: Cam16, cat16, and cam16-ucs. Color Research & Application, 42(6):703–718, 2017. doi: 10.1002/col. 22131. [16] Hao Li, Roy Qin, Zhengyu Zou, Diqi He, Bohan Li, Bingquan Dai, Dingewn Zhang, and Junwei Han. Langsurf: Language-embedded surface gaussians for 3d scene understanding. arXiv preprint arXiv:2412.17635, 2024. URL https://arxiv.org/abs/2412.17635. [17] Wanhua Li, Yujie Zhao, Minghan Qin, Yang Liu, Yuanhao Cai, Chuang Gan, and Hanspeter Pfister. Langsplatv2: High-dimensional 3D language Gaussian splatting with 450+ FPS. In Advances in Neural Information Processing Systems (NeurIPS), 2025. URL https://arxiv. org/abs/2507.07136. [18] Wanhua Li, Renping Zhou, Jiawei Zhou, Yingwei Song, Johannes Herter, Minghan Qin, Gao Huang, and Hanspeter Pfister. 4D langsplat: 4D language Gaussian splatting via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22001–22011, June 2025. [19] Yue Li, Qi Ma, Runyi Yang, Huapeng Li, Mengjiao Ma, Bin Ren, Nikola Popovic, Nicu Sebe, Ender Konukoglu, Theo Gevers, Luc Van Gool, Martin R. Oswald, and Danda Pani Paudel. Scenesplat: Gaussian splatting-based scene understanding with vision-language pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4961–4972, October 2025. [20] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061–7070, 2023. [21] Ming Ronnier Luo, Guihua Cui, and Bryan Rigg. The development of the cie 2000 colourdifference formula: Ciede2000. Color Research & Application, 26(5):340–350, 2001. doi: 10.1002/col.1049. [22] Hidenobu Matsuki, Riku Murai, Paul H. J. Kelly, and Andrew J. Davison. Gaussian splatting slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [23] R. McDonald and K. J. Smith. Cie94—a new colour-difference formula. Journal of the Society of Dyers and Colourists, 111(12):376–379, 1995. doi: 10.1111/j.1478-4408.1995.tb01688.x. [24] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405–421. Springer, 2020. [25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 18–24 Jul 2021. [26] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797, 2019. URL https://arxiv.org/abs/1906.05797. [27] Laszlo Szilagyi, Francis Engelmann, and Jeannette Bohg. Slag: Scalable language-augmented gaussian splatting. IEEE Robotics and Automation Letters, 2025. 11
[28] Osama Tasneem and Roel Pieters. Human–robot collaborative visual inspection with large language models. Robotics and Computer-Integrated Manufacturing, 98:103154, 2026. doi: 10.1016/j.rcim.2025.103154. [29] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600–612, 2004. doi: 10.1109/TIP.2003.819861. [30] Abdelrhman Werby, Chenguang Huang, Martin Büchner, Abhinav Valada, and Wolfram Burgard. Hierarchical open-vocabulary 3d scene graphs for language-grounded robot navigation. In Robotics: Science and Systems (RSS), 2024. [31] Yanmin Wu, Jiarui Meng, Haijie Li, Chenming Wu, Yahao Shi, Xinhua Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, and Jian Zhang. Opengaussian: Towards pointlevel 3D Gaussian-based open vocabulary understanding. In Advances in Neural Information Processing Systems, volume 37, pages 19114–19138. Curran Associates, Inc., 2024. doi: 10.52202/079017-0604. [32] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [33] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 586–595, 2018. doi: 10.1109/CVPR.2018.00068. [34] Wei Zhang, Qing Cheng, David Skuddis, Niclas Zeller, Daniel Cremers, and Norbert Haala. HI-SLAM2: Geometry-aware gaussian SLAM for fast monocular scene reconstruction. IEEE Transactions on Robotics, 41:6478–6493, October 2025. doi: 10.1109/TRO.2025. 3626627. URL https://doi.org/10.1109/TRO.2025.3626627. Originally available as arXiv:2411.17982. [35] Ruijie Zhu, Mulin Yu, Linning Xu, Lihan Jiang, Yixuan Li, Tianzhu Zhang, Jiangmiao Pang, and Bo Dai. Objectgs: Object-aware scene reconstruction and scene understanding via Gaussian splatting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8350–8360, October 2025.
12
A
Additional deblending proofs
This appendix gives the full derivations for the deblending formulation. We use v for a pixel coordinate and m for a general observation, e.g., a pixel or a pixel–frame pair. All semantic colors lie in R3 , and all vector equations hold channel-wise. Lemma A.1 (Semantic-independence of alpha). Statement. Fix a pixel v. Let i ∈ {1, . . . , N } index Gaussians ordered along the ray through v. Let the unknown semantic color of Gaussian i be si ∈ R3 . Let θi denote all non-semantic parameters of Gaussian i, such as mean, covariance, and opacity. Define αi (v) = gi (v; θi ) ∈ [0, 1), (19) and assume gi does not depend on {sk }N . Then, for all i, k, k=1 ∂αi (v) = 0. ∂sk
(20)
Proof. By assumption, αi (v) = gi (v; θi ) and θi is independent of the semantic colors. Hence, for any k, ∂ ∂αi (v) = gi (v; θi ) = 0. (21) ∂sk ∂sk Lemma A.2 (Semantic-independence of transmittance and weights). A.1. Define T1 (v) = 1, Ti (v) =
i−1 Y
(1 − αj (v))
for
Statement. Assume Lemma
i ≥ 2,
(22) (23)
j=1
wi (v) = αi (v)Ti (v), and
N Y
TN +1 (v) =
(24)
(1 − αj (v)) .
(25)
j=1
Then, for all i, k, ∂Ti (v) = 0, ∂sk
∂wi (v) = 0, ∂sk
∂TN +1 (v) = 0. ∂sk
(26)
Proof. For i ≥ 2, Ti (v) =
i−1 Y
(1 − αj (v)) .
(27)
j=1
Differentiating with respect to sk gives i−1 i−1 ∂Ti (v) X Y = ∂sk ℓ=1
j=1, j̸=ℓ
∂αℓ (v) (1 − αj (v)) − . ∂sk
(28)
By Lemma A.1, each derivative ∂αℓ (v)/∂sk is zero. Therefore ∂Ti (v) = 0. ∂sk
(29)
Also T1 (v) = 1, so ∂T1 (v)/∂sk = 0. Since wi (v) = αi (v)Ti (v), ∂wi (v) ∂αi (v) ∂Ti (v) = Ti (v) + αi (v) = 0. ∂sk ∂sk ∂sk
(30)
Finally, TN +1 (v) is also a product of terms that are independent of the semantic colors, so the same product-rule argument gives ∂TN +1 (v) = 0. (31) ∂sk 13
Theorem A.3 (Forward rendering is affine in semantic colors). Statement. Fix a pixel v. Let si ∈ R3 be unknown semantic colors and let bg ∈ R3 be a known background color. The forward alpha-compositing equation is C(v) =
N X
wi (v)si + TN +1 (v)bg.
(32)
i=1
Then the map (s1 , . . . , sN ) 7→ C(v) is affine. Proof. Let
s1 S = ... ∈ R3N . sN
(33)
Define A(v) = [ w1 (v)I3
w2 (v)I3
···
wN (v)I3 ] ,
(34)
and b(v) = TN +1 (v)bg.
(35)
Then N X
wi (v)si = A(v)S,
(36)
C(v) = A(v)S + b(v).
(37)
i=1
and therefore By Lemma A.2, A(v) and b(v) are independent of the unknown semantic colors. Thus the map from the semantic colors to C(v) is affine. Corollary A.4 (Linearity after background subtraction). Statement. Define the backgroundsubtracted color e C(v) = C(v) − TN +1 (v)bg. (38) Then e C(v) =
N X
wi (v)si .
(39)
i=1
Proof. Using Equation (32), e C(v) =
N X
! wi (v)si + TN +1 (v)bg
− TN +1 (v)bg.
(40)
i=1
The background terms cancel, giving e C(v) =
N X
wi (v)si .
(41)
i=1
Thus, after background subtraction, the inverse problem is linear in the unknown semantic colors. Corollary A.5 (Stacked linear system over many observations). Statement. Let I index observations, such as pixels or pixel–frame pairs. For each m ∈ I, define wm,i as the weight of Gaussian i at observation m and tm = TN +1,m . Stack the observations as Cm1 tm1 bg .. .. Y = B= (42) , . . . Cm|I| tm|I| bg Then there exists a matrix W with block rows Wm = [ wm,1 I3
wm,2 I3 14
···
wm,N I3 ]
(43)
such that Y − B = W S.
Y = W S + B,
(44)
Proof. For each observation m ∈ I, Theorem A.3 gives Cm =
N X
wm,i si + tm bg.
(45)
i=1
Stacking these equations over all observations gives Y = W S + B.
(46)
Y − B = W S.
(47)
Subtracting B from both sides gives
Statement. Fix a pixel v and define
Lemma A.6 (Recursive transmittance update). T1 (v) = 1,
Ti (v) =
i−1 Y
(1 − αj (v))
for
i ≥ 2.
(48)
j=1
Then, for all i ∈ {1, . . . , N }, Ti+1 (v) = Ti (v) (1 − αi (v)) .
(49)
Proof. By definition, Ti+1 (v) =
i Y
(1 − αj (v)) =
j=1
i−1 Y
(1 − αj (v)) (1 − αi (v)) .
(50)
j=1
The product in parentheses is Ti (v), so Ti+1 (v) = Ti (v) (1 − αi (v)) . Lemma A.7 (Weight equals transmittance drop).
(51)
Statement. Let
wi (v) = αi (v)Ti (v).
(52)
wi (v) = Ti (v) − Ti+1 (v).
(53)
Ti (v) − Ti+1 (v) = Ti (v) − Ti (v) (1 − αi (v)) .
(54)
Ti (v) − Ti+1 (v) = Ti (v)αi (v) = wi (v).
(55)
Then, for all i ∈ {1, . . . , N }, Proof. Using Lemma A.6,
Therefore,
Theorem A.8 (Partition of unity).
Statement. Let
TN +1 (v) =
N Y
(1 − αj (v)) .
(56)
j=1
Then N X
N X
wi (v) = 1 − TN +1 (v),
i=1
wi (v) + TN +1 (v) = 1.
(57)
i=1
Proof. By Lemma A.7, N X i=1
wi (v) =
N X
(Ti (v) − Ti+1 (v)) .
i=1
15
(58)
This sum telescopes: N X
wi (v) = (T1 (v) − T2 (v)) + · · · + (TN (v) − TN +1 (v)) = T1 (v) − TN +1 (v).
(59)
i=1
Since T1 (v) = 1, N X
wi (v) = 1 − TN +1 (v).
(60)
wi (v) + TN +1 (v) = 1.
(61)
i=1
Adding TN +1 (v) to both sides gives N X i=1
Corollary A.9 (Nonnegativity and boundedness).
Statement. If αi (v) ∈ [0, 1), then, for all i,
0 ≤ Ti+1 (v) ≤ Ti (v) ≤ 1,
0 ≤ TN +1 (v) ≤ 1.
0 ≤ wi (v) ≤ 1,
(62)
Proof. Since αi (v) ∈ [0, 1), each factor 1 − αi (v) lies in (0, 1]. Therefore every transmittance product is nonnegative and at most one: 0 ≤ Ti (v) ≤ 1.
(63)
Ti+1 (v) = Ti (v) (1 − αi (v)) ,
(64)
By Lemma A.6, so 0 ≤ Ti+1 (v) ≤ Ti (v). By Lemma A.7, wi (v) = Ti (v) − Ti+1 (v).
(65)
Thus wi (v) ≥ 0 and wi (v) ≤ Ti (v) ≤ 1. The same product argument gives 0 ≤ TN +1 (v) ≤ 1. Theorem A.10 (Exact mixing form of alpha compositing). C(v) =
N X
Statement. The rendered color
wi (v)si + TN +1 (v)bg
(66)
i=1
is an exact convex mixture of the Gaussian semantic colors and the background: wi (v) ≥ 0,
N X
TN +1 (v) ≥ 0,
wi (v) + TN +1 (v) = 1.
(67)
i=1
Proof. The nonnegativity of wi (v) and TN +1 (v) follows from Corollary A.9. The sum-to-one property follows from Theorem A.8. Therefore the coefficients in Equation (66) form a convex mixture over the semantic colors {si }N i=1 and the background bg. Lemma A.11 (Convex quadratic form). Statement. Fix a Gaussian index i. Let M index observations. For each m ∈ M, let ym ∈ R3 be a known target and wi,m ≥ 0 be the known contribution weight of Gaussian i. Define X 2 Ji (s) = wi,m ∥s − ym ∥2 . (68) m∈M
Then Ji is a convex quadratic in s. Its Hessian is ! 2
∇ Ji (s) = 2
X
wi,m
I3 .
m∈M
If
P
m∈M wi,m > 0, then Ji is strictly convex and has a unique minimizer.
16
(69)
Proof. We expand the squared norm: 2
∥s − ym ∥2 = (s − ym )⊤ (s − ym ).
(70)
Hence Ji (s) =
X
wi,m (s − ym )⊤ (s − ym ).
(71)
m
Differentiating gives ! ∇Ji (s) =
X
X
2wi,m (s − ym ) = 2
m
wi,m
s−2
m
X
wi,m ym .
(72)
m
Differentiating again gives ! X
2
∇ Ji (s) = 2
wi,m
I3 .
(73)
m
If
P
m wi,m > 0, the Hessian is positive definite, so Ji is strictly convex and the minimizer is unique.
Theorem A.12 (Weighted-average estimator). Statement. Assume X Wi = wi,m > 0.
(74)
m∈M
Then the unique minimizer of X
ŝi = arg mins∈R3
2
wi,m ∥s − ym ∥2
(75)
m∈M
is P m∈M wi,m ym ŝi = P . m∈M wi,m
(76)
Proof. By Lemma A.11, the objective is strictly convex when Wi > 0, so the unique minimizer satisfies ∇Ji (s) = 0. From Equation (72), X 0 = 2Wi s − 2 wi,m ym . (77) m
Thus Wi s =
X
wi,m ym .
(78)
m
Since Wi > 0, P s=
m wi,m ym
Wi
.
(79)
Therefore the minimizer is Equation (76). Corollary A.13 (Accumulation form). Statement. Define X X color_accum[i] = wi,m ym , weight_accum[i] = wi,m . m∈M
(80)
m∈M
If weight_accum[i] > 0, then ŝi =
color_accum[i] . weight_accum[i]
(81)
Proof. This follows directly from Theorem A.12 by substituting the two accumulator definitions from Equation (80). 17
Theorem A.14 (Exact recovery under one-hot mixing). Statement. Let M index observations. Assume the forward model N X Cm = wi,m si + TN +1,m bg (82) i=1
and define ym = Cm − TN +1,m bg.
(83)
Assume one-hot mixing: for every m ∈ M, there exists ι(m) ∈ {1, . . . , N } such that wι(m),m = 1,
wj,m = 0
for
j ̸= ι(m).
(84)
Let Mi = {m ∈ M : ι(m) = i}.
(85)
P m∈M wi,m ym ŝi = P m∈M wi,m
(86)
ŝi = si .
(87)
If |Mi | > 0, then the estimator
recovers the true semantic color: Proof. From Equations (82) and (83), N X
ym =
wi,m si .
(88)
X
wj,m sj = sι(m) .
(89)
Fix a Gaussian i with |Mi | > 0. Then X X X wi,m ym = ym = si = |Mi |si .
(90)
i=1
Under one-hot mixing, ym = wι(m),m sι(m) +
j̸=ι(m)
m∈M
m∈Mi
m∈Mi
Also, X
X
wi,m =
m∈M
1 = |Mi |.
(91)
m∈Mi
Therefore ŝi =
|Mi |si = si . |Mi |
(92)
Theorem A.15 (Overlap decomposition of the estimator). Statement. Assume the backgroundsubtracted forward model N X ym = wj,m sj . (93) j=1
Define Wi =
X
wi,m ,
X
Gij =
m∈M
wi,m wj,m .
(94)
m∈M
If Wi > 0, then the estimator P ŝi =
m∈M wi,m ym
Wi
(95)
satisfies ŝi =
Gii si + Wi
N X j=1, j̸=i
18
Gij sj . Wi
(96)
Proof. Substitute Equation (93) into Equation (95): 1 Wi
ŝi =
N X wi,m wj,m sj .
X
(97)
j=1
m∈M
Rearranging the sums gives !
N
1 X ŝi = Wi j=1
X
wi,m wj,m
sj .
(98)
m∈M
Using the definition of Gij , N
1 X Gij sj . Wi j=1
ŝi =
(99)
Separating the j = i term gives Equation (96). Corollary A.16 (Explicit bias under overlap).
Statement. Define
ei = ŝi − si . Then ei =
(100)
N X
Gii − 1 si + Wi
Gij sj . Wi
(101)
Gij sj , Wi
(102)
wi,m (1 − wi,m ).
(103)
j=1, j̸=i
Equivalently, ei = −
Wi − Gii si + Wi
where
X
Wi − Gii =
N X j=1, j̸=i
m∈M
Proof. Subtract si from Equation (96): ei = ŝi − si =
Gii − 1 si + Wi
N X
Gij sj . Wi
(104)
wi,m (1 − wi,m ).
(105)
j=1, j̸=i
Also, Wi − Gii =
X m
Therefore
wi,m −
X
2 wi,m =
m
X m
Gii Wi − Gii −1=− , Wi Wi
(106)
which gives Equation (102). Corollary A.17 (Norm bound in terms of overlap). ∥ei ∥2 ≤
Gii − 1 ∥si ∥2 + Wi
Statement. Under Corollary A.16, N X j=1, j̸=i
Proof. Apply the triangle inequality to Equation (101): N X Gii ∥ei ∥2 = − 1 si + Wi
j=1, j̸=i
Gij ∥sj ∥2 . Wi
(107)
Gij sj Wi
.
(108)
Gij ∥sj ∥2 . Wi
(109)
2
Thus, ∥ei ∥2 ≤
Gii − 1 ∥si ∥2 + Wi 19
N X j=1, j̸=i
Lemma A.18 (Normal equations of the global least-squares inverse). Statement. Let M index M observations. Stack the unknown semantic colors as ⊤ s1 .. S = . ∈ RN ×3 , (110) s⊤ N
and stack the background-subtracted targets as ⊤ ym1 Y = ... ∈ RM ×3 . ⊤ ym M
(111)
Let W ∈ RM ×N be the matrix with entries Wm,i = wm,i .
(112)
Consider
2
S ⋆ = arg minS∈RN ×3 ∥W S − Y ∥F .
(113)
(W ⊤ W )S = W ⊤ Y.
(114)
⋆
Then S satisfies Equivalently, defining
G = W ⊤ W,
B = W ⊤ Y,
(115)
the system is GS = B, with entries Gij =
X
wm,i wm,j ,
(116) X
Bi =
wm,i ym .
(117)
2 ∥W S − Y ∥F = tr (W S − Y )⊤ (W S − Y ) .
(118)
m∈M
m∈M
Proof. Expand the objective:
Differentiating with respect to S gives 2
∇S ∥W S − Y ∥F = 2W ⊤ (W S − Y ).
(119)
At a minimizer, the gradient is zero:
Therefore ⊤
W ⊤ (W S − Y ) = 0.
(120)
(W ⊤ W )S = W ⊤ Y.
(121)
⊤
Using G = W W and B = W Y gives GS = B. Entrywise, X X Gij = (W ⊤ W )ij = Wm,i Wm,j = wm,i wm,j , m
and
Bi = (W ⊤ Y )i =
X
(122)
m
Wm,i Ym =
m
X
wm,i ym .
(123)
m
Theorem A.19 (Approach 1 as a diagonal approximation). tions can be written row-wise as N X Gij sj = Bi .
Statement. The exact normal equa(124)
j=1
Using the diagonal-only approximation G ≈ D = diag(G11 , . . . , GN N ) 20
(125)
gives the decoupled estimator sei =
Bi = Gii
P m∈M wm,i ym P , 2 m∈M wm,i
Gii > 0.
2 If wm,i = wm,i for all m, then Gii = Wi and P m∈M wm,i ym sei = P = ŝi . m∈M wm,i
(126)
(127)
2 ≈ wm,i , then sei ≈ ŝi . More generally, if wm,i
Proof. From Equation (124), N X
Gii si +
Gij sj = Bi .
(128)
j=1, j̸=i
The diagonal approximation sets the off-diagonal terms Gij , j ̸= i, to zero, giving Gii si ≈ Bi .
(129)
Bi = sei . Gii
(130)
X
(131)
Thus si ≈ 2 If wm,i = wm,i for all m, then
Gii =
X
2 wm,i =
m
wm,i = Wi .
m
Substituting Gii = Wi into Equation (126) gives P m wm,i ym sei = P = ŝi . m wm,i
(132)
2 If wm,i ≈ wm,i , then
Gii =
X
2 wm,i ≈
X
m
wm,i = Wi ,
(133)
m
so the diagonal estimator is close to the weighted-average estimator. Theorem A.20 (Multi-frame weighted average as a single-index weighted LS). Statement. Let F index frames and let Pf index pixels in frame f . Define M = {(p, f ) : f ∈ F , p ∈ Pf }.
(134)
For each observation (p, f ), let yp,f ∈ R3 be a known target and let wi,p,f ≥ 0 be the known weight of Gaussian i. If X X Wi = wi,p,f > 0, (135) f ∈F p∈Pf
then the minimizer of ŝi = arg mins∈R3
X X
2
wi,p,f ∥s − yp,f ∥2
(136)
f ∈F p∈Pf
is
P ŝi =
f ∈F
P
P
f ∈F
p∈P
P f
wi,p,f yp,f
p∈Pf wi,p,f
.
Proof. Using the observation set M, the objective can be written as X 2 ŝi = arg mins∈R3 wi,p,f ∥s − yp,f ∥2 . (p,f )∈M
21
(137)
(138)
Differentiating and setting the derivative to zero gives X 0= 2wi,p,f (s − yp,f ).
(139)
(p,f )∈M
Therefore 0 = 2Wi s − 2
X
wi,p,f yp,f .
(140)
(p,f )∈M
Thus
X
Wi s =
wi,p,f yp,f .
(141)
(p,f )∈M
Since Wi > 0,
P
(p,f )∈M wi,p,f yp,f
. Wi Expanding M back into frames and pixels gives Equation (137). ŝi =
(142)
Corollary A.21 (Accumulation across frames). Statement. Define X X X X color_accum[i] = wi,p,f yp,f , weight_accum[i] = wi,p,f . f ∈F p∈Pf
(143)
f ∈F p∈Pf
If weight_accum[i] > 0, then ŝi =
color_accum[i] . weight_accum[i]
(144)
Proof. Substituting the accumulator definitions in Equation (143) into Equation (137) gives Equation (144). Corollary A.22 (Stacked multi-frame linear-system interpretation). Statement. Let S ∈ RN ×3 stack the Gaussian semantic colors, and let Y ∈ R|M|×3 stack the targets {yp,f }. Let W ∈ R|M|×N have entries W(p,f ),i = wi,p,f . (145) Then the multi-frame forward model is Y = W S, (146) and the estimator in Theorem A.20 is the weighted least-squares minimizer over the expanded index set M. Proof. By definition, each observation (p, f ) satisfies yp,f =
N X
wi,p,f si .
(147)
i=1
Stacking all observations gives Y = W S. (148) The multi-frame objective in Equation (136) is therefore the same per-Gaussian weighted leastsquares problem as before, but with the observation index set expanded to include all pixels across all frames.
B
Why Overlap Happens?
This appendix explains why semantic overlap appears in our pipeline and why it is a natural consequence of alpha-composited Gaussian rendering. The main point is that a rendered pixel is rarely produced by a single Gaussian. Instead, it is usually formed by several projected Gaussians whose contributions are blended in image space. Our deblending step tries to reverse this process from the rendered 2D observations, but when the same Gaussian contributes to pixels belonging to different semantic regions, its recovered color can become a mixture of those regions. Figure 3 illustrates common situations where overlap appears. 22
Figure 3: Typical overlap cases in semantic Gaussian deblending. Overlap is caused by image-space alpha blending. In Gaussian splatting, each 3D Gaussian is projected to a 2D elliptical footprint on the image plane. This footprint is not a hard region. Its opacity is highest near the projected center and decreases smoothly toward the boundary according to the projected covariance. Therefore, even if a Gaussian mainly belongs to one object, its projected footprint may still cover nearby pixels from another object, especially near object boundaries. For a pixel observation p, the background-subtracted semantic color can be written as yp =
N X
wi,p si ,
(149)
i=1
where si is the semantic color of Gaussian i and wi,p is its alpha-compositing weight at pixel p. Equation 149 shows that the observed color at a pixel is a weighted mixture of all Gaussians that influence that pixel. If only one Gaussian contributes to the pixel, then the observation directly represents that Gaussian. However, when several Gaussians overlap in the same pixel, the observation contains semantic evidence from multiple objects at once (Figure 4).
Figure 4: Visualization of the overlap effect in semantic deblending. Object boundaries are the most affected regions. Overlap is especially visible around object boundaries. At the boundary between two objects, the projected Gaussian footprints from both sides can cover the same pixels. For example, a Gaussian located on the edge of a chair may still splat 23
partly onto pixels labeled as the wall or the floor. Similarly, a Gaussian on the edge of a table may influence both table pixels and background pixels. Since the renderer blends these contributions smoothly, the resulting semantic observation near the boundary may lie between the valid object colors. This does not mean that the geometry is incorrect. It is a normal result of using soft Gaussian primitives rather than hard object surfaces. A Gaussian is a smooth basis function with spatial extent, so its influence naturally spreads beyond a single ideal pixel or object mask. The inverse problem is coupled. Our deblending step assigns a semantic color to each Gaussian by collecting all pixel observations in which that Gaussian participates: P p wi,p yp ŝi = P . (150) p wi,p This estimator is exact when the Gaussian only contributes to pixels of one semantic color. However, if Gaussian i also contributes to pixels where other Gaussians are visible, then the observations yp already contain the colors of those other Gaussians. Substituting Equation 149 into Equation 150 gives ! N X 1 X X wi,p wj,p sj , ŝi = Wi = wi,p . (151) Wi j=1 p p The term Gij =
X
wi,p wj,p
(152)
p
measures how often Gaussian i and Gaussian j influence the same observations. When Gij is large for j ̸= i, the recovered color of Gaussian i is pulled toward the color of Gaussian j. This is the overlap bias described in Theorem 4. In other words, the deblending formula is solving the best per-Gaussian average, but the rendered observations themselves are not purely per-Gaussian; they are already mixtures produced by the forward renderer. Multi-view observations can increase ambiguity. The same Gaussian is seen from many frames and viewing directions. In one frame, it may project mostly inside its own object mask. In another frame, due to a different viewpoint, camera pose, occlusion ordering, or segmentation boundary, the same Gaussian may partially overlap a neighboring object. Therefore, the semantic evidence collected for one Gaussian may not be perfectly consistent across all frames. This is important because our estimator uses all observations where the Gaussian has nonzero weight. A Gaussian that is mostly assigned to one object can still receive weaker evidence from another object if it repeatedly overlaps that object in some views. As a result, its averaged semantic color may drift away from the true palette color and move toward an intermediate value. Small opacity does not always mean small accumulated influence. Although a Gaussian may have only a small contribution to a single wrong pixel, this effect can accumulate across many frames. If the Gaussian appears in many views and repeatedly touches a neighboring semantic region, the total weighted evidence from that region can become noticeable. Thus, overlap is not only a single-frame boundary artifact; it is also a multi-view accumulation effect. Why quantization helps. The 2D segmentation maps provide a discrete set of valid semantic colors. However, after deblending, a Gaussian color ŝi may lie between two or more valid colors because of alpha blending. Palette quantization corrects this by snapping the recovered color back to the closest valid segmentation color (Figure 5): sqi = arg minc∈P dCMC (Lab(ŝi ), Lab(c)) .
(153)
This step does not remove the fact that overlap happened. Instead, it uses the discrete structure of the 2D segmentation labels to make the final Gaussian map semantically cleaner. The deblending step estimates where the Gaussian color lies from all rendered observations, and the quantization step maps this estimate back to a valid object-level label. 24
Figure 5: Visualization of CMC-based semantic palette quantization. The overall error metric shows 7.48% error across rendered images.
C
Further Experiments
In this appendix, we provide additional experiments and observations about the sensitivity of our method to 2D segmentation quality. Since our pipeline assigns semantics to 3D Gaussians from 2D segmentation evidence, the quality of the input segmentations directly affects the quality of the final semantic Gaussian map. These experiments are not meant to evaluate a specific segmentation model. Instead, they show what types of segmentation errors are harmful, why they affect the deblending and quantization stages, and what practical conditions make the method work reliably. Failure case from segmentation noise. Figure 6 shows a representative failure case caused by noisy 2D segmentations. The right image shows the semantic rendering obtained from clean segmentation inputs, while the left image shows the result when the input segmentations are corrupted by artificial noise and segmentation artifacts. The noisy input produces a less consistent semantic rendering from the 3D Gaussian map. This happens because our method relies on the 2D segmentation maps as semantic observations. During deblending, each Gaussian collects color evidence from all pixels that it helps render. If the segmentation color at a pixel is wrong, then that incorrect color is also accumulated as evidence for the Gaussians that contributed to that pixel. When this happens repeatedly across frames, the estimated Gaussian semantic color can be pulled away from the correct object color. The problem becomes more visible near object boundaries, where a Gaussian may already contribute to multiple neighboring regions due to alpha blending. Therefore, noisy segmentation does not only create local 2D errors. It can also propagate into the 3D map because the same Gaussian is observed in multiple frames. If a wrong semantic label appears consistently, or if boundary noise is present in many views, the final assigned color of a Gaussian 25
may become incorrect. This explains why segmentation quality is an important practical requirement for our pipeline.
Figure 6: Failure case caused by noisy 2D segmentation inputs. Left: semantic rendering when the input segmentation contains artificial noise and segmentation artifacts. Right: semantic rendering obtained from clean segmentation inputs. Since our method uses 2D segmentation colors as semantic evidence for the Gaussians, noisy labels can be accumulated during deblending and can lead to incorrect semantic assignments in the final 3D map. It is important to note that our method is not tied to a particular segmentation model. The user can choose any segmentation method as long as the resulting segmentation maps are reliable enough. In practice, better segmentation masks lead to cleaner semantic Gaussian assignments, while noisy masks naturally lead to noisier 3D semantics. Color-confusable object experiment. We also evaluate a more challenging setting where different object instances are assigned visually similar segmentation colors. This experiment is designed to test the palette quantization stage. Since palette quantization maps each deblended Gaussian color to the closest valid color in the segmentation palette, it can become difficult when two valid colors are very close in the chosen color space.
Figure 7: Example of color-confusable segmentation inputs. Different objects or object parts are assigned similar colors, making the palette quantization step more challenging. The experiment tests whether the recovered Gaussian colors can still be mapped back to the correct discrete semantic colors. Figure 7 shows examples of similar colors used for different parts of the scene, and Figure 8 shows the corresponding semantic Gaussian rendering. In our experiments, the method remains stable even 26
Figure 8: Semantic rendering result for the color-confusable experiment. The method remains reliable when similar colors are still sufficiently separated in the selected color space. When colors become too close, however, palette quantization can become ambiguous.
when the colors are close, as long as the color distance is still large enough for the quantization metric to separate them. This is because the deblending stage aggregates evidence over many pixels and frames, and the quantization stage then snaps the estimated color to one of the valid palette entries. However, this experiment also highlights an important limitation. If two object colors are almost identical in the quantization space, then palette quantization can fail because the nearest-color decision becomes ambiguous. In such cases, a Gaussian whose recovered color lies between two very similar palette entries may be assigned to the wrong object. This is not a limitation of a specific implementation, but a natural ambiguity caused by using colors as object identifiers but still our method can handle them at an acceptable margin. In our implementation, we use CMC distance in CIELAB space because it gives a perceptually meaningful comparison between colors. Nevertheless, the choice of quantization metric is modular. For special cases, the user may choose a different color space or distance function if it better separates the specific palette used by the segmentation method. In general, assigning more separated colors to different objects makes the quantization stage more robust. Ablation on segmentation quality. To better understand how segmentation quality affects the final semantic map, we perform an ablation study by changing the quality of the 2D segmentation inputs. The purpose of this ablation is to isolate the dependency of our method on the semantic observations. The 3D Gaussian geometry, camera poses, rendering procedure, and quantization method are kept fixed, while only the segmentation inputs are changed. 27
We consider several segmentation quality settings. In the clean setting, the original segmentation maps are used without additional corruption. In the boundary-noise setting, errors are added around object boundaries, where Gaussian overlap is already more likely. In the random-label-noise setting, a percentage of pixels is randomly assigned to incorrect palette colors. In the mixed-boundary setting, boundary pixels are replaced by intermediate RGB values between neighboring object colors. Finally, in the region-artifact setting, small holes, thin noisy structures, or isolated mislabeled regions are added to simulate typical segmentation artifacts. The results show that clean segmentations produce the most consistent semantic Gaussian maps. Boundary noise and mixed boundary colors are especially harmful because they occur in the same regions where Gaussian footprints naturally overlap. Random label noise also degrades the result, but its effect depends on how often the corrupted labels are observed and accumulated across frames. Region artifacts can also create incorrect local assignments, especially when small noisy structures are repeatedly visible from multiple viewpoints. This ablation suggests three practical rules of thumb for preparing segmentation inputs for our method. First, segmentation masks should avoid mixed RGB values at object boundaries. Some segmentation pipelines produce smooth transitions between two object colors near boundaries, but this is harmful for our method because those intermediate colors do not correspond to valid object labels. Second, object colors should be well separated in the color space used by the quantization algorithm. Larger distances between palette colors reduce the chance that a blended or noisy Gaussian color is snapped to the wrong object. Third, segmentation masks should avoid sharp isolated artifacts, such as thin noisy structures or small mislabeled regions, because these can introduce incorrect semantic evidence during the deblending stage.
Artificial corruption of 2D segmentations. Finally, we artificially corrupt the 2D segmentation maps and measure the drop in semantic quality. Starting from the clean segmentation maps, we add increasing levels of noise and then run the same semantic assignment pipeline. This allows us to measure how robust the method is when the segmentation input becomes less reliable. Let mIoUclean be the result obtained using the original segmentation maps, and let mIoUnoise be the result after corrupting the segmentations. We report the performance drop as ∆mIoU = mIoUclean − mIoUnoise .
(154)
We corrupt the segmentation maps using random pixel-level label noise. For a noise level ρ, a fraction ρ of pixels is randomly selected and replaced with another color from the valid segmentation palette. This keeps the corrupted image inside the same palette, but breaks the spatial consistency of the labels. We also test boundary-focused corruption, where noise is applied mostly around object boundaries. This setting is more challenging because boundary pixels are the most likely to influence multiple Gaussians through alpha blending. After corruption, the final semantic rendering quality decreases compared with the clean segmentation setting. The drop in mIoU becomes larger as the corruption level increases. Boundary-focused corruption causes a stronger drop than uniform random noise, because boundary regions already contain larger alpha-compositing ambiguity. In other words, errors near object edges are more likely to affect the semantic color assigned to nearby Gaussians. These results confirm that corrupted 2D segmentations lead to weaker final semantic maps. This is expected because the 2D segmentation maps are the semantic source used by the pipeline. When the input labels are corrupted, the deblending stage receives incorrect observations, and the quantization stage may snap some Gaussians to wrong palette colors. Taken together, these experiments show where the method works well and where it needs cleaner input. The pipeline can handle moderate noise and visually close palette colors when the masks are consistent and the object colors remain separable. However, errors that repeat across frames, especially near object boundaries, can be accumulated by the Gaussians and appear again in the final 3D semantic map. For this reason, the segmentation stage should be treated as an important input preparation step rather than a separate detail of the pipeline. 28
D
Broader Impact
This work is primarily methodological and operates on monocular videos provided by the user. Therefore, its societal impact depends strongly on the input data and deployment context. Potential positive uses include lightweight 3D scene understanding for robotics, augmented reality, inspection, and assistive systems. However, if applied to sensitive indoor spaces without consent, the same capability could raise privacy concerns or support unauthorized spatial mapping. Practical deployments should therefore use consented data, avoid privacy-sensitive scenes when possible, and follow local privacy and data-protection requirements. We do not release any new scraped dataset or high-risk generative model; any released code is intended to reproduce the proposed semantic mapping pipeline on consented or public benchmark data.
29