Depth-Guided Video Object Counting in Crowded Scenes

arXiv:2608.06236v1 [cs.CV] 6 Aug 2026

Depth-Guided Video Object Counting in Crowded Scenes Yuanjing Xu

Xinyan Liu∗

Weidong Chen

xuyuanjing@stu.hit.edu.cn Harbin Institute of Technology (Weihai) Weihai, Shandong, China

xinyliu@hit.edu.cn Harbin Institute of Technology (Weihai) Weihai, Shandong, China City University of Hong Kong Hong Kong, China

chenweidong@ustc.edu.cn University of Science and Technology of China Hefei, Anhui, China

Zixuan Zou

Linhao Zhang

Zhuangzhe Meng

zouzixuan@stu.hit.edu.cn Harbin Institute of Technology (Weihai) Weihai, Shandong, China

zhanglinhao@stu.hit.edu.cn Harbin Institute of Technology (Weihai) Weihai, Shandong, China

mengzhuangzhe@stu.hit.edu.cn Harbin Institute of Technology (Weihai) Weihai, Shandong, China

Antoni B. Chan

Weigang Zhang

abchan@cityu.edu.hk City University of Hong Kong Hong Kong, China

wgzhang@hit.edu.cn Harbin Institute of Technology (Weihai) Weihai, Shandong, China Harbin Institute of Technology Qingdao Research Institute Qingdao, Shandong, China

Abstract

1

Our primary objective is to advance video object counting in crowded scenes, aiming to robustly count all instances of a target category based on given text or visual prompts. Existing methods rely on RGB information, limiting their discriminative ability in crowded and occluded conditions. To address this, we propose a Depth-Guided Detector (DG-Det) along with a general post-processing pipeline. By integrating depth cues with multi-scale RGB-D cross-attention and explicit occlusion prediction, our method enhances spatial understanding and achieves robust detection in crowded and occluded scenes. Furthermore, we introduce a unified de-duplication framework to eliminate cross-frame redundant counting. To facilitate future research, we also release a new RGB-D Video Object Counting dataset featuring depth information and multiple object categories persequence. Extensive experiments demonstrate that our method achieves a 62.01% reduction in MAE compared to existing baselines, and also produces consistent improvements in RMSE. We provide the source code at https://github.com/streamer-AP/DG-Net and the dataset at https://huggingface.co/datasets/aerospace123/RGBDVideoCount.

We focus on the task of video object counting in crowded scenes, where the goal is to enumerate all instances of a specified target category within a video using either text or visual prompts. However, research on video counting in crowded and occluded scenes remains very limited. In real-world applications such as shelf inventory checks and warehouse inspections, accurate counting from dynamic video streams is a key prerequisite for automated inventory management and out-of-stock warnings, which have substantial practical value. Motivated by this, we focus on video counting in real crowded and occluded object scenes, and conduct the first research on video individual counting in crowded scenes. Unlike common targets in existing video tasks, such as pedestrians and vehicles, same-category objects in multi-category occluded scenes are often highly similar in packaging, texture, and scale. At the same time, these scenes involve dense arrangements, frequent occlusions, and viewpoint changes. Since RGB mainly captures appearance cues such as color and texture, relying solely on RGB features within a single frame makes it difficult to effectively separate tightly adjacent and mutually occluded objects. Across frames, viewpoint variation and short-term occlusion further disturb the stability of appearance representations, making instance matching more prone to confusion. As a result, existing detection-and-association strategies that mainly rely on RGB information face clear limitations in occluded scenes(see Fig. 1). Therefore, the core challenge of this task lies not only in cross-frame association and deduplication at the video level, but also in accurately distinguishing neighboring objects with highly ambiguous appearances within each frame.

CCS Concepts • Computing methodologies → Object detection.

Keywords Object Counting in videos, RGB-D, Object Tracking

∗ Corresponding author.

Introduction

for subsequent counting. Second, in the cross-frame association stage, we propose Depth-Guided Tracking and De-duplication. We incorporate depth constraints between objects into the trajectory matching cost, which improves the robustness of cross-frame matching. On top of this, Occlusion-Adaptive Temporal Voting adaptively adjusts the temporal filtering threshold using the predicted occlusion probability, enabling the recovery of heavily occluded target trajectories while maintaining high precision. Furthermore, since existing video counting datasets contain only RGB information, we construct and release the RGBD-VideoCount dataset. This dataset provides synchronized RGB-D video streams and contains 195 video clips across 6 categories, covering complex scenarios. Experimental results show that the proposed method achieves significant improvements over existing baselines on RGBD-VideoCount. Our contributions are as follows: Figure 1: Qualitative comparison in a dense shelf scene. (a) CountVID misses occluded objects and undercounts. (b) Naive RGB-D concatenation still struggles with overlapping targets. (c) Our method exploits depth cues to better resolve occlusion and achieve more accurate video-level counting.

• We introduce depth cues into video individual counting in crowded scenes, effectively alleviating the limitations of RGB-only methods in occluded scenarios. • We propose a depth-guided RGB-D detection and de-duplication framework that better distinguishes occluded instances while reducing missed detections and duplicate counting. • We release an RGBD-VideoCount dataset with synchronized depth information and multi-category coexisting scenes to facilitate research on video counting with depth.

To overcome the limitations of RGB features in crowded and occluded scenes, introducing depth information to provide front-back relations and relative distance is an effective solution. For tightly arranged or mutually occluded objects of the same category, they still exhibit distinguishable depth differences and spatial hierarchy in 3D space. Such information can not only alleviate occlusion among adjacent objects within a single frame, but also provide geometric constraints that are less sensitive to appearance variation across frames. However, introducing depth does not automatically lead to consistent gains; the key lies in how to fuse it effectively. Direct channel concatenation of RGB and depth features, as shown in Fig. 1(b), ignores the distribution gap between the two modalities and therefore leads to many missed detections. Conventional earlyfusion or late-fusion strategies also often suffer from insufficient feature interaction. Simple fusion methods usually fail to establish effective complementarity between the two modalities and cannot emphasize truly useful geometric information in critical regions. Therefore, for object-level video counting to truly benefit from depth, the key lies not in simply introducing depth, but in designing an effective fusion mechanism that can better handle occlusion, appearance similarity, and multi-scale geometric variation. To fully exploit the potential of depth information for this challenging counting task, we propose a unified framework consisting of depth-guided detection and depth-enhanced deduplication. First, in the detection stage, we design a Depth-Guided Detector (DGDet). Since RGB appearance and depth have different representation properties, DG-Det introduces multi-scale RGB-D cross-attention with Depth Affinity Bias at each level of the feature pyramid. By imposing geometric constraints on feature interaction through local depth relations, it explicitly models the complementarity between the two modalities and promotes feature aggregation in geometrically consistent regions, thereby suppressing background interference and cross-level mismatching. In addition, we design an occlusion-aware prediction head to explicitly estimate the occlusion probability of each target, providing reliable confidence cues

2

Related Work

This section reviews the development of object counting from static images to videos, and further discusses the role of depth information in detection and tracking frameworks. Existing image counting methods can be broadly divided into two categories: density map regression methods [4, 6, 15, 24] and detection-based instance counting methods [5, 9, 22, 23, 25]. The former perform well in highly crowded and homogeneous scenes, but the category is usually implicitly encoded in the network parameters, which limits their generalization to unseen categories and prevents instance-level localization [10, 28]. The latter provides explicit localization, but most of them rely on predefined categories and often degrade in crowded scenes [16, 31]. To overcome the limitation of closed-set categories, open-world counting has received increasing attention. Few-shot counting uses visual exemplars for matching, but it is sensitive to exemplar quality and appearance variation [11, 20, 29]. Zero-shot counting leverages the semantic transfer ability of vision-language models, but it is still easily affected by semantic ambiguity on fine-grained targets [1, 7, 14]. Recent works unify multi-modal prompts for open-world counting, either through density estimation (Lin et al. [19]) or detection-based frameworks (CountGD [2]). Related research in the video domain remains limited. As a representative method, CountVID [3] primarily enhances temporal counting uniqueness via promptable segmentation and pseudo-track filtering. However, in real-world occluded scenes, the high visual similarity and severe spatial overlap among objects make this approach highly susceptible to missed detections and track interruptions. Unlike previous video object counting methods, our approach innovatively incorporates depth to mitigate severe occlusion challenges faced by models that rely 2

solely on 2D RGB features. By leveraging spatial hierarchy information, it achieves more accurate separation of visually overlapping instances. In single-frame visual perception, depth has been introduced into detection frameworks through RGB-D encoding, feature interaction, or attention recalibration to enhance spatial modeling and occlusion awareness. For example, DGT [32] uses a Transformer to explicitly model depth-guided occlusion awareness for RGB-D occluded object detection. MonoDETR[34] further reformulates the Transformer into a depth-aware structure, allowing object queries to interact with scene depth and thus reducing reliance on local appearance cues. Unlike existing depth-aware detectors dedicated to 3D pose regression or salient object extraction, our depth-guided detector not only uses depth features to separate overlapping targets within a single frame, but also explicitly predicts occlusion to provide crucial cues for cross-frame trajectory de-duplication,making it highly suitable for the VIC task in real-world occluded scenes In video tasks, recent studies have started to use depth in the design of association costs, trajectory refinement, and occlusion reasoning for multi-object tracking, in order to improve temporal stability in complex scenes. For example, DepthMOT[30]uses depth cues and camera pose estimation to reduce ID switches caused by occlusion and camera motion. DepTR-MOT[8] introduces instancelevel depth supervision and depth consistency into a tracking-bydetection framework, thereby improving trajectory robustness under occlusion and close-range interactions.Although both MOT and our method use depth to reduce occlusion, MOT focuses on long-term ID tracking in real-world complex scenes, whereas video counting focuses on cross-frame de-duplication. Therefore, our approach, which effectively utilizes depth information during crossframe association and dynamically evaluates trajectory validity via Occlusion-Adaptive Temporal Voting, is suited for this task.

3

Based on the aligned RGB and depth feature pyramids, we design a Cross-Attention Fusion module to inject depth cues into the visual representation. Specifically, at each scale 𝑖, we project the rgb dep RGB feature 𝐹𝑖,𝑡 into queries 𝑄, and the depth feature 𝐹𝑖,𝑡 into keys 𝐾 and values 𝑉 . To suppress spurious associations between regions that are distant in 3D space but appear nearby in the image plane, we introduce a Depth Affinity Bias 𝐵(𝑍𝑡 ) into the attention computation. The resulting attention weights are defined as  ⊤  𝑞 𝑘 exp 𝑢√ 𝑣 + 𝐵𝑢𝑣 (𝑍𝑡 ) 𝑑 (𝑖,𝑡 )  ⊤ , 𝛼𝑢𝑣 = (1) 𝑞𝑢 𝑘 𝑣 ′ Í √ + 𝐵𝑢𝑣 ′ (𝑍𝑡 ) 𝑣 ′ exp 𝑑

where Depth Affinity Bias is given by 𝐵𝑢𝑣 (𝑍𝑡 ) = −𝛽 bias · 𝜙 (𝑍𝑡 )𝑢 − 𝜙 (𝑍𝑡 )𝑣 ,

where 𝜙 (·) is a depth normalization function (including log-inverse transformation and quantile normalization), and 𝛽 bias is a coefficient controlling the strength of depth suppression. Based on these weights, the depth-guided fusion feature at scale 𝑖 is computed as ∑︁ (𝑖,𝑡 ) 𝐹ˆ𝑖,𝑡 (𝑢) = 𝛼𝑢𝑣 𝑣𝑣, (3) 𝑣

In this way, feature interaction between regions with different depth values is effectively suppressed. The final multi-scale fused features 5 thus encode both RGB and depth information. {𝐹˜𝑖,𝑡 }𝑖=2 To address the mismatch between the Transformer’s requirement for four input scales and the backbone’s output of only three, previous methods often use simple downsampling, which lacks global depth information. To overcome this limitation, we propose a global feature generation strategy based on Feature-wise Linear Modulation (FiLM)[26]. SE blocks mainly perform multiplicative reweighting, while conditional normalization depends on normalization statistics and may disturb the pretrained feature distribution. In contrast, FiLM enables conditional affine modulation through both scaling and shifting, making it more suitable for injecting global depth priors to generate the additional coarse-scale feature. Specifically, we apply global average pooling (GAP) and global max pooling (GMP) on the depth map 𝑍𝑡 to extract scene-level depth statistics, and use 1 × 1 convolutions to generate channelwise scaling and shifting parameters 𝛾 and 𝛽:
𝛾 = 𝜎 𝑊𝛾 · GAP(𝑍 𝐿,𝑡 ) , 𝛽 = 𝑊𝛽 · GMP(𝑍 𝐿,𝑡 ). (4)

Method

We consider a synchronized RGB-D video sequence 𝑉 = {(𝐼𝑡 , 𝑍𝑡 )}𝑇𝑡=1 , where 𝐼𝑡 and 𝑍𝑡 denote the RGB image and the aligned depth map at frame 𝑡. Given an open-vocabulary text query 𝑞, a visual exemplar 𝑉 𝐸, and a video-level counting region (ROI), our goal is to output the individual number 𝐶𝑠𝑡𝑑 within the ROI. The proposed method consists of three components: a Depth-Guided Detector, Depth-Guided Tracking and De-duplication, and training strategy.

3.1

(2)

Depth-Guided Detector

We then perform a channel-wise affine transformation on the toplevel fused feature 𝐹˜𝐿,𝑡 , followed by downsampling to generate the fourth-level feature 𝐹˜6 . Finally, we feed {𝐹˜2, 𝐹˜3, 𝐹˜4, 𝐹˜5, 𝐹˜6 } into the subsequent multi-layer Transformer. Building upon the fused multi-scale features, we further introduce an occlusion-aware design to explicitly model dense-scene occlusion. Specifically, we generate a continuous occlusion score 𝑜𝑖 ∈ [0, 1] for each ground-truth object during data preprocessing, rather than using a simple binary label. Let 𝐵𝑖 denote the bounding box of the 𝑖-th object. If a neighboring object spatially overlaps with 𝐵𝑖 and is closer to the camera in depth, it is considered to contribute to the occlusion of object 𝑖. We jointly compute the occlusion score 𝑜𝑖 according to both the spatial coverage of neighboring objects over object 𝑖 and their relative depth relationships, where a larger

We begin by encoding the input query 𝑞 with a BERT encoder, and retain the tokens corresponding to the category name to guide cross-modal decoding. On the visual side, we adopt the backbone and FPN structure of GroundingDINO[21] to extract multi-scale rgb 5 RGB features {𝐹𝑖 }𝑖=2 from 𝐼𝑡 as visual inputs to the Transformer. To incorporate depth cues aligned with the RGB feature space, we design a depth encoder 𝐸 dep . This encoder produces a depth pyramid that matches the multi-scale RGB features of GroundingDINO. Its architecture is based on ResNet-18 + FPN [12, 17], and the first convolutional layer is modified to accept single-channel input, using the channel-wise mean of the RGB pretrained weights for initialization. Through top-down feature fusion and smoothing convolutions, we dep 5 obtain a four-level depth feature pyramid {𝐹𝑖 }𝑖=2 . 3

Figure 2: Overall architecture of our framework. Top: the Depth-Guided Detector (DG-Det) takes a text and visual query and synchronized RGB–D inputs, fuses depth and RGB features via cross-modal attention, and predicts class scores, bounding boxes, and occlusion probabilities. Bottom: the Depth-Guided Tracking and De-duplication(DG-Track) module refines detections within the ROI, performs depth-aware online association to form stable trajectories, and applies adaptive history-window voting to remove duplicates and obtain consistent video counting results. 𝑜𝑖 indicates a higher degree of foreground occlusion. The detailed construction of 𝑜𝑖 is provided in the supplementary material. Through the above process, we build a detector that jointly models semantic, depth, and occlusion information, providing reliable multi-modal foundation for subsequent video counting tracks.

3.2

depth 𝑑𝑖 , and occlusion probability 𝑝 occ,𝑖 , which together form the detection set D𝑡 of the current frame. To remove duplicate counts across frames, we perform online multi-object tracking based on ObjectTracker. Unlike conventional multi-object tracking methods that rely solely on 2D positions and appearance cues, we incorporate depth into trajectory association, yielding depth-guided association and de-duplication. For a trajectory 𝑡𝑟𝑖 ∈ 𝑇𝑡 −1 and a detection 𝑑𝑒𝑡 𝑗 ∈ 𝐷𝑡 , let Δp𝑖 𝑗 denote the center displacement, Δ𝑧𝑖 𝑗 denote the center-depth difference, and u𝑖 , u 𝑗 denote the normalized appearance features. We model the matching event 𝑀𝑖 𝑗 = 1 using the following modality-specific likelihoods, whose motivation and detailed derivation are provided in the Supplementary Material:   ∥Δp𝑖 𝑗 ∥ 2 𝑝 (Δp𝑖 𝑗 | 𝑀𝑖 𝑗 = 1) ∝ exp − , 𝜎𝑝   |Δ𝑧𝑖 𝑗 | (6) 𝑝 (Δ𝑧𝑖 𝑗 | 𝑀𝑖 𝑗 = 1) ∝ exp − , 𝜎𝑑
𝑝 (u 𝑗 | u𝑖 , 𝑀𝑖 𝑗 = 1) ∝ exp 𝜅 𝑓 u𝑖⊤ u 𝑗 .

Depth-Guided Tracking and De-duplication

Given a video sequence 𝑉 and an open-vocabulary query set 𝑞, we first run our DG-Det on each frame to obtain the raw detection set D̃𝑡 . These are then sequentially processed by detection refinement, online tracking, and temporal voting to produce the final count: Fdet

Ftrk

Fcnt

D̃𝑡 −−−−→ D𝑡 −−−−→ T −−−−→ 𝐶 std,

(5)

where D𝑡 is the refined detection set at frame 𝑡, T = {𝑇𝑘 } is the set of video-level object trajectories, and 𝐶 std is the final count. More details are provided in the Supplementary Material. Starting from the raw detections produced by DG-Det, we first perform per-frame detection refinement to obtain reliable instance candidates for subsequent association. For each frame and each query category, we first compute the phrase-level confidence score 𝑆 from the logits and obtain the corresponding occlusion probability 𝑃occ from the decoder. To avoid losing potential targets at the early filtering stage, we adopt a two-stage filtering strategy: 1) Candidate boxes satisfying 𝑆 > 𝜏𝑐 are retained. 2) Post-NMS, candidates are spatially filtered to retain those exceeding an overlap threshold with the video-level ROI. Finally, for each remaining candidate, we extract its bounding box 𝑏𝑖 , decoder feature vector 𝑓𝑖 , sampled

where 𝜎𝑝 and 𝜎𝑑 denote the effective noise scales of the position and depth modalities, respectively, and 𝜅 𝑓 denotes the concentration of appearance consistency. Assuming conditional independence across modalities given the matching event, and introducing an IoU-based geometric prior 𝑝 geo (𝑀𝑖 𝑗 = 1), the matching posterior can be written as 𝑝 (𝑀𝑖 𝑗 = 1 | 𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) ∝ 𝑝 (Δp𝑖 𝑗 | 𝑀𝑖 𝑗 = 1) 𝑝 (Δ𝑧𝑖 𝑗 | 𝑀𝑖 𝑗 = 1) · 𝑝 (u 𝑗 | u𝑖 , 𝑀𝑖 𝑗 = 1) 𝑝 geo (𝑀𝑖 𝑗 = 1), 4

(7)

Table 1: RGBD-VideoCount Dataset information.

We then define the association cost by minimizing the negative log-posterior: 𝐶𝑖 𝑗 = − log 𝑝 (𝑀𝑖 𝑗 = 1 | 𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) = 𝑤 𝑝 𝑑 pos (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) + 𝑤 𝑓 𝑑 feat (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) + 𝑤𝑑 𝑑 depth (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 )
+ 𝜆iou 𝐼 IoU(𝑏𝑖 , 𝑏 𝑗 ) < 𝜏iou + const, (8) where 𝑑 pos is the Euclidean distance between box centers, 𝑑 feat is the cosine distance between appearance features, and 𝑑 depth is the center-depth difference. The weights are related to modality reliability as 𝑤 𝑝 ∝ 𝜎𝑝−1,

𝑤𝑑 ∝ 𝜎𝑑−1,

𝑤𝑓 ∝ 𝜅𝑓 .

Cls.

Mod.

Vid.

MC

Ann.

CARPK[13] COCO[18] FSC147[27] VideoCount[3] RGBD-VideoCount

1 80 147 141 6

RGB RGB RGB RGB RGB-D

N N N Y Y

– – – – Y

B B, IM P, EB B, EB B, EB, ROI

RGBD-VideoCount category information Category

#Vid.

Min

Max

Avg.

beverage bag book plant box cabinet

94 8 42 12 55 42

1 3 28 5 1 12

208 23 209 93 143 132

48.33 13.50 114.20 25.92 20.42 69.86

Overall

195+

1

209

70.51

(9)

We normalize the cost matrix, use the Hungarian algorithm to obtain the minimum-cost assignment, and accept only matched pairs whose cost is below a predefined threshold. Matched tracks are updated with the new bounding box, depth, and features; unmatched tracks are deactivated if they are not observed for a consecutive number of frames; unmatched detections are initialized as new tracks. Finally, we obtain a set of stable target trajectories {track𝑘 }. To further suppress spurious trajectories caused by transient false positives and short-term occlusions, we perform temporalconsistency-based validity verification for each trajectory 𝑇𝑘 . Specifically, we first compute the trajectory lifespan 𝐿𝑘 , the confidence ¯ and the mean occlusion rate 𝑂¯𝑘 . After removing statistics (𝑆 max, 𝑆), noisy trajectories with excessively short lifespans (𝐿𝑘 < 𝐻 base ) or low confidence, we adaptively determine the minimum hit count 𝑁 min and the frame-density threshold 𝜌 according to 𝑂¯𝑘 .Next, for each candidate trajectory, we slide a temporal window of length 𝑊 over its time span and count the number of frames ℎ in which the object is successfully detected within the window. Let 𝑛𝑘 = min(𝑊 , 𝐿𝑘 ) denote the effective window length for the trajectory 𝑇𝑘 . To jointly evaluate the absolute number of hits and the local hit density, we define a trajectory support score as   ℎ ℎ/𝑛𝑘 , (10) 𝑠𝑘 = min , 𝑁 min 𝜌

B: boxes; P: points; EB: exemplar boxes; IM: instance masks; ROI: region of interest; MC: multiple object categories may coexist in one video.

occlusion prediction branch, we define the occlusion loss as 1 ∑︁ Locc = BCE(𝑝 occ,𝑖 , 𝑜𝑖 ), (12) 𝑁 pos 𝑖 ∈Pos During training, the total loss L is given by L = 𝜆1 Lcls + 𝜆2 Lgiou + 𝜆3 Lbbox + 𝜆4 Locc,

(13)

where Lcls and Lgiou are the classification loss and GIoU loss in CountGD_Box, and 𝜆1 , 𝜆2 , 𝜆3 , and 𝜆4 are hyper-parameters. For optimization, we initialize the model from CountGD_Box and freeze the pretrained backbone, while optimizing the newly introduced modules, including the depth encoder, the cross-attention fusion module, and the detection head. Training is conducted with AdamW using a batch size of 1, together with standard data augmentation such as scaling, cropping, and flipping.More details are provided in the Supplementary Material.

Here, the first term measures whether the trajectory has accumulated a sufficient number of detections, while the second term evaluates whether its detections are temporally dense enough within the local window. A larger 𝑠𝑘 therefore indicates stronger temporal support for the trajectory. We regard a trajectory as valid if 𝑠𝑘 ≥ 1. The total number of valid trajectories then constitutes the final standard counting result 𝐶 std .

3.3

Dataset

4

RGBD-VideoCount Dataset

Most existing video counting datasets only provide RGB modality, which makes it difficult to explicitly model occlusion relationships(see Table 1). To address this limitation, we release RGBDVideoCount, a synchronized RGB-D dataset for video individual counting in crowded scenes. Unlike previous datasets that only provide RGB videos and usually focus on a single target category, RGBD-VideoCount allows multiple categories to coexist in the same video, and also provides depth information, instance-level bounding box annotations, exemplar boxes, and video-level ROIs. This makes it a unified benchmark for depth-assisted video detection, cross-frame association, and de-duplication. Such a design is closer to real shelf and inventory scenarios, and it also provides a more effective testbed for evaluating the robustness of counting methods in the presence of distracting categories. RGBD-VideoCount contains 2,032 finely annotated frames and 77,638 instance boxes. The number of finely annotated frames exceeds that of CARPK, while the number of instance boxes is

Training

To train the proposed model, we introduce the training strategy. In addition to the original classification and GIoU losses in CountGD_Box, we further introduce a bounding-box regression loss and an occlusion prediction loss to enhance localization accuracy in overlapping regions and explicitly supervise the newly added occlusion branch. Specifically, to further improve localization accuracy under dense overlap, we employ an L1 bounding-box regression loss: 1 ∑︁ Lbbox = 𝑏𝑖 − 𝑏ˆ𝑖 1, (11) 𝑁 pos 𝑖 ∈Pos where 𝑁 pos is the number of matched boxes, and 𝑏𝑖 and 𝑏ˆ𝑖 denote the predicted and ground-truth boxes. To supervise the newly added 5

MAE, showing that strong single-frame detection cannot ensure accurate counting under severe occlusion. Without explicit occlusion modeling, frequent ID switches and fragmented trajectories lead to severe over-counting. Our method avoids this by unifying precise localization with occlusion-guided temporal de-duplication. To further verify that the gain of our method does not simply come from introducing an additional depth modality, we also compare with two RGB-D tracking methods (see Table 2), namely DepthMOT[30] and DepTR-MOT[8]. The results show that DepTRMOT effectively utilizes depth information in both the detection and tracking stages to outperform pure RGB baselines. In contrast, DepthMOT exploits depth solely during the tracking stage, resulting in overall performance comparable to pure RGB trackers. Furthermore, both tracking methods yield lower AP scores, suggesting that directly applying traditional tracking paradigms is challenging to maintain high localization accuracy under severe occlusions. In contrast, our method achieves the best performance under the same input setting. This result suggests that the key to the improvement lies not in naively adding depth, but in effectively modeling depth at the feature representation level and jointly exploiting occlusion cues during both detection and de-duplication. To investigate the impact of depth quality, we evaluated our model using pseudo-depth ( Table 2, Ours†). As shown in the red box (Fig 3 ), the predicted depth maps suffer from "edge smoothing," erasing spatial boundaries between overlapping objects. In contrast, although the ground-truth depth is sparse and noisy, it keeps sharp depth changes. This provides more accurate geometric cues for distinguishing crowded objects, leading to better performance. We further evaluate our method on FSCD-147[25] using the pseudo-depth estimated by Depth Anything V2[33]. Details and qualitative results are provided in the supplementary material.

Figure 3: Qualitative comparison between predicted pseudodepth (middle) and ground-truth depth (right).

of a similar scale. The data distribution is designed to be close to real-world scenes. Specifically, the densely arranged Beverage category serves as a challenging setting for heavy occlusion, while categories such as Book, which have relatively weaker occlusion but still dense layouts, are used to evaluate the model’s generalization ability in high-density localization scenarios. The data are collected using an Intel RealSense D455, with the RGB and depth streams strictly synchronized through SDK-level alignment. More details are provided in the Supplementary Material.

5

Experiments

We conduct comprehensive experiments to evaluate the effectiveness of our DG-Det and DG-Track. First, we introduce the evaluation metrics. Then, we compare our method with the baseline and several state-of-the-art approaches. Finally, we perform ablation studies to analyze the contribution of each key component.

5.1

Metrics

We adopt both counting and localization metrics to evaluate model performance. For each video 𝑖, we compute the discrepancy between the predicted count 𝑦ˆ𝑖 and the ground-truth count 𝑦𝑖 . We use the Mean Absolute Error (MAE) and the Root Mean Squared Error (RMSE) to measure counting accuracy. In addition, we evaluate localization performance using Average Precision (AP) and AP50 , where AP50 denotes the AP computed at an IoU threshold of 0.5.

5.2

5.3

Ablation Studies

Ablation Study on depth coverage. Table 3 compares different depth coverage settings. For the 25%, 50%, and 75% partial-depth settings, we report the mean and standard deviation over multiple random seeds. Compared with the RGB-only baseline, depth cues consistently reduce MAE, demonstrating their value for providing spatial constraints in crowded scenes. Partial depth improves MAE but can lead to RMSE fluctuations due to cross-modal inconsistency, while full real depth achieves the best and most stable results. Fine-grained Module Ablation. To further analyze the contribution of each component, we start from the RGB-only baseline and progressively add Cross-Attention Fusion, Depth Affinity Bias, and the Occlusion Head, as shown in Table 4. Adding Cross-Attention Fusion reduces the validation and test MAE/RMSE from 18.72/32.40 and 18.07/26.80 to 17.12/28.77 and 14.93/20.89, respectively, demonstrating that depth feature fusion improves target localization in crowded scenes. With Depth Affinity Bias, the results further improve to 14.26/26.08 on validation and 12.20/15.99 on test, indicating more effective use of local depth relations. Finally, adding the Occlusion Head achieves the best performance, with MAE/RMSE of 13.33/19.68 on validation and 7.98/11.45 on test. These progressive gains show that the proposed components jointly improve target reliability estimation and cross-frame de-duplication .

Comparison with Existing Methods

We first compare our method with CountVID on RGBD-VideoCount, and construct three representative RGB tracking models based on CountGD-Box, namely ByteTrack, OC-SORT, and SparseTrack. ByteTrack and OC-SORT emphasize low-confidence detection recovery and motion modeling, while SparseTrack performs cascaded matching using pseudo-depth estimated from single-frame RGB images. Together, these models cover representative temporal association strategies commonly used in crowded video counting. Under the challenging text-only setting, pure RGB baselines generally suffer from severe missed detections and false positives due to the lack of visual exemplars, leading to poor localization accuracy and large counting errors. In contrast, our method shows a clear advantage (see Table 2). This shows that text-only 2D features struggle in crowded scenes. By fusing multimodal features with spatial depth, our DG-Det achieves reliable localization for accurate counting. With the introduction of visual exemplars, the overall performance of all methods improves, and our approach still achieves the best results in both counting and detection metrics. Notably, the SAM 2.1 baseline achieves an excellent AP50 but a high 6

Table 2: Results on RGBD-VideoCount. Methods marked with † utilize estimated depth.

RGBD-VideoCount Val Method

Detector

RGBD-VideoCount Test

Tracker Counting MAE↓

RMSE↓

Detection

Counting

Detection

AP↑

AP50↑

MAE↓

RMSE↓

AP↑

AP50↑

11.59 4.44 6.81 5.89 21.95 20.16

25.76 13.54 18.17 16.89 45.16 43.26

23.69 25.74 22.29 24.86 13.75 12.68

30.78 37.28 36.34 36.51 20.81 15.87

9.38 4.87 7.40 7.06 16.97 17.52

23.21 15.19 18.85 20.31 44.79 45.68

47.65 32.25 30.29 39.03 48.99 50.12

20.98 23.17 25.26 23.02 13.63 7.97

27.73 31.95 38.89 32.17 33.51 11.44

17.08 11.55 13.35 15.75 22.92 20.78

48.42 34.49 31.68 42.84 53.00 50.39

7.87 11.47 50.12

23.57 14.33 7.97

32.65 25.75 11.44

4.08 3.40 20.78

9.30 13.28 50.39

Text-only prompt CountVID – – – Ours† Ours

COUNTGD-BOX COUNTGD-BOX COUNTGD-BOX COUNTGD-BOX DG-Det† DG-Det

SAM 2.1 ByteTrack OC-SORT SparseTrack† DG-Track† DG-Track

24.77 28.78 34.74 26.74 16.95 14.55

28.76 43.5 55.78 42.11 28.39 25.32

Text + exemplar prompt CountVID – – – Ours† Ours

COUNTGD-BOX COUNTGD-BOX COUNTGD-BOX COUNTGD-BOX DG-Det† DG-Det

SAM 2.1 ByteTrack OC-SORT SparseTrack† DG-Track† DG-Track

19.68 25.51 38.41 25.22 14.13 13.32

26.87 37.83 55.57 37.47 28.21 19.68

18.73 11.32 13.89 16.26 22.50 23.96

RGB-D tracking Model DepthMOT† DepTR-MOT† Ours

– – DG-Det

– – DG-Track

26.38 18.67 13.32

43.53 41.27 19.68

Table 3: Ablation study on depth coverage. Validation

Bias maintains good robustness across a relatively wide range of parameter values. Overall, the introduced bias enables the model to exploit depth hierarchy information more consistently, leading to improved detection and counting performance in complex scenes. We further evaluate the effectiveness of Occlusion-Adaptive Temporal Voting. Specifically, we compare the adaptive voting strategy with a fixed-threshold temporal filtering scheme. As shown in Table 5, introducing the adaptive voting mechanism reduces MAE and RMSE by 1.91 and 2.75, respectively. These results show that Occlusion-Adaptive Temporal Voting achieves a better balance between suppressing spurious trajectories and recovering heavily occluded targets. Compared with a fixed-threshold scheme, our method dynamically adjusts the trajectory retention criterion according to the predicted occlusion probability, leading to more stable and accurate video-level counting in occlusion scenarios. Considering that depth sensors in real-world applications may suffer from noise, blur, and missing values, we further corrupt the test-set depth maps with several perturbations, including Gaussian noise, Salt & Pepper noise, Edge Blur, and Random Holes. The results show that our method remains highly robust under Edge Blur and Random Holes, achieving test MAEs of 8.50 and 8.23, respectively, which are close to the 7.97 obtained with clean depth input. In contrast, both Gaussian noise and Salt & Pepper noise lead to noticeable performance degradation, with the MAE increasing to 11.16 and 10.66, respectively. We attribute this to the fact that both types

Test

Method MAE↓

RMSE↓

MAE↓

RMSE↓

RGB-only (0%) 19.88 28.46 15.47 25.26 Partial depth (25%) 16.15 ± 0.29 30.25 ± 1.57 14.74 ± 0.17 35.85 ± 0.82 Partial depth (50%) 16.52 ± 0.40 28.87 ± 0.61 12.60 ± 0.36 19.66 ± 0.93 Partial depth (75%) 14.18 ± 0.27 29.13 ± 1.10 12.91 ± 0.21 36.44 ± 0.82 Ours (100%) 13.33 19.68 7.98 11.45

Table 4: Fine-grained ablation of the proposed components. Validation

Test

Configuration

RGB-only + Cross-Attention Fusion + Depth Affinity Bias + Occlusion Head

5.4

MAE↓

RMSE↓

MAE↓

RMSE↓

18.72 17.12 14.26 13.33

32.40 28.77 26.08 19.68

18.07 14.93 12.20 7.98

26.80 20.89 15.99 11.45

2.80 3.05 23.96

Robustness and Sensitivity Analysis

We first analyze the sensitivity of the depth affinity bias coefficient 𝛽. As shown in the table 5, when 𝛽 varies within the range of 0.5–3.0, the model performance remains stable, with only minor fluctuations in MAE and RMSE. This indicates that the proposed Depth Affinity 7

Figure 4: Example predictions of our method in two typical challenging scenarios. For each video, several sampled frames and their corresponding depth maps are shown, together with the video-level prediction Pred(video) and the ground-truth count GT(video). (a) A shelf video with severe occlusions (testing robustness to occlusion); (b) a bookshelf video without obvious occlusion but with highly crowded objects (testing localization accuracy in dense scenes). More scene details are provided in the Supplementary Material. Table 5: Ablation Study and Robustness Analysis. Setting / Condition

Param.

MAE

RMSE

ΔMAE

ΔRMSE

– 10.2 -0.4 15.2 18.5

7.97 11.16 10.66 8.50 8.23

11.44 21.13 13.31 12.46 12.11

– +3.18 +2.68 +0.52 +0.25

– +9.68 +1.86 +1.00 +0.66

– – – –

8.02 7.97 7.86 8.59

11.69 11.44 11.05 12.34

-0.05 – -0.11 +0.61

+0.24 – -0.40 +0.89

9.88 7.97

14.19 11.44

+1.91 –

+2.75 –

SNR (dB)

6

We investigate depth information for video individual counting in crowded scenes. Unlike RGB-based methods that mainly rely on temporal deduplication, our approach introduces depth constraints for per-frame detection and video-level inference. Depth-guided fusion and occlusion-aware prediction reduce missed detections and duplicate counting, while stable depth cues improve cross-frame association and video-level counting accuracy.

Experimental Results on Depth Noise Robustness ours (clean) Gaussian Salt & Pepper Edge Blur Random Holes

– 𝜎 = 0.05 ratio=5% k=15 × 15 area≈ 5%

Conclusion

Sensitivity Analysis of Depth Bias 𝛽 = 0.5 𝛽 = 1.0† 𝛽 = 2.0 𝛽 = 3.0

– – – –

References [1] Niki Amini-Naieni, Kiana Amini-Naieni, Tengda Han, and Andrew Zisserman. 2023. Open-world text-specified object counting. arXiv preprint arXiv:2306.01851 (2023). [2] Niki Amini-Naieni, Tengda Han, and Andrew Zisserman. 2024. Countgd: Multimodal open-world counting. Advances in Neural Information Processing Systems 37 (2024), 48810–48837. [3] Niki Amini-Naieni and Andrew Zisserman. 2026. Open-world object counting in videos. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 2300–2308. [4] Carlos Arteta, Victor Lempitsky, J Alison Noble, and Andrew Zisserman. 2014. Interactive object counting. In European conference on computer vision. Springer, 504–518. [5] Olga Barinova, Victor Lempitsky, and Pushmeet Kohli. 2010. On detection of multiple object instances using hough transforms. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 2233–2240. [6] Siu-Yeung Cho, Tommy WS Chow, and Chi-Tat Leung. 1999. A neural-based crowd estimation by hybrid global learning algorithm. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 29, 4 (1999), 535–541.

Effectiveness of Occlusion-Adaptive Voting Fixed threshold Adaptive (ours)

– –

– –

of noise directly corrupt local depth structures and fine-grained depth ordering, thereby reducing the effectiveness of the bias in cross-modal fusion. Overall, these experiments suggest that our method is more robust to structured depth degradation such as blur and partial missing regions, while random noise remains more challenging for reliable counting. 8

conference on computer vision. Springer, 38–55. [22] Xinyan Liu, Guorong Li, Zhenjun Han, Weigang Zhang, Yifan Yang, Qingming Huang, and Nicu Sebe. 2021. Exploiting sample correlation for crowd counting with multi-expert network. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 3195–3204. [23] Xinyan Liu, Guorong Li, Yuankai Qi, Zhenjun Han, Anton van den Hengel, Nicu Sebe, Ming-Hsuan Yang, and Qingming Huang. 2024. Consistency-aware anchor pyramid network for crowd localization. IEEE transactions on pattern analysis and machine intelligence (2024). [24] Xinyan Liu, Guorong Li, Yuankai Qi, Ziheng Yan, Zhenjun Han, Anton Van Den Hengel, Ming-Hsuan Yang, and Qingming Huang. 2024. Weakly supervised video individual counting. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 19228–19237. [25] Thanh Nguyen, Chau Pham, Khoi Nguyen, and Minh Hoai. 2022. Few-shot object counting and detection. In European Conference on Computer Vision. Springer, 348–365. [26] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. [27] Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. 2021. Learning to count everything. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3394–3403. [28] Qingyu Song, Changan Wang, Zhengkai Jiang, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yang Wu. 2021. Rethinking counting and localization in crowds: A purely point-based framework. In Proceedings of the IEEE/CVF international conference on computer vision. 3365–3374. [29] Nikola Ðukić, Alan Lukežič, Vitjan Zavrtanik, and Matej Kristan. 2023. A lowshot object counting network with iterative prototype adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 18872–18881. [30] Jiapeng Wu and Yichen Liu. 2024. Depthmot: Depth cues lead to a strong multiobject tracker. arXiv preprint arXiv:2404.05518 (2024). [31] Shaokai Wu and Fengyu Yang. 2023. Boosting detection in crowd analysis via underutilized output features. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 15609–15618. [32] Kelei Xu, Chunyan Wang, Wanzhong Zhao, and Jinqiang Liu. 2025. DGT: Depthguided RGB-D occluded target detection with transformers. Applied Intelligence 55, 5 (2025), 300. [33] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. 2024. Depth anything v2. Advances in Neural Information Processing Systems 37 (2024), 21875–21911. [34] Renrui Zhang, Han Qiu, Tai Wang, Ziyu Guo, Ziteng Cui, Yu Qiao, Hongsheng Li, and Peng Gao. 2023. MonoDETR: Depth-Guided Transformer for Monocular 3D Object Detection. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV). 9121–9132. doi:10.1109/ICCV51070.2023.00840

[7] Siyang Dai, Jun Liu, and Ngai-Man Cheung. 2024. Referring expression counting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16985–16995. [8] Buyin Deng, Lingxin Huang, Kai Luo, Fei Teng, and Kailun Yang. 2025. DepTRMOT: Unveiling the Potential of Depth-Informed Trajectory Refinement for Multi-Object Tracking. arXiv preprint arXiv:2509.17323 (2025). [9] Chaitanya Desai, Deva Ramanan, and Charless C Fowlkes. 2011. Discriminative models for multi-class object layout. International journal of computer vision 95, 1 (2011), 1–12. [10] Guangshuai Gao, Junyu Gao, Qingjie Liu, Qi Wang, and Yunhong Wang. 2020. Cnn-based density estimation and crowd counting: A survey. arXiv preprint arXiv:2003.12783 (2020). [11] Shenjian Gong, Shanshan Zhang, Jian Yang, Dengxin Dai, and Bernt Schiele. 2022. Class-agnostic object counting robust to intraclass diversity. In European Conference on Computer Vision. Springer, 388–403. [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778. [13] Meng-Ru Hsieh, Yen-Liang Lin, and Winston H Hsu. 2017. Drone-based object counting by spatially regularized regional proposal network. In Proceedings of the IEEE international conference on computer vision. 4145–4153. [14] Ruixiang Jiang, Lingbo Liu, and Changwen Chen. 2023. Clip-count: Towards textguided zero-shot object counting. In Proceedings of the 31st ACM International Conference on Multimedia. 4535–4545. [15] Dan Kong, Douglas Gray, and Hai Tao. 2006. A viewpoint invariant approach for crowd counting. In 18th International Conference on Pattern Recognition (ICPR’06), Vol. 3. IEEE, 1187–1190. [16] Bo Li, Hongbo Huang, Ang Zhang, Peiwen Liu, and Cheng Liu. 2021. Approaches on crowd counting and density estimation: a review. Pattern Analysis and Applications 24, 3 (2021), 853–874. [17] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2117–2125. [18] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740–755. [19] Wei Lin and Antoni B Chan. 2024. A fixed-point approach to unified promptbased counting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 3468–3476. [20] Chang Liu, Yujie Zhong, Andrew Zisserman, and Weidi Xie. 2022. Countr: Transformer-based generalised visual counting. arXiv preprint arXiv:2208.13721 (2022). [21] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. 2024. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European

9

Appendix A RGBD-VideoCount Dataset Details

where 𝑜𝑖 ∈ [0, 1]. A larger 𝑜𝑖 indicates that instance 𝑖 is more heavily occluded by foreground objects, while a smaller 𝑜𝑖 indicates that it is largely unoccluded. During training, we use the continuous occlusion score 𝑜𝑖 as the supervision signal for the occlusion prediction branch, which learns the occlusion probability 𝑝 occ,𝑖 for each candidate instance. During inference, the predicted occlusion probability is not only used to characterize single-frame detection results, but also further participates in subsequent trajectory-level occlusion statistics and temporal de-duplication.

RGBD-VideoCount is designed to mimic more realistic scenarios, where a single video typically contains multiple co-occurring categories, leading to challenging, mixed, and dense scenes. The dataset covers six everyday categories. Beverage, Box, and Bag. These categories often appear in the same physical space and may exhibit similar appearance or containment relations. The beverage category includes three canonical scenarios: real drinks densely packed on shelves (high density and heavy occlusion), miniature beverages (medium occlusion), and vending machines (covering both severely occluded and almost unoccluded views). Multiple categories densely mixed in a limited space not only lead to severe occlusions but also impose higher demands on the model’s semantic discrimination ability. Book and Cabinet. These categories are used to evaluate the model in highly regular and densely arranged scenes. Book videos depict multi-layer bookshelves packed with books; cabinet videos feature grid-like cabinet structures and, in some videos, dynamic door opening and closing, which place stringent requirements on localisation accuracy and fine-grained counting. plant. This category provides diverse shapes and textures to assess the model’s generalisation ability further.

B

C

Probabilistic Interpretation of the Association Cost C.1 Modality-Specific Likelihood Models For a trajectory 𝑡𝑟𝑖 and a detection 𝑑𝑒𝑡 𝑗 , let the matching event be denoted by 𝑀𝑖 𝑗 = 1. We define the center displacement, centerdepth residual, and normalized appearance features as Δp𝑖 𝑗 = c𝑖 − c 𝑗 ,

u𝑖 , u 𝑗 ∈ S𝑑 −1 .

(18)

Positional Modality.Under the online association setting with short temporal intervals, we adopt a local motion continuity assumption, i.e., a true match is more likely to appear near the previous trajectory location. Therefore, positional consistency is mainly characterized by the displacement magnitude rather than the specific motion direction. Based on this property, we model the positional modality as a radial decaying distribution over the displacement norm:   ∥Δp𝑖 𝑗 ∥ 2 𝑝 (Δp𝑖 𝑗 | 𝑀𝑖 𝑗 = 1) ∝ exp − , (19) 𝜎𝑝

Occlusion Score Construction.

To characterize mutual occlusion among object instances in dense scenes, we define a continuous occlusion score for each instance, rather than using a simple binary occlusion label. Let 𝐵𝑖 denote the bounding box of instance 𝑖, and let 𝑑𝑖 denote the average depth value within 𝐵𝑖 . For any other instance 𝑗 ≠ 𝑖, if it spatially overlaps with instance 𝑖 and is closer to the camera in depth, it is considered a potential occluder of instance 𝑖. We first define the spatial occlusion strength of instance 𝑗 on instance 𝑖 as |𝐵𝑖 ∩ 𝐵 𝑗 | , (14) 𝑠𝑖 𝑗 = |𝐵𝑖 | where |𝐵𝑖 ∩ 𝐵 𝑗 | denotes the intersection area between the two bounding boxes, and |𝐵𝑖 | denotes the area of instance 𝑖. This term measures the fraction of instance 𝑖 covered by instance 𝑗. We then define a depth-ordering weight based on their depth difference:   𝑑𝑖 − 𝑑 𝑗 − 𝛿 𝑧 𝑧𝑖 𝑗 = 𝜎 , (15) 𝜏𝑧 where 𝜎 (·) is the Sigmoid function, 𝛿𝑧 denotes the minimum effective depth gap, and 𝜏𝑧 is a smoothing parameter. When instance 𝑗 has a smaller depth value, i.e., it is closer to the camera, 𝑧𝑖 𝑗 becomes larger, indicating that it is more likely to occlude instance 𝑖. Based on the above two terms, the occlusion contribution of instance 𝑗 to instance 𝑖 is defined as 𝑐 𝑖 𝑗 = 𝑠𝑖 𝑗 𝑧𝑖 𝑗 .

Δ𝑧𝑖 𝑗 = 𝑧𝑖 − 𝑧 𝑗 ,

where 𝜎𝑝 denotes the effective noise scale of the positional modality. This form depends only on ∥Δp𝑖 𝑗 ∥ 2 and is therefore isotropic. Depth Modality.For the depth modality, we consider the onedimensional residual consistency between a trajectory and a detection. In trajectory association, depth information is reflected by the temporal continuity of the target center depth across frames, so the absolute depth residual |Δ𝑧𝑖 𝑗 | serves as a natural observation. Meanwhile, depth estimation in real videos may be affected by occlusion, boundary ambiguity, and local distortions, causing the error distribution to typically exhibit a sharp peak with moderately heavy tails, i.e., small residuals dominate while occasional larger deviations may occur. Based on this characteristic, we adopt a Laplace residual model:   |Δ𝑧𝑖 𝑗 | 𝑝 (Δ𝑧𝑖 𝑗 | 𝑀𝑖 𝑗 = 1) ∝ exp − , (20) 𝜎𝑑 where 𝜎𝑑 denotes the effective noise scale of the depth modality. Appearance Modality.For the appearance modality, we use ℓ2 -normalized feature representations. As a result, the underlying geometry is no longer primarily characterized by Euclidean magnitude differences, but rather by directional consistency in the feature space. In this case, a more appropriate similarity measure is the angular relation on the unit hypersphere, namely the inner product or cosine similarity between normalized features. Accordingly, we write the appearance matching likelihood as
𝑝 (u 𝑗 | u𝑖 , 𝑀𝑖 𝑗 = 1) ∝ exp 𝜅 𝑓 u𝑖⊤ u 𝑗 , (21)

(16)

Optionally, to suppress interference from weakly related overlaps, 𝑐𝑖 𝑗 can be further multiplied by a smooth IoU-based gating term. Finally, we aggregate the contributions of all neighboring instances to obtain the final occlusion score of instance 𝑖: Ö 𝑜𝑖 = 1 − (1 − 𝑐𝑖 𝑗 ), (17) 𝑗≠𝑖 10

D Experiments on RGBD-VideoCount D.1 Supplementary experiments

where 𝜅 𝑓 denotes the concentration of appearance consistency. Since u𝑖⊤ u 𝑗 is equivalent to cosine similarity, the above form directly captures the relation that the more consistent the feature directions are, the higher the matching probability becomes.

C.2

We further supplement our comparisons by adopting the depthenhanced detector YOLOv8 RGB-D as an additional baseline. Specifically, YOLOv8 RGB-D is used as the detection front-end and integrated with the same downstream tracking and counting pipeline as in the main experiments, with evaluation conducted under the same data split and metrics. The results(see Tab 6) show that, after introducing depth information at the detection stage, the performance is slightly better than that obtained with the RGB detector, indicating that depth cues can provide auxiliary benefits for target localization in densely arranged and occluded scenes. Nevertheless, our proposed model still achieves the best overall performance. This suggests that improvements in video instance counting do not solely rely on expanding the detector input modality, but also depend on the effective discrimination of adjacent similar instances, as well as accurate cross-frame association and deduplication.

From the Matching Posterior to the Final Association Cost

Given the matching event 𝑀𝑖 𝑗 = 1, we assume that the three modalities are approximately conditionally independent, and introduce an IoU-based geometric prior 𝑝 geo (𝑀𝑖 𝑗 = 1). The matching posterior is written as 𝑝 (𝑀𝑖 𝑗 = 1 | 𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) ∝ 𝑝 (Δp𝑖 𝑗 | 𝑀𝑖 𝑗 = 1) 𝑝 (Δ𝑧𝑖 𝑗 | 𝑀𝑖 𝑗 = 1) (22) · 𝑝 (u 𝑗 | u𝑖 , 𝑀𝑖 𝑗 = 1) 𝑝 geo (𝑀𝑖 𝑗 = 1). Substituting the likelihoods of the three modalities gives     ∥Δp𝑖 𝑗 ∥ 2 |Δ𝑧𝑖 𝑗 | 𝑝 (𝑀𝑖 𝑗 = 1 | 𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) ∝ exp − exp − 𝜎𝑝 𝜎𝑑
· exp 𝜅 𝑓 u𝑖⊤ u 𝑗 𝑝 geo (𝑀𝑖 𝑗 = 1).

(23)

D.2

Taking the negative logarithm yields ∥Δp𝑖 𝑗 ∥ 2 |Δ𝑧𝑖 𝑗 | − log 𝑝 (𝑀𝑖 𝑗 = 1 | 𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) = + 𝜎𝑝 𝜎𝑑

We present two representative scene categories, Beverage & Box and Cabinet, in Fig. 5. Beverage & Box: This scene is a simulated densely arranged setting involving beverages and boxes. Multiple objects are placed within a limited space, resulting in evident occlusions, overlaps, and viewpoint changes, while different objects also exhibit high similarity in appearance, size, and packaging form. These factors make the boundaries between adjacent objects more difficult to distinguish, often leading to missed detections, mismatches, and duplicate counting. Nevertheless, our method can still stably distinguish closely adjacent instances in this scenario, demonstrating strong capability in dense-scene perception and counting. Cabinet: This category represents a typical dense setting, where cabinet compartments are tightly arranged with highly homogeneous textures. Our model exhibits strong fine-grained localization and can accurately discriminate individual storage units, demonstrating robust performance on densely packed and visually similar objects.

− 𝜅 𝑓 u𝑖⊤ u 𝑗 − log 𝑝 geo (𝑀𝑖 𝑗 = 1) + const. (24) Next, we rewrite the above terms using the distance forms adopted in the main text: 𝑑 pos (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) := ∥Δp𝑖 𝑗 ∥ 2, 𝑑 depth (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) := |Δ𝑧𝑖 𝑗 |, (25) 𝑑 feat (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) := 1 − u𝑖⊤ u 𝑗 . Then the appearance term can be rewritten as
−𝜅 𝑓 u𝑖⊤ u 𝑗 = 𝜅 𝑓 1 − u𝑖⊤ u 𝑗 − 𝜅 𝑓 (26) = 𝜅 𝑓 𝑑 feat (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) + const. Therefore, the negative log-posterior becomes 1 𝑖𝑗 1 𝑖𝑗 𝑖𝑗 − log 𝑝 (𝑀𝑖 𝑗 = 1 | 𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) = + 𝑑 𝑑 + 𝜅 𝑓 𝑑 feat 𝜎𝑝 pos 𝜎𝑑 depth − log 𝑝 geo (𝑀𝑖 𝑗 = 1) + const,

(27)

E

where 𝑖𝑗 𝑑 pos ≡ 𝑑 pos (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ),

𝑖𝑗 𝑑 feat ≡ 𝑑 feat (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ),

As a result, the final association cost is 𝐶𝑖 𝑗 = 𝑤 𝑝 𝑑 pos (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) + 𝑤 𝑓 𝑑 feat (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 )
+ 𝑤𝑑 𝑑 depth (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ) + 𝜆iou I IoU(𝑏𝑖 , 𝑏 𝑗 ) < 𝜏iou + const, (29) with 𝑤𝑑 ∝ 𝜎𝑑−1,

Experiments on FSD-147

Our method enhances CountGD-Box by fusing an additional depth 𝑖𝑗 𝑑 depth ≡ 𝑑 depth (𝑡𝑟𝑖 , 𝑑𝑒𝑡 𝑗 ). modality to better handle occlusions and scale variations. To evalu-

For the geometric prior, we adopt a hard IoU-gating constraint:
− log 𝑝 geo (𝑀𝑖 𝑗 = 1) = 𝜆iou I IoU(𝑏𝑖 , 𝑏 𝑗 ) < 𝜏iou . (28)

𝑤 𝑝 ∝ 𝜎𝑝−1,

Additional Qualitative Results on RGBD-VideoCount

𝑤𝑓 ∝ 𝜅𝑓 .

(30)

11

ate its generalization, we extend our experiments to the FSCD-147 dataset and use pseudo-depth maps generated by Depth Anything V2 as a surrogate input. Due to the limited quality of the estimated depth, incorporating this modality does not yield the expected substantial gains. In particular, the pseudo-depth often falls short in preserving object-boundary details, matching the noise characteristics of real depth sensors, and maintaining consistent absolute scale, which makes it difficult for the model to extract stable and accurate geometric cues. As shown in Fig. 6(c), while the pseudo-depth from Depth Anything V2 captures the coarse depth layout, it is over-smoothed and blurs depth discontinuities (e.g., box boundaries), failing to resolve instance-level depth differences in densely packed apples. Consequently, the model cannot reliably extract depth cues from such pseudo-depth, thereby weakening the discriminative power of the depth modality in crowded scenes and limiting the effectiveness of the resulting predictions.

Figure 5: (a) Beverage scene: Demonstrates the model’s performance in occluded scenarios. (b) Cabinet scene: Demonstrates the model’s performance in dense scenes with highly homogeneous textures. Table 6: Comparison with YOLOv8 RGB-D based baselines on RGBD-VideoCount.

RGBD-VideoCount Val Method YOLOv8 RGB-D YOLOv8 RGB-D Ours

Detector

RGBD-VideoCount Test

Tracker MAE↓

RMSE↓

AP↑

AP50↑

MAE↓

RMSE↓

AP↑

AP50↑

YOLOv8 RGB-D YOLOv8 RGB-D

SparseTrack SAM2.1

25.36 20.32

42.85 41.81

16.42 17.42

31.95 43.41

23.75 22.76

37.74 35.90

13.54 16.78

30.89 40.95

DG-Det

DG-Track

13.32

19.68

23.96

50.12

7.97

11.44

20.78

50.39

Figure 7: Failure cases in glass-door beverage cabinet scenes. The first and third rows show the RGB images overlaid with detection results, while the second and fourth rows present the corresponding raw depth maps and predicted depth maps.

12

Table 7: Results on FSCD-147. Best results are marked in the reference-only settings, while best and second-best results are marked in the unified both setting. Val Prompt

Test

Method MAE↓

RMSE↓

AP↑

AP50↑

MAE↓

RMSE↓

AP↑

AP50↑

text

Grounding DINO OWLv2 PSeCo DAVEprm CountGD CountGD-Box

54.45 49.02 23.90 14.87 12.46 12.24

137.12 131.41 100.30 59.60 67.52 66.24

6.66 11.39 × 16.31 13.36 27.02

10.26 20.33 × 47.12 43.90 60.73

54.16 41.83 16.58 15.52 15.19 15.01

157.87 149.82 129.77 114.10 119.40 118.16

11.60 22.84 41.14 18.50 18.10 30.44

17.80 35.76 69.03 50.24 52.90 61.56

exemplar

Counting-DETR PSeCo DAVE GeCo CountGD CountGD-Box

20.38 15.31 9.75 9.52 9.34 9.27

82.45 58.34 40.30 43.00 51.74 52.15

17.27 32.12 24.20 33.51 15.16 32.09

41.90 60.02 61.08 62.51 47.96 68.82

16.79 13.05 10.45 7.91 10.77 10.85

123.56 112.86 74.51 54.28 99.51 99.60

22.66 43.53 26.81 43.42 19.76 34.81

50.57 74.64 62.82 75.06 57.24 69.46

both

CountGD CountGD-Box Ours

8.69 8.76 14.85

43.89 44.24 51.86

14.74 33.10 20.45

47.95 71.38 47.22

10.18 10.29 11.91

96.20 96.33 93.78

20.50 36.20 26.58

59.40 72.39 55.95

Figure 6: Qualitative examples on FSCD-147 with pseudo-depth. Top row: input images with predicted counts (Pred) and ground-truth counts (GT). Bottom row: pseudo-depth maps generated by Depth Anything V2. In (a) and (b), the pseudo-depth provides relatively clear geometric cues and the predictions are accurate. In (c), the model yields a suboptimal prediction due to the influence of the pseudo-depth.

13

F

Failure Scenes analysis

conditions disrupt the stable correspondence between image appearance and true scene geometry. As a result, the model often fails to recover accurate depth hierarchies, and the predicted depth still suffers from structural deviations and blurred boundaries. Therefore, predicted depth cannot effectively compensate for the limitations of raw depth in glass scenes, and its contribution to dense instance separation and counting remains limited. This failure case indicates that, in scenes involving glass, transparent media, and strong reflections, depth information itself may lose its expected geometric discriminative power. Neither directly captured raw depth nor model-estimated predicted depth can provide stable and reliable support for video instance counting. Therefore, improving the robustness of the model under abnormal depth inputs, as well as more effectively integrating RGB appearance cues with unreliable depth signals in transparent and reflective scenes, remains an important direction for future research.

As shown in the figure 7, in beverage scenes with glass doors, the glass surface introduces reflections, refractions, and local highlights, which cause depth values acquired by the depth sensor to exhibit noise, missing regions, or structural distortions. This issue is particularly pronounced around the cabinet door boundaries, reflective glass regions, and densely arranged bottles and cans, where the raw depth map often fails to accurately represent the true spatial hierarchy of the objects, thereby affecting the effective separation between foreground targets and background structures. In such scenarios, if the model directly relies on raw depth information, it is easily disturbed by abnormal depth responses, which further leads to unstable target localization, ambiguous boundaries between adjacent instances, and even missed detections or duplicate counting. Meanwhile, the predicted depth information also performs poorly in this type of scene. Although the predicted depth maps can maintain relatively smooth global structures in some regions, the reflections from glass, transparent occlusions, and complex illumination

14

Related documents

• Counting Beyond Instances: A Benchmark for Group-Individual Object Counting#971393
• NumBench: Diagnosing Counting Failures in Text-to-Image Models#784959
• NumBench: Diagnosing Counting Failures in Text-to-Image Models#783369
• Cluster number counts in dark energy model with energy and momentum coupling to dark matter#677481
• Cluster number counts in dark energy model with energy and momentum coupling to dark matter#674284