Rethinking Object-Centric Representations for Video Dynamics Modeling
arXiv:2606.23436v1 [cs.CV] 22 Jun 2026
Amaury Wei , Ismail Nejjar , and Olga Fink Intelligent Maintenance and Operation Systems (IMOS) École Polytechnique Fédérale de Lausanne CH-1015 Lausanne, Switzerland {first.last}@epfl.ch
Fig. 1: STAITUS: Our method disentangles object appearance from spatial pose and enforces temporal alignment, enabling state-of-the-art unsupervised object tracking. Abstract. Unsupervised video object tracking aims to decompose dynamic scenes into persistent, object-centric entities without manual annotations. Many recent approaches rely on slot-based representations, where a fixed set of latent variables (“slots”) represent individual objects across frames. To preserve object identity, these models enforce temporal consistency on slot embeddings. However, when appearance and pose are entangled, this consistency objective conflicts with object motion and viewpoint changes. As a result, slots tend to lock onto static regions (e.g., background) to satisfy the consistency objective, while foreground objects become fragmented across multiple slots or frequently swap identities. To address these limitations, we propose STAITUS, a unified framework that explicitly disentangles each slot into appearance and geometric pose (position/scale). Leveraging this disentanglement, STAITUS enforces within-frame spatial separation and applies temporal alignment only in appearance space, yielding sharper masks and more persistent identities under motion, occlusion, and object entry/exit. Furthermore, to mitigate over-segmentation, we introduce an adaptive gating mechanism that dynamically adjusts the number of active slots to match scene complexity. Extensive experiments on synthetic and realworld benchmarks demonstrate that STAITUS substantially outperforms state-of-the-art baselines in segmentation quality and tracking stability.
2
A. Wei et al.
1
Introduction
Video understanding is a fundamental problem in computer vision, with applications ranging from robotics and autonomous driving [14,24,27] to captioning [43] and video question answering [2, 48]. Understanding videos requires identifying objects, tracking them over time, and reasoning about their interactions [48]. Object-centric representations provide a natural solution by decomposing scenes into individual entities. They have enabled progress in object-centric video prediction [32, 42, 45], causal reasoning [7, 38, 46], and planning [41, 50]. Learning these representations without supervision [40] is particularly attractive as it enables models to leverage raw video data without assuming predefined object categories or costly manual annotations. A common strategy for unsupervised object discovery is to model a scene using a fixed set of latent variables ("slots"), each intended to capture an individual object. Early methods such as MONet [4] and Slot Attention [26] demonstrated that scenes can be decomposed into permutation-invariant latent vectors that serve as object-centric representations. Subsequent extensions improved slot expressiveness [17, 22] and separation [3, 22, 28], enabling accurate object segmentation on synthetic benchmarks [12, 48]. More recently, models such as DINOSAUR [35] leverage self-supervised vision backbones [5] to extend object discovery in real-world images [47]. However, these advances focus primarily on single images, and extending object-centric representations to video introduces additional challenges. Beyond discovering objects, models must maintain consistent identities over time as objects move, become occluded, or interact with other objects. Several recent methods have attempted to bridge this gap by introducing temporal mechanisms into slot-based architectures. SAVi [9,21] propagates slots via recurrent updates conditioned on prior frames, while VideoSAUR [49] aligns slots by predicting DINO feature trajectories. More recently, SlotContrast [29] introduced a contrastive loss to align slot embeddings across time. While these approaches improve slot continuity, they assume that forcing stable embeddings is sufficient for object-level tracking. In practice, temporal consistency objectives are misaligned with the dynamic nature of object tracking because standard slot representations entangle object appearance with pose. As objects move, change scale, or become occluded, their pose must evolve and the embeddings should adapt, yet the training objective encourages them to remain constant. As a result, slots tend to attach to static background regions, while dynamic foreground objects drift between slots over time. This leads to background-foreground mixing, diffuse object masks, and identity switching. Moreover, using a fixed number of slots can fragment a single object across multiple slots, exacerbating over-segmentation. These effects produce poorly localized and temporally unstable object representations, limiting their usefulness for downstream tasks such as video prediction, dynamics modeling, and visual reasoning.
STAITUS
3
To address these challenges, we propose STAITUS (Sparse and Temporally Aligned InvarianT Unsupervised Slots), a unified slot-based video framework for unsupervised object tracking, designed to produce sharp object masks and stable object identities. STAITUS explicitly disentangles each object’s geometric pose (position and scale) from its visual appearance, enabling cleaner object separation and unambiguous segmentation. This disentanglement further allows the introduction of two video-specific regularization mechanisms: a spatial separation loss that prevents multiple objects from collapsing into the same slot, and a temporal alignment loss that enforces consistency of each slot’s visual appearance across frames. In addition, STAITUS incorporates adaptive slot usage, deactivating redundant slots as scenes evolve and thereby preventing over-segmentation. Through extensive experiments on synthetic and real-world video benchmarks, we demonstrate that STAITUS consistently outperforms recent slotbased baselines, producing sharper object masks and significantly more stable object identities (Fig. 1). Beyond overall performance improvements, targeted ablation studies demonstrate how geometric disentanglement, adaptive slot selection, and spatio-temporal regularization jointly contribute to robust objectcentric representations. Our main contributions are summarized as follows: 1. We analyze a key failure mode of existing slot-based video models and show that enforcing temporal consistency on pose-entangled representations inherently conflicts with object motion, leading to unstable tracking. 2. We propose STAITUS, a unified slot-based video model for unsupervised object tracking that disentangles object appearance from geometric pose, enabling temporally aligned appearance modeling, spatial separation between objects, and adaptive slot usage. 3. Through extensive experiments and ablations on synthetic and real-world benchmarks, we demonstrate that STAITUS yields substantially sharper object masks and more stable identities than prior slot-based approaches.
2
Related Work
Unsupervised Object-centric Learning. Early work on unsupervised objectcentric learning decomposed images into candidate objects using autoencoding architectures with iterative inference. MONet [4] introduced attention-based decomposition, extracting objects sequentially and demonstrating that meaningful object representations can emerge without supervision. Subsequent approaches [10, 13] improved stability and scalability. Although effective on simple scenes, these approaches often struggle to scale to visually complex images. Slot-based Object Discovery. Building on these foundations, Slot Attention (SA) [26] introduced an attention-based mechanism that groups image features into latent "slots", establishing a central paradigm for object-centric representation learning. Later work explored mixture-based decoders [22], compositional rendering approaches [17,37], and disentangled slot representations [3,28]. Other research focused on improved training objectives, including contrastive
4
A. Wei et al.
foreground-background separation [39], and slot-mixing strategies for novel scene synthesis [18]. SPOT [19] further improved robustness through a self-training objective with patch-order permutation. Recent advances leverage vision foundation models like DINO [5], enabling methods such as DINOSAUR [35] to scale to complex, real-world images. Additional directions incorporate diffusion models [1] or extend slot-based representations to multi-view 3D scenes [25]. However, these approaches are primarily designed for processing single images and do not explicitly address temporal consistency or object tracking. Unsupervised Object Tracking in Videos. Extending object-centric learning to videos is particularly challenging, as models must both discover objects and maintain consistent associations over time in dynamic scenes. Most slotbased models approaches address this by encouraging temporal consistency of slot representations. SAVi [21] and SAVi++ [9] propagate slots recurrently across frames, conditioning them on previous embeddings. VideoSAUR [49] instead aligns slots over time by predicting the temporal evolution of DINO patch embeddings. Other methods incorporate additional modalities, such as optical flow [23], to improve tracking performance. More recently, SlotContrast [29] introduced a contrastive objective that encourages slot embeddings within batches to remain distinct while promoting temporal consistency. Although these approaches improve temporal continuity compared to frame-by-frame inference, they often fail to preserve one-to-one correspondences between slots and objects. This limitation becomes particularly pronounced in dynamic scenes, where object motion, occlusions, and interactions conflict with consistency-based training objectives. Image Reconstruction and Slot Decoders. Most slot-based models are trained using reconstruction objectives, either in pixel space or feature space. The image decoder, which maps slot representations back to the visual signal, plays a critical yet often overlooked role in object-centric learning, as it determines how reconstruction errors are attributed to individual slots and therefore how scenes are decomposed into objects. Early approaches such as Slot Attention [26] relied on spatial broadcast decoders [44], which decode slots independently and combine them using transparency masks. Later work introduced more expressive decoders that enable interactions between slots, including autoregressive decoders [6] in SLATE [37], SlotMixer decoders [34], and Scene Representation Transformers [33]. Although these designs improve image reconstruction quality, they also allow multiple slots to jointly explain the same pixels. This directly conflicts with the objective of assigning each pixel to a single object, thereby weakening unsupervised object discovery.
3
Method
We propose STAITUS, a unified framework for unsupervised object tracking that decouples object identity from motion. As illustrated in Fig. 2, STAITUS integrates dense feature extraction, recurrent disentangled slot grouping, adaptive decoding, and temporal–spatial regularization to yield spatially precise and temporally consistent object-centric representations from unlabeled videos.
STAITUS Recurrent Adaptive Disentangled Slot Attention
5
Adaptive Spatial Broadcasting
Initialization
Slots
Dense Feature Encoding
Pos. Scale Appearance
Activ. 1 1
Reconstruction
1
MLP
MLP
DINO encoder
1 0 1
Image
Features
Individual masks
Fig. 2: Overview of STAITUS. Given a frame xt , an encoder extracts dense features ht , which are grouped by a recurrent module into disentangled slot representations consisting of position (pt ), scale (st ), and visual appearance (vt ) components. A learned gating mechanism Ggate determines slot activation zt dynamically adapting the number of active slots over time. Each active slot is decoded into an image x̂kt and an alpha mask αtk . The final reconstruction x̂t is obtained by compositing all decoded slots.
3.1
Problem Formulation
Given a video sequence V = {x1 , . . . , xT } of T frames, we denote the RGB frame at time t ∈ {1, . . . , T } by xt ∈ RH×W ×3 . Our objective is to decompose each frame into a set of K object-centric slot representations St = {St1 , . . . , StK }, where each slot corresponds to an object and one slot models the background. Existing slot-based video models [29,49] typically entangle object appearance and pose within a single slot representation. Although temporal consistency is required to preserve identities [29], enforcing it on pose-entangled slots conflicts with motion and viewpoint changes in dynamic scenes. As a result, slots often attach to static regions (e.g., background) to satisfy the temporal consistency objective, while foreground objects become fragmented or swap identities. To address this issue, we disentangle each slot Stk into appearance and geometry: S_t^k \;=\; (v_t^k, p_t^k, s_t^k), \qquad v_t^k \in \mathbb {R}^{D}, \;\;\; p_t^k, s_t^k \in \mathbb {R}^{2},
(1)
where vtk is an object-specific appearance embedding and (pkt , skt ) denote the 2D position and scale of the slot. This disentanglement allows STAITUS to enforce temporal alignment in appearance space while allowing geometric attributes to evolve freely under motion, occlusion, and object entry or exit. 3.2
Dense Feature Encoder
Given a video frame xt , we first extract N dense patch features gt of dimension Dfeat using a pretrained and frozen self-supervised DINO [5] encoder Ffeat : g_t = F_\mathrm {feat}(x_t),\quad g_t\in \mathbb {R}^{N\times D_\mathrm {feat}}.
(2)
While DINO features capture rich semantic information from static images, they are not optimized for object-centric tasks such as localization or tracking. We, therefore, project each feature vector gt into a task-specific embedding space using a lightweight two-layer MultiLayer Perceptron (MLP) Fproj : h_t = F_\mathrm {proj}(g_t),\quad h_t\in \mathbb {R}^{N\times D_\mathrm {feat}}.
(3)
6
3.3
A. Wei et al.
Recurrent Adaptive Disentangled Slot Attention Module
To discover and track objects consistently over time, we adopt a recurrent slot refinement mechanism similar to [21, 49]. Slots at time t are initialized from the previous state at t − 1 and iteratively refined using the dense features ht . Our grouping module produces K slot representations {Stk }K k=1 , corresponding to individual objects and the background. It is designed to maintain temporally consistent appearance representations while dynamically adapting the number of active slots according to the scene content. Disentangled Slot Grouping. To extract disentangled slot components (vtk , ptk , stk ), we build upon Invariant Slot Attention (ISA) [3]. In contrast to vanilla SA, ISA replaces fixed global positional encodings with slot-specific reference frames derived from the slot geometry. Each slot uses its estimated position and scale to center on an object and attends to pose-normalized appearance features. This produces an appearance embedding vtk that is invariant to object motion, while (pkt , skt ) capture the object’s evolving pose for decoding (Sec. 3.4). Formally, we apply an iterative attention-based grouping module Gslot for Tslot refinement steps. To incorporate temporal context, slots at time t are initialized from the previous frame using a predictor Gpred , yielding predicted slot states S̃tk = (ṽtk , p̃kt , s̃kt ) (detailed next). The grouping step then refines them as: \{(v^k_t, p^k_t, s^k_t)\}_{k=1}^K = G_\mathrm {slot}(h_t, \tilde S_{t}) \label {eq:isa_refine}
(4)
During training, we additionally enforce temporal alignment on appearance embeddings and apply a spatial separation loss to prevent slot redundancy (Sec. 3.5). Details and a pseudocode for Gslot are provided in Supplementary. Recurrent Slot Attention Formulation. To connect consecutive frames, we initialize slots at time t from the refined slot states at time t−1. We compute predicted states S̃tk = (ṽtk , p̃kt , s̃kt ), which serve as the initialization for the grouping step in Eq. (4). We treat appearance and pose differently. The appearance embedding is predicted to accommodate gradual appearance changes and the emergence of new objects [45], while pose is propagated forward under a smooth-motion assumption. Specifically, a residual MLP Gpred predicts a Gaussian distribution over the next-step appearance: (\mu ^k_t, \log \sigma ^k_t) &= G_\mathrm {pred}(v^k_{t-1}), \qquad \tilde {v}^k_t \sim \mathcal {N}(\mu ^k_t, \sigma ^k_t), \\ (\tilde p_t^k,\tilde s_t^k)&=(p_{t-1}^k,s_{t-1}^k). (6) Gradients are propagated using the reparameterization trick [20]. For t=1, all slot components are initialized from learned embeddings. Adaptive Slot Activation. Scenes contain a varying number of objects, yet most slot-based models enforce a fixed number of slots K for every frame, often leading to over-segmentation. Inspired by AdaSlot [11], STAITUS dynamically determines which slots are active at each timestep, adapting slot usage to the scene content.
STAITUS
7
Specifically, for each slot appearance embedding vtk , a lightweight MLP Ggate predicts a binary activation variable: z^k_t = G_\mathrm {gate}(v^k_t),\quad z^k_t \in \{0,1\}.
(7)
The gating decision depends solely on appearance, allowing the model to suppress visually redundant slots while remaining invariant to object motion and position. Only slots with ztk = 1 participate in image reconstruction (Sec. 3.4), while inactive slots are ignored. To enable end-to-end learning, we use the Gumbel–Softmax relaxation [16] during training. 3.4
Adaptive Spatial Broadcast Decoding
After slot decomposition, the model reconstructs the input RGB frame, which serves as the primary unsupervised learning signal. To obtain meaningful objectcentric representations, reconstruction should preserve sharp object boundaries and assign each pixel predominantly to a single slot. We therefore decode slots independently and avoid inter-slot feature mixing, which often leads to blurry masks. For this purpose, we adopt a spatial broadcast decoding strategy [44]. A shared decoder Dspatial maps each slot representation to a full-frame RGB reconstruction x̂kt ∈ RH×W ×3 and an alpha mask αtk ∈ RH×W . Internally, the decoder uses the disentangled geometric parameters (pkt , skt ) to construct a slotspecific coordinate grid as positional embedding, following ISA [3]. The per-slot decoding operation is: (\hat {x}^k_t, \alpha ^k_t) = D_{\mathrm {spatial}}\!\left (\mathrm {SB}(v^k_t) ,p_t^k,s_t^k\right ),
(8)
where Dspatial is a CNN-based decoder, SB is the spatial broadcast operation, and (pkt , skt ) parametrize the relative coordinate grid. Implementation details of Dspatial are provided in the Supplementary Material. The final frame reconstruction is obtained by compositing all active slots: \hat {x}_t = \sum _{k=1}^K z^k_t \cdot \alpha ^k_t \odot \hat {x}^k_t .
(9)
The final reconstruction x̂t is used to train using a pixel-wise end-to-end Mean Squared Error (MSE) loss, Lrecon = E ∥xt − x̂t ∥22 , averaged over pixels and color channels, which serves as the primary unsupervised learning objective. 3.5
Temporal Alignment and Spatial Separation Objectives
To guide unsupervised learning toward stable and well-separated object representations, we introduce two complementary objectives: a temporal alignment loss and a spatial separation loss. Both operate on the slot appearance embeddings {vtk }K k=1 , and Fig. 3 provides a visual illustration of their effects.
8 a)
A. Wei et al. Frame
b)
Frame
Frame
Slot visual appearances Spatial separation
... ...
...
...
c)
...
Temporal alignment
Push away
Temporal alignment ...
...
Image
Slots
Reconstruction ...
Reconstruction
Fig. 3: Illustration of the training objectives. a) Temporal alignment loss Ltime encourages consistent slot appearance across consecutive frames. b) Spatial separation loss Lsep encourages distinct slot appearances vtk in embedding space. c) Reconstruction loss Lrecon drives scene decomposition by minimizing the error between the input frame xt and its composited reconstruction x̂t .
Temporal Alignment Loss. For reliable object tracking, the same object should remain assigned to the same slot across consecutive frames. This requires each slot to maintain a consistent appearance appearance v k over time, even as its pose changes. To encourage this behavior, we introduce a temporal alignment loss that regularizes consecutive slot appearance embeddings using cosine similarity: \mathcal {L}_{\mathrm {time}} = \frac {\sum _{t,k} z^k_{t-1} z^k_t \, \left (1 - \cos \!\left (\tilde {v}^k_t, v^k_t \right )\right )} {\sum _{t,k} z^k_{t-1} z^k_t}
(10)
k The product zt−1 ztk ensures that only slots active at both timesteps contribute to the loss. Unlike SlotContrast [29], which aligns a fixed set of pose-entangled slots, our alignment operates exclusively in disentangled appearance space and only for active slots. This allows pose attributes to evolve freely under motion while naturally supporting slot deactivation when objects enter or leave the scene.
Spatial Separation Loss. Within each frame, distinct objects should be represented by different slots. Without explicit regularization, however, multiple slots may collapse onto the same object or encode overlapping regions. To promote slot specialization, we introduce a spatial separation loss that discourages similar appearance embeddings among active slots by penalizing positive cosine similarity. Slots representing distinct objects incur no penalty, whereas slots encoding similar visual content are pushed apart or encouraged to deactivate.
\mathcal {L}_{\mathrm {sep}} = \frac { \sum _{t} \sum _{i \neq j} z^i_t z^j_t \, \max \!\left (0,\, \cos \!\left (v^i_t, v^j_t \right )\right ) }{ \sum _{t} \sum _{i \neq j} z^i_t z^j_t }
(11)
The product zti ztj ensures that only pairs of active slots contribute to the loss. Unlike SlotContrast [29], our loss acts solely on appearance embeddings, preventing nearby objects with similar poses from being grouped into a single slot (see
STAITUS
9
Sec. 5.3). Furthermore, SlotContrast relies on a batch-wise contrastive objective that introduces global competition across videos and can destabilize optimization, whereas our separation loss is computed locally within each frame and integrates naturally with adaptive slot activation. Training Objective. The full training objective combines the reconstruction loss with the proposed regularization terms: (12) k where Lspars = Et,k zt is a sparsity regularizer that encourages the model to activate only the required number of slots at each timestep. \mathcal {L}=\mathcal {L}_{\mathrm {recon}}+\lambda _{\mathrm {time}} \mathcal {L}_{\mathrm {time}}+\lambda _{\mathrm {sep}} \mathcal {L}_{\mathrm {sep}}+\lambda _\mathrm {spars}\mathcal {L}_{\mathrm {spars}}
4
Experiments
We evaluate STAITUS on both unsupervised object discovery and segmentationbased object tracking across synthetic video benchmarks and real-world datasets. We evaluate the method along three dimensions: mask sharpness, foregroundbackground separation, and identity stability over time. Datasets. We evaluate our method on synthetic datasets, including CLEVRER and the MOVi benchmark suite (MOVi-A, MOVi-B, MOVi-C, MOVi-E) [12], which provide increasing scene complexity in terms of object count, appearance, and motion dynamics. To assess scalability to real-world videos, we additionally evaluate on the YouTube-VIS 2021 dataset [47], featuring unconstrained scenes with camera motion, occlusions, and background clutter. Representative video examples are provided in the Supplementary Material. Metrics. We report complementary metrics to evaluate both segmentation quality and tracking performance (Sec. 5). FG-ARI (Foreground Adjusted Rand Index) [15,31] measures foreground object discovery, while full ARI additionally reflects background segmentation quality. We further report mean Best Overlap (mBO) [30] to assess mask sharpness and boundary precision. All metrics are computed both per frame and across full video sequences. Baselines. We compare STAITUS with image-based methods Invariant Slot Attention (ISA) [3], Adaptive Slot Attention (AdaSlot) [11], and DINOSAUR [35], as well as video-based methods SAVi [21], VideoSAUR [49], and SlotContrast [29]. All baselines are evaluated using their official implementations, and feature encoders are matched whenever required to ensure a fair comparison. Implementation Details. We use a single set of hyperparameters across all datasets to highlight the robustness of our approach. We employ DINO ViT-B/16 as the feature encoder, set the slot dimension to Dslot =128, and use Tslots =2 slot attention iterations. All experiments are conducted at the native resolution of the MOVi datasets (256, 256), resulting in N =256 patch tokens. Additional training details and hyperparameters are provided in the Supplementary Material.
10
A. Wei et al.
5
Results
5.1
Unsupervised Object Discovery
Across all datasets, STAITUS consistently outperforms both image-based and video-based baselines in per-image object discovery, achieving the highest ARI and mBO on nearly all benchmarks (Tab. 1). The strong ARI reflects improved foreground–background separation, while the high mBO indicates sharper object boundaries. Notably, STAITUS maintains high FG-ARI while simultaneously improving full ARI, whereas prior methods typically trade foreground grouping against background separation. These improvements persist from simple to complex scenes (CLEVRER to MOVi-E), and extend to the challenging real-world YouTube-VIS benchmark, demonstrating the robustness of our disentangled and adaptive architecture. Qualitative comparisons are shown in Sec. 5.3. Table 1: Object discovery (per-image) results. Metrics are computed for entire video sequences (24 frames for CLEVRER and MOVi, up to 84 frames for YouTube-VIS). Best results are shown in bold, and second-best are underlined. CLEVRER
Method
MOVi-A
MOVi-B
ARI↑ FG-ARI↑ mBO↑ ARI↑ FG-ARI↑ mBO↑ ARI↑ FG-ARI↑ mBO↑ DINOSAUR [35] ISA [3] AdaSlot [11] SAVi [21] VideoSAUR [11] SlotContrast [29] STAITUS (Ours)
0.05 0.03 0.02 0.01 0.37 0.05 0.86
DINOSAUR [35] ISA [3] AdaSlot [11] SAVi [21] VideoSAUR [11] SlotContrast [29] STAITUS (Ours)
0.15 0.15 0.02 0.06 0.24 0.11 0.50
0.97 0.85 0.34 0.41 0.28 0.96 0.91
0.15 0.16 0.06 0.07 0.18 0.12 0.84
0.04 0.23 0.07 0.02 0.07 0.19 0.88
0.40 0.39 0.04 0.22 0.43 0.23 0.44
0.13 0.15 0.07 0.01 0.25 0.20 0.38
MOVi-C
5.2
0.65 0.61 0.21 0.47 0.73 0.71 0.77
0.95 0.81 0.40 0.41 0.50 0.93 0.84
0.17 0.59 0.09 0.07 0.20 0.20 0.65
0.07 0.63 0.45 0.18 0.22 0.15 0.65
0.38 0.31 0.09 0.29 0.33 0.30 0.34
0.18 0.18 0.08 0.05 0.16 0.16 0.25
MOVi-E 0.61 0.70 0.20 0.61 0.66 0.77 0.81
0.86 0.56 0.38 0.48 0.53 0.89 0.65
0.25 0.55 0.21 0.26 0.22 0.27 0.57
YouTube-VIS 0.22 0.23 0.14 0.18 0.23 0.28 0.32
0.50 0.53 0.26 0.23 0.44 0.40 0.46
Unsupervised Segmentation-Based Object Tracking
Building on its strong per-image decomposition, STAITUS also excels in segmentationbased object tracking. Across all datasets, it preserves consistent slot–object identities over video sequences, avoiding slot swapping and background mixing. As shown in Tab. 2, ARI and mBO remain high with only minor degradation relative to per-frame results (Tab. 1), even under object motion, occlusion, scene changes, and camera movements. While SAVi and VideoSAUR achieve reasonable per-frame decompositions, they struggle to maintain object identities over time, leading to substantial drops (often exceeding 30%) in ARI or FG-ARI. SlotContrast improves temporal consistency but often merges nearby objects into a single slot (see Sec. 5.3).
STAITUS
11
Table 2: Object tracking results. Metrics are computed on 24-frame video clips (using the first 24 frames for YouTube-VIS). Best results are shown in bold, and second-best are underlined. CLEVRER
Method
MOVi-A
MOVi-B
ARI↑ FG-ARI↑ mBO↑ ARI↑ FG-ARI↑ mBO↑ ARI↑ FG-ARI↑ mBO↑ SAVi [21] VideoSAUR [11] SlotContrast [29] STAITUS (Ours)
0.00 0.00 0.04 0.88
SAVi [21] VideoSAUR [11] SlotContrast [29] STAITUS (Ours)
0.01 0.22 0.10 0.48
0.24 0.16 0.62 0.76
0.05 0.03 0.09 0.65
0.00 0.19 0.18 0.88
0.08 0.33 0.27 0.35
0.00 0.24 0.20 0.25
MOVi-C 0.14 0.61 0.57 0.65
0.08 0.30 0.91 0.78
0.03 0.15 0.19 0.63
0.14 0.18 0.15 0.73
0.10 0.25 0.26 0.28
0.03 0.14 0.14 0.23
MOVi-E 0.23 0.59 0.74 0.78
0.27 0.27 0.83 0.54
0.15 0.14 0.27 0.50
YouTube-VIS 0.11 0.19 0.23 0.25
0.18 0.38 0.36 0.44
In contrast, STAITUS preserves both object separation and identity across simple CLEVRER scenes, dense MOVi-E sequences, and real-world YouTube-VIS videos, demonstrating that appearance–pose disentanglement provides a strong inductive bias for unsupervised tracking. 5.3
Qualitative Results
We provide qualitative comparisons to illustrate how STAITUS improves object discovery and tracking beyond what is captured by aggregate metrics. Sharp and Compact Masks. Figure 4 shows example decompositions on a CLEVRER frame. DINOSAUR and SlotContrast successfully extract foreground objects in slots 0−4, but their masks exhibit substantial background leakage and diffuse boundaries. Moreover, the background is split between slots 4−5, explaining the high FG-ARI yet low full ARI results observed in Tab. 1. In contrast, STAITUS produces compact masks with sharp boundaries: each slot cleanly isolates a single object, and unused slot are automatically deactivated (slot 6). This visual evidence supports the ARI and mBO improvements reported in Tab. 1, confirming that the proposed appearance-pose disentanglement and the spatial separation loss promote precise object-specific segmentation. Temporal Identity Consistency. Figure 5 presents an object tracking example on a MOVi-C video. Beyond background leakage in the masks, VideoSAUR suffers from over-segmentation in several frames (e.g., t = {0, 4, 8}) and fails to maintain object identity between t = 12 and t = 16. In comparison, SlotContrast preserves identities more consistently but merges distinct objects into a single slot (e.g., t = {8, 12, 16}). In contrast, STAITUS maintains stable slot–object correspondence throughout the entire sequence, preserving object identities under motion, overlap, and partial occlusion. These observations are consistent with the strong tracking performance in Tab. 2 and demonstrate that aligning appearance space while allowing pose to evolve freely provides a robust inductive bias for unsupervised object tracking.
12
A. Wei et al. Slot 0
Slot 1
Slot 2
Slot 3
Slot 4
Slot 5
Slot 6
SlotContrast
DINOSAUR
Input
STAITUS (Ours)
Deactivated
Fig. 4: Unsupervised scene decomposition on CLEVRER. We compare the segmentation masks generated by DINOSAUR, SlotContrast, and STAITUS on a sample frame. STAITUS produces significantly sharper and more precise masks, successfully isolating individual objects with minimal background leakage. t=4
t=8
t=12
t=16
t=20
t=23
Identity swap
Over segmentation
SlotContrast
VideoSAUR
t=0
Object grouping
STAITUS (Ours)
Identity swap
Fig. 5: Qualitative comparison of unsupervised object tracking in MOVi-C. The baselines exhibit identity swaps, over-segmentation (splitting single objects), and object grouping (merging distinct objects). In contrast, STAITUS maintains robust object identities and sharp segmentation boundaries across the entire sequence.
These visual results highlight that STAITUS not only improves quantitative metrics but also produces cleaner decompositions and more reliable long-term object identity tracking in challenging dynamic scenes. 5.4
Ablation Studies
Ablation of Loss Components. We systematically ablate each regularization term to assess its necessity (Tab. 3). Removing any component degrades performance, confirming that STAITUS relies on complementary constraints rather than a single dominant loss. Using only reconstruction leads to a collapse in performance (e.g., CLEVRER ARI: 0.88 → 0.21), showing that reconstruction alone is insufficient to learn stable object-centric representations. Removing temporal alignment mainly affects tracking under motion and occlusion, with a drastic drop on MOVi-C (0.48 → 0.06 ARI) but only minor changes on YouTube-VIS, where errors are dominated by visual complexity
STAITUS
13
Table 3: Ablation of loss components on the object tracking task. Each row removes one component from the full objective. Metrics are computed over 24-frame video clips. CLEVRER
Method
MOVi-C
YouTube-VIS
ARI↑ FG-ARI↑ mBO↑ ARI↑ FG-ARI↑ mBO↑ ARI↑ FG-ARI↑ mBO↑ No regularization No alignment Ltime No separation Lsep No sparsity Lspars Complete model
0.21 0.16 0.84 0.86 0.88
0.76 0.72 0.76 0.76 0.76
0.42 0.59 0.57 0.65 0.65
0.12 0.06 0.15 0.11 0.48
0.59 0.36 0.47 0.63 0.65
0.29 0.12 0.26 0.34 0.35
0.06 0.23 0.22 0.14 0.23
0.18 0.23 0.25 0.21 0.25
0.26 0.43 0.42 0.36 0.44
rather than identity drift. Removing Lsep causes slot collapse and degraded decomposition, notably on MOVi-C (0.48 → 0.15 ARI). Finally, removing Lspars causes over-segmentation and weaker foreground–background separation, most evident on YouTube-VIS (0.23 → 0.14 ARI). These results highlight complementary roles: temporal alignment is critical under motion and occlusion (MOVi-C), whereas separation and sparsity are most important in visually complex and cluttered scenes (YouTube-VIS). The full model consistently achieves the best performance across datasets. Qualitative visualizations are provided in the Supplementary Material. Sensitivity to Feature Encoder. STAITUS remains robust to the choice of pretrained visual backbone. Table 4 shows that performance varies only moderately across encoders, while the relative ranking of datasets and metrics remains stable. This indicates that STAITUS does not depend on a specific pretrained representation yet can still benefit from stronger visual features. In particular, DINOv3 achieves the best overall ARI and mBO scores, demonstrating that improved encoders translate into measurable gains. For fair comparison with prior slot-based work, we adopt DINOv1 in the main experiments. Table 4: Performance of STAITUS using different pretrained DINO feature encoders on the object tracking task. Results are reported on 24-frame video clips. CLEVRER
Method
MOVi-C
YouTube-VIS
ARI↑ FG-ARI↑ mBO↑ ARI↑ FG-ARI↑ mBO↑ ARI↑ FG-ARI↑ mBO↑ DINOv1 [5] 0.88 DINOv2 [8] 0.85 DINOv3 [36] 0.90
6
0.76 0.74 0.76
0.65 0.64 0.68
0.48 0.42 0.25
0.65 0.53 0.68
0.35 0.31 0.37
0.23 0.20 0.20
0.25 0.21 0.28
0.44 0.35 0.43
Discussion
STAITUS exposes a core misalignment in prior slot-based video models: enforcing temporal consistency on pose-entangled representations creates an inherent conflict between identity preservation and motion. Our findings indicate that stable object-centric learning depends on structuring the latent space so that appearance and pose are disentangled, allowing temporal constraints to operate in a semantically meaningful subspace. Crucially, restructuring alone is insufficient: explicit regularization during training is required to guide the slot representations toward stable solutions.
A. Wei et al.
STAITUS (Ours)
Raw Image
14
Fig. 6: Qualitative slot decomposition on MOVi-E and YouTubeVIS. While STAITUS accurately segments foreground objects, it occasionally decomposes complex backgrounds into distinct semantic regions (e.g., separating forest from snow).
The qualitative behavior of STAITUS further reveals interesting structural properties. On photo-realistic and real-world data, STAITUS occasionally splits visually distinct background regions into separate slots (Fig. 6). For instance, the snowboarder background is partitioned into snow and forest regions, while the tennis background is split into court, fence, and trees. In MOVi-E, background regions are often separated according to depth. This behavior reflects the model’s bias toward grouping regions with consistent appearance and geometry, even when such regions belong to the semantic background. On real-world videos, mask boundaries are less precise under strong texture or illumination changes, revealing limits in the current feature representation and decoder. More broadly, the reconstruction objective introduces an inherent tradeoff: encouraging high-fidelity reconstruction promotes sharp masks, but leads to over-segmentation of heterogeneous background regions. Finally, while STAITUS maintains identity over moderate temporal horizons, extending stability to very long sequences remains an open challenge. Future work could explore explicit background modeling to prevent background fragmentation, and more expressive decoding strategies to refine realworld mask boundaries.
7
Conclusion
We introduce STAITUS, a unified framework for unsupervised video objectcentric learning that resolves a key misalignment in prior slot-based models between temporal consistency and object motion. By disentangling appearance from pose and aligning slots only in appearance space, STAITUS preserves identity without constraining geometric evolution. Combined with adaptive slot usage and spatial separation, this structured formulation yields sharper decompositions and more stable tracking across both synthetic and real-world datasets. More broadly, our results indicate that stable unsupervised object tracking depends less on increasing architectural complexity and more on imposing semantically meaningful structure on the latent space.
STAITUS
15
References 1. Akan, K., Yemez, Y.: Slot-guided adaptation of pre-trained diffusion models for object-centric learning and compositional generation. In: ICLR (2025) 4 2. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual question answering. In: ICCV. pp. 2425–2433 (2015) 2 3. Biza, O., van Steenkiste, S., Sajjadi, M.S., Mahendran, A., Kipf, T.: Invariant slot attention: Object discovery with slot-centric reference frames. In: ICML. pp. 2507–2527 (2023) 2, 3, 6, 7, 9, 10 4. Burgess, C.P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., Lerchner, A.: MONet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390 (2019) 2, 3 5. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV. pp. 9650– 9660 (October 2021) 2, 4, 5, 13 6. Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., Sutskever, I.: Generative pretraining from pixels. In: ICML. pp. 1691–1703 (2020) 4 7. Chen, Z., Dong, S., Yi, K., Li, Y., Ding, M., Torralba, A., Tenenbaum, J.B., Gan, C.: Compositional physical reasoning of objects and events from videos. IEEE TPAMI 47(9), 7689–7703 (2025) 2 8. Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. In: ICLR (2024) 13 9. Elsayed, G.F., Mahendran, A., van Steenkiste, S., Greff, K., Mozer, M.C., Kipf, T.: SAVi++: Towards end-to-end object-centric learning from real-world videos. In: NeurIPS (2022) 2, 4 10. Engelcke, M., Kosiorek, A.R., Jones, O.P., Posner, I.: GENESIS: Generative scene inference and sampling with object-centric latent representations. In: ICLR (2020) 3 11. Fan, K., Bai, Z., Xiao, T., He, T., Horn, M., Fu, Y., Locatello, F., Zhang, Z.: Adaptive slot attention: Object discovery with dynamic slot number. In: CVPR. pp. 23062–23071 (2024) 6, 9, 10, 11 12. Greff, K., Belletti, F., Beyer, L., Doersch, C., Du, Y., Duckworth, D., Fleet, D.J., Gnanapragasam, D., Golemo, F., Herrmann, C., Kipf, T., Kundu, A., Lagun, D., Laradji, I., Liu, H.T.D., Meyer, H., Miao, Y., Nowrouzezahrai, D., Oztireli, C., Pot, E., Radwan, N., Rebain, D., Sabour, S., Sajjadi, M.S.M., Sela, M., Sitzmann, V., Stone, A., Sun, D., Vora, S., Wang, Z., Wu, T., Yi, K.M., Zhong, F., Tagliasacchi, A.: Kubric: A scalable dataset generator. In: CVPR. pp. 3749–3761 (2022) 2, 9 13. Greff, K., Kaufman, R.L., Kabra, R., Watters, N., Burgess, C., Zoran, D., Matthey, L., Botvinick, M., Lerchner, A.: Multi-object representation learning with iterative variational inference. In: ICML (2019) 3 14. Hamdan, S., Güney, F.: CarFormer: Self-driving with learned object-centric representations. In: ECCV. pp. 177–193. Springer (2024) 2 15. Hubert, L., Arabie, P.: Comparing partitions. Journal of classification 2(1), 193– 218 (1985) 9 16. Jang, E., Gu, S., Poole, B.: Categorical reparameterization with Gumbel-Softmax. In: ICLR (2017) 7 17. Jiang, J., Deng, F., Singh, G., Ahn, S.: Object-centric slot diffusion. In: NeurIPS. vol. 37 (2023) 2, 3 18. Jung, W., Yoo, J., Ahn, S., Hong, S.: Learning to compose: Improving object centric learning by injecting compositionality. In: ICLR (2024) 4
16
A. Wei et al.
19. Kakogeorgiou, I., Gidaris, S., Karantzalos, K., Komodakis, N.: SPOT: Self-training with patch-order permutation for object-centric learning with autoregressive transformers. In: CVPR. pp. 22776–22786 (2024) 4 20. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: ICLR (2014) 6 21. Kipf, T., Elsayed, G.F., Mahendran, A., Stone, A., Sabour, S., Heigold, G., Jonschkowski, R., Dosovitskiy, A., Greff, K.: Conditional object-centric learning from video. In: ICLR (2022) 2, 4, 6, 9, 10, 11 22. Kirilenko, D., Vorobyov, V., Kovalev, A., Panov, A.: Object-Centric Learning with Slot Mixture Module. In: ICLR (2024) 2, 3 23. Lee, M., Cho, S., Lee, D., Park, C., Lee, J., Lee, S.: Guided slot attention for unsupervised video object segmentation. In: CVPR. pp. 3807–3816 (2024) 4 24. Liu, D., Cui, Y., Chen, Y., Zhang, J., Fan, B.: Video object detection for autonomous driving: Motion-aid feature calibration. Neurocomputing 409, 1–11 (2020) 2 25. Liu, Y., Jia, B., Chen, Y., Huang, S.: SlotLifter: Slot-guided feature lifting for learning object-centric radiance fields. In: ECCV. pp. 270–288 (2024) 4 26. Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., Kipf, T.: Object-centric learning with slot attention. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) NeurIPS. vol. 33, pp. 11525–11538 (2020) 2, 3, 4 27. Maddern, W., Pascoe, G., Linegar, C., Newman, P.: 1 year, 1000 km: The Oxford RobotCar dataset. The International Journal of Robotics Research 36(1), 3–15 (2017) 2 28. Majellaro, R., Collu, J., Plaat, A., Moerland, T.M.: Explicitly disentangled representations in object-centric learning. TMLR (2025) 2, 3 29. Manasyan, A., Seitzer, M., Radovic, F., Martius, G., Zadaianchuk, A.: Temporally consistent object-centric learning by contrasting slots. In: CVPR. pp. 5401–5411 (2025) 2, 4, 5, 8, 9, 10, 11 30. Pont-Tuset, J., Arbelaez, P., Barron, J.T., Marques, F., Malik, J.: Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE transactions on pattern analysis and machine intelligence 39(1), 128–140 (2016) 9 31. Rand, W.M.: Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association 66(336), 846–850 (1971) 9 32. Rezazadeh, A., Badithela, A., Desingh, K., Choi, C.: SlotGNN: Unsupervised discovery of multi-object representations and visual dynamics. In: ICRA. pp. 17508– 17514 (2024) 2 33. Sajjadi, M.S.M., Meyer, H., Pot, E., Bergmann, U., Greff, K., Radwan, N., Vora, S., Lucic, M., Duckworth, D., Dosovitskiy, A., Uszkoreit, J., Funkhouser, T., Tagliasacchi, A.: Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. In: CVPR (2022) 4 34. Sajjadi, M.S., Duckworth, D., Mahendran, A., Van Steenkiste, S., Pavetic, F., Lucic, M., Guibas, L.J., Greff, K., Kipf, T.: Object scene representation transformer. In: NeurIPS. vol. 35, pp. 9512–9524 (2022) 4 35. Seitzer, M., Horn, M., Zadaianchuk, A., Zietlow, D., Xiao, T., Simon-Gabriel, C.J., He, T., Zhang, Z., Schölkopf, B., Brox, T., Locatello, F.: Bridging the Gap to RealWorld Object-Centric Learning. In: ICLR (2023) 2, 4, 9, 10 36. Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., Massa, F., Haziza, D., Wehrstedt, L., Wang, J., Darcet, T., Moutakanni, T., Sentana, L., Roberts, C., Vedaldi, A., Tolan, J., Brandt, J., Couprie, C., Mairal, J., Jégou, H., Labatut, P., Bojanowski, P.: DINOv3. arXiv preprint arXiv:2508.10104 (2025) 13
STAITUS
17
37. Singh, G., Deng, F., Ahn, S.: Illiterate DALL-E learns to compose. In: ICLR (2022) 3, 4 38. Tang, Q., Zhu, X., Lei, Z., Zhang, Z.: Intrinsic physical concepts discovery with object-centric predictive models. In: CVPR. pp. 23252–23261 (2023) 2 39. Tian, P., Yang, S., Yu, H., Kot, A.: Pay attention to the foreground in object-centric learning. In: CVPR. pp. 30281–30290 (2025) 4 40. Tuytelaars, T., Lampert, C.H., Blaschko, M.B., Buntine, W.: Unsupervised object discovery: A comparison. IJCV 88(2), 284–302 (2010) 2 41. Villar-Corrales, A., Behnke, S.: PlaySlot: Learning inverse latent dynamics for controllable object-centric video prediction and planning. In: ICML (2025) 2 42. Villar-Corrales, A., Wahdan, I., Behnke, S.: Object-centric video prediction via decoupling of object dynamics and interactions. In: ICIP. pp. 570–574. IEEE (2023) 2 43. Wang, B., Ma, L., Zhang, W., Liu, W.: Reconstruction network for video captioning. In: CVPR. pp. 7622–7631 (2018) 2 44. Watters, N., Matthey, L., Burgess, C.P., Lerchner, A.: Spatial broadcast decoder: A simple architecture for learning disentangled representations in VAEs. arXiv preprint arXiv:1901.07017 (2019) 4, 7 45. Wu, Z., Dvornik, N., Greff, K., Kipf, T., Garg, A.: SlotFormer: Unsupervised visual dynamics simulation with object-centric models. In: ICLR (2023) 2, 6 46. Xiao, J., Yao, A., Li, Y., Chua, T.S.: Can i trust your answer? visually grounded video question answering. In: CVPR. pp. 13204–13214 (June 2024) 2 47. Yang, L., Fan, Y., Xu, N.: Video instance segmentation. In: ICCV. pp. 5188–5197 (2019) 2, 9 48. Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A., Tenenbaum, J.B.: CLEVRER: Collision events for video representation and reasoning. In: ICLR (2020) 2 49. Zadaianchuk, A., Seitzer, M., Martius, G.: Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities. In: NeurIPS. vol. 37 (2023) 2, 4, 5, 6, 9 50. Zou, J., Zhu, X., Zhang, Z., Lei, Z.: Top-down guidance for learning object-centric representations. In: IJCAI (2025) 2