U NI E GO: Proxies as Mediators for Unified Egocentric Video Representation Learning
arXiv:2606.20559v1 [cs.CV] 18 Jun 2026
Wenhao Chi, Arkaprava Sinha, Dominick Reilly, Hieu Le, Srijan Das University of North Carolina at Charlotte
Abstract Egocentric video understanding is inherently limited by the narrow perspective of wearable cameras: a single viewpoint, a single modality, a single model cannot capture the full richness of human action. We argue that a truly expressive egocentric representation must subsume complementary knowledge across viewpoints, modalities, and foundation model representations, yet remain deployable from egocentric video alone. To this end, we introduce a hierarchical multi-teacher distillation framework that produces U NI E GO, a unified egocentric encoder trained with nine teachers spanning ego-exo viewpoints, RGB, depth, and skeleton modalities, and four foundation models. Rather than distilling directly from heterogeneous teachers whose incompatible architectures and feature geometries induce conflicting gradients, our framework interposes a layer of representation-specific Proxy models that translate diverse teacher knowledge into a homogeneous egocentric space. A second distillation stage, Selective Proxy Distillation (SPD), then adaptively selects, for each training sample, the subset of proxies that are both correct and confident, distilling exclusively from reliable supervision and suppressing erroneous signals. SPD is further stabilized by initializing U NI E GO as a learned convex combination of proxy parameters, placing the unified model in a well-conditioned region of the loss landscape before distillation begins. U NI E GO achieves stateof-the-art performance across three egocentric video understanding tasks - action recognition, video retrieval, and action segmentation on three challenging egoexo benchmarks, outperforming naive multi-teacher distillation baselines and demonstrating that structured, proxy-mediated knowledge transfer yields richer and more discriminative egocentric representations. We release code and models at https://github.com/Wenhao-Chi/UNIEGO.
1
Introduction
Understanding human actions from egocentric video is a fundamental challenge in visual perception, with broad applications spanning augmented reality, assistive robotics, and procedural activity analysis [45, 7, 42]. Yet, learning a truly expressive egocentric representation from a single model remains elusive. Wearable cameras impose a narrow field of view and suffer from persistent selfocclusions, obscuring the actor’s body and surrounding scene context [13]. Complementary modalities such as depth and skeleton which encode the geometric structure of human motion are discarded entirely [49, 30, 4]. Moreover, the rich, diverse representational knowledge encapsulated within large-scale foundation models (FMs) remains untapped [52, 5, 50, 17]. Consequently, a standalone egocentric model is fundamentally limited. We argue that overcoming these limitations requires a unified egocentric representation, a single, comprehensive embedding that subsumes complementary knowledge across modalities (RGB, depth, skeleton), viewpoints (egocentric and exocentric), and diverse representations from heterogeneous foundation models, yet operates solely on egocentric video at inference. Such a representation would unlock richer action understanding across a broad spectrum of downstream egocentric tasks. Preprint.
These diverse perceptual signals spanning viewpoints, modalities, and FM representations naturally suggest a multi-teacher knowledge distillation framework, a paradigm explored extensively in concurrent work [34, 40, 46, 1]. However, existing methods distill homogeneous representations into a single student, sidestepping the deeper structural challenges that arise in the egocentric setting. First, teachers here are fundamentally heterogeneous: skeleton-based models operate over graph-structured neural architectures incompatible with video encoders [9, 8], while exocentric RGB teachers encode scene geometry from an entirely different viewpoint, introducing a substantial “representational gap" [27, 21]. Second, naïvely forcing an egocentric student to reconcile these incompatible feature spaces simultaneously leads to “conflicting gradient signals" and degrades optimization [25, 51, 41]. Thus, the ego student is asked not merely to learn, but to simultaneously bridge modality gaps, close viewpoint gaps, and absorb diverse representational priors, an ill-posed objective for a single distillation stage. This motivates a more principled approach to unified egocentric representation learning. We address these challenges by introducing a hierarchical multi-teacher distillation framework that consolidates diverse perceptual signals across viewpoints, modalities, and FM representations into U NI E GO, a single unified egocentric encoder (see Figure 1). Rather than distilling directly from heterogeneous teachers, our framework first translates each teacher’s knowledge into a studentcompatible space through a set of representation-specific Proxy models. Each proxy shares the architecture of U NI E GO and operates on egocentric video. Therefore, the proxy learning diminishes the representational gap induced by heterogeneous teacher architectures and viewpoints by converting incompatible feature geometries into a homogeneous egocentric embedding space. Further, the first distillation level naturally exposes the reliability of each supervision signal on a perinstance basis: a proxy that cannot correctly classify a given sample carries no trustworthy knowledge to transfer. Our proposed hierarchical framework exploits this signal through Selective Proxy Distillation (SPD), which selects, for each training sample, the subset of proxies whose predictions are both correct and confident, distilling exclusively from this reliable subset and suppressing erroneous supervision entirely. To further stabilize optimization, SPD is initialized via a learned convex combination of proxy parameters, placing U NI E GO in a flat, well-conditioned region of the loss landscape prior to distillation. Together, proxy merging and proxy selection collectively mitigate the conflicting gradient problem inherent to naive multi-teacher distillation. The outcome of our hierarchical distillation framework is U NI E GO, a unified egocentric encoder trained with 9 teachers spanning ego-exo viewpoints, RGB, depth, and skeleton modalities, and four FMs. U NI E GO outperforms naive multi-teacher distillation baselines across three egocentric video understanding tasks - action recognition, video retrieval, and action segmentation on three challenging ego-exo benchmarks [38, 22, 14]. Moreover, U NI E GO generalizes across video backbone architectures, including compact models with as few as 22M parameters. We summarize our contributions as follows: • We introduce U NI E GO, a unified egocentric encoder trained via a novel hierarchical distillation framework with nine teachers across ego-exo viewpoints, three modalities, and four foundation models, using representation-specific proxies as structured mediators. • We propose three tightly coupled components: (i) Proxy Learning, which converts heterogeneous, multi-modal teacher supervision into a pool of architecturally homogeneous egocentric proxies, bridging the modality and viewpoint gap; (ii) Selective Proxy Distillation (SPD), a sample-wise selective distillation mechanism that dynamically routes supervision from the most reliable proxies, mitigating conflicting gradients across heterogeneous teachers; and (iii) Proxy Merging, a learned convex initialization of U NI E GO that places the unified model in a well-conditioned region of the loss landscape, stabilizing SPD’s optimization. • U NI E GO achieves state-of-the-art performance across three egocentric video understanding tasks on three challenging benchmarks, demonstrating that hierarchical proxy-mediated distillation yields richer, more discriminative egocentric representations than direct multiteacher supervision.
2
Exo ViT Ego Depth
Exo Skeleton
Student parameter Gradient direction
Ego DINOv2
…
Ego DINOv2
…
Student
Exo Skeleton
Ego Depth
Ego DINOv2 Exo ViT
(a)
Unified Proxy Space
(b)
Pool of Proxies
Exo Skeleton
UNIEGO Student
Exo ViT
(c)
Ego Depth
(d)
Figure 1: (a) Naive multi-teacher distillation with heterogeneous teachers for learning unified egocentric representations results in representational gaps and conflicting gradients, as illustrated in (b). (c) In contrast, our proposed U NI E GO adopts a hierarchical distillation framework that mitigates these limitations through proxy-mediated learning, as shown in (d). Black dashed arrows illustrate the effects of this framework, shifting teachers into a unified representation space.
2
Related Work
2.1
Egocentric Representation Learning
Over the past few years, egocentric representation learning has become a central problem in video understanding. Early works, such as EgoVLP [24] and LaViLa [54], focused on learning egocentric representations from egocentric videos alone [7, 13, 42, 31]. This is challenging: egocentric cameras move with the person, hands and objects frequently occlude one another, and the same action can look very different depending on the wearer and environment. More recent works augment egocentric representation learning with signals beyond the raw ego video, typically leveraging synchronized exocentric viewpoints [21, 48, 32, 10, 27] or additional modalities [12, 47, 44, 33]. For example, ViewpointRosetta [27] uses diffusion models to learn a mapping between egocentric and exocentric representation spaces, EgoDTM [47] learns 3D-aware egocentric representations through distillation from a depth-modality trained teacher. These methods typically exploit a single specific auxiliary signal (viewpoint/modality). In contrast our work aims to consolidate many heterogeneous teachers, spanning auxiliary viewpoints and modalities, into a single egocentric encoder. 2.2
Multi-teacher knowledge distillation
Knowledge distillation was originally introduced as a way to compress the knowledge of a large model, or an ensemble of models, into a single deployable student [15]. This idea has since been extended to multi-teacher distillation, where a student learns from several teachers rather than a single teacher. For example, approaches like AMTML-KD [26] and CA-MKD [53] adaptively distill from an ensemble of teachers operating on a single input modality, learning instance-level importance weights for each teacher. More recent work has considered a more heterogeneous form of multi-teacher distillation, where teachers are differentiated by their architecture or input modalities. For example, AM-RADIO [34] and Theia [40] both distill multiple vision foundation models with different architectures and pretraining objectives into a single universal model. The key differences between these approaches and U NI E GO are that: (1) we learn egocentric representation learning that has not been explored by existing works, (2) we leverage a more diverse set of teachers spanning various architectures, modalities, and viewpoints; and (3) we selectively choose which teachers to distill from rather than treating all teacher as equally useful.
3
Method: Hierarchical Distillation Framework
In this section, we present U NI E GO, a hierarchical multi-teacher distillation framework for egocentric N e video understanding. Let D = {(xei , {xri }R r=1 , yi )}i=1 denote the training set, where xi is the r egocentric video clip, xi denotes the input to the r-th teacher model Tr , and yi ∈ Y is the groundtruth action label. In practice, xri may coincide with xei , correspond to its exocentric counterpart, 3
Level-II: Proxy Merging and Selective Proxy Distillation
Level-I: Proxy Learning Exo ViT
𝒯!
𝒫!
Ego Depth
𝒯"
𝒫"
𝒯#
𝒫#
Exo Skeleton
𝒫!
𝛼!∗
𝒫$
𝒯$
…
𝒫$
Top-K Proxy Selection 𝛼#∗
𝒫!
𝛼$∗ Weighted Merging
…
…
…
… Ego DINOv2
𝒫#
𝒫"
𝜃%
𝒫$ Distillation
UNIEGO
Action Classification
𝛼%∗ Ego RGB Input
Figure 2: Overview of U NI E GO. U NI E GO learns a unified egocentric encoder through a twolevel proxy-mediated distillation framework. In Level-I (left), heterogeneous teachers spanning viewpoints, modalities, and foundation representations independently supervise egocentric proxy models, converting diverse teacher signals into a homogeneous proxy space. In Level-II (right), the proxy parameters are first merged to initialize the unified model, after which Selective Proxy Distillation (SPD) performs sample-wise reliability filtering and distills only from proxies that are correct and confident. At inference, only the final U NI E GO model operates using egocentric input. or constitute an entirely different modality (e.g., skeleton or depth), each paired with its associated teacher Tr . Each teacher thus encodes a distinct representation, modality, or viewpoint. The objective is to learn a unified egocentric encoder f (·) such that the resulting representation f (xe ) subsumes the complementary knowledge of all teachers {Tr (xr )}R r=1 during training, while requiring only the egocentric stream xe at inference. Overview. U NI E GO employs a hierarchical knowledge distillation strategy comprising two levels, as illustrated in Figure 2. In the first level, we train a set of representation-specific Proxy models by independently distilling knowledge from each teacher Tr . Each proxy specializes in transferring representation- or modality- or viewpoint-specific information from its corresponding teacher, while bridging the domain gap between heterogeneous teacher representations and the egocentric input space, converting diverse supervisory signals into a homogeneous egocentric proxy space. In the second level, we perform Selective Proxy Distillation (SPD), which aggregates knowledge from all proxy models into the final U NI E GO model, initialized via a principled proxy merging strategy. The resulting Unified Egocentric Model effectively aggregates complementary knowledge from multiple teachers, with proxies acting as intermediaries for structured knowledge transfer. In the following subsections, we detail each component of U NI E GO’s learning paradigm. 3.1
Level-I: Proxy Learning
In the first distillation level, we train R proxy models {Pr }R r=1 by independently distilling featurelevel knowledge from each teacher Tr into an egocentric student. Thus, for an egocentric video xei and its time-synchronized auxiliary inputs {xri }R r=1 comprising RGB, depth, and skeleton data from both egocentric and exocentric viewpoints, we extract representations via R teacher networks {Tr }R r=1 , each providing viewpoint- and modality-specific supervision signals. All proxies share the same architecture but are trained with independent parameters. Since some teachers are foundation models that yield only feature embeddings rather than action logits, we adopt feature-level distillation throughout the entirety of this stage. Specifically, let hi , zi = f (xei ) denote the feature embedding and action logits of the proxy student, and hri = Tr (xri ) the teacher’s feature embedding. Each proxy Pr is optimized via: N
LrI =
1 X (λI Dcos (hi , hri ) + λcls CE(zi , yi )) , N i=1
(1)
where Dcos denotes cosine distance and CE is the cross-entropy loss. Despite being induced by heterogeneous teachers, all proxies share the same egocentric architecture. This converts heterogeneous teacher supervision into a homogeneous egocentric proxy space {Pr }R r=1 , partitioned as P = Pego ∪ Pexo according to viewpoint. Each proxy Pr subsequently serves as a 4
mediator in the second distillation level, reducing the modality and viewpoint gap between the original teachers and the final unified model. 3.2
Level-II: Selective Proxy Distillation (SPD)
Given the learned proxy set {Pr }R r=1 , the goal of this level is to consolidate their complementary knowledge into a single unified egocentric model f (θU ; ·). We first introduce a principled initialization of θU via proxy merging, followed by SPD, which selectively transfers knowledge from the most reliable proxies for each training sample. Proxy Merging Initialization. We initialize θU as an optimally weighted combination of the proxy parameters. Let θr denote the parameters of proxy Pr . We solve for merging coefficients α∗ ∈ ∆R that minimize the action classification loss over the training set: ! ! N R R X X X 1 ∗ ∗ θU ← θmerge , θmerge = αr∗ θr , α∗ = arg min CE f αr θr ; xei , yi , (2) α∈∆R N i=1 r=1 r=1 PR e where ∆R = {α ∈ RR ≥0 | r=1 αr = 1} is the probability simplex and f (θ; xi ) denotes the e model with parameters θ evaluated on xi . This initialization places θU in a favourable region of the optimization landscape, providing a stable starting point for subsequent distillation. Proposition 1 (Proxy Merged Initialization as a Loss Upper Bound for U NI E GO) Let L(θ) = PN 1 e i=1 CE(f (θ; xi ), yi ) denote the classification loss. Assume L is convex in θ in a neighbourhood N R containing {θr }r=1 . Then for any α ∈ ∆R : ! R R X X L αr θr ≤ αr L(θr ), (3) r=1
r=1
∗ and consequently the optimally merged initialization satisfies: L(θmerge ) ≤ minr L(θr ).
The first inequality follows directly from Jensen’s inequality P[3] applied to the convex loss L. The second follows by noting that α∗ is chosen to minimize L( r αr θr ), and the degenerate solution αr = 1 for any single r is feasible in ∆R , so the optimum is no worse than the best individual proxy. ∗ This establishes that θmerge achieves lower classification loss than any individual proxy under local convexity, placing U NI E GO in a flatter, better-generalizing region of the loss landscape [16, 11]. This favorable initialization reduces the optimization burden of SPD, as distillation begins from a point that already encodes the consensus of all R proxy representations rather than the bias of any single one. Proxy Selection. Rather than distilling from all proxies uniformly, SPD selects a reliable subset Si ⊆ {1, . . . , R} for each sample xei . We adopt a correctness-filtered small-loss criterion: a proxy Pr is considered a reliable candidate only if it correctly predicts the action class, i.e., ŷir = yi , where ŷir = arg maxc (zir )c . The candidate set is thus: Ci = {r ∈ {1, . . . , R} | ŷir = yi }.
(4)
Among candidates, proxy reliability is quantified by the cross-entropy loss sri = CE(zir , yi ), where a lower loss indicates higher predictive confidence. When |Ci | > 0, the top-k proxies with the lowest sri are selected to form Si . When Ci = ∅, we set Si = ∅ and skip distillation entirely for sample i, preventing the unified model from absorbing erroneous supervision. Selective Proxy Distillation. For samples where Si ̸= ∅, SPD transfers knowledge from the selected proxies to U NI E GO. Since all proxies are trained for action classification, they provide both feature embeddings and action logits. We therefore combine feature-level and logit-level distillation. Let Pr Pr U e e hU i , zi = f (θU ; xi ) and hi , zi = f (θr ; xi ) denote the feature embeddings and action logits of U NI E GO and the r-th proxy Pr , respectively. The per-sample SPD loss is: βlogit X βfeat X Pj Pj U U D (h , h ) + D σ(z )∥σ(z ) , (5) LU (i) = cos i KL distill i i i |Si | |Si | j∈Si
j∈Si
5
where Dcos and DKL denote cosine distance and KL divergence, σ(·) is the softmax function, and βfeat , βlogit are loss weights. The total Level-II objective is: LII =
N 1 X U I(|Si | > 0) LU distill (i) + βcls CE(zi , yi ) , N i=1
(6)
where βcls is the classification loss weight and I(·) is the indicator function. During inference, only U NI E GO with egocentric input xe is required: ŷ = arg max zcU . c
(7)
No teacher network, exocentric stream, or proxy is needed at test time.
4
Experiments
We evaluate U NI E GO on three egocentric datasets across three video understanding tasks: action recognition, video retrieval, and action segmentation. In all experiments, U NI E GO requires only egocentric RGB video at inference. 4.1
Experimental Setting
Datasets. We evaluate on three publicly available ego-exo datasets: EgoExo-Fitness [22], Table 1: Teacher pool for Level-I proxy learning. Assembly101 [38], and EgoExo4D [14]. For Modality Viewpoint Teacher Model Feature Dim. EgoExo-Fitness and EgoExo4D, we follow the official evaluation splits [22, 14]. For AssemEgo DINOv2 [29] 1024 RGB Ego SigLIP [52] 1152 bly101, we pair egocentric videos from the Depth Ego DepthAnything [50] 1024 helmet-mounted ego04 camera with exocentric Exo TimeSformer [2] 768 videos from the frontal exo03 camera, yielding Exo DINOv2 [29] 1024 RGB 46,202 training and 15,307 test samples across Exo Sk-Ego [37] 512 Exo SigLIP [52] 1152 24 action classes. Further dataset details are Skeleton Exo ST-GCN [49] 256 provided in the Appendix. Depth Exo DepthAnything [50] 1024 Implementation Details. We use TimeSformer [2] as the egocentric student backbone for U NI E GO f (·) unless stated, which takes 8 frames sampled at 224 × 224 and produces a 768-dimensional video representation. During training, U NI E GO learns from multiple teacher representations Tr which are detailed in Table 1. We apply a linear projection layer to align all teacher representations into the 768-dimensional student space to handle the dimensional mismatch between the student (hi ) and various teacher features (hri ). All proxies are trained, following TimeSformer [2] default configuration, for 15 epochs using SGD with a base learning rate of 0.005, momentum of 0.9, and weight decay of 1 × 10−4 . We scale the learning rate by factors of 0.1 and 0.01 at epochs 11 and 14, respectively. All models are trained with a total batch size of 8 distributed across 4 NVIDIA RTX A5000 GPUs. In Level-I, we train 9 proxies across 6 teacher architectures and ego-exo viewpoints (Table 1), with loss weights λI = 5 and λcls = 1. In Level-II, the proxy merging coefficients α∗ are optimized on the training set using Adam [18] for 2 epochs with a learning rate of 0.02 and weight decay of 0.01. For proxy selection, we set K = 1 for EgoExo-Fitness and Assembly101, and K = 2 for EgoExo4D (ablated in Table 8). The Level-II loss weights are βfeat = 5, βcls = 1 throughout, and βlogit = 1 for EgoExo-Fitness and EgoExo4D and 5 for Assembly101, with KL divergence temperature τ = 1. All hyperparameters are selected on the respective validation sets. 4.2
Comparison with State-of-the-Art
Table 2 compares U NI E GO with recent methods across three egocentric action recognition benchmarks under two inference protocols: exocentric inference, where exocentric streams are available at test time, and egocentric inference, where only egocentric video is provided. The former serves as a privileged-view upper bound [22, 38], as exocentric cameras in these datasets capture global body 6
Table 2: Comparison with state-of-the-art methods on three egocentric action recognition datasets (EgoExo-Fitness, Assembly101, EgoExo4D). Dist. indicates whether the model performs distillation and Acc. indicates Top-1 accuracy. (a) EgoExo-Fitness. Method Dist. Exocentric inference TimeSformer [2] ✗ ST-GCN [49] ✗ Egocentric inference I3D [6] ✗ EgoVLP [24] ✗ ViFi-CLIP [35] ✗ π -ViT [36] ✓ TimeSformer [2] ✗ Multiteacher Dist. [2] ✓ U NI E GO (Ours) ✓
(b) Assembly101. Acc. 88.9 87.5 74.7 74.7 81.8 80.1 80.3 81.5 84.7
Method Dist. Exocentric inference TimeSformer [2] ✗ ST-GCN [49] ✗ Egocentric inference TSM+TA [23, 39] ✗ ViFi-CLIP [35] ✗ π -ViT [36] ✓ TimeSformer [2] ✗ Multiteacher Dist. [2] ✓ U NI E GO (Ours) ✓
Method Timesformer Multiteacher Dist. U NI E GO (Ours)
Assembly101 mAP R@1 0.226 0.410 0.228 0.413 0.253 0.424
Acc. 62.7 46.2 40.5 46.6 47.8 47.6 48.2 50.7
Method
Dist. Exocentric inference TimeSformer [2] ✗ ST-GCN [49] ✗ Egocentric inference VI Encoder [28] ✓ EgoVLPv2 [31] ✓ Ego-Exo MAE [21] ✓ Viewpoint Distillation [15] ✓ TimeSformer [2] ✗ Multiteacher Dist. [2] ✓ U NI E GO (Ours) ✓
Acc. 26.0 42.9 40.3 39.1 37.2 38.2 39.9 40.6 41.1
Table 4: Performance on Egocentric Temporal Action Segmentation on Assembly101.
Table 3: Performance on Video Retrieval EgoExo-Fitness mAP R@1 0.474 0.712 0.486 0.720 0.543 0.748
(c) EgoExo4D.
EgoExo4D mAP R@1 0.167 0.326 0.178 0.331 0.182 0.340
Feature Backbone (Method) TimeSformer (Ego only) TimeSformer (Multiteacher Dist.) U NI E GO (Ours)
F1@10
F1@25
F1@50
Edit
Acc
16.2 15.3 19.6
14.1 13.2 16.9
10.4 9.8 12.3
18.7 18.4 19.4
34.4 34.2 34.7
pose and scene layout that are largely occluded in the egocentric field of view. All baselines except ST-GCN [49] utilize RGB input at inference. U NI E GO consistently outperforms the TimeSformer backbone by +4.4%, +3.1%, and +1.2% on EgoExo-Fitness, Assembly101, and EgoExo4D respectively, demonstrating the benefit of consolidating diverse multi-teacher supervision. Against the strongest distillation baseline π-ViT [36], U NI E GO achieves gains of +4.6% and +2.9% on EgoExo-Fitness and Assembly101. Furthermore, U NI E GO consistently surpasses naive multi-teacher distillation [34, 40], confirming that proxymediated hierarchical distillation is essential for reconciling heterogeneous teacher representations into a unified egocentric encoder. The largest gains are observed on EgoExo-Fitness, where exocentric proxies Pexo are particularly strong as many actions in this dataset involve full-body motion that is inherently occluded from the egocentric viewpoint, making exocentric supervision especially informative. Conversely, on EgoExo4D, exocentric proxies are weaker, as evidenced by the baseline TimeSformer achieving only 26.0% under exocentric inference, below its egocentric counterpart. Nevertheless, U NI E GO remains robust to this proxy inconsistency across viewpoints, as the adaptive selection mechanism in SPD suppresses unreliable proxies and routes supervision from the most discriminative sources available. Across all three benchmarks, U NI E GO achieves state-of-the-art action recognition performance under egocentric inference. Generalization Across Backbone Architectures. We verify Table 5: Backbone robustness. that U NI E GO is architecture-agnostic by replacing the proxy U NI E GO improves across diverse and unified model backbone with UniFormer-S [20] and ViFi- egocentric backbones. CLIP [35] (Table 5). Across both alternatives, our hierarchical Backbone Method EEF distillation framework consistently outperforms naive multibaseline 80.3 teacher distillation, confirming that the gains of U NI E GO are TimeSformer [2] Multiteacher Dist. 81.5 U NI E GO (Ours) 84.7 not tied to a specific video encoder. Notably, UniFormer-S, a baseline 68.4 compact 22M model yields significant classification improve- Uniformer-S [20] Multiteacher Dist. 69.0 U NI E GO (Ours) 73.5 ments, demonstrating that our framework is equally effective baseline 81.8 for learning efficient unified egocentric representations which ViFi-CLIP [35] Multiteacher Dist. 81.7 U NI E GO (Ours) 83.8 is crucial for deployment in resource-constrained egocentric applications. Video Retrieval & Temporal Action Segmentation. Table 3 evaluates U NI E GO on video retrieval, performed by extracting features from action recognition trained backbones and computing their pairwise similarity across the test set. On EgoExo-Fitness, naive multi-teacher distillation yields only a marginal improvement of +0.012 mAP over the baseline, whereas SPD achieves a substantially 7
Table 6: Ablation of components Table 7: Alternative proxy merg- Table 8: Strategies for Selective Proxy distillation. of U NI E GO. ing strategies. Proxy Learning ✗ ✓ ✓ ✓ ✓
Proxy Merging ✗ ✗ ✗ ✓ ✓
SPD
EEF
A101
✗ ✗ ✓ ✗ ✓
80.3 82.1 82.3 81.4 84.7
47.6 48.7 48.9 48.3 50.7
Merging Strategy Best Proxy Average Layer-level Parameter-level Proxy Merging (Ours)
EEF 83.7 83.4 84.2 83.6 84.7
A101 50.6 50.2 50.6 50.4 50.7
Distillation Strategy No Distillation All Proxies Top-1(Ego) + Top-1(Exo) Top-3 Top-2 Top-1
EEF 80.3 82.8 83.2 83.2 84.3 84.7
A101 47.6 49.9 49.5 50.0 50.1 50.7
larger gain of +0.057 mAP. This trend remains consistent across datasets, confirming that proxymediated distillation yields highly discriminative egocentric representations. For temporal action segmentation on Assembly101, we extract features from three egocentric TimeSformer backbones: i) trained from scratch, ii) naive multi-teacher distillation, and iii) U NI E GO, and feed them into a fixed temporal encoder [43]. We evaluate the performance via F1 score, Edit distance, and frame accuracy. As shown in Table 4, U NI E GO features yield the strongest performance across all metrics. Notably, naive multi-teacher distillation degrades performance on all metrics relative to the scratch baseline, confirming that directly distilling heterogeneous teachers disrupts fine-grained temporal representations. In contrast, U NI E GO’s hierarchical distillation preserves the local temporal structure essential for frame-wise discrimination and boundary-sensitive segmentation.
4.3
Ablation Studies and Model Diagnosis
We perform all the ablations and diagnosis of U NI E GO on the EgoExo-Fitness (EEF) and Assembly101 (A101) datasets. Effect of Each Component. Table 6 ablates the three key components of U NI E GO: Proxy Learning, Proxy Merging Initialization, and SPD. Starting from the egocentric-only baseline, adding Level-I proxy learning followed by simultaneous distillation from all proxies yields gains of +1.8% and +1.1% on EEF and A101, respectively, highlighting the role of proxies as mediators. Then, replacing the simultaneous distillation in level-II with SPD further improves performance by +0.2% on both datasets, demonstrating that instance-adaptive proxy selection yields more reliable supervision than distilling from all proxies indiscriminately. We also observe that proxy merging initialization provides a strong starting point for SPD, as the merged model already achieves higher accuracy than the baseline. Finally, prepending proxy merging initialization before SPD achieves 84.7% and 50.7% on EEF and A101 resulting in overall improvements of +4.4% and +3.1% over the baseline. These results confirm the contribution of each component in our hierarchical distillation framework. Alternative proxy merging strategies Table 7 compares different strategies for initializing U NI E GO before SPD. Best Proxy uses the strongest individual proxy as the initialization, while Average uniformly averages the weights of all trained proxies. We also compare against finegrained learnable merging strategies: Layer-level learns separate merging weights for each layer, and Parameter-level learns separate merging weights for individual parameters. Although fine-grained merging offers greater flexibility, it does not improve performance in practice. Both layer-level and parameter-level merging outperform uniform averaging yet fall short of our global merging strategy, suggesting that a globally consistent combination of proxy parameters is more effective than local merging at the level of individual layers or parameter groups, likely due to the importance of maintaining parameter consistency across the full model. Alternative SPD strategies. In Table 8, we investigate various selective distillation strategies following the proxy merging initialization. SPD outperforms distillation from all proxies simultaneously, confirming that selective supervision mitigates conflicting gradients across homogeneous proxies. Enforcing viewpoint diversity via a Top-1(Ego) + Top-1(Exo) selection strategy degrades performance, particularly on Assembly101, indicating that global proxy selection is preferable, i.e., not all training samples contain discriminative information from every viewpoint, and forcing viewpoint-balanced selection introduces noisy supervision into U NI E GO. Among K ∈ {1, 2, 3}, all top-K variants outperform naive proxy distillation, demonstrating the robustness of the selection mechanism, with K = 1 yielding the best overall performance. 8
Ego DINO
Ego SigLip
Ego Depth
Exo SigLip
Exo SkEgo
Exo Depth
Exo DINO
0.0
Eg o D Eg INO o Eg SigL i o De p Ex pth o RG Ex B o Ex SKL o S Ex igL i o S p Ex kEg o De o Ex pt o DI h NO
60
Conflict Rate (%)
0.4
0.100
70
Exo SKL
Selective Proxy Distillation (Top 3) Selective Proxy Distillation (Top 1)
Direct Multimodal Distillation All Proxy Distillation
Avg Cosine Similarity
Exo RGB
Selective Proxy Distillation (Top 3) Selective Proxy Distillation (Top 1)
Direct Multimodal Distillation All Proxy Distillation
0.8
50 40 30 20
0.075 0.050 0.025 0.000 - 0.025 - 0.050 - 0. 075
10
1
2
3
4
5
6
7
8 9 10 11 12 13 14 15
Epoch
- 0.100
1
2
3
4
5
6
7
8 9 10 11 12 13 14 15
Epoch
Figure 3: Teacher vs Proxy Centered Kernel Alignment Figure 4: Gradient Conflict Rate Figure 5: Cosine Similarity among teacher and proxy gradients across distillation strategies scores
5
Does U NI E GO Mitigate Representational Gap and Conflicting Gradients?
The answer is “Yes". We provide a model analysis to confirm that our hierarchical distillation framework addresses the two core failure modes of naive multi-teacher distillation: representational gap and conflicting gradients. Representational Gap. To quantify the representational gap among the R teachers, we visualize their pairwise linear Centered Kernel Alignment (CKA) [19] in Figure 3. The lower triangle displays teacher-pair similarities and the upper triangle displays proxy-pair similarities on EgoExo-Fitness. The substantially higher pairwise CKA among proxies than among teachers confirms that LevelI proxy learning projects heterogeneous teacher representations into a homogeneous egocentric embedding space, directly alleviating the representational gap that impedes naive multi-teacher distillation. Conflicting Gradients. Figures 4 and 5 report the gradient conflict rate which is defined as the fraction of distillation gradients that oppose the classification gradient, i.e., cos(∇cls , ∇kd ) < 0, and the average cosine similarity across training epochs, respectively. Direct multi-teacher distillation exhibits severe gradient interference, confirming that heterogeneous teacher supervision actively opposes the primary task objective. Distilling from all proxies simultaneously reduces the conflict rate to 42–50% and shifts the cosine similarity toward zero, indicating partial mitigation. SPD further suppresses conflicts by routing supervision exclusively through reliable proxies, yielding consistently cooperative distillation and classification gradients. Together, these results confirm that proxy learning and SPD collectively transform a conflicting multi-teacher optimization landscape into a coherent one.
6
Conclusion
We presented U NI E GO, a unified egocentric encoder that consolidates diverse perceptual knowledge across viewpoints, modalities, and foundation model representations into a single model. The core of U NI E GO lies a hierarchical distillation framework in which representation-specific proxies serve as structured mediators, dissolving the representational gap between heterogeneous teachers before selective, reliability-guided distillation assembles their collective knowledge into a coherent unified representation. The result is an egocentric encoder that sees further, knows more, and generalizes better without requiring more than a single egocentric camera at inference. While we hope U NI E GO inspires broader exploration into systematically harvesting and reconciling diverse supervisory signals for richer egocentric perception, the current framework has an important limitation. Specifically, U NI E GO relies on a small-loss criterion for proxy selection, a heuristic that, while effective, does not exploit the full potential of the proxy pool. A learned selection mechanism, one that dynamically weighs proxy reliability as a function of both the input and the training state could yield richer and more adaptive supervision. Designing such an adaptive proxy selection strategy is non-trivial, and we leave this as a promising direction for future work. 9
Acknowledgements This work was supported in part by the National Science Foundation (IIS-2245652) and the University of North Carolina at Charlotte. Computational resources were provided by the NSF National AI Research Resource Pilot (NAIRR240338) and NCShare.
References [1] Varun Belagali, Saarthak Kapse, Pierre Marza, Srijan Das, Zilinghan Li, Sofiène Boutaj, Pushpak Pati, Srikar Yellapragada, Tarak Nath Nandi, Ravi K Madduri, Joel Saltz, Prateek Prasanna, Stergios Christodoulidis, Maria Vakalopoulou, and Dimitris Samaras. Ticon: A slide-level tile contextualizer for histopathology representation learning, 2025. [2] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML), July 2021. [3] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004. [4] Zhe Cao, Gines Hidalgo Martinez, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019. [5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021. [6] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Conference on Computer Vision and Pattern Recognition, pages 4724–4733. IEEE, 2017. [7] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018. [8] Srijan Das, Rui Dai, Di Yang, and Francois Bremond. Vpn++: Rethinking video-pose embeddings for understanding activities of daily living. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021. [9] Srijan Das, Saurav Sharma, Rui Dai, Francois Bremond, and Monique Thonnat. Vpn: Learning video-pose embedding for activities of daily living. In European Conference on Computer Vision, pages 72–90. Springer, 2020. [10] Zi-Yi Dou, Xitong Yang, Tushar Nagarajan, Huiyu Wang, Jing Huang, Nanyun Peng, Kris Kitani, and Fu-Jen Chu. Unlocking exocentric video-language data for egocentric video representation learning, 2024. [11] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carlin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning (ICML), 2020. [12] Xinyu Gong, Sreyas Mohan, Naina Dhingra, Jean-Charles Bazin, Yilei Li, Zhangyang Wang, and Rakesh Ranjan. Mmg-ego4d: Multi-modal generalization in egocentric action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. [13] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, 10
Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022. [14] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zachary Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, María Escobar, Cristhian Forigua, Abrham Kahsay Gebreselasie, Sanjay Haresh, Jing Huang, Md Mohaiminul Islam, Suyog Dutt Jain, Rawal Khirodkar, Devansh Kukreja, Kevin J. Liang, Jia-Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh K. Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Jinxu Zhang, Ángela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina González, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anushk Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Romy Mi Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatumininu Oguntola, Xiaqing Pan, Penny Peng, Shraman Pramanick, Merey Ramazanova, Fiona Ryan, W. Shan, Kiran Somasundaram, Chenan Song, Audrey Southerland, Masatoshi Tateno, Huiyu Wang, Yuchen Wang, Takuma Yagi, Mingfei Yan, Xitong Yang, Ze Yu, Shengxin Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, J. F. Zhuo, Pablo Arbeláez, Gedas Bertasius, David J. Crandall, Dima Damen, Jakob Julian Engel, Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C. V. Jawahar, Richard A. Newcombe, Hyun Soo Park, James M. Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng Shou, and Michael Wray. Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19383–19400, 2023. [15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. [16] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Uncertainty in Artificial Intelligence (UAI), 2018. [17] Qiuhong Ke, Mohammed Bennamoun, Senjian An, Ferdous Sohel, and Farid Boussaid. Learning clip representations for skeleton-based 3d action recognition. IEEE Transactions on Image Processing, 27(6):2842–2855, June 2018. [18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. [19] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019. [20] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Jiao Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:12581–12600, 2022. [21] Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual representations from third-person to first-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10995–11005, 2021. 11
[22] Yuan-Ming Li, Wei-Jin Huang, An-Lan Wang, Ling-An Zeng, Jing-Ke Meng, and Wei-Shi Zheng. Egoexo-fitness: towards egocentric and exocentric full-body action understanding. In European Conference on Computer Vision, pages 363–382. Springer, 2024. [23] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE International Conference on Computer Vision, 2019. [24] Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric videolanguage pretraining. arXiv preprint arXiv:2206.01670, 2022. [25] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 18873–18885. Curran Associates, Inc., 2021. [26] Yuang Liu, Wei Zhang, and Jun Wang. Adaptive multi-teacher multi-level knowledge distillation. Neurocomputing, 2020. [27] Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. Viewpoint rosetta stone: Unlocking unpaired ego-exo videos for view-invariant representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. [28] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. [29] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), 2024. [30] Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d human pose estimation in video with temporal convolutions and semi-supervised training. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. [31] Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pretraining with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5285–5297, 2023. [32] Camillo Quattrocchi, Antonino Furnari, Daniele Di Mauro, Mario Valerio Giuffrida, and Giovanni Maria Farinella. Synchronization is all you need: Exocentric-to-egocentric transfer for temporal action segmentation with unlabeled synchronized video pairs. In European Conference on Computer Vision (ECCV), 2024. [33] Gorjan Radevski, Dusan Grujicic, Marie-Francine Moens, Matthew Blaschko, and Tinne Tuytelaars. Multimodal distillation for egocentric action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. [34] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024. [35] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Finetuned clip models are efficient video learners. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. [36] Dominick Reilly and Srijan Das. Just add π! pose induced video transformers for understanding activities of daily living. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024. 12
[37] Dominick Reilly, Manish Kumar Govind, Le Xue, and Srijan Das. From my view to yours: Ego-augmented learning in large vision language models for understanding exocentric daily living activities, 2025. [38] Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21064–21074, 2022. [39] Fadime Sener, Dipika Singhania, and Angela Yao. Temporal aggregate representations for long-range video understanding. In European conference on computer vision, pages 154–171. Springer, 2020. [40] Jinghuan Shang, Karl Schmeckpeper, Brandon B May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, and Laura Herlant. Theia: Distilling diverse vision foundation models for robot learning. arXiv preprint arXiv:2407.20179, 2024. [41] Guangyuan Shi, Qimai Li, Wenlong Zhang, Jiaxin Chen, and Xiao-Ming Wu. Recon: Reducing conflicting gradients from the root for multi-task learning. In International Conference on Learning Representations (ICLR), 2023. [42] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626, 2018. [43] Arkaprava Sinha, Monish Soundar Raj, Pu Wang, Ahmed Helmy, and Srijan Das. Mstemba: Multi-scale temporal mamba for efficient temporal action detection. arXiv preprint arXiv:2501.06138, 2025. [44] Shuhan Tan, Tushar Nagarajan, and Kristen Grauman. Egodistill: Egocentric head motion distillation for efficient video understanding. arXiv preprint arXiv:2301.02217, 2023. [45] Anirudh Thatipelli, Shao-Yuan Lo, and Amit K. Roy-Chowdhury. Egocentric and exocentric methods: A short survey. Computer Vision and Image Understanding, 257:104371, 2025. [46] Thomas Wimmer, Prune Truong, Marie-Julie Rakotosaona, Michael Oechsle, Federico Tombari, Bernt Schiele, and Jan Eric Lenssen. Anyup: Universal feature upsampling. In Proceedings of the International Conference on Learning Representations (ICLR), 2026. [47] Boshen Xu, Yuting Mei, Xinbi Liu, Sipeng Zheng, and Qin Jin. Egodtm: Towards 3d-aware egocentric video-language pretraining. In Advances in Neural Information Processing Systems (NeurIPS), 2025. [48] Zihui Xue and Kristen Grauman. Learning fine-grained view-invariant representations from unpaired ego-exo videos via temporal alignment. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [49] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Thirty-second AAAI conference on artificial intelligence, 2018. [50] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. In Advances in Neural Information Processing Systems, 2024. [51] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5434–5445. Curran Associates, Inc., 2020. [52] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 2023. 13
[53] Hailin Zhang, Defang Chen, and Can Wang. Confidence-aware multi-teacher knowledge distillation. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022. [54] Yue Zhao, Ishan Misra, Philipp Krahenbuhl, and Rohit Girdhar. Learning video representations from large language models. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6586–6597, 2022.
14
A
Overview
The appendix is categorized into the following parts: • Section B: Detailed Data Description • Section C: Proxy Performances • Section D: Analysis of Proxy Selection • Section E: Analysis of Proxy Merging • Section F: Action Analysis
B
Detailed Dataset Description
EgoExo-Fitness [22] is a full-body action understanding dataset containing synchronized egocentric and exocentric videos of fitness activities. It consists of 12 action categories and approximately 32 hours of video. Following the official split, we use 3,522 training samples and 912 test samples, with action sequences ranging from 10 to 30 seconds in duration. Assembly101 [38] is a large-scale procedural activity dataset comprising 167 hours of video, recording subjects performing object assembly tasks from multiple viewpoints. In our experiments, we pair the egocentric videos from the helmet-mounted ego04 camera with the corresponding exocentric videos in front exo03 camera. The resulting split contains 46,202 training samples and 15,307 test samples over 24 action classes. EgoExo4D [14] is a large-scale egocentric-exocentric video dataset covering a diverse range of skilled human activities. For each sample, we pair the egocentric video with the exocentric view annotated as the best view by human annotators. The dataset split used in our experiments contains 30,660 training samples and 9,356 test samples across 665 action classes.
C
Proxy Performances
Table 9: Performance of the base ego model and different ego/exo proxy models across EgoExoFitness (EEF), Assembly101 (A101), and Ego-Exo4D (EE4D). Modality
Viewpoint
RGB [2]
Ego
Model
EEF
A101
EE4D
Base Ego Model TimeSformer
80.3
47.6
39.9
Depth [50] DINOv2 [29] SigLip [52]
Ego Proxies (Pego ) Ego DepthAnything Ego DINOv2 Ego SigLip
80.3 80.9 80.7
48.2 48.4 48.4
40.1 40.2 40.6
RGB [2] Skeleton [49] Depth [50] DINOv2 [29] Sk-Ego [37] SigLip [52]
Exo Exo Exo Exo Exo Exo
Exo Proxies (Pexo ) TimeSformer ST-GCN DepthAnything DINOv2 Sk-Ego SigLip
81.9 81.9 80.3 80.7 81.9 80.3
49.5 48.4 48.5 48.2 48.3 48.0
40.0 40.3 39.9 39.5 40.2 40.6
In Table 9, we report the action classification performance of all proxies (Pr ) after Level-I distillation. We observe that not all proxies outperform the base ego model, motivating the need for proxy selection. Nevertheless, the final performance of U NI E GO surpasses that of all individual proxies, corroborating the effectiveness of our proposed hierarchical distillation framework.
D
Analysis of Proxy Selection.
In Figure 6, we analyze the selection frequency of each proxy during the Level-II selective proxy distillation (SPD). First, every proxy is selected to some extent across all benchmarks, indicating that SPD does not collapse to a single teacher. Unsurprisingly, we find that the routing distributions are 15
Ego Depth Exo Skeleton
Ego DINOv2
Exo SkEgo
Ego SigLIP
Exo SigLIP
Exo Depth
Exo RGB
Exo DINOv2
A101
EEF
EE4D
Figure 6: Selective Proxy Distillation Statistics. Table 10: Effect of different proxy initialization strategies before and after SPD on EgoExoFitness (EEF) and Assembly101 (A101). EEF
Method Best Proxy Average Layer-level Parameter-level Model-level
A101
Before SPD
After SPD
Before SPD
After SPD
81.9 79.9 80.9 79.8 81.4
83.7 83.4 84.2 83.6 84.7
49.5 42.9 43.5 43.4 48.3
50.6 50.2 50.6 50.4 50.7
not consistent across benchmarks, and that proxy selection is strongly correlated with the distinct dataset characteristics. On EgoExo-Fitness, the SPD primarily selects Exo Skeleton and Exo RGB. This is consistent with the nature of the dataset, which consists largely of full-body fitness motions: exocentric and skeleton-based proxies provide supervision for body pose and motion patterns that are often only partially visible from a head-mounted camera. In Assembly101, the selection is more diverse, assigning a larger weight to egocentric proxies. This suggests that, compared to EgoExo-Fitness, more actions in Assembly101 rely on egocentric cues such as local hand-object interactions and subtle manipulation dynamics. On EgoExo4D, there is a notable increase in the selection of the Exo SkEgo proxy. We attribute this to the high-resolution, spatially localized nature of the benchmark: action-relevant regions can occupy only a small part of the full frame, and the SkEgo crop provides a more focused view of these regions.
E
Analysis of Proxy Merging
Table 10 compares action classification performance of U NI E GO under different proxy merging initialization strategies, before and after SPD, on EgoExo-Fitness and Assembly101. Across all initialization strategies, SPD consistently improves performance, confirming the importance of the second distillation level. Notably, the best-performing initialization prior to SPD, i.e., the single Best Proxy, does not yield the strongest post-SPD accuracy. Instead, the model soup, i.e., the learned convex combination of all proxy parameters, provides the most effective initialization for SPD. This suggests that the advantage of proxy merging lies not in its standalone classification performance but in its ability to place U NI E GO in a well-conditioned region of the loss landscape that is maximally amenable to subsequent distillation.
F
Action Analysis
As shown in Table 11 and Table 12, exo_rgb is the most frequently selected proxy on both datasets, with a sample-weighted selection rate of 0.349 on EEF and 0.182 on A101, followed by exo_skl. However, the class-level separation between the top-ranked and second-ranked proxies differs substantially across datasets. The average per-class gap between the top-1 and top-2 proxies is 0.186 on EEF, but only 0.069 on A101. This suggests that EEF classes often have a more clearly preferred 16
Table 11: Per-class top-1 proxy selection rates on EEF (12 classes) class_name
none
exo_rgb
exo_skl
exo_siglip
ego_siglip
exo_skego
ego_depth
exo_depth
exo_dino
ego_dino
Kneeling pushing-ups Push-ups Kneeling Torso Twist Knee Raise & Abd. Contract Shoulder Bridge Sit-ups Leg Reverse Lunge Leg Lunge With Knee Lift Sumo Squat Jumping Jacks High Knee Clap Jacks
0.053 0.000 0.046 0.055 0.000 0.057 0.038 0.118 0.112 0.042 0.054 0.011
0.139 0.500 0.431 0.290 0.987 0.172 0.106 0.300 0.388 0.242 0.264 0.345
0.172 0.400 0.218 0.428 0.013 0.261 0.477 0.291 0.233 0.347 0.271 0.414
0.053 0.000 0.030 0.014 0.000 0.070 0.068 0.073 0.017 0.126 0.085 0.029
0.053 0.025 0.005 0.028 0.000 0.013 0.068 0.036 0.017 0.016 0.078 0.023
0.139 0.042 0.056 0.048 0.000 0.108 0.114 0.109 0.198 0.153 0.078 0.063
0.013 0.008 0.117 0.055 0.000 0.076 0.023 0.036 0.009 0.016 0.008 0.029
0.179 0.017 0.041 0.034 0.000 0.146 0.045 0.018 0.009 0.005 0.054 0.057
0.106 0.000 0.036 0.021 0.000 0.045 0.053 0.009 0.000 0.021 0.054 0.011
0.093 0.008 0.020 0.028 0.000 0.051 0.008 0.009 0.017 0.032 0.054 0.017
Table 12: Per-class top-1 proxy selection rates on A101 (24 classes). class_name
none
exo_rgb
exo_skl
exo_siglip
ego_siglip
exo_skego
ego_depth
exo_depth
exo_dino
ego_dino
pick up put down inspect rotate unscrew position screw remove position screw on remove screw from pass tilt up attempt to position push tilt down pull attempt to remove attempt to pick up clap attempt to unscrew attempt to put down spin attempt to screw shake
0.049 0.051 0.186 0.307 0.042 0.107 0.048 0.187 0.134 0.201 0.165 0.291 0.413 0.336 0.224 0.370 0.595 0.754 0.007 0.515 0.988 0.452 0.951 0.783
0.256 0.204 0.136 0.123 0.196 0.139 0.250 0.125 0.141 0.115 0.142 0.143 0.084 0.063 0.179 0.082 0.067 0.040 0.883 0.095 0.012 0.170 0.019 0.075
0.125 0.122 0.108 0.085 0.124 0.081 0.078 0.138 0.086 0.117 0.089 0.041 0.097 0.098 0.116 0.123 0.031 0.022 0.047 0.075 0.000 0.064 0.010 0.000
0.089 0.107 0.080 0.056 0.100 0.075 0.099 0.064 0.104 0.092 0.059 0.058 0.036 0.076 0.078 0.042 0.029 0.025 0.010 0.037 0.000 0.053 0.000 0.025
0.079 0.106 0.066 0.049 0.100 0.086 0.087 0.067 0.084 0.068 0.067 0.082 0.025 0.096 0.068 0.062 0.012 0.020 0.005 0.050 0.000 0.027 0.000 0.000
0.104 0.103 0.092 0.090 0.123 0.143 0.088 0.098 0.088 0.071 0.123 0.057 0.088 0.102 0.108 0.071 0.084 0.047 0.015 0.025 0.000 0.059 0.010 0.017
0.079 0.091 0.085 0.088 0.124 0.125 0.128 0.090 0.111 0.083 0.086 0.102 0.089 0.061 0.046 0.076 0.061 0.025 0.005 0.079 0.000 0.037 0.000 0.008
0.042 0.045 0.052 0.037 0.044 0.048 0.042 0.044 0.039 0.043 0.040 0.042 0.035 0.054 0.041 0.027 0.006 0.017 0.007 0.008 0.000 0.027 0.000 0.000
0.076 0.076 0.091 0.083 0.064 0.088 0.087 0.085 0.107 0.141 0.130 0.087 0.048 0.078 0.079 0.120 0.027 0.017 0.012 0.017 0.000 0.032 0.010 0.042
0.102 0.094 0.105 0.082 0.085 0.108 0.091 0.102 0.106 0.071 0.098 0.097 0.085 0.036 0.059 0.029 0.088 0.032 0.007 0.100 0.000 0.080 0.000 0.050
proxy, whereas A101 exhibits a flatter proxy distribution, where several proxies can have similar selection rates for the same class. Interestingly, exo_depth shows a highly class-specific behavior. Although it has the lowest aggregate selection rate among all proxies (0.042 on A101 and 0.052 on EEF), it is the most frequently selected proxy for Kneeling push-ups on EEF, with a selection rate of 0.179, exceeding the second-ranked proxy, exo_rgb, at 0.139. This indicates that depth can provide useful supervision for specific classes where geometric cues, such as body-to-floor distance, are discriminative. However, its low aggregate selection rate also suggests that this benefit is not broadly shared across classes. The behavior of ego-side proxies also differs across the two datasets. On EEF, all ego-side proxies have relatively low selection rates, around 0.03, suggesting that third-person supervision is more reliable for large whole-body fitness motions. In contrast, on A101, ego_dino and ego_depth reach selection rates of approximately 0.091 and rank among the middle group of proxies. Each of them is also the most frequently selected proxy for at least one attempt-to-* class. This difference is consistent with the task characteristics of A101: fine-grained assembly actions are often handand object-centric, and egocentric observations may capture local manipulation details that are less visible from exocentric views.
17