Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates

Technical Article

H UMAN -JEPA: A H UMAN -C ENTRIC V ISION M ODEL THAT P ERCEIVES AND A NTICIPATES

arXiv:2608.21160v1 [cs.CV] 21 Aug 2026

Hui Wei1,2 , Licai Sun1,2 , Guoying Zhao1,2 ∗ 1 ELLIS Institute Finland, Finland 2 Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland {hui.wei, licai.sun, guoying.zhao}@oulu.fi

A BSTRACT Machines that understand humans should perceive the present and anticipate the future. Existing human-centric vision model are pretrained on human images, set the state of the art in static dense perception, so motion and anticipation are out of reach. Here we present Human-JEPA, a human-centric vision model trained on video by anchored forecasting: dense targets are pinned to a frozen copy of the initialization, preventing a silent collapse of dense perception, and block masks are replaced by a pure past-to-future split, avoiding a five-point action tax and a seventeen-point re-identification collapse. Under frozen probes, HumanJEPA leads the pixel-anchored specialists on pose and person re-identification at 2.7 times fewer parameters, conceding high-resolution dense parsing, and its released predictor head is the first that does not degrade anticipation. A single safely adapted model thus serves both halves of understanding humans.

1

I NTRODUCTION

The human visual system perceives and anticipates simultaneously (Rao & Ballard, 1999; Summerfield et al., 2006). Watching someone reach across a table, we register the body, the pose, and who the person is, and at the same time we predict where the hand will land, well before it arrives. Machine vision for humans needs both halves. Perception of the present segments bodies (Zhang et al., 2025), localizes joints (Zheng et al., 2023), decides whether two views show the same person (Shahreza & Marcel, 2025), etc. Anticipation of the future matters just as much, because decisions are made before an action completes (Zatsarynna et al., 2025). Existing strongest human-centric vision model comes from pixel-anchored image encoders pretrained on human corpora: Sapiens2 (Khirodkar et al., 2026) sets the state of the art in parsing and pose at high resolution, HAP (Yuan et al., 2023) shows that body-part priors improve masked person modeling, and supervised generalists such as UniHCP (Ci et al., 2023) and Hulk (Wang et al., 2025) unify human tasks behind shared heads. However, these models that consume human images can be the strongest possible reader of a photograph and still represent no motion and predict no future, so the second half of understanding humans is unreachable. To address the limitation, we propose Human-JEPA, the first human-centric vision model that perceives and anticipates simultaneously. Human-JEPA built from the released V-JEPA 2.1 (MurLabadia et al., 2026) by anchored forecasting with two changes and one omission. The anchor pins dense context targets to a frozen copy of the initialization, removing the target drift that causes the collapse; an image co-training branch complements it and turns out to be where the identity ability originates. The forecasting mask family replaces block inpainting with a pure past-to-future split, so the only route to the target is a model of how the scene evolves; this specializes the temporal representation to humans at almost no cost and trains the predictor as a same-space forecaster. The experimental results, measured under frozen probes with an identical protocol per track (Figure 1), leads on every axis the image specialists cannot contest and on two they can. Pose reaches 0.620 AP over two pretraining seeds, ahead of every pixel-anchored specialist including the 2.7× ∗

Corresponding author.

1

Technical Article

base

ATR .7392

DensePose .6298

.7659

.6608

.7667

.6662

.7479 .73 .75 .78

DPT mIoU

.6490

Sapiens2

Pose .6202

.5639 .5914 .6142

DINOv3

ReID .4635

AP

NTU

Mutual

68.2

Anticipation

77.5

80.0

.5148 .4522 .4370

.63 .66 .69 .55 .60 .65 .40 .48 .56

DPT mIoU

Human-JEPA (ours)

mAP

70.8 68

70

top-1 (%)

79.6 72 76

78

80

top-1 (%)

75.9 76

80

top-1 (%)

Figure 1: What human specialization buys, and what it costs. Seven frozen-probe tracks under one protocol. Human-JEPA (blue, outlined) leads the human specialists on pose, identity, and anticipates.

larger Sapiens2-0.8B by 2.9 AP, of which the base family position contributes 2.3 and the specialization 0.6, an attribution we keep explicit throughout. Person re-identification reaches 0.4635 mAP on Market-1501, which turns the base’s deficit against Sapiens2 into a lead and gains 2.7 mAP over the base itself. Anticipation gains 4.1 points over the base’s released bundle, of which 1.0 comes from our encoder while the base’s own shipped head costs it 3.0. Overall, the key contributions of Human-JEPA are as follows: • We propose Human-JEPA, the first human-centric vision model that perceives and anticipates simultaneously: under frozen probes it leads the pixel-anchored specialists on pose and person re-identification at 2.7× fewer parameters, and its predictor head does not degrade anticipation. • We develop an anchored forecasting that specializes a video JEPA without destroying it: a frozen-initialization dense anchor with an image co-training branch prevents the silent collapse of continued pretraining, and a past-to-future masking removes the action and identity tax. • We conduct systematic evaluations under one frozen-probe protocol, including a causal partnerablation probe that falsifies all nine person-level objectives we construct.

2

R ELATED W ORK

2.1

H UMAN -C ENTRIC V ISION M ODELS

Human-centric vision has gradually moved from task-specific designs toward general-purpose representations that can be shared across diverse tasks (Tang et al., 2025). Early unified models, such as UniHCP (Ci et al., 2023) and PATH (Tang et al., 2023), jointly train a single Vision Transformer (ViT) (Dosovitskiy et al., 2021) across multiple human-centric datasets with task-specific queries or projectors, demonstrating that shared body semantics can benefit pose estimation, human parsing, re-identification, etc. Hulk (Wang et al., 2025) further extends this direction with a multimodal generalist framework that formulates heterogeneous 2D, 3D, skeleton-based, and vision-language tasks as translations between discrete and continuous output spaces, reducing the need for task-specific heads. In parallel, large-scale self-supervised pretraining has been explored to build high-fidelity human-centric backbones. Sapiens (Khirodkar et al., 2024) pretrains masked autoencoders on hundreds of millions of in-the-wild human images and achieves strong dense prediction performance at native 1K resolution, while Sapiens2 (Khirodkar et al., 2026) further combines masked reconstruction with a self-distilled contrastive objective to preserve both low-level appearance cues and high-level semantics, scaling to billion-parameter models and 4K-resolution inputs. All of these operate on single images or labeled multi-task data; none represents how people move or interact over time, which is the axis Human-JEPA targets. 2.2

S ELF -S UPERVISED L EARNING

Self-supervised learning (SSL) has emerged as a powerful paradigm for learning transferable representations from massive unlabeled data (Gui et al., 2024). Existing SSL methods can be broadly grouped into three paradigms: contrastive learning, distillation, and masked data modeling. 2

Technical Article

The contrastive line can be traced back to instance discrimination (Wu et al., 2018), which treats each image as an individual class and learns representations by separating each instance from others using a memory bank. Building on this idea, CMC (Tian et al., 2020) maximizes mutual information across multiple views of the same scene to encourage view-invariant representations. SimCLR (Chen et al., 2020a) further shows that strong augmentations, a nonlinear projection head, and large in-batch negatives are crucial for effective representation learning. To reduce the dependence on large batch sizes, MoCo (He et al., 2020) introduces a queue-based dynamic dictionary updated by a momentum encoder. MoCov2 (Chen et al., 2020b) further improves this framework with stronger augmentations and an MLP projection head, and MoCov3 (Chen et al., 2021) extends it to ViT and addresses the training instability encountered in this setting. Beyond explicit negative pairs, distillation-based methods learn representations by matching the outputs of two networks. BYOL (Grill et al., 2020) trains an online network to predict a momentum target network using only positive pairs, while SimSiam (Chen & He, 2021) shows that a simple Siamese framework with stop-gradient can avoid collapse without negatives, momentum encoders, or large batches. Along this direction, DINO (Caron et al., 2021) aligns the output distributions of a student and a momentum teacher through centering and sharpening, yielding ViT attention maps with emergent object-level segmentation. DINOv2 (Oquab et al., 2024) scales this recipe with curated data and optimized training to obtain general-purpose frozen features, and DINOv3 (Siméoni et al., 2025) further improves dense representations by scaling model and data and introducing a Gram-anchoring objective. Finally, masked data modeling learns representations by reconstructing partially corrupted inputs. MAE (He et al., 2022) masks a large proportion of image patches and reconstructs the missing parts with an asymmetric encoder–decoder architecture, enabling scalable visual pretraining. Data2vec (Baevski et al., 2022) provides a unified self-supervised framework for speech, vision, and language by predicting latent representations of the full input from a masked view, and the subsequent data2vec 2.0 (Baevski et al., 2023) substantially improves the efficiency of this paradigm. 2.3

J OINT E MBEDDING P REDICTIVE A RCHITECTURE

Joint Embedding Predictive Architecture (JEPA) learns representations by predicting target-region embeddings from context embeddings in latent space, avoiding both augmentation-dependent positive pairs in contrastive learning and low-level biases of pixel/token reconstruction. I-JEPA (Assran et al., 2023) first demonstrated this paradigm for images, and subsequent variants extended it to video (V-JEPA (Bardes et al., 2023a), V-JEPA 2 (Assran et al., 2025), V-JEPA 2.1 (Mur-Labadia et al., 2026)), motion (MC-JEPA (Bardes et al., 2023b)), point clouds and 3D scenes (Point-JEPA (Saito et al., 2025), 3D-JEPA (Hu et al., 2024)), LiDAR (AD-L-JEPA (Zhu et al., 2026)), and audio (AudioJEPA (Tuncay et al., 2025)). Recent work further simplifies anti-collapse objectives with distributional regularization (LeJEPA (Balestriero & LeCun, 2025), LeWorldModel (Maes et al., 2026)) and introduces structured semantic priors through object-level masking or vision-language guidance (Causal-JEPA (Nam et al., 2026), ThinkJEPA (Zhang et al., 2026)). These advances establish JEPAs as a general framework for learning structured representations across modalities and tasks. For human-centric learning, S-JEPA (Abdelfattah & Alahi, 2024) applies latent prediction to 3D skeletons for action recognition.

3

H UMAN -JEPA

Human-JEPA takes a released video JEPA, an encoder f and a predictor g pretrained on general internet video (Mur-Labadia et al., 2026), and returns a human-centric model of the same architecture and size, so that adopting it is a checkpoint swap for an existing user. The input is a curated corpus of human video and person crops, the output is a frozen-transferable encoder together with a predictor trained in that encoder’s own representation space, and the objective is self-supervised throughout, with no human labels and no pose annotations at any stage. Specializing such a model by simply training it longer on human data does not work, and the two reasons organize this section. The dense features that make the family attractive collapse during adaptation, silently enough that the training objective never reveals it. And the standard masked objective spends the adaptation budget on a copy task, taxing exactly the temporal representation a 3

Technical Article

video JEPA exists for. Human-JEPA therefore keeps the architecture and the initialization, whose abilities are the asset being specialized, and makes two critical changes, both at the masking and anchoring level rather than the architecture level: an anchored context stream that pins dense perception to the initialization, and a forecasting mask family that specializes the temporal representation without taxing it. A third decision is an omission, no person-level masking and no pose input, and a fourth is optional, a two-phase schedule for the start of training. 3.1

A NCHORED C ONTINUED P RETRAINING

Given a video x tokenized into a (T ′ , H ′ , W ′ ) grid of spatiotemporal patches, a student encoder fθ sees only unmasked (context) tokens and a predictor gϕ predicts the latent representation of masked tokens, supervised by an EMA teacher fθ̄ . A context loss additionally supervises predictions at visible positions with multi-layer targets, the mechanism behind V-JEPA 2.1’s dense features: L = ẑmsk − sg [fθ̄ (x)]msk 1 + λ ẑctx − sg [ A(x) ]ctx 1 , ẑ = gϕ (fθ (xctx )), (1) | {z } | {z } masked prediction

context (dense) loss

where sg is stop-gradient and A is the context-target network. Stock training uses A = fθ̄ , which lets the dense targets drift with the student. We instead set A = f0 , a frozen copy of the initialization, so the dense stream is distilled from the one network whose dense quality we want to keep. An image branch trains the same objective on LUPerson-T person crops (Shao et al., 2023) rendered as one-frame clips, carrying full-body appearance at a scale video cannot. 3.2

F ORECASTING M ASKS , AND M IXING THE FAMILIES

Standard multiblock masks repeat a spatial block across every frame. The visible context then always contains the same spatial surround at other instants, so the shortest path to a correct prediction is to copy appearance across space and time. Under continued pretraining this is destructive rather than merely uninformative, because the adaptation budget is spent strengthening a copy task rather than a model of motion, at a cost we measure on both action recognition and person identity. Our recipe replaces the blocks with a pure past-to-future split: the context is the first half of the tubelets and the target the second half,
(2) L = gϕ fθ (xt<T /2 ) − sg [fθ̄ (x)]t≥T /2 1 + λ ẑctx − sg [f0 (x)]ctx 1 . The first term now demands a continuation rather than an interpolation, since nothing about the future is visible and the only route to the target is a model of how the scene evolves, while the second term is unchanged from Eq. 1 and keeps the dense stream anchored throughout. The design principle is that the temporal structure of what is hidden matters more than its spatial placement on the person, which is why the recipe uses no human prior in the mask and why we test the natural alternative, part-guided masking (Yuan et al., 2023), as a control rather than adopting it. Because the two families sit at opposite ends of an appearance-versus-dynamics trade-off, we also train per-batch mixtures. At step i a batch draws the forecasting family with probability pi and the block family otherwise; pi is constant or follows a two-phase schedule (forecasting first, blocks late). Mixing acts across batches, never within one, because batched training requires a fixed maskedtoken count per batch; the family draw is seeded by the step index so all data workers agree. The two ends of this sweep are exactly the block recipe and the forecasting recipe, so a single knob traverses the whole space between them. The released recipe is the pure forecasting endpoint of Eq. 2; mixing ships as a configuration knob rather than as part of the objective.

4

E XPERIMENTS

The experiments establish four claims: frozen-probe leadership on pose and identity, a released bundle that anticipates better than the base’s, causal attribution of every design decision, and quantified costs. Figure 2 shows what a single frozen checkpoint produces across this range. 4.1

E XPERIMENTAL S ETUP

Data. A person detector inspects four sampled frames of each of 536,699 Kinetics-700 clips (Carreira et al., 2019); a loose gate keeps candidates and records the keypoints of every detected per4

Technical Article

Parsing Body parts

Pose

Identity

given only these frames

Anticipation

0.90

agreement with what followed

Human-JEPA forecast

0.80 if nothing moved another clip

0.70 0.5

1.0 1.5 seconds ahead

2.0

Figure 2: One frozen checkpoint covers the human perception tasks and anticipates past what it has seen. All outputs are from this paper’s probes on the frozen seed-1 encoder: parsing and surface-part maps, COCO keypoints, and the top Market-1501 match (green frames carry the query identity). For anticipation the released predictor forecasts the latents of the unseen second half of held-out clips; the curve is cosine agreement with the true continuation over 20 clips, above the static reference at every horizon out to two seconds.

son in a per-shard manifest; the final full-body gate then selects 164,431 clips (30.6 percent) by replaying thresholds over the manifests. A second offline gate over the same manifests keeps 82,696 clips where at least two solid people are spatially engaged, of which 24,524 were recovered from the source archives after the full-body pass had pruned them, again without any re-detection. AIST++ (Li et al., 2021) adds dance motion at 5 percent weight, and the image branch trains on 958k LUPerson-T person crops (Shao et al., 2023); video sampling weights are 0.6 full-body, 0.35 interaction, 0.05 AIST++. Every evaluation subset was frozen before any model saw it, the released manifests distribute clip lists and gate parameters rather than video, and the corpus is Kineticsbounded by construction. Comparison Method. We compare against the strongest pixel-anchored human specialists, Sapiens2-0.8B (Khirodkar et al., 2026) and HAP ViT-B (Yuan et al., 2023), the general contrastive encoder DINOv3-L (Siméoni et al., 2025) as the capacity-matched dense reference, and the frozen V-JEPA 2.1-L (Mur-Labadia et al., 2026) that we initialize from. The Sapiens2 comparison is capacity-unmatched in its favor (0.8B against our 0.3B). Image baselines process frames independently and have no native video-action probe, so their action and anticipation cells print n/a rather than a per-frame approximation. Protocol. Every number comes from a frozen backbone under a probe protocol identical across models: the same head, training budget, frozen evaluation subsets, and feature interface, with no number copied from another paper, so absolute values are not comparable to published fine-tuned results. Dense parsing uses ATR (Liang et al., 2015) with linear and DPT (Ranftl et al., 2021) heads at 512 pixels, surface correspondence uses DensePose (Güler et al., 2018) with a DPT head, pose uses a COCO keypoint probe (Lin et al., 2014) reporting AP, and action uses attentive probes on NTU RGB-D 120 (Liu et al., 2019), with its 26-class two-person mutual slice scored separately, and on the Kinetics-700 validation split, whose probe head trains on an uncurated subset so our own curation cannot bias the column. ReID uses a frozen BNNeck probe on Market-1501: the concatenated token means of four evenly spaced encoder depths on 256 by 128 crops feed a 512dimensional reduction head trained with label-smoothed cross-entropy and batch-hard triplet under PK sampling, with cosine retrieval, identical for every backbone. One honesty note: both the base model and ours saw Kinetics video during pretraining, so the K700 column measures general-action retention, on which frozen top-1 drops from 54.7 to 51.8, a cost we report rather than a preserved property. Implementation Detail. Every arm continues pretraining from the released V-JEPA 2.1-L for 200 epochs (60k steps) at global batch 192 video clips plus 576 person crops, in bfloat16 on 4 GPUs, at learning rate 10−4 with warmup and cosine decay, gradient clipping at 1.0, and a pre-backward 5

Technical Article

loss-spike guard that rejects anomalous batches before they reach the optimizer. Video input is 16 frames at 4 fps and 256 pixels; the forecast mask keeps the first half of the tubelets, and the block family uses standard multiblock geometry. Pilot arms use the identical recipe at 17 or 34 epochs, and the released pair trains single-phase, so the two-phase start enters only as an ablation. 4.2

R ESULTS OF P ERCEPTION

Overall profile. Table 1 sizes both halves of the trade. Against the base, averaged over two pretraining seeds, the recipe buys 0.6 AP of pose and 2.7 mAP of ReID and pays 0.9 points of dense parsing, 1.9 of DensePose, a residual 2.7 point action tax on NTU-120 (2.1 on the mutual slice), well under the block-mask recipe’s 4.8 but not zero, and 2.9 points of general action on K700. Against the image specialists, the family position does most of the work: the frozen base already leads Sapiens2-0.8B on pose by 2.3 AP at 2.7 times fewer parameters, and the specialization extends that lead to 2.9. Dense parsing is conceded (2.3 points of ATR to Sapiens2), and ReID is the one column specialization flips outright. The two-seed spread is larger than our pilots predicted; we report it on every headline cell, defend the mean rather than the better seed, and quantify the schedule that removes it in the ablation study. The conceded dense gap resists every anchor, auxiliary, and capacity repair we tested, so we treat it as a division of labor rather than a deficit to chase: pixel-anchored single-image pretraining owns static appearance at high resolution, and latent video prediction owns dynamics. Table 1: Human-JEPA leads the human specialists on pose and identity while conceding dense parsing and paying on both action columns. Frozen probes measured by us under one protocol; bold best, underline second best; ± is the two-seed spread (ReID is seed 1); n/a marks image models, which have no video-action probe.

Params

ATR mIoU ↑

DensePose mIoU ↑

Pose AP ↑

NTU top-1 ↑

Mutual top-1 ↑

ReID mAP ↑

Human-centric image encoders Sapiens2-0.8B 0.8B DINOv3-L 0.3B HAP ViT-B 0.09B

.767 .766 .685

.666 .661 .492

.591 .564 .490

n/a n/a n/a

n/a n/a n/a

.4522 .5148 n/a

Video JEPA V-JEPA 2.1-L (base)

0.3B

.748

.649

.614

70.8

79.6

.4370

Human-JEPA (ours)

0.3B

.739±.004

.630±.004

.620±.003

68.2±2.3

77.5±1.6

.4635

Model

Person re-identification. ReID is the largest dividend of the specialization and the only column where a deficit against the pixel-anchored specialist turns into a lead. Table 2 gives the result: Human-JEPA beats the pixel-anchored specialist (.4635 versus .4522 mAP) and improves 2.7 mAP over its own frozen base, so the specialization helped rather than merely survived, and the identity signal originates in the LUPerson-T image branch, a capability no video self-supervised baseline reports. DINOv3 keeps the overall lead, which we report without qualification: contrastive objectives are instance-discriminative by construction, and the honest claim is best human-video model, not best model. The sharpest row is the last one: the block-mask variant of our own recipe, identical in data, anchor, and budget, collapses to .2666 mAP, seventeen points below the base it started from, while the forecasting recipe gains. The action tax and the ReID collapse are the same phenomenon observed on two task families that share nothing but the encoder. 4.3

R ESULTS OF A NTICIPATION

A video JEPA is released as an encoder plus a predictor, and a user who anticipates deploys the bundle, so we evaluate the bundle directly: NTU-120 early action, where the probe observes only the first half of each clip, with the same frozen attentive-probe protocol in every cell and two feature interfaces per model, encoder only and encoder plus shipped head, whose predictions for the unobserved window the probe attends over. Table 3 contains one expected result and one unexpected one. Expected: our encoder anticipates better than the base encoder (+1.02), and the full released bundle is +4.06 over the base’s released bundle. Unexpected, and to our knowledge unreported: the base’s own released head hurts it, costing 2.98 points against its encoder-only reading, consistent with its 6

Technical Article

Table 2: The mask family decides the identity cost: forecasting gains 2.7 mAP over the frozen base and passes the human specialist, while block masks lose 17 mAP from the same base, data, and budget. Frozen BNNeck probe on Market-1501, identical head and budget per backbone; bold best, underline second best. Model

mAP ↑

Rank-1 ↑

Rank-5 ↑

DINOv3-L Sapiens2-0.8B V-JEPA 2.1-L (base)

.5148 .4522 .4370

.7586 .7055 .6974

.9074 .8744 .8702

Human-JEPA (ours) block-mask variant (A1)

.4635 .2666

.7043 .5235

.8798 .7271

upstream training as a cross-space distillation head rather than a same-space forecaster. Our head, trained by forecasting in its own representation space, is the only released head in the comparison that does no harm (+0.06). We state the boundary of the claim plainly: at this observation fraction the rollout adds essentially nothing over our encoder alone, so the value demonstrated is representation consistency, not extra anticipatory signal, and a pre-registered variable-horizon test, reported in the ablation study, does not change that verdict at shorter horizons. The released head can also be read directly, without a probe in the way. On 20 held-out Kinetics700 validation clips the encoder sees the first half of each clip and the predictor is asked for the latents of the second half: its predictions agree with the latents of the frames that actually followed at 0.873 cosine, against 0.798 for holding the last observed latent fixed and 0.735 for scoring the same predictions against a different clip’s future. The margin over the static reference holds on every clip and widens with distance, from 0.057 half a second ahead to 0.087 two seconds ahead, and the forecast retrieves the exact moment it was asked to predict 40 percent of the time against a 0.6 percent chance rate. That is the signature of a forecast rather than a copy. Table 3: The shipped head is not neutral: the base’s released head costs it 2.98 points, while ours is the only one that does no harm. NTU-120 early action with the probe observing the first half of each clip. Encoder only top-1 ↑

Encoder + shipped head top-1 ↑

Effect of the head ∆ top-1

V-JEPA 2.1-L (base)

78.91

75.93

−2.98

Human-JEPA (ours)

79.93

79.99

+0.06

Model

4.4

A BLATION S TUDY

The mask family sets the cost, on two unrelated task families. At a matched 34-epoch pilot horizon, replacing block masks with the pure past-to-future split and touching nothing else moves action by +8.6 NTU points overall and +7.0 on the mutual slice, at no dense or pose cost. At full scale (Table 4), every block-masked arm pays about five points of action, spread evenly across twoperson and single-person classes, while the forecasting recipe returns to within noise of the base (70.4 versus 70.8; 79.0 versus 79.6 mutual). The same decision controls ReID, a task family the recipe never saw: block masks collapse it by seventeen mAP while forecasting gains (Table 2). The tax is a mask-family artifact, not a data or anchor effect: block inpainting spends the adaptation budget on a temporally local copy task, and forecasting keeps the pressure to model dynamics. Table 4: Forecasting returns action to the base level, block masks pay a uniform five-point tax, and wholeperson deletion collapses at scale. Frozen NTU-120 probe at full scale; mutual is the 26 two-person classes. NTU top-1 ↑

mutual ↑

non-mutual ↑

A0: V-JEPA 2.1-L (frozen) A1: anchored block-mask adaptation whole-person deletion (200 ep)

70.8 65.6 45.5

79.6 74.7 57.0

68.1 62.8 42.0

Human-JEPA

70.4

79.0

67.7

Model

7

Technical Article

Person-level objectives fail, with a causal probe as judge. Table 5 isolates the omitted personlevel family at matched pilot scale. Masking whole people costs dense parsing (0.7357, below the random-mask band of 0.7423 ± 0.0010), and the cost is the mask, not the data, since a random-mask control on the same blended interaction corpus stays inside the band (0.7418). Deletion does not teach partner reading (mutual 56.4 versus 56.5), because deleting a person from all of space-time removes them from the target side of every token the loss touches, so two-person co-motion is never a training signal, and the damage compounds with horizon: at 200 epochs dense falls to 0.6996, pose to 0.5403, and action to 45.5 (Table 4). Any pose input makes it worse: a four-anchor skeleton trajectory scores 52.9 and even a single-anchor identity skeleton 51.5 against the 56.5 unconditioned control, because a masked person’s latent target is dominated by body configuration, so the skeleton supplies most of the answer. Person-level forecasting, the Causal-JEPA-style intervention, forfeits the scene-level gain (57.8 versus 69.7) by leaving part of the future visible, reopening the temporal copy leak. The judge throughout is the partner-ablation probe: on 256 held-out two-person clips per checkpoint, it predicts the masked person’s tokens with the partner visible and again with the partner’s exact token support removed from the context, and reads the relative error increase as a direct causal measure of partner use. Across nine designs, two representation levels (token grid and person slots), and three loss families (point L1, per-person L1, contrastive), partner use never exceeds 0.72 percent on wild clips, and a person-slot InfoNCE arm that solves its retrieval task emphatically (78 percent over roughly 768 candidates, chance 0.13 percent) shows the least partner use of any arm, so benchmark-shaped proxies dissociate completely from relational learning. Calibration on scripted NTU mutual validation clips, which no wild-trained arm ever saw, shows the ceiling is a property of the data: given coupling, our scene-forecasting model reads the largest partner use of any wildtrained arm (2.02 percent), while supervised mutual-action co-training suppresses partner use to the floor (0.37 percent) even as it solves its training task. The probe, not any benchmark, is the instrument that makes these distinctions visible. Table 5: Making people the unit of prediction never pays: every person-level variant sits at or below the random-mask band on dense parsing and at or below its matched control on interaction, and only scene-level forecasting lifts interaction. Matched-recipe pilots; rows compare within a horizon block; bold best mutual; n/a marks unprobed cells. Masking

Dense mIoU ↑

Pose AP ↑

Mutual top-1 ↑

random multiblock (default) random multiblock, blended corpus part tubes (HAP-style control) whole person, |M |=1 whole person + trajectory prior whole person + traj. prior, |M |=2 whole person + identity prior

0.7423 ± 0.0010 0.7418 0.7357 0.7357 0.7363 0.7354 0.7340

n/a 0.6185 n/a 0.6132 0.6105 0.6105 0.6132

56.5 55.8 n/a 56.4 52.9 54.4 51.5

random multiblock, blended (34 ep) whole person, |M |=1 (34 ep) whole person, |M |=1 (200 ep)

0.7439 0.7391 0.6996

0.6204 0.6224 0.5403

62.7 58.5 57.0

scene forecasting, no person mask (34 ep) person forecasting, t0 anchor (34 ep)

0.7413 0.7394

0.6223 0.6223

69.7 57.8

The collapse is silent and selective, and the anchors are complementary. Naive continued pretraining (A = fθ̄ , video only; row C of Table 6) destroys dense human perception, pose AP falling from 0.614 to 0.110 and DensePose from 0.649 to 0.353, while the training loss decreases throughout and clip-level action drops only four points. Matched-epoch probes (Figure 3) show the control indistinguishable from the anchored run for 90 epochs before collapsing over the next 40, with nothing anomalous in any trainer-visible signal, and the collapse is selective for appearance: under a DPT readout the control’s body and geometry classes recover while garment and appearance classes remain collapsed. Counterintuitively, lower learning rates make it worse (canary 0.520 at lr 10−4 versus 0.328 at 2×10−5 , both unanchored), and within each family the lower-rate run reaches a lower final training loss and a worse canary: the optimizer succeeds at the wrong thing, drifting toward features that are easy to predict and less informative. Our reading is that the freshly initialized predictor converges slowly at low learning rate and feeds structured but wrong gradients 8

training loss

ATR mIoU (frozen probe)

Technical Article

frozen baseline (0.600)

0.6 0.5 0.4 0.3 0.6 0.4

silent collapse

anchored + image branch naive continued pretraining anchor only (no image branch)

no anomaly at the collapse 0

25

50

75 100 125 150 175 continued-pretraining epoch Figure 3: The collapse is silent, and the two preservers play different roles. Top: frozen ATR linear canaries at matched epochs; the full recipe (green) rises to the frozen baseline, the naive control (orange) collapses after 90 epochs, and the anchor-only run (gray) never collapses but plateaus. Bottom: the training losses, indistinguishable and anomaly-free.

into the encoder for longer; anchoring works at both rates because it removes the easy basin rather than racing the descent into it, and the same mechanism motivates the two-phase schedule. Table 6 then completes a two-by-two factorization of the two candidate preservers. With neither, the model collapses (C). The image branch alone holds the frozen baseline and matches the anchored arm’s pose gain (Cimg ), despite training on uncurated video, which also shows curation is not the active ingredient at matched objective, with the caveat that uncurated Kinetics is still human-dominated. The anchor alone never collapses, its canary holding a plateau through the exact window where C loses 25 points, but it erodes slowly to 0.709 dense and 0.554 pose, concentrating on garment and appearance classes, the same selectivity in slow motion. The verdict is asymmetric and the mechanisms complementary: the anchor removes the drifting context target that causes the collapse, and the image stream continually re-grounds the appearance features that video-only prediction lets fade, so Human-JEPA uses both. Table 6: Naive continued pretraining collapses dense perception while its training loss improves, and each ingredient recovers a specific part of it. Full-scale arms under identical frozen probes; gray marks collapsed cells, n/a unprobed metrics. Arm

Params

ATR-lin ↑

ATR-DPT ↑

DensePose ↑

Pose AP ↑

NTU ↑

A0: V-JEPA 2.1-L (frozen) C: naive continued PT (uncurated) Cimg : C + image branch anchor only (no image branch) A1: anchored block-mask adaptation Human-JEPA

0.3B 0.3B 0.3B 0.3B 0.3B 0.3B

.6230 .2947 .6223 n/a .6235 .6210

.7479 .5907 .7415 .709 .7452 .7433

.6490 .3534 .6367 n/a .6384 .6333

.6142 .1105 .6262 .554 .6279 .6233

70.8 66.5 65.4 61.9 65.6 70.4

You keep what you anchor. Every continued-pretraining arm pays an NTU action tax under block masks, about five points for the arms with an image branch or naive objective and more still for the anchor-only arm (61.9), because the context anchor pins only the dense spatial features it supervises while temporal dynamics follow the moving teacher and drift with the data distribution. This is precisely the opening for the masking change: rather than anchoring a temporal stream, which would freeze what we want to improve, the forecasting split redirects the masked stream to demand temporal prediction, and Table 4 shows this recovers the tax nearly in full. No repair closes the dense gap. Treating the context target A as a design variable at matched pilot scale: a re-layered self-anchor does not help (0.739); a cross-model DINOv3 anchor routed onto the predictor leaves dense unchanged while dragging pose toward DINOv3’s weakness (0.737/0.595); routed onto the encoder it collapses the representation at every weight tried, since latent-predictive and contrastive dense spaces are incompatible; and an auxiliary pixel decoder preserves but does not compound with a longer schedule. Capacity does not close it either: under identical frozen probes the 2B-parameter V-JEPA 2.1-G, seven times our model, gains only 0.006 ATR-DPT and 0.011 DensePose mIoU and remains below Sapiens2-0.8B, while its pose advantage widens. Across self, cross-model, pixel, and capacity repairs the dense metric plateaus near 0.74 against the pixel9

Technical Article

anchored references near 0.767, which is why we treat the gap as a division of labor rather than a deficit to chase. Knobs and nulls. Mixing the two mask families per batch traverses a clean monotone trade-off at 17 epochs: pose and action rise with the forecast probability while dense parsing and DensePose ease down, and no interior point dominates both endpoints, so the pure forecasting objective of Eq. 2 stands and mixing ships as a configuration knob that maps the appearance-versus-dynamics frontier. The two-phase schedule lifts the collapse-sensitive linear canary by 1.1 points at pilot scale, exactly where the fresh-predictor mechanism predicts; at 200 epochs the canary gain does not compound into higher means, and what the schedule buys instead is reproducibility, shrinking the released pair’s dense and pose spread roughly tenfold and halving the action spread at a mean cost of 0.5 AP of pose and 1.1 mAP of ReID, so we release the higher-mean single-phase pair and document the schedule as the variance control. Training the predictor across observed fractions {1/4, 3/8, 1/2} posts the best dense and pose of any 17-epoch arm yet fails its pre-registered horizon test, leading its fixed-horizon control by only 0.26 points at a held-out quarter-clip fraction against a bar of +2, so it ships as a zero-cost knob and no horizon-generalization claim is made. A motion-guided variant that reweights the forecast loss per token by frame-difference magnitude is a null despite provably shifting the objective, its training loss holding 3.3 percent above the control throughout an identical data order: no probe moves by more than 0.1, because frame-difference weights are barely humanselective on curated clips. The pattern is consistent: changing what the encoder sees moves the capability profile, while reweighting errors over the same targets does not. One further negative worth stating: a 17-epoch schedule control already matches the 200-epoch model on ReID (.4634 versus .4635 mAP), so identity features saturate early under forecasting masks and long budgets pay elsewhere.

5

C ONCLUSION

Human-JEPA is, to our knowledge, the first human-centric vision model that perceives the present and anticipates the immediate future. Two findings made it possible and should transfer to anyone specializing a video JEPA: continued pretraining silently collapses dense perception through target drift, prevented by anchoring dense targets to the frozen initialization with an image branch as the complementary preserver, and the specialization tax lives in the mask family, removed by forecasting rather than inpainting. The natural next step, making people the unit of prediction, is falsified in all nine forms we constructed by a causal partner-ablation probe that ships with the model. Pixelanchored pretraining owns static appearance at high resolution; Human-JEPA owns people in time.

R EFERENCES Mohamed Abdelfattah and Alexandre Alahi. S-jepa: A joint embedding predictive architecture for skeletal action recognition. In European Conference on Computer Vision, pp. 367–384. Springer, 2024. Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619–15629, 2023. Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025. Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. In Proceedings of the International Conference on Machine Learning, pp. 1298–1312. PMLR, 2022. Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In Proceedings of the International Conference on Machine Learning, pp. 1416–1429. PMLR, 2023. 10

Technical Article

Randall Balestriero and Yann LeCun. Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025. Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual representation learning. 2023a. Adrien Bardes, Jean Ponce, and Yann LeCun. Mc-jepa: A joint-embedding predictive architecture for self-supervised learning of motion and content features. arXiv preprint arXiv:2307.12698, 2023b. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660, 2021. Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning, pp. 1597–1607. PmLR, 2020a. Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750–15758, 2021. Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b. Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640–9649, 2021. Yuanzheng Ci, Yizhou Wang, Meilin Chen, Shixiang Tang, Lei Bai, Feng Zhu, Rui Zhao, Fengwei Yu, Donglian Qi, and Wanli Ouyang. Unihcp: A unified model for human-centric perceptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17840–17852, 2023. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pp. 21271–21284, 2020. Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self-supervised learning: Algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9052–9071, 2024. Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7297–7306, 2018. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738, 2020. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000–16009, 2022. 11

Technical Article

Naiwen Hu, Haozhe Cheng, Yifan Xie, Shiqi Li, and Jihua Zhu. 3d-jepa: A joint embedding predictive architecture for 3d self-supervised representation learning. arXiv preprint arXiv:2409.15803, 2024. Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, and Shunsuke Saito. Sapiens: Foundation for human vision models. In European Conference on Computer Vision, pp. 206–228. Springer, 2024. Rawal Khirodkar, He Wen, Julieta Martinez, Yuan Dong, Zhaoen Su, and Shunsuke Saito. Sapiens2. In International Conference on Learning Representations, 2026. URL https:// openreview.net/forum?id=IVAlYCqdvW. Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13381–13392, 2021. Xiaodan Liang, Si Liu, Xiaohui Shen, Jianchao Yang, Luoqi Liu, Jian Dong, Liang Lin, and Shuicheng Yan. Deep human parsing with active template regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(12):2402–2414, 2015. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740–755. Springer, 2014. Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot. Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(10):2684–2701, 2019. Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026. Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran, Koustuv Sinha, Mike Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-jepa 2.1: Unlocking dense features in video self-supervised learning. arXiv preprint arXiv:2603.14482, 2026. Heejeong Nam, Quentin Le Lidec, Lucas Maes, Yann LeCun, and Randall Balestriero. Causaljepa: Learning world models through object-level latent interventions. arXiv preprint arXiv:2602.11389, 2026. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, 2024. René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12159–12168. IEEE, 2021. Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79–87, 1999. Ayumu Saito, Prachi Kudeshia, and Jiju Poovvancheri. Point-jepa: A joint embedding predictive architecture for self-supervised learning on point cloud. In IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 7348–7357. IEEE, 2025. Hatef Otroshi Shahreza and Sébastien Marcel. Foundation models and biometrics: A survey and outlook. IEEE Transactions on Information Forensics and Security, 2025. Zhiyin Shao, Xinyu Zhang, Changxing Ding, Jian Wang, and Jingdong Wang. Unified pre-training with pseudo texts for text-to-image person re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11140–11150. IEEE, 2023. 12

Technical Article

Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. Transactions on Machine Learning Research, 2025. Christopher Summerfield, Tobias Egner, Matthew Greene, Etienne Koechlin, Jennifer Mangels, and Joy Hirsch. Predictive codes for forthcoming perception in the frontal cortex. Science, 314(5803): 1311–1314, 2006. Shixiang Tang, Cheng Chen, Qingsong Xie, Meilin Chen, Yizhou Wang, Yuanzheng Ci, Lei Bai, Feng Zhu, Haiyang Yang, Li Yi, et al. Humanbench: Towards general human-centric perception with projector assisted pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21970–21982, 2023. Shixiang Tang, Yizhou Wang, Lu Chen, Yuan Wang, Sida Peng, Dan Xu, and Wanli Ouyang. Human-centric foundation models: Perception, generation and agentic modeling. In Proceedings of the International Joint Conference on Artificial Intelligence, 2025. Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision, pp. 776–794, 2020. Ludovic Tuncay, Etienne Labbé, Emmanouil Benetos, and Thomas Pellegrini. Audio-jepa: Jointembedding predictive architecture for audio representation learning. In ICME, 2025. Yizhou Wang, Yixuan Wu, Weizhen He, Xun Guo, Feng Zhu, Lei Bai, Rui Zhao, Jian Wu, Tong He, Wanli Ouyang, et al. Hulk: A universal knowledge translator for human-centric tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via nonparametric instance-level discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3733–3742, 2018. Junkun Yuan, Xinyu Zhang, Hao Zhou, Jian Wang, Zhongwei Qiu, Zhiyin Shao, Shaofeng Zhang, Sifan Long, Kun Kuang, Kun Yao, et al. Hap: Structure-aware masked image modeling for human-centric perception. In Advances in Neural Information Processing Systems, volume 36, pp. 50597–50616, 2023. Olga Zatsarynna, Emad Bahrami, Yazan Abu Farha, Gianpiero Francesca, and Juergen Gall. Manta: Diffusion mamba for efficient and effective stochastic long-term dense action anticipation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3438– 3448, 2025. Haichao Zhang, Yijiang Li, Shwai He, Tushar Nagarajan, Mingfei Chen, Jianglin Lu, Ang Li, and Yun Fu. Thinkjepa: Empowering latent world models with large vision-language reasoning model. arXiv preprint arXiv:2603.22281, 2026. Xiaomei Zhang, Xiangyu Zhu, Ming Tang, and Zhen Lei. Deep learning for human parsing: A survey. ACM Computing Surveys, 58(1):1–33, 2025. Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah. Deep learning-based human pose estimation: A survey. ACM Computing Surveys, 56(1):1–37, 2023. Haoran Zhu, Zhenyuan Dong, Kristi Topollai, Beiyao Sha, and Anna Ewa Choromanska. Ad-l-jepa: Self-supervised representation learning with joint embedding predictive architecture for automotive lidar object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 13925–13933, 2026.

13

Related documents

• Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates#662323
• Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates#661273
• EgoForge: Goal-Directed Egocentric World Simulator#1155148
• Multimodal perceptual curation for cognitive autonomy in Generation Z decision making.#295341
• Ambiguous figures, attention, and perceptual content: reply to Jagnow#756089