UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following Kun Yu1 , Jianhua Yang2 , Yixiang Chen2 , Changwei Wang3 , Hongyuan Yu4 , Yan Huang2 , Fushuo Huo6 , Ya Jing5 , Zhumin Chen1 , Keji He1 1 Shandong University
2 Institute of Automation, Chinese Academy of Sciences
4 The Multimedia Department, Xiaomi Inc
arXiv:2607.13621v1 [cs.AI] 15 Jul 2026
(a) Seek-and-Follow Task Overview
5 Beijing University Of Technology
3 Qilu University of Technology
6 Hong Kong Polytechnic University
(b) Core Challenges 1.
2.
3.
Semantic-Spatial Seek in Open Environments
Implicit Seek-to-Follow Transition and
Stage Ambiguity
Delayed Identity Grounding
under Distractors
The target is initially out of view, requiring the agent to actively seek the person using language and visual context. Instruction+Visual Seek
No explicit signal indicates when to switch from seeking to following. After target disappearance, the agent must infer whether to recover following or re-seek. Instruction+Visual Transition
With the target initially out of view, the agent cannot form a reliable visual-semantic representation and identify the target among distractors. Instruction+Visual Follow
Figure 1: Overview of the embodied visual seek-and-follow task, its core challenges.
Abstract
1
Language-guided human following is an important capability for embodied agents, but existing benchmarks typically assume that the target person is visible at the start of an episode. This setting simplifies the problem and overlooks a more realistic requirement: an agent often needs to first find a language-described target and then persistently follow that target in a dynamic environment. While recent work has started to study human search, existing settings are typically evaluated in task-specific scenarios and often rely on stronger prior knowledge of the environment. Moreover, they usually treat searching and following as separate tasks and still lack a unified benchmark for systematic evaluation. To address these limitations, we introduce the Unified Embodied Seeking and Following Benchmark (UESF-Bench), a large-scale and diverse benchmark for embodied human seeking and following. The benchmark requires agents to handle semantic-guided exploration, reliable behavior switching and recovery, and delayed identity grounding. To this end, we propose SeekFollow-VLA, a vision-language-action framework with a task-driven routing mechanism for latent phase inference and transition modeling between seeking and following. Experimental results show that SeekFollow-VLA achieves clear improvements over both single-head and dual-head baselines across single-person and multi-person environments, establishing a baseline for unified embodied seek-and-follow.
Embodied human search [7, 20] and following [6, 14, 28, 30, 34] are fundamental tasks for intelligent robots and have received increasing attention in the robotics community. In practical applications such as service robotics [11, 17, 18], elderly assistance [5, 12, 19], human-centered embodied interaction is often required. In such scenarios, an embodied agent is required to first locate a specific person and then seamlessly transition to persistent following in dynamic environments. This search-to-follow paradigm represents a more realistic form of embodied interaction, as it requires the agent not only to accomplish multiple subtasks, but also to maintain task continuity and behavioral consistency across different phases. Existing language-guided human-following benchmark EVTBench [28] has made important progress in embodied human intelligence. However, it relies on a key assumption that is often unrealistic in real-world scenarios, namely, the target person is initially visible. This assumption restricts the agent to learning localized perception-action mappings and consequently limits the capacity for modeling environmental structure and global spatial relationships. In real-world scenarios, the target is often not immediately observable within the agent’s initial field of view. For example, an instruction such as “Find the blond-haired man wearing a fitted black leather jacket in the bedroom and follow him.” inherently requires an initial search phase, as the agent cannot begin following until the target has first been localized. This indicates that the following task alone is insufficient to capture the core functional requirements of real-world scenarios. This observation naturally leads to a more fundamental question: how can an agent autonomously localize a target in an unexplored environment? Although recent studies have begun to explore the human search problem, this line of research remains at an early stage. Existing
Keywords Embodied AI, Embodied Visual Seek-and-Follow, Vision-LanguageAction Models, Human-Robot Interaction
Introduction
Trovato et al.
methods [7, 20] are typically studied in simplified environments or under additional structured priors, such as explicit target cues and candidate regions. These settings limit their ability to systematically model complex search processes in open environments. Moreover, the lack of a unified evaluation benchmark makes it difficult to comprehensively assess an agent’s ability in semantic-guided exploration, spatial reasoning, and long-horizon decision-making. As a result, human-like search behavior, which relies on semantic and spatial commonsense for active target search in complex environments, remains insufficiently modeled and systematically evaluated. More importantly, simply treating search and follow as two independent tasks still fails to capture the continuity of real-world interaction. In practical scenarios, an agent must make decisions continuously under uncertainty: whether to keep searching, when to switch to following, and whether to resume search or continue following once the target is temporarily lost. This process fundamentally involves dynamic switching and coupling between different behavioral modes, rather than a static composition of separate tasks. Therefore, the core question is not how to evaluate searching and following independently, but how to unify them within a single framework and systematically assess the agent’s ability to switch between them during continuous embodied decision-making. To fill this gap, we construct the Unified Embodied Seeking and Following Benchmark (UESF-Bench), a large-scale and diverse benchmark that covers a wide range of difficulty levels, resulting in a dataset of 1.43 million embodied seeking-and-following samples. UESF-Bench is designed to evaluate language-guided embodied seek-and-follow under a single fused instruction, covering both single-person and crowded multi-person environments. To the best of our knowledge, UESF-Bench is the first large-scale benchmark for unified language-guided human seeking and following in embodied settings. Under this unified setting, the agent must autonomously complete the full process from target search to persistent following in an unseen environment based solely on a single fused naturallanguage instruction. As illustrated in Fig. 1, this setting introduces three core challenges. First, semantic-spatial search in open environments requires the agent to actively infer potential target regions and perform goal-directed exploration from language descriptions and visual context when the target is initially out of view. Second, implicit seek-to-follow transition and recovery ambiguity requires the agent not only to make a reliable transition from search to follow when the target is first observed, but also to determine whether to perform local recovery or resume search when the target is temporarily lost. Third, the target is initially out of view, so the agent cannot establish a reliable visual-semantic representation of the true target at the beginning. It must therefore identify the correct person among distractors using only the language description before establishing persistent following and maintaining identity consistency. To address these challenges, we propose SeekFollow-VLA, a vision-language-action framework for the unified embodied seekand-follow task under a single fused instruction. The framework is designed to support semantic-guided target search in open environments, enable reliable behavior switching under implicit phase transitions, and handle delayed identity grounding under distractors. Specifically, SeekFollow-VLA adopts a task-driven routing
design to reduce mode confusion between seeking and following behaviors. The proposed router enables dynamic task inference and transition modeling without relying on explicit task separation, thereby promoting more consistent behavior across different phases of the task. The contributions of this work can be summarized as follows: • We formulate embodied visual seek-and-follow as a unified embodied task where an agent must first seek a languagedescribed target person who is initially out of camera view and then seamlessly transition to following that person in dynamic environments. • We first construct UESF-Bench, a large-scale, diverse, and general benchmark for unified language-guided human seeking and following in embodied settings. UESF-Bench contains 1.43 million embodied visual seek-and-follow samples that cover a wide range of difficulty levels in both single-person and multi-person scenarios. • We propose SeekFollow-VLA, a vision-language-action framework for unified embodied seek-and-follow under a single fused instruction. By explicitly modeling dynamic phase inference and seek-to-follow transition, SeekFollow-VLA provides a more effective solution to mode confusion between seeking and following behaviors. • Extensive experiments in both single-person and multiperson environments demonstrate the effectiveness of the proposed method.
2 Related Work 2.1 VLA Models for Embodied Tasks Recent advances in Vision-Language Models (VLMs) [13, 27, 32] have driven the development of Vision-Language-Action (VLA) models [23, 31, 37], which augment pre-trained VLMs with action generation capabilities. VLA models integrate perception, language understanding, and action generation in an end-to-end framework, showing strong potential for general-purpose embodied control [2, 10, 38]. Although recent VLA models can support multiple tasks within a unified architecture, they largely depend on explicit task separation. Specifically, tasks are commonly specified either through one-to-one instruction–task mappings [14] or through the addition of special task tokens [28, 34] to distinguish predefined task types. As a result, these tasks are generally formulated and handled independently. In contrast, our task requires a single model to first seek a target person and then transition seamlessly to human following under a single fused instruction. This setting poses a new challenge for VLA design because the model must manage behaviorally distinct stages without any explicit task boundary. In particular, when the target person temporarily disappears from view, the model must infer from the evolving visual context alone whether it should continue following or return to seeking.
2.2
Language-Guided Target Search
Prior work has largely studied language-conditioned target search in neighboring embodied settings, particularly in object-goal navigation [4, 8, 24, 29]. In these settings, the agent must ground natural language into embodied exploration in order to localize a target.
UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following
1. Dataset Composition
2. Statistics & Advantages
Task Illustration Seek
Global Statistics Transition
4800+
Avatars
Following
1.43M+
Total Samples
continues
770+
Environments
15900+
Instruction: In the kitchen, find the short brown-haired man in the red-and-dark gray suit and follow him.
Instruction Description
Episodes
Single vs. Multi-person sample Ratio Single (52%)
Find the blond-haired man wearing a fitted black leather jacket in the bedroom and follow him.
Multi (48%)
Comparison with EVT-Bench Feature
EVT-Bench UESF-Bench (Ours)
Human Avatars Samples Semantic-Spatial Target Seek Implicit Seek-to-Follow Transition Delayed Identity Grounding
under Distractors
100
4800+
0.885M
1.43M+
× ×
✓ ✓
×
✓
Figure 2: Overview and statistics of UESF-Bench. UESF-Bench contains 4800+ diverse humanoid avatars and 1.43M+ embodied visual seek-and-follow samples from 770+ environments under both single-person and multi-person settings. However, these tasks are largely object-centric and do not truly address the retrieval of a specific person. More recently, text-driven target person retrieval has begun to receive attention, where language and visual cues are typically combined with map guidance and target-related prior information to localize the target person [7, 20]. Although these studies move closer to person-centered embodied search, they are still typically validated in specific environments and scenario instances, and they do not provide a unified and general benchmark for systematic evaluation. Their evaluation settings are often limited to narrow task formulations and do not sufficiently reflect the diversity and continuity of real embodied interaction. More importantly, they mainly focus on the retrieval stage itself and stop once the target person has been found, without considering the subsequent requirement of persistent human following. Consequently, they do not address how an agent should transition from target search to target-centered following, nor how it should maintain coherent behavior when the target is temporarily lost or reappears in the scene. As a result, existing work does not capture the full embodied interaction process in which an agent must first identify and locate a language-described target person and then continue to follow that person over time. This limitation makes it difficult to systematically study unified policies that must couple semantic target grounding, behavioral transition, and long-horizon target-centered interaction within a single task formulation. This gap motivates a benchmark that jointly evaluates target search, behavioral transition, and sustained target-centered interaction within a unified embodied setting.
2.3
Embodied Human Following
Human following [6, 9, 14, 28, 34, 36] investigates how an embodied agent continuously follows a dynamic target using different sensing modalities. Recent end-to-end VLA-based following models further demonstrate the effectiveness of directly mapping multimodal observations to actions. Uni-NaVid [34] introduces a visionlanguage-action (VLA) model that improves human-following capability through large-scale training in simulation. However, its reliance on a discrete action space limits adaptability in real-world environments. Building on this direction, TrackVLA [28] advances this line of research by jointly modeling recognition and planning within a unified framework, achieving strong performance in real-world tracking tasks. TrackVLA++ [14] further improves embodied visual tracking by incorporating explicit spatial reasoning and long-horizon target memory, leading to stronger robustness under severe occlusions and visually similar distractors. However, existing methods generally assume that the target is already visible at the beginning of the task. As a result, these methods mainly focus on maintaining continuous following after the target has been identified, rather than first searching for the target or handling the transition from search to follow. Overall, prior work treats person retrieval and human following as separate tasks, thereby overlooking a more practical real-world requirement in which an embodied agent must first find a language-specified target person and then continue to follow that person within a unified interaction process.
Trovato et al.
To address this gap, we introduce a large-scale, diverse, and generalizable benchmark for the unified task of language-guided human seeking and persistent following.
3 UESF-Bench Construction 3.1 Data Collection Existing embodied benchmarks mainly focus on human following with an initially visible target, while largely overlooking the humanlike target search process that is often required before following can begin in real-world scenarios. As a result, a benchmark for unified language-guided human seeking remains lacking. To fill this gap, we construct a large-scale, diverse, and general benchmark, termed UESF-Bench, that unifies human seeking and following in a single continuous task setting. We collect a total of 1.43M embodied visual seek-and-follow samples for UESF-Bench. Unlike existing benchmarks that typically treat these two tasks separately, UESFBench requires an agent to first seek a language-described target person who is initially out of camera view and then seamlessly transition to following that person in dynamic environments.
3.2
UESF-Bench Simulation Environment
We build the simulation environment of UESF-Bench on Habitat 3.0 [21], which provides a ready-to-use engine for rendering, navigation, and collision handling in indoor scenes. Our environment construction pipeline consists of five stages: human avatar generation, appearance description generation, motion generation, scene recognition, and instruction generation. Specifically, we use the SMPL-X human model to create diverse humanoid avatars with randomized body shapes and UV texture maps from the ATLAS dataset [15]. We then employ the vision-language model GPT-5.2 [26] to produce a one-sentence textual description for each avatar based on its visible appearance, and adopt MotionConverterSMPLX from Habitat 3.0 to generate realistic human motion sequences. To provide scene-aware context for language instruction generation, we further perform scene recognition by randomly sampling the main human’s position and using GPT-5.2 [26] to infer the room type from the corresponding views of the main human. Finally, based on the generated appearance description and recognized scene context, we use Gemini-3-Pro [16] to produce the seek-follow instruction for each episode.
3.3
Embodied Visual Seek-and-Follow Benchmark
Based on our simulation environment, we construct the Unified Embodied Visual Seek-and-Follow Benchmark (UESF-Bench) for systematic evaluation of this task, as illustrated in Fig. 2. We create 4859 humanoid avatars with diverse appearances and generate corresponding appearance descriptions for each of them. UESFBench is built from 777 of the 890 scenes available in HM3D [22] and MP3D [3]; the remaining scenes are excluded because they do not support valid search-to-follow episode generation under our construction criteria. UESF-Bench contains a total of 15905 episodes, which are partitioned into training, validation, and testing splits with no overlap in scenes. Specifically, the training split contains 11,310 episodes from 543 scenes, the validation split contains
1510 episodes from 77 unseen scenes, and the test split contains 3085 episodes from 157 unseen scenes. To evaluate model performance under different social complexities, UESF-Bench includes both single-person and multi-person scenarios. Overall, we collect 1.43M embodied visual seek-and-follow samples. Each sample consists of a navigation history (RGB sequence), a target description, and the corresponding expert trajectory.
4 Methodology 4.1 Problem Formulation We formulate the embodied visual seek-and-follow task as a partially observable sequential decision problem. At each time step 𝑡, the agent receives a fused language instruction 𝑙 and an egocentric RGB observation history 𝑂𝑡 = {𝑜 1, . . . , 𝑜𝑡 }, where the instruction specifies the appearance of a target person together with the seekand-follow goal. The agent is required to transform the waypoints predicted by the model into control actions 𝑎𝑡 ∈ A = {𝑣, 𝜔 }, where 𝑣 and 𝜔 denote the linear and angular velocities of the agent, respectively. Unlike conventional embodied visual tracking, the target person is typically outside the camera view at the beginning of the task. As a result, the agent must first find a language-described target person who is initially out of camera view, and then transition seamlessly to following that person in dynamic environments. Since no explicit signal is provided to indicate the transition between the seeking and following phases, the model must handle both tasks under a unified formulation. The task is considered successful only if the agent first seeks the correct target person and then continuously maintains an appropriate following distance (1–3 m) while facing toward the target.
4.2
SeekFollow-VLA Overview
As shown in Fig. 3, SeekFollow-VLA adopts a pre-trained visionlanguage model, Qwen3-4B [16], as the backbone. For waypoint prediction, SeekFollow-VLA first encodes historical and current egocentric RGB observations into visual tokens, and then concatenates these tokens with the fused language instruction as the input to the multimodal backbone. Based on the shared multimodal representation, SeekFollowVLA predicts a sequence of future waypoints for agent control. To study how action prediction should be designed under the implicit transition between seeking and following, we instantiate three alternative action head designs under the same backbone: (1) a single shared action head, which directly maps the multimodal representation to waypoint predictions; (2) dual action heads with a phase-aware router, where separate seeking and following heads are combined through learned routing weights; and (3) dual action heads with a task-driven router, where the routing process is further guided by stage supervision to better distinguish search-oriented and follow-oriented behaviors. This design allows us to systematically investigate how architectural inductive bias influences policy learning in embodied visual seek-and-follow. In particular, the task-driven dual-head variant explicitly addresses the mode confusion caused by the implicit stage boundary, enabling the model to switch more reliably between human seeking and human following behaviors within a unified framework.
UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following
Alternative Action Head Designs Three alternative designs under the same backbone
Embodied Observation Encoding History frames oT-k:T-1 Current frame oT
History visual tokens
Current visual tokens
Instruction tokens
Qwen3-4B Vision Encoder
(a) Single head
(b) Dual head+
Phase-aware Router
(c) Dual head+
Task-driven Router
Single Head
Phase-aware
Router
Task-driven
Router
Seeking
head
Seeking
head
Cross-modality Projector
4×k tokens
1×64 tokens
Predicted
waypoints
Following
head
Stage
label
Predicted
waypoints
Following
head
Predicted
waypoints
Figure 3: Architecture of SeekFollow-VLA. The model encodes historical and current observations into visual tokens, projects them into the language space, and combines them with instruction tokens in a shared Qwen3-4B backbone. On top of this backbone, we study three alternative action head designs for embodied seek-and-follow: a single head, dual heads with a phase-aware router, and dual heads with a task-driven router.
4.3
SeekFollow-VLA Architecture
Given the egocentric RGB observation history 𝑂𝑇 = {𝑜 1, . . . , 𝑜𝑇 }, we encode each frame with a dual-tower visual encoder composed of DINO-V3 [25] and SigLIP [33]. For each frame, the two vision towers extract spatial patch-level features under a unified input resolution of 384 × 384. Let the resulting token grids be denoted as 𝑉 dino ∈ R𝑃 ×𝐶𝑑 and 𝑉 siglip ∈ R𝑃 ×𝐶𝑠 , 𝑉 = Concat 𝑉 dino, 𝑉 siglip ∈ R𝑃 ×𝐶 , (1) where 𝑃 = 𝐻𝑝𝑊𝑝 denotes the number of aligned visual patches (set to 576); 𝐶𝑠 and 𝐶𝑑 denote the embedding dimensions of the SigLIP and DINO features, respectively; and 𝐶 = 𝐶𝑠 +𝐶𝑑 denotes the total embedding dimension after feature concatenation. To obtain compact multi-scale visual representations, we further apply a gridpooling operation [34, 35] to the fused patch tokens. Specifically, we generate two levels of visual tokens: 𝑉 fine = GridPool(𝑉 , 64),
(2)
𝑉 coarse = GridPool(𝑉 , 4),
(3)
where 𝑉 fine ∈ R64×𝐶 provides a fine-grained representation and 𝑉 coarse ∈ R4×𝐶 provides a compact coarse-grained representation. To ensure computational efficiency during inference, the observation history uses a sliding-window mechanism that retains the most recent 𝑘 historical frames together with the current frame, where T denotes the current time step and 𝑘 = 31 in our experiments. Within this temporal window, we use the fine-grained tokens 𝑉𝑇fine
for the latest observation and the coarse-grained tokens for historical observations to balance visual detail and token efficiency. Accordingly, the visual token sequence is organized as: coarse fine V𝑇 = {𝑉𝑇coarse −𝑘 , . . . , 𝑉𝑇 −1 , 𝑉𝑇 }.
(4)
Following established vision-language models (VLMs) [16], we use a cross-modality projector 𝑃 (·), implemented as a two-layer MLP, to project the visual token sequence V𝑇 into the latent space of the large language model: 𝐸𝑇𝑣 = 𝑃 (V𝑇 ).
(5)
We concatenate the projected visual embeddings 𝐸𝑇𝑣 with the text embeddings of the fused instruction and feed them into the multimodal backbone to obtain the output representation 𝐸𝑇LLM . This representation is then used as the input to the action prediction module for waypoint generation. Depending on the action head architecture, 𝐸𝑇LLM is processed by a single action head, dual action heads with routing, or dual action heads with a task-driven router.
4.4
Action Head Design
A key challenge of the embodied visual seek-and-follow task is that it unifies human seeking and human following within a single task, while the transition between the two stages is implicit. The objectives of its two stages are fundamentally different. During the seeking stage, the agent aims to quickly locate the target person described by language, whereas during the following stage, it must continuously track and follow that dynamic target. Existing VLA
Trovato et al.
designs with either a single action head or dual action heads may suffer from mode confusion under our setting. To systematically study how action prediction should be structured under implicit seek-to-follow transitions, we investigate three action head architectures. 4.4.1 Single Action Head. As the unified seek-and-follow task is formulated end-to-end, a straightforward formulation is to use a single shared action head that maps the multimodal representation 𝐸𝑇LLM to the waypoint sequence. The action head is implemented as a three-layer MLP. Specifically, the input is first normalized by LayerNorm, then passed through two hidden linear layers with GELU activations, followed by a final linear projection to produce an output vector of dimension 𝑀 ×𝑑𝑎 , where 𝑀 (set to 10) is the number of predicted waypoints and 𝑑𝑎 (set to 3) is the action dimension of each waypoint. A tanh function is applied to bound the output within [−1, 1]. The resulting vector is then reshaped into Ŵ ∈ 𝑀 consists of the agent’s R𝑀 ×𝑑𝑎 . Each waypoint 𝑤𝑖 = (𝑥𝑖 , 𝑦𝑖 , 𝜃 𝑖 )𝑖=1 position 𝑥, 𝑦 in the plane, and 𝜃 , which represents the heading or orientation of the agent, Ŵ𝑇 = SingleActionHead E𝑇LLM .
(6)
The single action head is simple and parameter-efficient, but it does not explicitly account for the behavioral differences between seeking and following. As a result, modeling both behaviors with a single action head may lead to mode confusion. This observation motivates our dual-head design with adaptive routing. 4.4.2 Dual Action Heads with Phase-Aware Router. To better capture the behavioral differences between seeking and following, we replace the shared action head with two separate heads: a seeking head and a following head. The two heads share the same architecture as the single action head, but maintain separate parameters for seeking and following. In this way, seeking and following are modeled by different prediction branches, Ŵ𝑇seek = SeekingActionHead E𝑇LLM , (7) Ŵ𝑇follow = FollowingActionHead E𝑇LLM .
(8)
Furthermore, since UESF is a continuous task without explicit phase boundaries, we introduce a Phase-Aware Router to adaptively determine which head should contribute more to the final prediction. The Phase-Aware Router is implemented as a three-layer MLP that takes the multimodal representation 𝐸𝑇LLM as input and outputs two routing weights for the seeking and following heads. These scores are normalized by a softmax function to obtain routing weights, which are then used to combine the outputs of the two action heads, 𝜶 𝑇 = PhaseAwareRouter E𝑇LLM , (9) where 𝜶 𝑇 = [𝛼𝑇seek, 𝛼𝑇follow ] denotes the routing weights assigned to the two heads. Using these routing weights, the final waypoint sequence is obtained by fusing the outputs of the two heads, Ŵ𝑇 = 𝛼𝑇seek Ŵ𝑇seek + 𝛼𝑇follow Ŵ𝑇follow,
(10)
where Ŵ𝑇 denotes the final predicted waypoint sequence obtained by fusing the outputs of the seeking and following heads.
Compared with the single-head formulation, this design explicitly decouples the two behavior modes into separate prediction branches, allowing each head to specialize in predicting a more coherent trajectory. At the same time, this design ensures that the two behaviors remain within the same unified task framework, enabling seamless transitions between them. Nevertheless, the router is still optimized solely through the final waypoint prediction objective, without explicit supervision on which behavior mode should dominate at each time step. Under partial observability, this lack of supervision may lead to ambiguous routing decisions. 4.4.3 Dual Action Heads with a Task-Driven Router. To improve the adaptability of the model, we propose a Task-Driven Router. Like the Phase-Aware Router, it is implemented as a three-layer MLP that incorporates task-specific cues to guide the routing process. This task-driven router enables the model to more effectively determine which head should dominate at each time step, ensuring a more accurate and stage-aware prediction. The two action heads remain the same as in the previous design: Ŵ𝑇seek = SeekingActionHead E𝑇LLM , (11) (12) Ŵ𝑇follow = FollowingActionHead E𝑇LLM . Different from the Phase-Aware Router, the Task-Driven Router outputs both routing weights and routing logits: (𝜶 𝑇 , g𝑇 ) = TaskDrivenRouter 𝐸𝑇LLM , (13) where 𝜶 𝑇 = [𝛼𝑇seek, 𝛼𝑇follow ] denotes the routing weights assigned to the two heads, and g𝑇 = [𝑔𝑇seek, 𝑔𝑇follow ] denotes the corresponding task-type logits. The final prediction is given by, Ŵ𝑇 = 𝛼𝑇seek Ŵ𝑇seek + 𝛼𝑇follow Ŵ𝑇follow .
4.5
(14)
Training Objective
For all model variants, the primary training objective is waypoint prediction. Given the predicted waypoint sequence Ŵ ∈ R𝐵×𝑀 ×𝐷 and the ground-truth waypoint sequence W ∈ R𝐵×𝑀 ×𝐷 , we compute a masked mean squared error over the selected waypoints: 2 ∑︁ 1 Ŵ𝑏,𝑚,𝑑 − W𝑏,𝑚,𝑑 , (15) Lwp = |Ω| (𝑏,𝑚,𝑑 ) ∈Ω
where Ω denotes the set of valid waypoint entries selected by the waypoint mask, 𝐵 is the batch size, 𝑀 is the number of predicted waypoints, and 𝐷 is the waypoint dimension. If no valid waypoint is selected, the loss is set to zero. For the Single Action Head and Dual Action Heads with Router variants, the training objective is defined as follows: L = Lwp .
(16)
For the Dual Action Heads with a Task-Driven Router model variant, we further introduce an auxiliary task supervision on the predicted task logits. Specifically, the Task-Driven Router outputs both routing weights and task logits, where the routing weights are used for waypoint fusion and the task logits are supervised by a task-related label 𝑧𝑇 ∈ {0, 1}. Here, 𝑧𝑇 is the ground-truth stage label, where 𝑧𝑇 = 0 indicates the seeking stage and 𝑧𝑇 = 1 indicates the following stage. The label is defined by task progression. Specifically, 𝑧𝑇 remains 0 until the agent first successfully finds the
UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following
target person, where success is defined as the target being within 1.7 m and the agent facing the target. After this first successful discovery, 𝑧𝑇 remains 1 for all subsequent time steps. An auxiliary stage-aware loss is defined as a cross-entropy loss: Lstage = CrossEntropy(g𝑇 , 𝑧𝑇 ),
(17)
where g𝑇 = [𝑔𝑇seek, 𝑔𝑇follow ] denotes the predicted task logits, and 𝑧𝑇 indicates the ground-truth behavior mode at the current step. The final objective for the task-driven variant is defined as the weighted sum of the waypoint prediction loss and the auxiliary stage-aware loss, L = 𝛽 nav Lwp + Lstage,
(18)
where, 𝛽 nav controls the relative weight of the waypoint prediction loss. In all experiments, we set 𝛽 nav = 10. In this way, the waypoint regression term optimizes trajectory prediction, while the auxiliary gate loss explicitly encourages correct task-aware routing between the seeking and following modes.
5 Experiments 5.1 Experimental Setup We follow the architecture described in Sec. 4.2. Unless otherwise specified, the three compared model variants differ only in the action prediction head. For both the single-person and multi-person settings, we construct training, validation, and test splits. All models are trained on the training split, selected based on validation performance, and finally evaluated on the corresponding test split. Training is conducted on five NVIDIA H100 GPUs using distributed mixed-precision. We use an initial learning rate of 2 × 10−5 , a batch size of 22, a waypoint horizon of 10, and a temporal history length of 31. Validation is performed every 1000 training steps. We set the maximum number of training epochs to 10 and apply early stopping based on validation performance. In practice, the best checkpoints are obtained within 2–3 epochs.
5.2
Metrics
To comprehensively evaluate performance on the embodied visual seek-and-follow task, we adopt four metrics covering task success, safety, following quality, and search efficiency. These metrics are defined as follows: • Task Success Rate (TSR): the ratio of episodes that successfully complete the overall seek-and-follow task to the total number of episodes. • Collision Rate (CR) [28]: the ratio of episodes with at least one robot-human collision during the following phase to the total number of episodes. • Following Ratio (FR) [28]: the ratio of successful following steps to the total number of following-phase steps. • Search SPL [1]: the success-weighted path length in the seeking phase, reflecting both search success and path efficiency, with values ranging from 0 to 1.
Table 1: Single-person results on UESF-Bench test set. TSR: Task Success Rate; CR: Collision Rate; FR: Following Ratio; SPL: Search SPL. Method
TSR
CR
FR
SPL
SingleHead DualHead-PA DualHead-TD
0.04 0.05 0.35
0.07 0.06 0.32
0.85 0.85 0.92
0.30 0.27 0.53
5.3
Single-Person: Main Comparison and Ablation
We first evaluate the three action head architectures in the singleperson setting. Specifically, we compare the Single Action Head (SingleHead), Dual Action Heads with Phase-Aware Router (DualHeadPA), and Dual Action Heads with a Task-Driven Router (DualHeadTD) on the single-person test split using TSR, CR, FR, and Search SPL as evaluation metrics. As shown in Table 1, the quantitative results of the three methods are compared under the single-person setting. In the single-person setting, DualHead-TD achieves the highest TSR of 0.35, compared with 0.04 for SingleHead and 0.05 for DualHead-PA. It also attains the highest FR of 0.92, whereas both SingleHead and DualHead-PA remain at 0.85. In addition, DualHeadTD achieves the best Search SPL of 0.53, clearly outperforming SingleHead (0.30) and DualHead-PA (0.27). These results indicate that task-driven routing effectively improves both search efficiency and following stability, leading to stronger end-to-end performance in the unified seek-and-follow task. By contrast, SingleHead and DualHead-PA perform similarly in the single-person setting, especially in terms of TSR, where both remain at a very low level. This suggests that neither using a single shared action head nor introducing a dual-head structure without explicit task-driven supervision is sufficient to address the core challenges of this task. Specifically, the model must not only handle the implicit transition between seeking and following, but also resolve the behavioral ambiguity that arises when the target temporarily leaves the field of view during the following stage.
5.4
Multi-Person: Main Comparison and Ablation
To evaluate the three action head architectures in the more challenging multi-person setting, we place 2–7 distractor persons in each scene, with trajectories that frequently intersect with that of the target person. This setting not only makes target identification more difficult during the seeking stage, but also introduces persistent identity ambiguity during following. As a result, the task becomes substantially more challenging than in the single-person setting. Table 2 summarizes the quantitative results on the multi-person test split. Compared with the single-person setting, performance on most metrics declines in the multi-person setting, indicating that distractor interference substantially increases the difficulty of embodied visual seek-and-follow. For DualHead-TD, TSR drops from 0.35 to 0.20, while FR decreases from 0.92 to 0.82. For SingleHead, FR also
Trovato et al.
TSR
CR
FR
SPL
0.04 0.04 0.20
0.06 0.06 0.33
0.73 0.85 0.82
0.31 0.26 0.55
decreases from 0.85 to 0.73. Although the metric changes are not entirely consistent across models, the overall trend shows that the multi-person setting is clearly more challenging. When multiple distractors are present, the model must not only identify the target during seeking, but also keep verifying whether the currently observed person is the intended one during following. This becomes even harder under occlusion, trajectory intersections, and close parallel movement. These results suggest that multi-person interference is not simply a matter of increased scene complexity; it also introduces additional challenges for target identification, stage transition, and sustained following. Even under this more challenging setting, DualHead-TD achieves the highest TSR of 0.20, whereas both SingleHead and DualHeadPA remain at 0.04. This result indicates that explicit task-driven routing is particularly important for end-to-end task success in the presence of multiple distractors. In terms of Search SPL, DualHeadTD achieves 0.55, again significantly outperforming the other two methods. This result indicates that it not only identifies the correct target more reliably, but also performs the seeking process more efficiently. It also suggests that, when multiple distractors are present, selecting an appropriate behavior mode according to the current task stage is more important than merely improving local capability within a single stage. For FR, DualHead-PA achieves the highest value of 0.85, while DualHead-TD reaches 0.82. Although DualHead-TD does not achieve the best FR, its clear advantages in TSR and Search SPL better reflect its overall suitability for the unified seek-and-follow task. In both the single-person and multi-person settings, DualHeadTD consistently achieves the best TSR and Search SPL, showing that task-driven routing effectively reduces mode confusion between seeking and following under implicit stage transitions. In contrast, SingleHead entangles the two behaviors within a shared action head, making it difficult to learn clear stage-specific policies. Although DualHead-PA adopts a dual-head structure, it lacks explicit task-driven supervision and therefore cannot reliably separate the demands of seeking and following. While DualHead-TD also exhibits a higher collision rate, this trend should be interpreted together with its stronger task success and search performance rather than viewed in isolation. Overall, these results validate the effectiveness of the proposed task-driven dual-head design for the unified seek-and-follow task.
5.5
Routing Dynamics Analysis
Fig. 4 visualizes the router weights of the task-driven variant in a representative episode. A clear two-stage routing pattern can be observed. During the early part of the episode, the router assigns nearly all weight to the search head, indicating that the model
Follow
Method SingleHead DualHead-PA DualHead-TD
Search
Table 2: Multi-person results on UESF-Bench test set. Same metrics as Table 1.
1.00 0.75 0.50 0.25 0.00 1.00 0.75 0.50 0.25 0.00
Raw Smoothed
Raw Smoothed
0
20
40
60
80
Step
100
120
140
160
Figure 4: Router weights 𝑝 seek and 𝑝 follow over episode time. primarily performs exploratory target seeking. Although slight oscillations appear in intermediate steps, the routing remains consistently dominated by the search branch. Once the target is found, the routing undergoes a sharp transition: the search weight quickly collapses to nearly zero, while the follow weight rises to nearly one and remains stable thereafter. Such behavior shows that the router learns an explicit functional separation between search and follow behaviors, despite the overall task being defined under a unified instruction without an explicit phase boundary at inference time. This supports our claim that task-driven routing improves behavioral allocation and helps the model handle implicit seek-tofollow transitions more reliably. Additional experimental results and visualizations are provided in the supplementary material.
6
Conclusion
In this work, we introduce embodied visual seek-and-follow as a unified task in which an agent must first locate a target person described by language and then persistently follow that person under an implicit phase boundary. This setting extends beyond conventional human-following tasks by jointly requiring semantic-guided target search, reliable behavior switching and recovery when the target appears or is temporarily lost, and delayed identity grounding under distractors. To enable systematic study of this problem, we construct UESF-Bench, a large-scale and diverse benchmark that covers both single-person and multi-person scenarios. We further propose SeekFollow-VLA, a vision-language-action framework with a task-driven routing design for latent phase inference and behavior coordination across the seeking and following stages. Experimental results show that SeekFollow-VLA consistently outperforms the compared baselines across different settings, with especially clear advantages in more challenging multi-person environments.
References [1] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. 2018. Vision-andlanguage navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3674–3683. [2] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. 2024. 𝜋 0 : A Vision-Language-Action Flow Model for General Robot Control. arXiv preprint arXiv:2410.24164 (2024).
UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following
[3] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 (2017). [4] Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. 2020. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems 33 (2020), 4247– 4258. [5] Andrea Fortuna, Marta Lorenzini, Mattia Leonori, Juan M Gandarias, Pietro Balatti, Younggeol Cho, Elena De Momi, and Arash Ajoudani. 2024. A personalizable controller for the walking assistive omni-directional exo-robot (WANDER). In 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 3212–3218. [6] Angus Fung, Beno Benhabib, and Goldie Nejat. 2025. Ldtrack: dynamic people tracking by service robots using diffusion models. International Journal of Computer Vision 133, 6 (2025), 3392–3412. [7] Angus Fung, Aaron Hao Tan, Haitong Wang, Bensiyon Benhabib, and Goldie Nejat. 2025. MLLM-search: A zero-shot approach to finding people using multimodal large language models. Robotics 14, 8 (2025), 102. [8] Sourav Garg, Dustin Craggs, Vineeth Bhat, Lachlan Mares, Stefan Podgorski, Madhava Krishna, Feras Dayoub, and Ian Reid. 2025. Objectreact: Learning object-relative control for visual navigation. arXiv preprint arXiv:2509.09594 (2025). [9] Meenakshi Gupta, Swagat Kumar, Laxmidhar Behera, and Venkatesh K Subramanian. 2016. A novel vision-based tracking algorithm for a human-following mobile robot. IEEE Transactions on Systems, Man, and Cybernetics: Systems 47, 7 (2016), 1415–1427. [10] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. 2024. Openvla: An open-source vision-language-action model, 2024. URL https://arxiv. org/abs/2406.09246 1, 2 (2024), 4. [11] Jin Joo Lee, Amin Atrash, Dylan F Glas, and Hanxiao Fu. 2023. Developing autonomous behaviors for a consumer robot to be near people in the home. In 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 197–204. [12] Lei Li, Ming Jeat Foo, Jiaye Chen, Kuan Yuee Tan, Jiaying Cai, Rohini Swaminathan, Karen Sui Geok Chua, Seng Kwee Wee, Christopher Wee Keong Kuah, Huiting Zhuo, et al. 2023. Mobile Robotic Balance Assistant (MRBA): a gait assistive and fall intervention robot for daily living. Journal of NeuroEngineering and Rehabilitation 20, 1 (2023), 29. [13] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024). [14] Jiahang Liu, Yunpeng Qi, Jiazhao Zhang, Minghan Li, Shaoan Wang, Kui Wu, Hanjing Ye, Hong Zhang, Zhibo Chen, Fangwei Zhong, et al. 2025. Trackvla++: Unleashing reasoning and memory capabilities in vla models for embodied visual tracking. arXiv preprint arXiv:2510.07134 (2025). [15] Yufei Liu, Junwei Zhu, Junshu Tang, Shijie Zhang, Jiangning Zhang, Weijian Cao, Chengjie Wang, Yunsheng Wu, and Dongjin Huang. 2024. Texdreamer: Towards zero-shot high-fidelity 3d human texture generation. In European conference on computer vision. Springer, 184–202. [16] Xingjun Ma, Yixu Wang, Hengyuan Xu, Yutao Wu, Yifan Ding, Yunhan Zhao, Zilong Wang, Jiabin Hua, Ming Wen, Jianan Liu, et al. 2026. A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Doubao 1.8, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5. arXiv preprint arXiv:2601.10527 (2026). [17] Nidhi Mishra, Teena Bharti, Aviral Kumar Tiwari, and Gregor Pfajfar. 2024. Public and scholarly interest in social robots: An investigation through Google Trends, bibliometric analysis, and systematic literature review. Technological Forecasting and Social Change 206 (2024), 123578. [18] Anastasia K Ostrowski, Cynthia Breazeal, and Hae Won Park. 2022. Mixedmethod long-term robot usage: Older adults’ lived experience of social robots. In 2022 17th ACM/IEEE international conference on human-robot interaction (HRI). IEEE, 33–42. [19] Akhil Padmanabha, Jessie Yuan, Janavi Gupta, Zulekha Karachiwalla, Carmel Majidi, Henny Admoni, and Zackory Erickson. 2024. Voicepilot: Harnessing LLMs as speech interfaces for physically assistive robots. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–18. [20] Jeongeun Park, Jefferson Silveria, Matthew Pan, and Sungjoon Choi. 2023. SOCRATES: Text-based Human Search and Approach using a Robot Dog. arXiv preprint arXiv:2302.05324 (2023). [21] Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, et al. 2023. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724 (2023). [22] Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. 2021. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv:2109.08238
(2021). [23] Ranjan Sapkota, Yang Cao, Konstantinos I Roumeliotis, and Manoj Karkee. 2025. Vision-language-action models: Concepts, progress, applications and challenges. arXiv preprint arXiv:2505.04769 (2025). [24] Dhruv Shah, Błażej Osiński, Sergey Levine, et al. 2023. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on robot learning. pmlr, 492–504. [25] Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. 2025. DINOv3. arXiv:2508.10104 [cs.CV] https://arxiv.org/abs/2508.10104 [26] Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267 (2025). [27] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). [28] Shaoan Wang, Jiazhao Zhang, Minghan Li, Jiahang Liu, Anqi Li, Kui Wu, Fangwei Zhong, Junzhi Yu, Zhizheng Zhang, and He Wang. 2025. Trackvla: Embodied visual tracking in the wild. arXiv preprint arXiv:2505.23189 (2025). [29] Karmesh Yadav, Arjun Majumdar, Ram Ramrakhya, Naoki Yokoyama, Alexei Baevski, Zsolt Kira, Oleksandr Maksymets, and Dhruv Batra. 2023. Ovrl-v2: A simple state-of-art baseline for imagenav and objectnav. arXiv preprint arXiv:2303.07798 (2023). [30] Hanjing Ye, Kuanqi Cai, Yu Zhan, Bingyi Xia, Arash Ajoudani, and Hong Zhang. 2025. Rpf-search: Field-based search for robot person following in unknown dynamic environments. IEEE/ASME Transactions on Mechatronics (2025). [31] Zhaoshu Yu, Bo Wang, Pengpeng Zeng, Haonan Zhang, Ji Zhang, Lianli Gao, Jingkuan Song, Nicu Sebe, and Heng Tao Shen. 2025. A survey on efficient vision-language-action models. arXiv preprint arXiv:2510.24795 (2025). [32] Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. 2025. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471 (2025). [33] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-Training. arXiv:2303.15343 [cs.CV] [34] Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, and He Wang. 2024. Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks. arXiv preprint arXiv:2412.06224 (2024). [35] Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. 2024. Navid: Video-based vlm plans the next step for vision-and-language navigation. arXiv preprint arXiv:2402.15852 (2024). [36] Fangwei Zhong, Kui Wu, Hai Ci, Churan Wang, and Hao Chen. 2024. Empowering embodied visual tracking with visual foundation models and offline rl. In European Conference on Computer Vision. Springer, 139–155. [37] Yifan Zhong, Fengshuo Bai, Shaofei Cai, Xuchuan Huang, Zhang Chen, Xiaowei Zhang, Yuanfei Wang, Shaoyang Guo, Tianrui Guan, Ka Nam Lui, et al. 2025. A survey on vision-language-action models: An action tokenization perspective. arXiv preprint arXiv:2507.01925 (2025). [38] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. 2023. Rt-2: Vision-languageaction models transfer web knowledge to robotic control. In Conference on Robot Learning. PMLR, 2165–2183.