Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision–Language–Action Models Nikita Kachaev*,1 Andrey Moskalenko*,2,3,4,5 Matvey Skripkin2,6 Nikita Kurlaev7 Daria Pugacheva7,8 Albina Burlova7,9 Mikhail Kolosov10 Denis Shepelev2,5 Andrey Kuznetsov2 Elena Tutubalina7,11 Aleksandr I. Panov1,10 Alexey K. Kovalev1,10 Vlad Shakhuro2,4,5
arXiv:2606.19297v1 [cs.LG] 17 Jun 2026
1
CogAI Lab 2 FusionBrain Lab 3 IAI MSU 4 Lomonosov MSU 5 NUST MISIS 6 Applied AI Institute 7 HSE University 8 Generalizable AI Systems 9 ISP RAS 10 MIRAI 11 Domain-specific NLP Group * Equal contribution
Abstract Embodied Vision–Language–Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation. Failures on knowledge-sensitive tasks are ambiguous, conflating missing knowledge with poor generalization of low-level control. We introduce Act2Answer, a lightweight protocol that adapts VLM knowledge benchmarks to VLA evaluation by requiring agents to answer through action. Each question becomes a short tabletop episode where the agent performs a single object-placement action to select among candidate answers, yielding an actiongrounded success rate with reduced control confounds. We curate a test suite of such environments across diverse commonsense and worldknowledge categories and introduce layerwise intent probing to localize answer-relevant information across the VLM backbone and action head. In a large-scale study of 7 VLA models and 9 VLM baselines, we systematically rank models across categories, finding that VLAs show solid performance on simple concepts while exhibiting larger gaps on richer semantic categories relative to their source VLMs, that VQA co-training is associated with better knowledge retention, and that answer-relevant signals peak in middle VLA layers but attenuate in upper layers. ACT 2A NSWER is available at tttonyalpha.github.io/act2answer.
1
Figure 1: Knowledge evaluation results for seven stateof-the-art VLA models across diverse knowledge domains on the Act2Answer Task Suite. The bottom panel shows model performance averaged across all environments on both Act2Answer and LIBERO (Liu et al., 2023) (averaging details are provided in Appendix D).
the world - what objects are, how they are typically used, and which behaviors are appropriate in a given situation - tightly linking low-level motor control with commonsense, world-level reasoning. Vision–Language–Action (VLA) models are widely proposed as the foundation for such agents and are often advertised as open-worldgeneralizable (Black et al., 2024; Kim et al., 2024; Yang et al., 2025; Qu et al., 2025; Patratskiy et al., 2025), so we naturally expect them to handle previously unseen environments and objects with at
Introduction
Embodied agents are increasingly studied as candidates for deployment in everyday environments, such as households (Shukla et al., 2024; Mees et al., 2022; Liu et al., 2023) and retail (Soshin et al., 2025; Liu et al., 2025) settings. To operate effectively in such contexts, their actions must be grounded in a rich semantic understanding of 1
least a basic level of appropriate generalization. Yet, the rapidly expanding VLA literature has largely centered on manipulation-centric success. Current benchmarks (Liu et al., 2023; Mees et al., 2022; Shukla et al., 2024; Soshin et al., 2025; Li et al., 2024a; Zhang et al., 2024) ask whether agents can complete complex tasks under perturbations, domain shifts, or new layouts, but they rarely ask whether agents remain able to act on even basic commonsense distinctions about objects, scenes, and goals after robotics training. Many VLA models are obtained by fine-tuning strong Vision-Language-Models (VLM) backbones on control tasks, and it is often implicitly assumed that the underlying world knowledge is preserved, even though we lack systematic ways to measure how much commonsense and factual knowledge is kept or catastrophically forgotten. In contrast, the VLM community has developed a rich ecosystem of benchmarks that explicitly probe world knowledge and commonsense. VLA evaluation, however, remains almost entirely task-success–centric: once a VLM backbone is fine-tuned into a policy that outputs actions, performance is usually reduced to success rates in manipulation or navigation domains, with little attention to whether the original knowledge is still present and accessible. As a result, we currently lack a principled way to test what a VLA model still knows after robotics fine-tuning. We introduce ACT 2A NSWER, an embodied evaluation protocol that adapts VLM knowledge benchmarks to VLA models by requiring action-based answer selection instead of text generation. Each question becomes a short simulated episode with a simple selection action, reducing confounds from long-horizon planning and low-level control. Prior action-based semantic evaluations (Zitkovich et al., 2023; Kim et al., 2024) are limited in scope; in contrast, ACT 2A NSWER enables systematic benchmark adaptation across diverse commonsense and world-knowledge categories and supports layerwise analyses that localize answer-relevant information within VLA models. Specifically, our key contributions are as follows:
VLAs text decoding on standard VLM QA benchmarks. 2. By adapting existing VLM benchmarks, we curate a diverse embodied benchmark suite for systematically evaluating commonsense and world knowledge in VLA models. In total, we collect 1,720 unique binary questions across 12 categories, including attribute, state, color, symmetry, shape, emotion, celebrity, living world, counting, time, traffic, public info. 3. We present a large-scale empirical study of 7 modern VLA systems and 9 strong VLM baselines, systematically ranking models across knowledge categories (Figure 1). We find that current VLA models perform strongly on simple perceptual categories but exhibit substantially larger gaps on richer semantic categories relative to their source VLMs, and that VQA co-training is associated with stronger performance on knowledge-sensitive tasks. 4. We introduce layerwise intent probing: linear classifiers trained on per-layer representations to predict the correct answer for each episode, to quantify how answer-relevant information is distributed across model depth. We show that relevant information can remain internally represented even when the model fails to translate it into the correct action.
2
Related Work
2.1
Evaluation of VLA Models
Current VLA benchmarks primarily evaluate manipulation success, emphasizing control generalization across tasks, scenes, embodiments, and language variations rather than explicit knowledge assessment. Benchmarks such as LIBERO (Liu et al., 2023), CALVIN (Mees et al., 2022), VLABench (Zhang et al., 2024), RoboBenchMart (Soshin et al., 2025), and BEHAVIOR-1K (Li et al., 2024a) measure whether agents can complete language-conditioned tasks, while MIKASARobo (Cherepanov et al., 2025), (Pugacheva et al., 2025), and VL-Think (Kachaev et al., 2025) probe memory, robustness, and transfer. However, across these settings, evaluation remains largely grounded in manipulation success and mostly tests only shallow semantics or primitive concepts, leaving commonsense and world knowledge largely unmeasured.
1. We propose ACT 2A NSWER, an embodied evaluation benchmark suite that adapts VLM knowledge tasks into action-based simulated episodes, providing a controlled protocol for probing knowledge-sensitive behavior in VLAs through action rather than assessing 2
Recent works have increasingly considered knowledge transfer from VLMs to VLA models. A common evaluation strategy, used in works such as (Cai et al., 2026a) and (Chen et al., 2025), is to test the VLM component on VQA-style benchmarks by decoding textual answers. This approach remains indirect: it measures whether the VLM component can still produce the correct textual answer, but not whether the VLA can use that knowledge when choosing an action. Consistent with this, Zhang et al. (2026) show that strong VQA performance of VLM models does not necessarily translate into stronger VLA embodied behavior. In contrast, our Act2Answer protocol brings knowledge-sensitive evaluation into an embodied action setting, making failures more directly informative about missing task-relevant knowledge or the model’s inability to use it for correct action selection in context, rather than merely indicating whether the same information can still be verbalized by the underlying VLM: instead of answering in text, the agent must express its choice through action.
tors. An agent must perceive the scene correctly, know which action is appropriate, execute that action reliably, and operate under the constraints of a particular environment. We use these four components: perception, knowledge, control, and environment - as a coarse conceptual decomposition of embodied task success. This decomposition matters because the same observed success rate may arise from very different underlying causes: strong knowledge but weak control, weak knowledge but strong motor routines, or even shortcut exploitation and favorable dynamics. As a result, end-to-end task success is often not diagnostic of what a VLA actually knows. Current benchmarks typically collapse these factors into a single outcome, making it difficult to tell whether failure reflects missing knowledge, perceptual error, motor difficulty, or environmental complexity. If the goal is to study knowledge in embodied agents, evaluation must therefore move beyond undifferentiated task success and more carefully isolate the contribution of knowledge from other sources of performance.
2.2
3.2
Knowledge and Commonsense in VLM
What matters for embodied agents is not knowledge in the abstract, the question is what kinds of knowledge should actually be evaluated in embodied agents. Prior work such as Cosmos-Reason1 (Azzolini et al., 2025) has highlighted the importance of physically grounded reasoning over space, time, and causal constraints. However, real-world decision making often depends on a broader range of knowledge, including social roles, norms, quantities, biological constraints, and cultural context. In this work, we use the term commonsense knowledge to refer to knowledge of this kind that can affect which action is appropriate in a given situation. Our goal is not to propose a universal taxonomy of embodied knowledge, but to introduce a practical set of knowledge categories that helps structure benchmark design and analysis. These categories are used to guide task selection, improve coverage across different types of commonsense knowledge, and support finer-grained error analysis. Because many embodied decisions are compositional, individual tasks may involve multiple factors at once. The proposed categories are therefore intended as a practical framework for organizing evaluation, rather than as a strict partition of embodied knowledge. Following this principle, we group the knowledge categories covered by our benchmark into seven broad domains:
The VLM community has developed a broad set of benchmarks for evaluating multimodal understanding. Examples include GQA (Hudson and Manning, 2019) for compositional visual reasoning, TextVQA (Singh et al., 2019) and DocVQA (Mathew et al., 2021) for question answering that requires reading text in images, AI2D (Kembhavi et al., 2016) for diagram understanding, ScienceQA (Lu et al., 2022) for multimodal science questions and MMMU (Yue et al., 2024) for large-scale, multidisciplinary multimodal evaluation. These benchmarks have become standard tools for assessing the knowledge and reasoning capabilities of VLMs. It is therefore essential to use such established benchmarks when evaluating knowledge in VLA systems. Our Act2Answer protocol enables this by converting suitable items into an embodied binary-decision format, making it possible to test/probe knowledge-sensitive behavior in an action-based setting while retaining the benchmark grounding provided by existing VLM evaluations.
3
Methodology
3.1
Decomposing Embodied Task Success
Commonsense Knowledge
Task success in embodied settings is not a single ability, but the outcome of several interacting fac3
Figure 2: Act2Answer episodes examples for testing VLA models, built on top of VLM benchmark questions. In each episode, the embodied agent must interpret a natural-language instruction and control the robot arm to move the cube onto the correct answer plate.
Physical world knowledge covers properties of objects and materials, object states and state changes, object identity under occlusion or motion, spatial relations, affordances, intuitive mechanics, visibility and optics, thermodynamic processes, and physical causality or plausibility. These categories capture whether an agent can understand what objects are, what can be done with them, and what physical outcomes are possible. Temporal knowledge covers action semantics, event segmentation, temporal order, duration, delayed effects, goal–subgoal structure, and shorthorizon prediction or planning. These categories capture whether an agent can interpret what is happening over time and choose an action that is appropriate not only for the current frame, but for the unfolding event. Quantitative knowledge covers counting, magnitude comparison, measurement, rates or proportions, and basic resource accounting such as available space, remaining time, or sufficient quantity. Many embodied decisions are not purely semantic: they depend on whether there is enough room, enough material, or the right relative amount to make an action succeed safely and correctly. Biological knowledge covers distinctions between living and non-living entities, bodily vulnerability, injury risk, food safety, allergy or toxicity,
animal and plant needs, and age- or developmentdependent constraints. Such knowledge is often necessary for acting appropriately around humans, animals, and biologically meaningful objects. Social knowledge covers agent roles, emotions, intentions, beliefs, relationships, communication, joint attention, cooperation, and conflict. In embodied settings, correct action often depends not only on physical state, but also on who the agents are, what they know, what they want, and how they are interacting. Normative knowledge covers moral and conventional norms, safety rules, institutional rules, role-based obligations, contextual appropriateness, customs, symbols, and culture-dependent interpretation. These categories matter whenever multiple physically possible actions exist, but only some of them are socially acceptable, safe, or contextually appropriate. Cultural knowledge covers shared cultural references and identities, such as well-known public figures, symbols, and conventions whose interpretation depends on broadly shared world knowledge rather than on the immediate physical or social scene. Such knowledge is what lets an agent recognize culturally salient entities and reason about them appropriately. 4
Figure 3: Overview of the data curation pipeline used to construct the Act2Answer task suite from VLM benchmarks, including selection, filtering and normalization, and conversion
4
Act2Answer: Embodied Evaluation of Knowledge
images are placed at known positions in the scene, and the textual instruction is provided to the agent in the standard way. The agent must indicate its answer through a minimal embodied response by moving its end-effector and placing a cube on the selected image. The prediction is scored as correct if the chosen image matches the correct answer. The protocol is designed to reduce the extent to which success is dominated by control difficulty or incidental environmental variation, and thereby make the outcome more directly informative about knowledge. ACT 2A NSWER is constructed so that success depends primarily on whether the agent can perceive, interpret the instruction, and use the required knowledge to select the correct answer, while minimizing the contribution of motor complexity and other low-level execution challenges. To this end, we deliberately reduce motor complexity: the required action is short-horizon, physically simple, and does not depend on difficult grasping or specialized manipulation skills. The resulting evaluation therefore does not fully isolate knowledge, but is intended to make the observed outcome more directly informative about whether the relevant knowledge remains available for action, rather than whether the agent can solve a difficult control problem.
The problem of knowledge evaluation in VLA models calls for a more direct way to assess actionrelevant world knowledge. Current benchmarks often entangle knowledge with perception, control, and environmental complexity, while text-based evaluation of the VLM part does not reveal whether that knowledge can still guide action. The key unresolved issue is whether a model can not only possess relevant knowledge, but also act on it. To address this, we introduce ACT 2A NSWER, a simple embodied evaluation protocol designed to evaluate whether an agent not only retains relevant world knowledge, but can also use it when selecting an action in a physically grounded setting. A useful parallel comes from cognitive science, where knowledge in nonverbal agents is often evaluated through knowledge-conditioned actions rather than verbal reports. Inspired by this paradigm, ACT 2A NSWER replaces textual answering with a minimal action that reveals the model’s choice. 4.1
Evaluation Protocol
The protocol is intentionally simple. Each episode is built from a VLM-style question together with one or more candidate images, such as a pair or a small grid, corresponding to possible answers (Figure 2 shows representative episodes). These VLM Benchmark
Domain
MLLM-CompBench IconQA MMBench OK-VQA VL-Think
Emotion, Attribute, State Time, Shape, Symmetry, Counting Celebrity Living World Public Info, Traffic, Color
4.2
Tasks Description
To evaluate VLA performance on the proposed knowledge-sensitive categories, we construct the ACT 2A NSWER task suite by adapting existing, community-established VLM benchmarks to an embodied setting rather than creating new question sets from scratch. This choice allows us to ground evaluation in widely used and already validated benchmark sources, while probing whether related knowledge-sensitive distinctions remain available for action after adaptation from VLM to VLA. For
Table 1: VLM benchmarks and knowledge domains used for adaptation to the Act2Answer protocol.
5
tile within a tolerance region ϵ around it (Figure 3). To formally define this, let p ∈ R2 be the final 2D position of the cube at the end of an episode. We define a tolerance radius ϵ around the target image center p+ and the incorrect option image center p− to partition the workspace W into target (Z + ), incorrect (Z − ), and out-of-bounds (OOB) (Z ∅ ) regions as follows:
each broad knowledge category, we select a representative subcategory, Table 1 summarizes the correspondence between the source VLM benchmarks and the knowledge categories used in our evaluation. Each benchmark item is then converted into a standardized ACT 2A NSWER episode: the original question is reformulated as an instruction to the agent, and the candidate answers are instantiated as visual options placed in the scene. The agent must answer by performing the same minimal action defined in our protocol, namely selecting the correct option through object placement. 4.3
Z + = {p ∈ W : ∥p − p+ ∥ ≤ ϵ},
Data Curation
(2)
Z ∅ = W \ (Z + ∪ Z − ).
(3)
= {p ∈ W : ∥p − p ∥ ≤ ϵ},
Then the Soft Success Rate (SR) over N binarychoice tasks is the empirical estimate of the probability of landing in the target region:
Starting from a diverse pool of existing VLM benchmarks, we first select tasks that match the selected knowledge categories. Because current VLA models remain limited in long-context instruction following, we further filter the selected items by instruction length. We also apply imagelevel filtering: since many VLA models operate on relatively low visual resolution, human annotators remove examples in which the relevant objects are too small or visually ambiguous to be reliably perceived. After this selection stage, we unify the remaining tasks into a common embodied format. The original benchmark items may appear as openended questions or multiple-choice problems, so we convert them into binary decision tasks that are more suitable for action-based evaluation. To do so, we use an LLM to rewrite each example into a standardized two-option question while preserving its underlying knowledge requirement. Finally, these curated tasks are wrapped into an embodied environment built on the Simpler (Li et al., 2024b), where each instance can be executed under the ACT 2A NSWER protocol. The result is a standardized task suite for evaluating whether VLA models retain and use Commonsense knowledge. In total, the suite covers 12 categories and contains 1,720 unique binary-choice items, corresponding to 3,440 evaluation episodes after including both original and swapped left/right configurations. Figure 3 illustrates the data curation pipeline. For more details see Appendix B 4.4
(1)
−
Z
−
N
1 X (i) SR = I p ∈ Z+ . N
(4)
i=1
Based on the probability mass distribution among these regions, performance can be interpreted in terms of three regimes defined around the random-guessing baseline of 0.5. A score is treated as statistically distinguishable from chance only if it lies outside a chance-level interval of half-width ∆, defined from binomial sampling fluctuations for the corresponding category. Details and categoryspecific values of ∆ are given in Appendix E. 1. Instruction or perceptual failure (SR < 0.5 − ∆): The model fails to ground the visual options or the instruction. This manifests either as a dominant probability of outof-bounds placement that dwarfs the valid regions (P(p ∈ Z ∅ ) ≫ max{P(p ∈ Z + ), P(p ∈ Z − )}), or as a systematic semantic misunderstanding where the model consistently favors the incorrect option (P(p ∈ Z − ) ≫ P(p ∈ Z + )). In both cases, the probability mass in the target zone drops significantly below random chance. 2. No reliable usable knowledge (|SR − 0.5| ≤ ∆): The model correctly grounds the actionable regions (P(p ∈ Z ∅ ) ≈ 0) but lacks the specific knowledge required to select the correct option. This results in a near-uniform random guess or a severe positional bias between the two candidate regions, yielding P(p ∈ Z + ) ≈ P(p ∈ Z − ) ≈ 0.5 when evaluated across swapped spatial configurations.
Soft Success Rate
To support consistent interpretation of VLA performance in ACT 2A NSWER, we define a simple interpretation methodology based on Soft Success Rate (SR). An episode is counted as successful if the agent places the cube on the correct answer 6
Table 2: Results across knowledge-sensitive categories. VLAs (bottom) answer by embodied action selection under ACT 2A NSWER, VLM baselines (top) use the action-free text probe (RQ3). Model
Social
Physical
Quantitative Temporal
Emotion Attribute State Color Shape Symmetry
Counting
Time
Normative
Cultural
Biological
Public Info Traffic Celebrity Living World
InternVL3.5-8B InternVL3.5-38B Ovis2.5-9B Qwen2.5-7B Qwen2.5-32B Qwen3-8B Qwen3-32B Prismatic-VLM-7B Paligemma-3B
95% 99% 89% 89% 99% 86% 92% 82% 53%
68% 73% 69% 64% 69% 68% 67% 59% 50%
64% 68% 69% 68% 69% 67% 66% 61% 52%
100% 100% 100% 100% 100% 100% 100% 96% 47%
89% 96% 98% 90% 93% 97% 98% 85% 48%
69% 83% 83% 78% 83% 81% 83% 67% 49%
52% 59% 59% 62% 61% 65% 59% 52% 49%
99% 100% 99% 99% 99% 98% 100% 96% 51%
85% 94% 88% 80% 85% 83% 87% 75% 48%
75% 81% 85% 86% 86% 93% 90% 76% 49%
99% 100% 100% 100% 100% 100% 100% 99% 48%
91% 96% 97% 94% 96% 95% 86% 82% 49%
OpenVLA OpenVLA (SFT) OpenVLA (RL) SpatialVLA π0 Magma Xiaomi-Robotics-R0 InternVLA-M1
48% 41% 46% 47% 51% 72% 63% 53%
51% 45% 50% 48% 50% 63% 52% 49%
49% 44% 47% 50% 48% 59% 50% 53%
89% 82% 88% 87% 86% 89% 91% 90%
64% 53% 61% 83% 49% 81% 82% 66%
45% 38% 44% 45% 46% 37% 58% 43%
48% 46% 50% 52% 50% 51% 48% 48%
49% 37% 48% 46% 48% 77% 52% 49%
49% 47% 46% 51% 46% 88% 64% 54%
46% 46% 48% 57% 48% 80% 57% 53%
50% 42% 47% 55% 38% 94% 68% 52%
52% 45% 52% 49% 45% 77% 56% 58%
Formally, let sbb n denote probe accuracy at backbone layer n, and let sexp n denote probe accuracy at Action Expert layer n. Since our main interest is in above-chance recoverable signal, we summarize this relationship using chance-normalized quantities. First, we define the Chance-Normalized Retention as
3. Evidence of usable knowledge (SR > 0.5 + ∆): The model successfully grounds the actionable regions and correctly leverages its internal semantic knowledge to skew the action distribution toward the target zone, such that P(p ∈ Z + ) > P(p ∈ Z − ). Finally, to reduce positional bias, we evaluate each example in both its original and swapped left/right versions and report the average score across the two. This makes the metric more robust to systematic side preferences. 4.5
maxn (sexp n − c) Retention = , bb maxn (sn − c) + ε where c is the chance-level accuracy and ε is a small constant for numerical stability. This metric compares the strongest above-chance probing signal in the Action Expert to the strongest abovechance probing signal in the backbone.
Linear Intent Probing
Beyond task success, we measure whether answerrelevant information is linearly recoverable from a model’s internal representations using layerwise intent probing. For each episode, we define a label y ∈ {0, 1} indicating the correct answer option, i.e., the option region corresponding to the groundtruth answer rather than the model’s own selection. For a given VLA model, we extract hidden states from all tokens of every transformer layer, including both the VLM and Action Expert parts. Since the Action Expert can be iterative (for example, 10 steps of flow-matching), we extract activations from every iteration. For each VLA, we then train a linear probe independently on the activations of each layer to predict y and report probe accuracy as a function of layer number. For Action Expert layers, we perform this procedure independently for all iterations. Rather than focusing on the absolute probe accuracies of the VLM and Action Expert parts in isolation, we focus on their relative relationship, which serves as an implicit indicator of how much answer-relevant information becomes attenuated along the path to action selection.
4.6
Evaluation and Results
Our goal is not to establish a new leaderboard, but to obtain an initial picture of current VLA performance on knowledge-sensitive categories under a controlled Act2Answer protocol. We evaluate several popular VLAs, including π0 (Black et al., 2024), OpenVLA (Kim et al., 2024), Magma (Yang et al., 2025), Xiaomi-RoboticsR0 (Cai et al., 2026b), InternVLA-M1 (Chen et al., 2025), SmolVLA (Shukor et al., 2025) and SpatialVLA (Qu et al., 2025), and compare them to strong VLM baselines such as Qwen2.5-VL, Ovis, PaliGemma, and InternVL (Table 2). Unless noted otherwise, we evaluate all VLA models using their original released checkpoints, without additional task-specific fine-tuning, the only exception is a separate set of supplementary ablations for OpenVLA. We then analyze the results to answer a set of research questions about what kinds of knowledge current VLAs preserve, lose, and remain able to use in action. 7
RQ1: How well do current VLA models handle simple primitives? We begin by testing whether VLA models can solve tasks built around basic perceptual concepts such as C OLOR and S HAPE. These categories serve as a useful lower bound: if models fail here, it would suggest a broad inability to use even simple visual distinctions in action. In practice (Table 2), nearly all evaluated models perform strongly on such tasks, with high success rates across primitive categories, although π0 is a notable exception on S HAPE, where its success rate is close to chance. This indicates that simple basic physical and perceptual knowledge remains behaviorally accessible in current VLA systems.
Figure 4: Probing results for internal representations of VLA models on four tasks from the Act2Answer task suite. In the legend, Prefix labels indicate representations from the VLM component, whereas Action labels indicate representations from the Action component.
RQ2: Can VLA models handle more complex semantic concepts? We next turn to categories that place greater demands on abstract semantic interpretation than primitive visual matching. Across the full set of non-primitive categories, current VLAs mostly remain at or near the random threshold, suggesting that once correct behavior depends on richer semantic, quantitative, temporal, normative, cultural, or biological distinctions, performance becomes highly unstable (Table 2). On E MOTION, ATTRIBUTE, and S TATE, T IME, nearly all models are near chance, with Magma as the only clear exception. Particularly strikingly, no evaluated VLA reaches above-random performance on S YMMETRY or C OUNTING, suggesting that these categories remain uniformly challenging for all tested VLA models. Normative, cultural, and biological categories show the same overall pattern. Most models remain close to chance on P UBLIC I NFO, T RAFFIC, C ELEBRITY, and L IVING W ORLD , whereas Magma again stands out with a large margin above threshold on all of them. Outside of Magma, gains are sparse and categoryspecific, such as SpatialVLA on T RAFFIC (57%) and C ELEBRITY (55%), or InternVLA-M1 on L IVING W ORLD (58%). Overall, these results suggest that current VLAs struggle once correct action depends on more than shallow perceptual cues, and that richer semantic information is often not reliably available for action selection in the evaluated models.
first frame, we ask the VLM: “Do you see the <board_name>? Answer yes or no. If yes, specify where: left, center, or right.” A prediction is counted as correct only if both the board identity and its position match the ground truth, yielding a rough estimate of semantic grounding that can be related to Act2Answer performance. The results (Table 2) reveal a substantial gap: across most domains, the original VLMs exceed their VLA counterparts by roughly 20-40 points. This provides evidence consistent with a marked drop in performance on knowledge-sensitive tasks after adaptation from vision-language pretraining to embodied policy learning. RQ4: Where does the knowledge go? A natural follow-up is whether answer-relevant information is truly erased or whether it remains somewhere in the hidden representations but is no longer accessible for action. To study this, we perform linear probing over all layers of VLA models on categories such as ATTRIBUTE, S TATE, E MOTION, and C OUNTING. The results in Figure 4 reveal a consistent pattern: intermediate layers of the VLM backbone are often above chance, suggesting that task-relevant information is still present, but performance declines toward the final layers used for action prediction, often approaching random guessing. This suggests a bottleneck between semantic representation and action generation: the model may retain answer-relevant information in intermediate representations, but fail to reliably translate it into the correct action.
RQ3: How large is the VLM-VLA gap? To obtain an upper-bound estimate of how much performance on knowledge-sensitive tasks may remain available after the transition from VLM to VLA, we compare each VLA to its original VLM checkpoint in an action-free probing setup. Given the 8
Table 3: Averaged probing-based retention metrics by model. VLM and Action report the maximum probing accuracies over backbone and Action Part layers, respectively. Model Magma Xiaomi-Robotics-R0 SpatialVLA OpenVLA SmolVLA π0
general knowledge-sensitive performance.
5
We introduced ACT 2A NSWER, a simple protocol for evaluating knowledge-sensitive behavior in VLA models. In our setup, a VLA operates in a table-top environment with several candidate images and a natural-language prompt, and must answer by performing a minimal embodied action, such as placing a dummy object onto the image it believes to be correct. This design keeps interaction embodied, but makes the choice itself as close as possible to the multiple-choice formats used in VLM benchmarks, so that failures are more directly informative about knowledge-sensitive behavior than about low-level motor difficulty. Our investigation reveals a consistent gap between strong performance on simple perceptual categories and substantially weaker performance on richer semantic categories in current VLA systems. The transition from VLM to VLA tends to preserve low-level visual discrimination (e.g., color, shape, coarse object identity), yet performance drops markedly on higher-level categories. This pattern suggests that current training pipelines often preserve the ability to act on shallow perceptual cues while weakening performance on tasks that require richer task-relevant semantic distinctions. A robot that can reliably grasp a cup but cannot distinguish a “dirty” cup from a “clean” one, or a “sad” human from a “neutral” one, is fundamentally limited in its usefulness as an assistant in everyday environments. These results indicate that simply fine-tuning VLMs on action data is insufficient: the next generation of embodied agents will require architectures and training objectives that better maintain and align the backbone’s actionrelevant semantic understanding with its learned motor policies, rather than allowing stronger control adaptation to come with weaker performance on broader knowledge-sensitive categories.
Probing Accuracy Prefix
Action
Retention
75.23 68.04 65.70 68.71 63.18 64.99
72.60 64.98 62.60 64.61 57.73 55.40
0.8702 0.8159 0.7808 0.7697 0.5809 0.3620
Conclusion
RQ5: Does vision-language supervision improve knowledge-sensitive performance? We also ask whether continued vision-language supervision during VLA training is associated with stronger performance on knowledge-sensitive categories. To examine this, we explicitly include two groups of VLA baselines: models trained with joint vision-language and robotics supervision, such as Magma, Xiaomi-Robotics-R0, and InternVLA-M1, and models trained primarily or almost exclusively on robotics data, such as OpenVLA (Kim et al., 2024), SpatialVLA (Qu et al., 2025), and π0 (Black et al., 2024). The overall trend is positive: models in the first group perform better on average across most categories, especially on higher-level semantic, temporal, normative, cultural, and biological tasks, than models in the second group. While the gains are not uniform in every domain, this pattern suggests that continued vision-language supervision is associated with stronger action-grounded performance on knowledge-sensitive tasks and may help maintain task-relevant semantic information during embodied training. RQ6: How does downstream fine-tuning affect knowledge-sensitive performance? Finally, we study whether additional downstream fine-tuning improves or harms performance on knowledgesensitive categories. Using OpenVLA as a case study, we fine-tune the model with both SFT and RL on a small pick-and-place dataset that is not directly drawn from our benchmark, but does include visual and semantic perturbations. The results do not show consistent improvements. On the contrary, some categories, including S TATE and C OLOR, exhibit noticeable drops after SFT fine-tuning. This suggests that standard downstream adaptation may further bias the model toward task-specific action optimization, sometimes at the expense of more
References Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, and 1 others. 2025. pi*0.6: a vla that learns from experience. arXiv preprint arXiv:2511.14759. Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, and 1
9
others. 2025. Cosmos-reason1: From physical common sense to embodied reasoning. arXiv preprint arXiv:2503.15558.
In Advances in Neural Information Processing Systems, volume 37, pages 28798–28827. Curran Associates, Inc.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, and 5 others. 2024. π0 : A vision-language-action flow model for general robot control. Preprint, arXiv:2410.24164.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2024. Openvla: An open-source vision-language-action model. Preprint, arXiv:2406.09246.
Rui Cai, Jun Guo, Xinze He, Piaopiao Jin, Jie Li, Bingxuan Lin, Futeng Liu, Wei Liu, Fei Ma, Kun Ma, and 1 others. 2026a. Xiaomi-robotics-0: An open-sourced vision-language-action model with real-time execution. arXiv preprint arXiv:2602.12684.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, and 16 others. 2024a. Behavior-1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. Preprint, arXiv:2403.09227.
Rui Cai, Jun Guo, Xinze He, Piaopiao Jin, Jie Li, Bingxuan Lin, Futeng Liu, Wei Liu, Fei Ma, Kun Ma, and 1 others. 2026b. Xiaomi-robotics-0: An open-sourced vision-language-action model with real-time execution. arXiv preprint arXiv:2602.12684.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. 2024b. Evaluating real-world robot manipulation policies in simulation. Preprint, arXiv:2405.05941.
Xinyi Chen, Yilun Chen, Yanwei Fu, Ning Gao, Jiaya Jia, Weiyang Jin, Hao Li, Yao Mu, Jiangmiao Pang, Yu Qiao, and 1 others. 2025. Internvla-m1: A spatially guided vision-language-action framework for generalist robot policy. arXiv preprint arXiv:2510.13778.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. 2023. Libero: Benchmarking knowledge transfer for lifelong robot learning. Preprint, arXiv:2306.03310.
Egor Cherepanov, Nikita Kachaev, Alexey K. Kovalev, and Aleksandr I. Panov. 2025. Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning. Preprint, arXiv:2502.10550.
Weiheng Liu, Yuxuan Wan, Jilong Wang, Yuxuan Kuang, Xuesong Shi, Haoran Li, Dongbin Zhao, Zhizheng Zhang, and He Wang. 2025. Fetchbot: Object fetching in cluttered shelves via zero-shot sim2real. arXiv preprint arXiv:2502.17894.
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision.
Anastasia Ivanova, Bakaeva Eva, Zoya Volovikova, Alexey Kovalev, and Aleksandr Panov. 2025. AmbiK: Dataset of ambiguous tasks in kitchen environment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33216–33241, Vienna, Austria. Association for Computational Linguistics.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, KaiWei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS).
Nikita Kachaev, Mikhail Kolosov, Daniil Zelezetsky, Alexey K. Kovalev, and Aleksandr I. Panov. 2025. Don’t blind your vla: Aligning visual representations for ood generalization. Preprint, arXiv:2510.25616.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2021. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In Advances in Neural Information Processing Systems. Datasets and Benchmarks Track.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. Preprint, arXiv:1603.07396. Jihyung Kil, Zheda Mai, Justin Lee, Arpita Chowdhury, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, and Wei-Lun Chao. 2024. Mllm-compbench: A comparative reasoning benchmark for multimodal llms.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowl-
10
edge. In IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, and Jianfeng Gao. 2025. Magma: A foundation model for multimodal ai agents. Preprint, arXiv:2502.13130.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556– 9567.
Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. 2022. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. Preprint, arXiv:2112.03227.
Jianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, and Jianyu Chen. 2026. Vlm4vla: Revisiting vision-language-models in vision-language-action models. arXiv preprint arXiv:2601.03309.
Maxim A Patratskiy, Alexey K Kovalev, and Aleksandr I Panov. 2025. Spatial traces: Enhancing vla models with spatial-temporal understanding. Optical Memory and Neural Networks, 34(Suppl 1):S72–S82. Alisa Petrova and Alexey Kovalev. 2026. Steering large language models toward clarification through sparse autoencoders. In Agentic AI in the Wild: From Hallucinations to Reliable Autonomy.
Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2024. Vlabench: A large-scale benchmark for languageconditioned robotics manipulation with long-horizon reasoning tasks. Preprint, arXiv:2412.18194.
Daria Pugacheva, Andrey Moskalenko, Denis Shepelev, Andrey Kuznetsov, Vlad Shakhuro, and Elena Tutubalina. 2025. Bring the apple, not the sofa: Impact of irrelevant context in embodied ai commands on vla models. Preprint, arXiv:2510.07067.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, and 1 others. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR.
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, and Xuelong Li. 2025. Spatialvla: Exploring spatial representations for visual-languageaction model. Preprint, arXiv:2501.15830. Arth Shukla, Stone Tao, and Hao Su. 2024. Maniskillhab: A benchmark for low-level manipulation in home rearrangement tasks. arXiv preprint arXiv:2412.13211. Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, Simon Alibert, Matthieu Cord, Thomas Wolf, and Remi Cadene. 2025. Smolvla: A vision-language-action model for affordable and efficient robotics. Preprint, arXiv:2506.01844. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326. Konstantin Soshin, Alexander Krapukhin, Andrei Spiridonov, Denis Shepelev, Gregorii Bukhtuev, Andrey Kuznetsov, and Vlad Shakhuro. 2025. Robobenchmart: Benchmarking robots in retail environment. arXiv preprint arXiv:2511.10276. Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai,
11
A
Evaluation and Setup Ablations
We also evaluate an alternative prompt that is closer to the VLA instruction format:
This appendix reports ablations that test whether the main ACT 2A NSWER conclusions are sensitive to evaluation choices rather than knowledgesensitive action selection. We consider two ablations for the VLM comparison, image resolution and prompt formulation, and three ablations for the embodied evaluation setup, texture rendering, answer-tile size, and lighting intensity. These experiments help verify that the central findings are not driven by a single prompt, image preprocessing choice, or simulator rendering condition. A.1
Action-Style Prompt (p2) <image> Put cube on {object} A left B right Answer with the option’s letter from the given choices directly.
As shown in Table 4, the action-style prompt slightly reduces VLM performance for some models and categories. This effect is most visible for Qwen3-8B and Qwen2.5-7B, but the overall comparison remains qualitatively similar across prompt styles. Nevertheless, VLM performance under the action-style prompt remains substantially above most VLA results in many knowledge-sensitive categories.
Effect of Image Resolution
The VLM–VLA comparison in the main paper is intended as an action-free estimate of how much task-relevant information remains accessible to the VLM backbone. To check whether this comparison is dominated by image preprocessing, we evaluate VLM baselines under two image resolutions: 224 × 224 and 560 × 480. As shown in Table 4, changing image resolution slightly affects some categories and models, especially fine-grained categories such as ATTRIBUTE, S TATE, and S YM METRY . However, the qualitative interpretation remains unchanged: strong VLM baselines continue to perform well on many categories where most VLA policies remain near chance, suggesting that the VLM–VLA gap is not solely an artifact of the chosen image resolution.
A.3
Robustness to Texture Rendering
<image> Do you see the {object}? If yes, specify where: “left” or “right”.
We next evaluate whether the visual appearance of the simulated environment changes the main conclusions. In the Raw Sim setting, models are evaluated with the default simulator rendering, using the original simulated textures, materials, and backgrounds. In the Visual Matching setting (Li et al., 2024b), we follow the general visual-gap reduction strategy used in Simpler: simulated observations are made closer to real-world robot scenes by combining real-background compositing with foreground texture tuning for salient assets. This includes matching or baking object textures from real images where possible and adjusting robot or scene textures that otherwise create a noticeable real-to-sim appearance gap. Table 5 shows that this change has only limited effect on the selected categories. The same category-level pattern is preserved: C OLOR and S HAPE remain easier than E MOTION and ATTRIBUTE, and Magma remains stronger than OpenVLA and π0 on the tested semantic categories.
A left B right
A.4
A.2
Influence of Prompt Formulation
We use a standardized prompt template to evaluate the vision-language model (VLM) in a constrained multiple-choice setting. The prompt consists of three components: (i) an image placeholder, (ii) a natural language question, and (iii) a set of answer options. Perceptual Prompt (p1)
Effect of Answer-Tile Size
We also test whether performance is sensitive to the physical size of the answer tiles. This ablation is important because tile size may act as an out-ofdistribution shift for VLA models. As shown in Table 5, moderate changes preserve the main qualitative trends. Increasing tile size can reduce performance for some models and categories, especially
Answer with the option’s letter from the given choices directly.
Answer Format. The model is instructed to output only the corresponding option letter (e.g., “A” or “B”), which ensures consistent and easily comparable predictions. 12
Table 4: VLM ablations over image resolution and prompt formulation. Prompt p1 is the perceptual QA-style prompt, and p2 is the VLA-style action prompt. These ablations test whether the VLM–VLA comparison is sensitive to image preprocessing or prompt wording. Model
Resolution Prompt
Social
Physical
Quantitative Temporal
Emotion Attribute State Color Shape Symmetry
Counting
Time
Normative
Cultural
Biological
Public Info Traffic Celebrity Living World
InternVL3.5-8B Ovis2.5-9B Qwen2.5-7B Qwen3-8B
224×224
p1
95% 89% 89% 86%
68% 69% 64% 68%
64% 69% 68% 67%
100% 100% 100% 100%
89% 98% 90% 97%
69% 83% 78% 81%
52% 59% 62% 65%
99% 99% 99% 98%
85% 88% 80% 83%
75% 85% 86% 93%
99% 100% 100% 100%
91% 97% 94% 95%
InternVL3.5-8B Ovis2.5-9B Qwen2.5-7B Qwen3-8B
560×480
p1
97% 92% 72% 78%
62% 65% 58% 60%
57% 67% 60% 54%
99% 96% 98% 100%
89% 97% 92% 97%
63% 82% 70% 74%
55% 57% 57% 57%
97% 98% 98% 99%
80% 80% 73% 82%
72% 88% 77% 90%
99% 100% 99% 99%
91% 93% 85% 84%
InternVL3.5-8B Ovis2.5-9B Qwen2.5-7B Qwen3-8B
224×224
p2
88% 73% 78% 55%
69% 65% 57% 62%
61% 67% 56% 58%
100% 100% 94% 100%
79% 97% 79% 88%
73% 70% 63% 50%
57% 57% 55% 60%
100% 100% 85% 98%
80% 87% 74% 80%
70% 83% 67% 80%
100% 100% 73% 100%
90% 96% 88% 95%
Table 5: Environment setup ablations on representative categories. Abl.
Setting
Texture
Tile Size
Lighting
Model
mains comparatively accessible, and Magma remains the strongest of the tested models on these categories. These results suggest that the evaluation setup is not tied to a single lighting condition, while also confirming that perceptual rendering choices can affect some categories and should be controlled in future evaluations.
Emo. Attr. Color Shape
OpenVLA 48% 52% 86% 50% 48% 86% Raw Sim π0 Magma 71% 63% 90%
62% 51% 79%
51% 89% Visual OpenVLA 48% 51% 50% 86% 0 Matching π Magma 72% 63% 89%
64% 49% 81%
default
OpenVLA 48% 51% 89% 51% 50% 86% π0 Magma 72% 63% 89%
64% 49% 81%
+40%
OpenVLA 51% 50% 71% π0 53% 48% 67% Magma 70% 59% 92%
56% 52% 78%
-20%
OpenVLA 52% 49% 86% π0 52% 50% 85% 69% 60% 89% Magma
65% 52% 77%
default
OpenVLA 48% 51% 89% π0 51% 50% 86% 72% 63% 89% Magma
64% 49% 81%
+30%
OpenVLA 51% 48% 86% 47% 50% 78% π0 Magma 71% 60% 91%
59% 50% 81%
-30%
OpenVLA 49% 47% 90% π0 50% 49% 83% Magma 64% 58% 89%
54% 50% 80%
B
Act2Answer is constructed from five source benchmarks: MLLM-CompBench (Kil et al., 2024), IconQA (Lu et al., 2021), MMBench (Liu et al., 2024), OK-VQA (Marino et al., 2019), and VLThink (Kachaev et al., 2025). We selected these benchmarks because together they provide established and complementary coverage of the knowledge domains targeted by Act2Answer. Our goal was to map these heterogeneous source benchmarks into a single evaluation setup: a short actioncompatible instruction, two visual answer options, and a common embodied binary action-selection protocol. The degree of adaptation therefore differs by source benchmark and is driven by source-format mismatch rather than arbitrary redesign. In practice, we prioritized source tasks that could be converted into embodied binary choice without substantially changing what the original task was meant to test. We further restricted the final pool to examples whose relevant visual evidence remains perceivable under the lower effective resolution typical of current VLA systems and whose answer can be expressed through a short action-conditioned instruction. Representative examples of the resulting embodied tasks are shown in Figures 5, 6, and 7. For readability, we group them into three panels: (i) so-
OpenVLA and π0 on C OLOR, but the broader pattern remains stable: simple perceptual categories are still easier than the more semantic categories, and the relative ranking among the tested models is largely unchanged. A.5
Benchmark Sources and Data Construction
Effect of Lighting Intensity
Finally, we vary lighting intensity to test whether results are driven by a narrow rendering condition. Table 5 shows that lighting perturbations can cause localized changes, especially under darker lighting and in harder categories. However, the main conclusions are preserved: E MOTION and ATTRIBUTE remain difficult for OpenVLA and π0 , C OLOR re13
Figure 5: Additional ACT 2A NSWER environment examples from the E MOTION, C ELEBRITY, and L IVING W ORLD categories.
Figure 6: Additional ACT 2A NSWER environment examples from the T IME, T RAFFIC, and P UBLIC INFO categories.
cial, biological, and culturally grounded categories, (ii) temporal and public-convention categories, and (iii) physical and quantitative categories.
14
Figure 7: Additional ACT 2A NSWER environment examples from the ATTRIBUTE, S TATE, C OLOR, S YMMETRY, S HAPE, and C OUNTING categories.
B.1
Near Format-Preserving Adaptations
retained the original image pair and converted the question into a short action-compatible instruction. Because almost no additional restructuring was required, these categories provide one of the most direct comparisons between VLM and VLA behavior in our study.
MLLM-CompBench. MLLM-CompBench was the cleanest source benchmark for Act2Answer. Its native format already consists of two candidate images and a comparative question, making it closely aligned with our final embodied answer-selection setup. We therefore used it as the primary source for E MOTION, ATTRIBUTE, and S TATE. Adaptation in this case was near format-preserving: we
VL-Think. VL-Think was also a natural fit for our protocol because its task structure is already close to an embodied semantic selection problem. We 15
Source benchmark
Domain
Native format
Transfer type
Initial pool
Final eval
MLLM-CompBench
Emotion Attribute State
image pair + comparative question
near
5254 5386 1066
300 300 300
IconQA
Time Shape Symmetry Counting
multi-image choice
moderate
508 12798 362 5001
300 300 300 300
MMBench
Celebrity
multiple choice
moderate
316
140
OK-VQA
Living World
open-ended VQA
heavy
2336
300
VL-Think
Public Info Traffic Color
embodied board selection
near
14 concepts 24 concepts 11 concepts
300 300 300
Table 6: Per-category curation statistics for the ACT 2A NSWER suite. ‘Final eval’ reports the total number of evaluation episodes after including both the original and swapped left/right configurations for each selected item. For VL-Think, the ‘Initial pool’ column reports the number of source concepts rather than the number of candidate image examples.
used it for P UBLIC I NFO, T RAFFIC, and C OLOR, where the relevant concepts are encoded in compact visual symbols or simple public conventions. Since the source benchmark already operates in a board-selection style setting, adaptation remained near format-preserving and mainly consisted of unifying the instruction style and episode format with the rest of the Act2Answer suite. Because VL-Think is specified in terms of small closed concept vocabularies rather than large candidate image pools, the corresponding ‘Initial pool’ entries in Table 6 report the number of source concepts rather than the number of raw image examples. B.2
tion. We kept the same identity-recognition task, but re-curated the set of public figures to focus on more broadly recognizable identities. The aim was to better target shared world knowledge without changing the underlying recognition problem. The resulting subset is therefore better understood as a curated derivative of MMBench rather than a verbatim extraction. B.3
Open-Ended to Binary Adaptation
OK-VQA. OK-VQA was used for the L IVING W ORLD category, covering animal identity, flora identity, and living-versus-nonliving distinctions. This benchmark required the most benchmark-specific adaptation because its native format is single-image open-ended VQA, whereas Act2Answer evaluates knowledge through binary embodied selection. We first retained only examples with short, visually grounded answers and stable annotator agreement. We then selected biologically relevant cases through question-pattern and answer-level filtering. To convert these items into the common format, each source image was paired with a second image from the same category that did not contain the correct answer. This allowed the same biological knowledge to be tested through embodied binary choice. Ambiguous cases were removed, and the final embodied instructions were edited accordingly. The resulting OKVQA-derived subset should be understood as a benchmark-specific adaptation from open-ended VQA to embodied binary evaluation.
Moderately Adapted Benchmarks
IconQA. IconQA was used for T IME, S HAPE, S YM METRY , and C OUNTING . It required moderate adaptation because its native format is multi-option visual choice rather than binary embodied selection. We therefore converted each item into binary choice by pairing the correct answer with one distractor at a time while preserving the original semantic target. Because IconQA uses schematic icon diagrams rather than cluttered natural images, it remains relatively robust under the lower visual resolution typical of current VLA systems. We retained only those subsets that fit the Act2Answer protocol most naturally, including geometry-based subsets for S HAPE and S YMMETRY, and a visually grounded clock-style subset for T IME. MMBench. We used MMBench for the C ELEBRITY category. This subset required moderate adapta16
B.4
Table 7: Preliminary mitigation ablations on a subset of representative categories.
Instruction Rewriting and Binary Conversion
Our goal was to preserve the original knowledge requirement while changing the response modality from text to action. We therefore used an LLM as a first-pass rewriting tool to convert source benchmark questions into short imperative instructions for VLA models, followed by human review and manual editing. This rewriting was limited to template normalization rather than semantic reformulation. For example, comparative prompts from MLLM-CompBench were converted into instructions of the form “Put the cube on . . . ” while preserving the same comparison target and candidate images. Prompts followed a simple instruction style, e.g., “Put the cube on the more smiling person” (E MOTION), “Put the cube on dryer grass” (S TATE), and “Put the cube on the picture of sheep” (L IVING W ORLD). Throughout the suite, instructions were kept short, visually grounded, and close to the original semantic target.
C
Model π0 π0 + paraphrases π0 + distill
Emotion Attribute Color Shape 51% 47% 49%
50% 48% 50%
86% 89% 88%
49% 53% 56%
varied through action-verb substitutions, spatialexpression variants, sentence-structure changes, descriptive noun replacements, and robot-directed commands. Second, we test a latent-distillation baseline that adds a representation-preservation loss between mid-layer VLA hidden states and final patch embeddings from a frozen vision foundation teacher. Both variants are obtained by fine-tuning π0 on BridgeDataV2 pick-and-place data and do not use ACT 2A NSWER evaluation examples for training. The results suggest that these lightweight interventions are useful but limited. Language rephrasing slightly improves C OLOR and S HAPE, but does not improve the more semantic E MOTION and ATTRIBUTE categories. Latent distillation shows a similar pattern: it improves S HAPE and preserves strong C OLOR performance, but still leaves E MO TION and ATTRIBUTE near chance. This pattern is consistent with the main findings of the paper: simple perceptual distinctions are easier to preserve or recover, whereas richer semantic distinctions require stronger mechanisms than shallow instruction diversity or a single representation-matching objective. At the same time, the distillation result points to a promising direction for future work, especially representation-preservation losses during VLA pretraining, better choices of teacher representations and distillation data, and comparisons between rehearsal, regularization, and parameterefficient adaptation.
Discussion
How to prevent the erosion of world knowledge? Finally, it is not yet clear how to systematically prevent the erosion of world knowledge during VLA training. Developing effective training procedures that retain the backbone’s world knowledge while acquiring strong control policies remains an open challenge. Promising directions include multi-task and continual-learning schemes that interleave action prediction with general question answering (Zitkovich et al., 2023; Amin et al., 2025; Ivanova et al., 2025), regularization or distillation techniques that explicitly preserve VLM representations, and architectural decoupling of knowledge and control components. Another complementary direction is inference-time intervention on interpretable internal features, such as SAEbased steering toward clarification behavior in ambiguous embodied-instruction settings (Petrova and Kovalev, 2026). To make this discussion more concrete, we also ran preliminary mitigation experiments on a small subset of representative ACT 2A NSWER categories, shown in Table 7. These experiments are not intended as full mitigation methods, but as simple probes of two plausible directions. First, we test language-rephrasing augmentation during downstream fine-tuning, where instructions are
However, large-scale multi-task or continual training that keeps many skills active can be prohibitively compute-intensive for real-world systems, especially when combined with highresolution perception and long-horizon control. This makes it essential to explore more efficient paradigms and lightweight mechanisms to prevent forgetting - such as parameter-efficient finetuning, targeted regularization, selective rehearsal, or sparsely updated knowledge modules - so that VLAs can become more capable without repeatedly sacrificing the commonsense they started with. 17
Table 8: Left/right split results for selected VLA models with heatmap formatting. Model
Social Emotion
SpatialVLA Magma Xiaomi-Robotics-R0 InternVLA-M1
D
Physical Attribute
State
Color
Shape
Quantitative
Temporal
Symmetry
Counting
Time
Cultural
Biological
Celebrity
Living World
L
R
L
R
L
R
L
R
L
R
L
L
R
L
R
L
R
L
R
L
R
L
R
58% 68% 60% 54%
55% 65% 46% 50%
42% 61% 58% 49%
52% 62% 54% 59%
48% 56% 46% 47%
80% 91% 86% 93%
95% 87% 95% 88%
82% 81% 84% 80%
85% 86% 79% 52%
90% 0% 50% 46% 28% 52% 46% 70% 45% 13% 74% 50%
55% 52% 50% 47%
47% 77% 65% 40%
48% 78% 39% 58%
68% 90% 69% 83%
35% 86% 59% 26%
61% 89% 63% 87%
53% 71% 51% 20%
66% 94% 70% 54%
44% 94% 65% 50%
67% 81% 62% 54%
32% 73% 51% 63%
configurations), which gives ∆ ≈ 0.057 (about 5.7 percentage points). The smaller C ELEBRITY set uses N = 140 episodes, giving a wider band of ∆ ≈ 0.083. The two swapped views of a single item are not fully independent, so treating each episode as an independent trial is mildly anticonservative. Using the number of unique items as the effective sample size (e.g., N = 150 rather than 300) yields a slightly wider, more conservative margin (∆ ≈ 0.08). Our qualitative conclusions are unchanged under either choice, since the gaps we report between primitive and richer semantic categories are substantially larger than ∆.
For LIBERO, all values for the compared VLA models were taken directly from the original papers. For Act2Answer, we used the results from Table 2. For each model, we first averaged accuracy across all environments, and then normalized this mean relative to random guessing (50%) using a linear rescaling: Normalized Score =
E
Traffic
36% 76% 66% 52%
Details of Score Averaging
R
Normative Public Info
Mean Accuracy − 50 ×100. 50 (5)
Chance Margin ∆ and Significance Thresholds
F
The three interpretation regimes of the Soft Success Rate (Section 4.4) are defined relative to a chance margin ∆ around the random-guessing baseline of 0.5. This appendix defines ∆ and explains how its value is chosen. Each ACT 2A NSWER item is a binary forced choice, so under a knowledge-free policy the perepisode outcome is a Bernoulli trial with success probability p0 = 0.5. For a category evaluated over N episodes, the empirical SR is a sample p proportion whosepstandard error at chance is p0 (1 − p0 )/N = 0.25/N . We set ∆ to the corresponding two-sided Wald confidence halfwidth, r r p0 (1 − p0 ) 0.25 ∆ = z1−α/2 = z1−α/2 , N N (6) so that the band |SR − 0.5| ≤ ∆ contains exactly those scores that are not significantly different from chance at level α. We use α = 0.05 (z1−α/2 = 1.96) throughout. A category result is then counted as evidence of usable knowledge when SR > 0.5 + ∆, as instruction or perceptual failure when SR < 0.5 − ∆, and as no reliable usable knowledge otherwise. Because ∆ depends on the number of evaluated episodes, it is computed per category. Most categories are evaluated over N = 300 episodes (150 items in both the original and swapped left/right
Left/Right Swapped-Layout Analysis
Act2Answer uses a binary tabletop layout, so a model with a fixed spatial preference could obtain misleading scores if each question were evaluated in only one candidate arrangement. To control for this, each item is evaluated in two spatial configurations: the original left/right placement of the candidate answer images and a swapped configuration in which the two candidates exchange positions. The main Act2Answer score averages over these two configurations, so systematic side preferences have less influence on the final category-level result. Table 8 reports the left/right breakdown for selected VLA models. The split is useful because it makes positional effects visible rather than hiding them inside a single averaged score. When a model does not have a reliable knowledge-conditioned signal for a category, performance can vary substantially across the two layouts, indicating either a side preference, a layout-specific failure, or nearrandom action selection. For example, several models show large left/right asymmetries on harder categories such as S YMMETRY, P UBLIC I NFO, T RAF FIC , and L IVING W ORLD . In contrast, categories where performance remains high in both configurations, such as C OLOR for most evaluated models, provide stronger evidence that the model is responding to the intended visual-semantic content rather than simply exploiting position. 18
This analysis supports the role of swappedlayout evaluation as a built-in robustness control in Act2Answer. It does not by itself eliminate all spatial or perceptual confounds, since layout can still interact with model-specific perception and control behavior. However, it reduces the effect of fixed side preferences on the reported score and provides a diagnostic view of when apparent success or failure may be driven by position rather than by knowledge-sensitive action selection.
19