Key-Gram: Extensible World Knowledge for Embodied Manipulation
arXiv:2605.18556v1 [cs.RO] 18 May 2026
1
Jingjing Fan1,∗, Siyuan Li2, ∗ , Botao Ren1 , and Zhidong Deng1,† Department of Computer Science and Technology 2 Department of Automation Tsinghua University [email protected]
Abstract Embodied control increasingly requires models to follow compositional language instructions while reasoning over dynamic visual states. However, current visionlanguage-action policies and world-action models often couple linguistic knowledge with visual computation in a shared backbone or conditioning pathway, leading to modality competition and making knowledge extension dependent on backbone updates. In this paper, we introduce Key-Gram, a conditional-memory framework that separates language-derived world knowledge from visual-state reasoning for embodied control. At its core is a memory module that decomposes an instruction into task-specific key-grams, retrieves static linguistic priors through deterministic hashed lookup, and injects the retrieved entries into selected hidden layers through context-aware gating and lightweight convolutional fusion. This design allows the backbone to devote its main capacity to visual reasoning and action inference, while reusable instruction knowledge is stored in an extensible external memory. The logical memory table can be conveniently partitioned during training and, due to its O(1) lookup pattern, efficiently placed on host memory during inference. Across RoboTwin2.0, LIBERO/LIBERO-Plus, and real-world dual-arm manipulation, Key-Gram consistently improves both π0 and π0.5 backbones, with average relative gains of 29.5%/9.9% on RoboTwin2.0, 35.8%/4.5% on LIBERO-Plus transfer without target-domain fine-tuning, and 15.4%/8.1% on real-world long-horizon tasks. These results demonstrate that externalized linguistic memory provides an effective and extensible mechanism for improving compositional grounding, transfer, and real-world manipulation.
1
Introduction
Embodied manipulation places two fundamentally different demands on a single model: it must preserve reusable world knowledge about objects, skills, relations, constraints, and task structure, while continuously adapting its internal model of how the visual world evolves under action. Recent vision-language-action (VLA) models have made instruction-sensitive control practical by bringing language priors into end-to-end policies [1, 2, 4, 5, 3, 6, 8, 9, 7], while recent World Action Models (WAM) have highlighted the value of forecasting future world states for physical generalization [10–15]. Yet both trends sharpen the same unresolved tension: static world knowledge and online physical reasoning remain tightly entangled inside the policy computation itself. Existing efforts largely approach this tension in two ways. The first is dense fusion, dominant in mainstream VLA formulations, where language and visual observations are projected into a shared token space and processed by the same backbone [1, 2, 4]. This design is simple and scalable, ∗ Equal contribution. † Corresponding author.
Preprint.
but it forces a small number of instruction tokens to compete with a much larger and continuously changing set of visual tokens inside the same dense attention stream. The second is conditional generative modeling, increasingly adopted in diffusion-based robot policies and WAMs, where language is encoded separately and injected as conditioning to guide the generation of future visual states, actions, or both [16, 15]. Community efforts such as FiLM-style modulation, cross-attention, and AdaLN-based conditioning should be understood as local refinements within this broader line rather than as a distinct paradigm of their own [27–30]: they acknowledge that naive fusion is problematic, but they still leave unresolved a deeper question—what role language should play in embodied control. In dense fusion, language is drawn into token-level competition at the input interface; in conditional generative models, it is reduced to a prompt-like steering signal, while compositional world knowledge remains implicitly baked into the generative backbone. In both cases, the architecture still fails to separate reusable knowledge from online scene reasoning, but merely entangles them at different locations. Consequently, when such models adapt to new physical regimes, the gradient updates that improve online reasoning can also overwrite previously acquired world knowledge, making continual adaptation and modular extension inherently fragile [31, 32]. We take a different view. In embodied control, visual computation should primarily reason over scene dynamics: what state the world is currently in, what future should be reached, and how action should evolve accordingly. Language, by contrast, does not primarily specify low-level physical evolution; rather, it serves as a compact and reusable index over abstract task priors. This functional asymmetry suggests that embodied models should separate information by function rather than by modality. Motivated by conditional-memory formulations [23], we introduce Key-Gram, in which the instruction is decomposed into a small set of task-specific key-grams that retrieve an external linguistic memory. The retrieved memory are then injected into the backbone as reusable priors, while its dominant computation remains focused on future-state reasoning and control-relevant physical evolution. More importantly, this design induces a principle of extensibility. When new knowledge is acquired, it can be appended as new entries in the external memory rather than rewritten into backbone weights. The backbone may still update to learn how newly retrieved knowledge should be grounded and used in physical interaction. What no longer needs to change, however, is the previously acquired world knowledge itself. In this way, knowledge growth becomes modular, while existing memory is protected from gradient interference during backbone adaptation, making the architecture naturally compatible with open-world deployment and continual accumulation. Empirically, this functional decoupling brings consistent improvements across simulated and realworld manipulation. Across RoboTwin2.0 [17], LIBERO [18], LIBERO-Plus [19], and real-world manipulation, Key-Gram improves both π0 and π0.5 backbones, with average relative gains of 29.5%/9.9% on RoboTwin2.0, 35.8%/4.5% on LIBERO-Plus transfer without target-domain finetuning, and 15.4%/8.1% on real-world long-horizon tasks. The gains are especially strong in instruction-sensitive settings where similar visual scenes require different linguistic grounding. In real-world expansion tasks, Key-Gram substantially improves unseen compositional pairings, increasing Task 3 by 34.6%/18.8% and Task 4 by 41.7%/21.2%, and also improves sequential adaptation in Task 5 by 10.0%/4.7%. These results suggest that separating language-side worldknowledge retrieval from vision-side physical reasoning provides a more effective inductive bias for embodied intelligence than either dense token fusion or coarse generative conditioning. Our contributions are summarized as follows: • A functionally decoupled embodied framework. We propose Key-Gram, which separates instruction-side world-knowledge retrieval from vision-side physical reasoning, reducing modality competition in embodied control. • An extensible external memory for world knowledge. Key-Gram stores reusable linguistic priors in a structured memory accessed by deterministic hashed lookup, enabling modular knowledge expansion without rewriting the backbone. • Consistent gains in instruction-sensitive manipulation. Experiments across simulated and real-world benchmarks show improved compositional grounding, transfer, and adaptation under diverse linguistic formulations.
2
(a) Pipeline
(b) Key-Gram Conv
put the yellow and white mug in the microwave and close it
Transformer Block Scaled Dot Product
VAE
FFN
action
Input Hidden
Linear
Linear
decompose
Concat
VLM
KV Cache
Extensible World Knowledge
Action Expert
Attention
noise
Transformer Block
Language Retrieve
Vision Reasoning
Key-Gram Embedding
Key-Gram Embedding
Key-Gram Embedding
Key-Gram Embedding
Hash
Hash
Hash
Hash
put mug in microwave
close microwave door
yellow and white mug
both mug inside microwave
Key-Gram
Action Inference put the yellow and white mug in the microwave and close it
Figure 1: Overview of Key-Gram. (a) The framework separates language-derived knowledge retrieval, visual reasoning, and action inference. (b) The Key-Gram module retrieves external linguistic priors from decomposed key-grams via multi-head hashing, and injects them into selected Transformer layers through context-aware gated fusion.
2
Related Work
2.1
Vision-Language-Action Models
VLA models have become a major paradigm for language-conditioned robot control by transferring large-scale vision-language priors into visuomotor policies. Systems such as RT-2, OpenVLA, π0 , π0.5 , and X-VLA show that scaling vision-language and robot data improves semantic grounding, instruction following, cross-embodiment transfer, and long-horizon manipulation [1, 2, 4, 5, 7]. Despite these advances, most VLA models adopt dense multimodal fusion: language, visual, and sometimes action tokens are embedded into a shared token space and processed by a common backbone [1, 2, 4]. This simple recipe creates a structural bottleneck for manipulation, as short instructions must compete with dense visual tokens within the same attention stream. Such modality competition can dilute instruction-specific information in cluttered or compositional scenes. In contrast, Key-Gram separates instruction-side world-knowledge retrieval from scene-side visual reasoning, thereby bypassing input-level token competition. 2.2
World Action Models and Predictive Manipulation
A parallel line of work argues that robust manipulation requires not only semantic grounding, but also explicit prediction of future observations, latent world states, or action trajectories. This view motivates diffusion-based visuomotor policies and the recent WAM paradigm [16, 13–15]. For example, DreamZero adapts a large pretrained video diffusion model into a real-time closed-loop controller, showing that future-state prediction can provide a strong inductive bias for long-horizon manipulation [15]. However, WAM-style methods mainly strengthen dynamic scene prediction, while leaving world knowledge implicit in the same monolithic model. In embodied settings, such knowledge is openended, as new objects, skills, and relations continually emerge. Once static knowledge and dynamic prediction are entangled in a single backbone, extension becomes costly and brittle. Key-Gram addresses this limitation by decoupling expandable world knowledge from visual future-state reasoning. 3
2.3
World Memory, Layer-wise Memory Scaling, and Lifelong Expansion
Recent LLM research increasingly treats world knowledge as an explicit memory primitive rather than as information fully absorbed by dense backbone computation. Classical memory-augmented models, such as product-key memories and retrieval-augmented language models, decouple storage from computation for greater capacity and efficiency [20, 21, 33, 22]. Recent work further develops this direction through hashed conditional memory, large static embedding tables, layer-wise memory modules, and per-layer embedding tables [35, 34, 23–26]. Yet these methods are primarily designed for language modeling, scaling, or efficient inference, not for embodied lifelong learning. In manipulation, the key issue is continual knowledge expansion: new objects, skills, and relations should be added without repeatedly rewriting the policy backbone. When such knowledge is entangled with core parameters, adaptation becomes expensive and prone to catastrophic forgetting [31, 32]. Key-Gram addresses this limitation by organizing world knowledge as an incrementally expandable external memory, enabling modular knowledge growth for embodied control without full-model retraining.
3
Method
3.1
Overview
We introduce Key-Gram, an external conditional memory module that augments visual reasoning models with structured linguistic world knowledge. Given a language instruction I and initial visual observations V0 , the model predicts both an action trajectory and a compact future visual state. As shown in Fig. 1, Key-Gram separates instruction-side knowledge retrieval from scene-side visual reasoning. It first decomposes I into a small set of task-specific key-grams and maps them to static dense memory embeddings through deterministic lookup. The retrieved embeddings are then injected into selected backbone layers via context-adaptive modulation and lightweight convolutional refinement. The memory-guided backbone supports two downstream prediction pathways: final-layer hidden states are adapted into VAE-encodable future latents, while the backbone KV cache conditions an action expert for trajectory decoding. This design keeps the main visual backbone focused on futurestate reasoning and control, while world knowledge is stored in an expandable external memory. The following sections describe key-gram retrieval, memory fusion, prediction heads, extensibility, and the training–inference system design. 3.2
Sparse Key-Gram Retrieval
The first phase maps instruction-level semantic units to static memory entries through key-gram extraction and deterministic hashed retrieval. Key-gram extraction. Rather than relying on tokenization or contiguous phrase segmentation, we use a lightweight language model or API-based parser to extract a fixed number of short, knowledgebearing key-grams from each instruction. The parser is prompted to compose reusable task-level units under a maximum word length. For example, “put the yellow and white mug in the microwave and close it” can be decomposed into {put mug in microwave, close microwave door, yellow and white mug, mug inside microwave}. The key-gram budget is set according to the instruction complexity of each environment, and the extraction prompt is provided in Appendix A.2. Hashed memory mapping. Let G = {gi }K i=1 be the extracted key-grams, where each gi contains at most M words. We convert each word into an integer identifier and right-pad shorter key-grams with zeros to obtain a fixed-length key ḡi = (ai,1 , . . . , ai,M ). Following multiplicative-XOR hashing, each key is mapped to H hash heads. For the h-th head at layer ℓ, the memory index is computed as M M (ℓ,h) (ℓ,h) zi = ai,j rj mod P (ℓ,h) , (1) j=1
4
(ℓ,h)
where ⊕ denotes bitwise XOR, rj
are deterministic odd multipliers, and P (ℓ,h) is a prime table (ℓ,h)
= E (ℓ,h) [zi
(ℓ,h)
.
size. The corresponding embedding is retrieved as ei concatenated to form (ℓ)
ei
H
= ∥ ei
(ℓ,h)
], and all head outputs are (2)
h=1 (ℓ)
The retrieved embeddings {ei }K i=1 are used as static linguistic memory for the subsequent fusion stage. 3.3
Context-Adaptive Key-Gram Fusion
As illustrated in Fig. 1 (b), the fusion phase injects retrieved key-gram memories into the visual backbone through a residual module inserted before the attention operation of selected Transformer blocks. Let H (ℓ) ∈ RB×L×d denote the hidden states before attention at layer ℓ, where L is the number of visual tokens. The retrieved embeddings from Sec. 3.2 are first concatenated into a memory vector M (ℓ) ∈ RB×1×dm , where dm aggregates both the K extracted key-grams and their H hash heads. Specifically, if each hash head returns a dh -dimensional vector, then dm = KHdh . We therefore project the memory into the backbone space through two learnable matrices, yielding (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) (ℓ) Km = M (ℓ) WK and Vm = M (ℓ) WV , where Km , Vm ∈ RB×1×d . The incoming hidden states are directly used as visual queries. A token-wise gate is computed as √ (ℓ) ⊤ A(ℓ) = σ H (ℓ) Km / d , A(ℓ) ∈ RB×L×1 . (3) This gate measures the relevance of the retrieved linguistic memory to each visual token, enabling the model to inject knowledge priors only where they are useful. The gated memory feature is then refined by a lightweight long-span convolution: ∆H (ℓ) = Convspan A(ℓ) ⊙ Vm(ℓ) .
(4) (ℓ)
Since M (ℓ) is formed by concatenating multiple key-gram memories, the projected value Vm contains heterogeneous pieces of instruction-level knowledge. The long-span convolution is used to mix these memory components over a wider receptive field, allowing relations among different key-grams to be jointly modeled before being injected into each visual token. This operation provides an explicit interaction mechanism among retrieved knowledge entries beyond independent gating. e (ℓ) = H (ℓ) + The Key-Gram output is finally added to the original hidden states in residual form, H (ℓ) (ℓ) e ∆H , and H is passed to the standard attention computation. We instantiate this module only at selected backbone layers, following the placement strategy validated by the ablation study in Sec. 4.4. 3.4
Coupled Visual and Action Prediction
As illustrated in Fig. 1 (a), the memory-guided backbone is followed by two prediction pathways for visual foresight and action decoding. Vision reasoning. Let H (L) denote the final-layer hidden states of the backbone. We introduce a short sequence of learnable queries Qv and use a cross-attention block to extract compact futurevision representations from H (L) . The resulting query features are projected into the visual latent space and passed through two lightweight upsampling stages to match the spatial resolution of the VAE latent. The visual pathway therefore predicts visual foresight in a compact latent space. Action inference. For action prediction, we reuse the KV cache accumulated by the preceding backbone as conditional context for a lightweight action expert. The action expert is architecturally isomorphic to the backbone but smaller in scale, and performs a flow-matching denoising process over action trajectories. This design allows action decoding to remain tightly coupled with the same memory-guided visual reasoning process that produces future visual latents. 5
(b) Scaling with instruction length
(a) Logical Table from Multiple Physical Sub-Tables logical concatenation
Logical Table
Shorter Instruction
Hash Head h E1,1
E1,H
Row/Addr Embedding
1 2
1 2
V
V
...
ES,1
ES,H
Row/Addr Embedding
1 2
1 2
...
key grams 1
...
key grams 2
...
key grams 3
...
key grams 4
...
(c) Scaling with task complexity
...
...
...
V
single sub-table ES,H (size: V * d)
Logical Table Slot 1
key grams 2
Row/Addr Embedding
V
Slot 1
...
...
Longer Instruction
...
...
...
Key-Gram Slot s
Row/Addr Embedding
key grams 1
...
Physical Sub-Table
Logical Table
Slot 1
Slot S
Collision Pressure Increases
New Embedding
Expand ES,H (size: 2V * d)
Row/Addr Embedding Row/Addr Embedding
1 2
V
...
V
1 2
...
V
Expand With Parallel Sub-Tables
...
1 2
...
...
...
...
...
...
Row/Addr Embedding
Task Complexity Increases
Figure 2: Extensible memory allocation of Key-Gram. The memory is a logical table composed of slot-head physical sub-tables rather than a single monolithic table. Longer instructions are handled by adding key-gram slots, while more complex task distributions are supported by appending new sub-tables to enlarge the effective capacity and reduce hash collisions. 3.5
Extensible and System-Efficient Memory Allocation
As illustrated in Fig. 2, Key-Gram organizes memory as a logical table composed of multiple subtables rather than a single monolithic table. Each key-gram slot and hash head owns an independent sub-table Es,h ∈ RV ×de , where V is the row capacity and de is the embedding width. Retrieved entries from all slot-head sub-tables are concatenated before fusion. This structure supports both flexible memory expansion and efficient system realization: during training, sub-tables can be partitioned across devices under the same deterministic lookup interface; during inference, indices are determined solely by the instruction and fixed hash functions, enabling O(1) lookup with respect to total memory size and allowing large knowledge tables to reside in CPU memory. Scaling with instruction length. Longer instructions typically require more key-grams to represent their world knowledge. Key-Gram handles this by increasing the number of key-gram slots, where each added slot introduces a group of H hash-head sub-tables. For example, a short instruction may activate key-grams such as {put mug in microwave, close microwave door, yellow and white mug}, whereas a longer instruction may require additional entries such as {pick hamburg and fries, French fries in red packaging, smooth glossy orange tray}. Thus, longer tasks can retrieve richer linguistic priors while keeping the visual backbone unchanged. Scaling with task complexity. More complex environments introduce additional entities, relations, and affordances, increasing hash-collision pressure under a fixed memory capacity. Key-Gram expands capacity by enlarging V or adding parallel sub-tables to existing slot-head groups, whose outputs are concatenated with the original retrieved entries. This increases the addressable knowledge space without modifying the visual backbone, allowing new knowledge to be appended while preserving existing memory structure and backbone parameters.
4
Experiment
4.1
Experiments Setup
Benchmarks. We evaluate π0 -Key-Gram and π0.5 -Key-Gram on RoboTwin2.0, LIBERO, and LIBERO-Plus [17–19]. Following the RoboTwin2.0 protocol, we test all 50 tasks under both clean 6
Table 1: RoboTwin2.0 results (%). Gains in parentheses for Key-Gram variants are relative improvements over their corresponding base backbones. π0
X-VLA
π0.5
Metric
Easy Hard Easy Hard Easy Hard
Avg. H=1 Avg. H=2 Avg. H=3 Avg. 50 Tasks
81.6 59.3 61.2 72.9
82.5 55.9 66.0 72.8
66.5 66.1 61.6 65.9
61.6 54.7 50.2 58.4
85.1 79.3 78.6 82.7
80.2 73.0 67.4 76.8
π0 -KG (ours) Easy
Hard
π0.5 -KG (ours) Easy
Hard
80.4 (+20.9%) 78.0 (+26.4%) 90.8 (+6.6%) 87.0 (+8.5%) 80.5 (+21.7%) 72.7 (+33.0%) 86.3 (+8.7%) 80.3 (+10.0%) 79.4 (+28.9%) 70.2 (+39.8%) 86.8 (+10.4%) 80.8 (+19.9%) 80.3 (+21.9%) 75.6 (+29.5%) 89.0 (+7.6%) 84.4 (+9.9%)
Table 2: LIBERO and LIBERO-Plus results (%). Gains in parentheses for Key-Gram variants are relative improvements over their corresponding base backbones. Model
LIBERO fine-tuned LIBERO-Plus from LIBERO LIBERO-Plus fine-tuned
OpenVLA-OFT π0 π0.5
95.3 94.2 96.9
69.6 53.6 83.9
79.6 84.0 90.4
π0 -KG (ours) π0.5 -KG (ours)
94.6 (+0.4%) 96.7 (-0.2%)
72.8 (+35.8%) 87.7 (+4.5%)
88.5 (+5.4%) 92.6 (+2.4%)
and randomized settings, yielding 100 configurations with 100 trials each in unseen environments. For LIBERO, we report results on the four standard task suites. For LIBERO-Plus, we follow its distribution-expansion and robustness setting, evaluating both zero-shot transfer from LIBEROtrained models and further fine-tuned models, measuring in-domain performance and generalization to expanded task variations. Model Configuration. We instantiate Key-Gram on two open-source VLA backbones, π0 and π0.5 [4, 5], while keeping the original backbone and action expert unchanged. Key-Gram is inserted as a lightweight residual module into Transformer Layers (1, 8, 13), following the layer-placement ablation in Sec. 4.4. This setting isolates the effect of external linguistic memory from changes to the base VLA architecture. Detailed memory size, hashing, and fusion configurations are provided in Appendix A.3. Real-world robot. We conduct real-world experiments on a Piper dual-arm platform, covering three long-horizon tasks and five expansion-task settings. For long-horizon evaluation, we collect 300 trajectories per task, including plate-based object picking, object sorting into boxes, and compositional object assembly. For expansion evaluation, we collect two 50-trajectory training sets, pick Pen and Charger and pick Matchbox car and Lighter. We then test in-distribution execution, compositional transfer to unseen pairings pick Pen and Lighter and pick Charger and Matchbox car, and sequential adaptation by re-evaluating pick Pen and Charger after further training on pick Matchbox car and Lighter. This protocol evaluates instruction grounding, object-concept recombination, and robustness to knowledge updates. 4.2
Results on Simulation
As shown in Table 1, Key-Gram consistently improves both π0 and π0.5 on RoboTwin2.0. The gains are larger for π0 , where π0 -KG increases the average success rate from 65.9 to 80.3 under the easy setting and from 58.4 to 75.6 under the hard setting. The improvement also grows with task horizon, suggesting that external linguistic memory helps preserve object relations and action priors across multi-step manipulation. On the stronger π0.5 backbone, Key-Gram still yields stable gains, improving average performance from 82.7 to 89.0 on easy tasks and from 76.8 to 84.4 on hard tasks. Detailed per-task results are provided in Appendix A.1. Table 2 further shows that Key-Gram improves LIBERO-Plus generalization while preserving strong LIBERO performance. Under direct transfer from LIBERO-trained models, π0 -KG improves over π0 by 35.8%, and π0.5 -KG improves over π0.5 by 4.5%. After fine-tuning on LIBERO-Plus, both variants continue to outperform their backbones, reaching 88.5 and 92.6, respectively. These results 7
Table 3: Real-world long-horizon task results (%). Gains in parentheses for Key-Gram variants are relative improvements over their corresponding base backbones. Model
Picking
Sorting
Assembly
Avg.
π0 π0.5
82.0 90.0
74.0 86.0
52.0 70.0
69.3 82.0
88.0 (+7.3%) 94.0 (+4.4%)
86.0 (+16.2%) 90.0 (+4.7%)
66.0 (+26.9%) 82.0 (+17.1%)
80.0 (+15.4%) 88.7 (+8.1%)
π0 -KG (ours) π0.5 -KG (ours)
Prompt: Sort all the bread into the white box and the remaining items into the brown basket.
Prompt: Put all the bread and the frok onto the plate in the middle.
Prompt: Assemble the bread and cutlery onto plates, then transfer the bread-plate to the white box and the cutlery-plate to the brown basket.
Figure 3: Demonstrations show the execution process of π0.5 -KG. indicate that Key-Gram is especially effective under distribution expansion, where language-derived world priors help bridge original training tasks and more diverse manipulation variations. 4.3
Results on Real-world Robot
Long-horizon Tasks. As shown in Table 3, Key-Gram consistently improves both π0 and π0.5 on real-world long-horizon tasks. The largest gains appear on the assembly task, where π0 -KG improves over π0 by 26.9% and π0.5 -KG improves over π0.5 by 17.1%. This suggests that external linguistic memory is particularly useful for maintaining object relations and sub-goal structure across multiple manipulation stages. Qualitative rollouts in Fig. 3 show successful execution of sorting, picking, and compositional assembly instructions. Expansion Tasks. Table 4 reports the real-world expansion results. KG has little effect on indistribution tasks where the base policies are already strong, but substantially improves unseen object-pair recombinations. It raises the average score from 72.4 to 81.6 for π0 and from 80.0 to 86.8 for π0.5 , with the largest gains on compositional transfer tasks. Fig. 4 shows the same trend qualitatively: π0.5 succeeds on familiar pairings but fails after object-target relations are recombined, whereas π0.5 -KG completes the expanded instructions. This indicates that Key-Gram mainly benefits real-world generalization through object-concept recombination. 4.4
Ablation Study
We conduct a layer-placement ablation on RoboTwin2.0 with the π0.5 backbone to determine where Key-Gram should be injected. The ablation uses four representative tasks, hanging mug, move stapler pad, pick dual bottles, and stack blocks three, covering hanging, pushing, dual-object selection, and 8
Table 4: Real-world expansion-task results (%). Gains in parentheses for KG variants are relative improvements over their corresponding base backbones. Task 1–5 correspond to two in-distribution settings, two unseen compositional pairings, and one sequential-adaptation setting. Model
Task 1
Task 2
Task 3
Task 4
Task 5
Avg.
π0 π0.5
92.0 92.0
90.0 92.0
52.0 64.0
48.0 66.0
80.0 86.0
72.4 80.0
90.0 (-2.2%) 96.0 (+4.3%)
92.0 (+2.2%) 92.0 (+0.0%)
70.0 (+34.6%) 76.0 (+18.8%)
68.0 (+41.7%) 80.0 (+21.2%)
88.0 (+10.0%) 90.0 (+4.7%)
81.6 (+12.7%) 86.8 (+8.5%)
π0 -KG (ours) π0.5 -KG (ours)
In-distribution evaluation Prompt: pick Pen and Charger, then place in black bookshelf.
Prompt: pick Matchbox car and Lighter, place them in box.
�0.5 �0.5 KG
Out-distribution evaluation
�0.5
Prompt: pick Pen and Lighter, then place Pen in bookshelf and Lighter in box.
Prompt: pick Charger and Matchbox car, then place Matchbox car in box and Charger in bookshelf .
�0.5 KG
Figure 4: Qualitative examples from real-world expansion tasks. Both π0.5 and π0.5 -KG succeed on in-distribution object pairs, while π0.5 fails on unseen object-pair recombinations. In contrast, π0.5 -KG correctly follows the expanded instructions by grounding compositional object relations with retrieved linguistic priors. multi-object stacking. The final score is computed by weighting Easy and Hard settings with a ratio of 1:9. As shown in Table 5 and Fig. 5, early insertion is most critical. A single insertion at Layer 1 improves the weighted score from 51.0 to 71.5, indicating that language-derived memory is most useful before the backbone spends substantial depth on instruction-related feature construction. Greedily adding Layer 8 further improves the score to 76.8, while adding Layer 13 yields the best score of 77.2. The marginal gain from (1, 8) to (1, 8, 13) suggests diminishing returns from deeper memory injection. Gate probing in Fig. 5 supports this trend. In the vanilla layer sweep, performance peaks at shallow layers and decreases for deeper insertions, although normalized gates remain above a small floor around 0.1, indicating that deeper memories are still active but less influential. After one layer is selected, gates in nearby subsequent layers are strongly suppressed in the next probing stage, suggesting that the inserted module has already absorbed much of the useful linguistic prior for downstream blocks. This explains why the greedy search selects increasingly separated layers and finally adopts the (1, 8, 13) configuration.
5
Conclusion
We presented Key-Gram, an external conditional memory framework for language-conditioned robot manipulation. Key-Gram separates instruction-side world-knowledge retrieval from scene-side visual reasoning by decomposing instructions into task-specific key-grams, retrieving static linguistic priors 9
Figure 5: Layer-placement ablation on RoboTwin2.0. Shaded curves denote the weighted task score, while diamond markers denote the normalized gate activation of the probed Key-Gram module. The left and right axes correspond to score and normalized gates, respectively. Table 5: Layer-placement ablation results on RoboTwin2.0(%). Numbers in parentheses denote the Transformer layers where Key-Gram is inserted. The score is computed as the average over the four tasks, with Easy and Hard settings weighted by a ratio of 1:9. Hanging Mug Move Stapler Pad Pick Dual Bottles Stack Blocks Three Model
Easy
Hard
Easy
Hard
Easy
Hard
Easy
Hard
Score
π0.5 (Vanilla) π0.5 -KG (1) π0.5 -KG (1, 8) π0.5 -KG (1, 8, 13)
18 30 35 38
17 32 36 35
56 83 88 90
42 79 85 85
93 97 99 98
63 86 95 98
91 94 96 89
76 87 90 90
51.0 71.5 76.8 77.2
through deterministic hashed lookup, and injecting them into selected Transformer layers. This design reduces dense token-level modality competition and allows the visual backbone to focus on future-state reasoning and control. Experiments on RoboTwin2.0, LIBERO, LIBERO-Plus, and real-world dual-arm tasks show that Key-Gram consistently improves both π0 and π0.5 , with larger gains in long-horizon manipulation, distribution-expanded settings, and compositional object-pair transfer. Ablations further show that early memory injection is most effective, supporting the role of external linguistic priors in reducing instruction-related burden on the backbone. Overall, Key-Gram suggests a lightweight and extensible direction for organizing world knowledge in embodied control.
10
References [1] Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., Vuong, Q., Vanhoucke, V., Tran, H., Soricut, R., Singh, A., Singh, J., Sermanet, P., Sanketi, P.R., Salazar, G., Ryoo, M.S., et al. (2023) RT-2: Vision-language-action models transfer web knowledge to robotic control. In J. Tan, M. Toussaint and K. Darvish (eds.), Proceedings of The 7th Conference on Robot Learning, pp. 2165–2183. PMLR. [2] Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E.P., Sanketi, P.R., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P. & Finn, C. (2025) OpenVLA: An open-source vision-language-action model. In P. Agrawal, O. Kroemer and W. Burgard (eds.), Proceedings of The 8th Conference on Robot Learning, pp. 2679–2713. PMLR. [3] Kim, M.J., Finn, C. & Liang, P. (2025) Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645. [4] Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Shi, L.X., Tanner, J., et al. (2024) π0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. [5] Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Galliker, M.Y., Ghosh, D., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., LeBlanc, D., Levine, S., Li-Bell, A., et al. (2025) π0.5 : A vision-language-action model with open-world generalization. In Proceedings of The 9th Conference on Robot Learning, pp. 17–40. PMLR. [6] Liu, S., Wu, L., Li, B., Tan, H., Chen, H., Wang, Z., Xu, K., Su, H. & Zhu, J. (2024) RDT-1B: A diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864. [7] Zheng, J., Li, J., Wang, Z., Liu, D., Kang, X., Feng, Y., Zheng, Y., Zou, J., Chen, Y., Zeng, J., Zhang, Y.-Q., Pang, J., Liu, J., Wang, T. & Zhan, X. (2026) X-VLA: Soft-prompted transformer as scalable crossembodiment vision-language-action model. In International Conference on Learning Representations. [8] NVIDIA, Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L., Fang, Y., Fox, D., Hu, F., Huang, S., Jang, J., Jiang, Z., Kautz, J., Kundalia, K., Lao, L., Li, Z., Lin, Z., Lin, K., Liu, G., Llontop, E., et al. (2025) GR00T N1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734. [9] Cheang, C.L., Chen, S., Cui, Z., Hu, Y., Huang, L., Kong, T., Li, H., Li, Y., Liu, Y., Ma, X., Niu, H., Ou, W., Peng, W., Ren, Z., Shi, H., Tian, J., Wu, H., Xiao, X., Xiao, Y., Xu, J. & Yang, Y. (2025) GR-3 technical report. arXiv preprint arXiv:2507.15493. [10] Cheang, C.L., Chen, G., Jing, Y., Kong, T., Li, H., Li, Y., Liu, Y., Wu, H., Xu, J., Yang, Y., Zhang, H. & Zhu, M. (2024) GR-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158. [11] Huang, S., Chen, L., Zhou, P., Chen, S., Jiang, Z., Hu, Y., Liao, Y., Gao, P., Li, H., Yao, M. & Ren, G. (2025) EnerVerse: Envisioning embodied future space for robotics manipulation. arXiv preprint arXiv:2501.01895. [12] Liao, Y., Zhou, P., Huang, S., Yang, D., Chen, S., Jiang, Y., Hu, Y., Cai, J., Liu, S., Luo, J., Chen, L., Yan, S., Yao, M. & Ren, G. (2025) Genie Envisioner: A unified world foundation platform for robotic manipulation. arXiv preprint arXiv:2508.05635. [13] Lu, G., Jia, B., Li, P., Chen, Y., Wang, Z., Tang, Y. & Huang, S. (2025) GWM: Towards scalable Gaussian world models for robotic manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. [14] Li, L., Zhang, Q., Luo, Y., Yang, S., Wang, R., Han, F., Yu, M., Gao, Z., Xue, N., Zhu, X., Shen, Y. & Xu, Y. (2026) Causal world modeling for robot control. arXiv preprint arXiv:2601.21998. [15] Ye, S., Ge, Y., Zheng, K., Gao, S., Yu, S., Kurian, G., Indupuru, S., Tan, Y.L., Zhu, C., Xiang, J., Malik, A., Lee, K., Liang, W., Ranawaka, N., Gu, J., Xu, Y., Wang, G., Hu, F., Narayan, A., Bjorck, J., et al. (2026) World action models are zero-shot policies. arXiv preprint arXiv:2602.15922. [16] Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R. & Song, S. (2023) Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems.
11
[17] Chen, T., Chen, Z., Chen, B., Cai, Z., Liu, Y., Li, Z., Liang, Q., Lin, X., Ge, Y., Gu, Z., Deng, W., Guo, Y., Nian, T., Xie, X., Chen, Q., Su, K., Xu, T., Liu, G., Hu, M., Gao, H., et al. (2025) RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088. [18] Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y. & Stone, P. (2023) LIBERO: Benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems 36, pp. 44776– 44791. [19] Fei, S., Wang, S., Shi, J., Dai, Z., Cai, J., Qian, P., Ji, L., He, X., Zhang, S., Fei, Z., Fu, J., Gong, J. & Qiu, X. (2025) LIBERO-Plus: In-depth robustness analysis of vision-language-action models. arXiv preprint arXiv:2510.13626. [20] Lample, G., Sablayrolles, A., Ranzato, M.A., Denoyer, L. & Jégou, H. (2019) Large memory layers with product keys. In Advances in Neural Information Processing Systems 32. [21] Guu, K., Lee, K., Tung, Z., Pasupat, P. & Chang, M.-W. (2020) REALM: Retrieval-augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, pp. 3929– 3938. PMLR. [22] Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., van den Driessche, G.B., Lespiau, J.-B., Damoc, B., Clark, A., de Las Casas, D., Guy, A., Menick, J., Ring, R., Hennigan, T., Huang, S., Maggiore, L., Jones, C., Cassirer, A., Brock, A., et al. (2022) Improving language models by retrieving from trillions of tokens. In Proceedings of the 39th International Conference on Machine Learning, pp. 2206–2240. PMLR. [23] Cheng, X., Zeng, W., Dai, D., Chen, Q., Wang, B., Xie, Z., Huang, K., Yu, X., Hao, Z., Li, Y., Zhang, H., Zhang, H., Zhao, D. & Liang, W. (2026) Conditional memory via scalable lookup: A new axis of sparsity for large language models. arXiv preprint arXiv:2601.07372. [24] Liu, H., Zhang, J., Wang, C., Hu, X., Lyu, L., Sun, J., Yang, X., Wang, B., Li, F., Qian, Y., Si, L., Sun, Y., Li, R., Pei, P., Xie, Y. & Cai, X. (2026) Scaling embeddings outperforms scaling experts in language models. arXiv preprint arXiv:2601.21204. [25] Ding, N., Liu, F., Kim, K., Hao, L., Lee, K.-H., Ko, H. & Tang, Y. (2026) MeKi: Memory-based expert knowledge injection for efficient LLM scaling. arXiv preprint arXiv:2602.03359. [26] Google (2026) Gemma 4 model overview. Google AI for Developers Documentation. Accessed May 7, 2026. [27] Perez, E., Strub, F., de Vries, H., Dumoulin, V. & Courville, A. (2018) FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence 32(1). [28] Dumoulin, V., Perez, E., Schucher, N., Strub, F., de Vries, H., Courville, A. & Bengio, Y. (2018) Featurewise transformations. Distill. doi:10.23915/distill.00011. [29] Peebles, W. & Xie, S. (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195–4205. [30] Dasari, S., Mees, O., Zhao, S., Srirama, M.K. & Levine, S. (2024) The ingredients for robotic diffusion transformers. arXiv preprint arXiv:2410.10088. [31] McCloskey, M. & Cohen, N.J. (1989) Catastrophic interference in connectionist networks: The sequential learning problem. In G.H. Bower (ed.), Psychology of Learning and Motivation, Vol. 24, pp. 109–165. Academic Press. [32] French, R.M. (1999) Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences 3(4):128–135. [33] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S. & Kiela, D. (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33, pp. 9459–9474. [34] Wang, W., Dong, L., Cheng, H., Liu, X., Yan, X., Gao, J. & Wei, F. (2023) Augmenting language models with long-term memory. In Advances in Neural Information Processing Systems 36, pp. 74530–74543. [35] Wu, Y., Rabe, M.N., Hutchins, D. & Szegedy, C. (2022) Memorizing transformers. In International Conference on Learning Representations.
12
A
Technical appendices and supplementary material
A.1
Full RoboTwin2.0 Results Table 6: Full RoboTwin2.0 results (%). Gains in parentheses for KG variants are relative improvements over their corresponding base backbones. π0
π0.5
π0 -KG (ours)
π0.5 -KG (ours)
Task
Easy Hard Easy Hard
Easy
Hard
Easy
Hard
adjust_bottle beat_block_hammer blocks_ranking_rgb blocks_ranking_size click_alarmclock click_bell dump_bin_bigbin grab_roller handover_block handover_mic hanging_mug lift_pot move_can_pot move_pillbottle_pad move_playingcard_away move_stapler_pad open_laptop open_microwave pick_diverse_bottles pick_dual_bottles place_a2b_left place_a2b_right place_bread_basket place_bread_skillet place_burger_fries place_can_basket place_cans_plasticbox place_container_plate place_dual_shoes place_empty_cup place_fan place_mouse_pad place_object_basket place_object_scale place_object_stand place_phone_stand place_shoe press_stapler put_bottles_dustbin put_object_cabinet rotate_qrcode scan_object shake_bottle_horizontally shake_bottle stack_blocks_three stack_blocks_two stack_bowls_three stack_bowls_two stamp_seal turn_switch
99 79 80 14 77 71 88 98 47 97 14 80 68 67 74 41 71 4 69 59 43 39 62 66 81 55 63 97 59 91 66 20 67 57 82 49 76 44 65 73 74 55 98 94 72 93 77 94 46 41
99 92 87 55 75 88 90 100 70 97 31 92 71 75 88 73 90 69 85 91 79 74 70 76 94 86 72 95 78 90 78 52 77 73 88 77 89 64 82 82 88 71 92 95 91 98 82 99 67 40
98 90 81 36 78 80 85 98 58 90 29 91 67 63 80 67 93 78 66 83 64 57 65 72 81 73 63 94 79 94 86 41 70 68 83 72 91 75 76 75 83 66 93 92 83 91 75 95 70 42
100 100 94 71 99 98 94 100 75 98 38 95 88 96 94 91 92 80 92 100 85 86 83 80 92 92 94 96 86 96 93 75 80 87 91 84 99 90 87 89 86 80 97 99 95 100 87 98 89 60
99 94 89 60 92 87 89 98 61 96 36 97 74 81 86 83 96 84 79 97 85 87 79 77 88 77 85 95 83 98 88 70 78 82 87 83 95 85 85 80 84 70 98 96 91 99 79 98 83 56
95 84 63 5 68 48 83 94 31 97 11 72 48 46 65 24 81 32 31 37 47 34 46 49 76 46 45 92 51 85 71 20 70 52 68 53 76 37 56 60 70 42 92 91 52 79 75 95 33 42
100 96 92 49 98 99 92 100 66 98 18 96 51 84 96 56 90 34 81 93 87 87 77 85 94 62 94 99 75 100 87 60 80 86 91 81 92 87 84 80 89 72 99 99 91 97 77 95 79 62
99 93 85 26 89 66 97 100 57 97 17 85 55 61 84 42 96 77 71 63 82 84 64 66 87 62 84 95 75 99 85 39 76 80 85 81 93 83 79 79 87 65 99 97 76 100 71 96 55 54
13
π0
π0.5
Task
Easy Hard Easy Hard
Avg. H=1 Avg. H=2 Avg. H=3 Avg. 50 Tasks
66.5 66.1 61.6 65.9
A.2
61.6 54.7 50.2 58.4
85.1 79.3 78.6 82.7
80.2 73.0 67.4 76.8
π0 -KG (ours) Easy
Hard
π0.5 -KG (ours) Easy
Hard
80.4 (+20.9%) 77.9 (+26.4%) 90.8 (+6.6%) 87.0 (+8.5%) 80.5 (+21.7%) 72.7 (+33.0%) 86.3 (+8.7%) 80.3 (+10.0%) 79.4 (+28.9%) 70.2 (+39.8%) 86.8 (+10.4%) 80.8 (+19.9%) 80.3 (+21.9%) 75.6 (+29.5%) 89.0 (+7.6%) 84.4 (+9.9%)
Key-Gram Extraction Prompt
For the RoboTwin2.0 dataset, we use an external language-model parser to convert each naturallanguage manipulation instruction into a fixed-size set of key-grams. Since RoboTwin2.0 contains diverse object-centric and long-horizon manipulation commands, the prompt is designed to encourage short, compositional, and action-centered phrases that preserve task-relevant semantic roles, such as actions, objects, attributes, sources, targets, and spatial relations. Unless otherwise specified, we extract exactly eight key-grams for each RoboTwin2.0 instruction. The complete prompt used in our experiments is shown below. SYSTEM_PROMPT = """ You are an expert parser for embodied manipulation instructions. Your task is to convert one instruction into exactly 8 composed keywords. These keywords must be short, natural, semantically meaningful phrases for embodied actions, not
Rules: 1. Output exactly 8 keywords. 2. Each keyword must contain 2 to 4 words. 3. Prefer high-information phrases that combine multiple semantic roles in one phrase. 4. Prefer action-centered phrases over static descriptive phrases whenever possible. 5. At least 3 of the 8 keywords must explicitly contain an action verb. 6. Prefer these phrase types, in this priority order: a. verb + object + relation/target/source b. verb + particle + object c. verb + prep + object d. object + prep + object e. attribute + object 7. A good keyword should ideally compress 2 or more semantic elements, such as: - action + object - action + object + source - action + object + target - object + attribute - object + location 8. Use standalone static noun phrases only when they add important information that is not alread 9. Use at most 5 standalone noun phrases. 10. If a static phrase can be replaced by a more informative action phrase, prefer the action phr 11. Prefer phrases like: - "pick up" - "pick bowl from drawer" - "pick up bowl" - "place bowl on plate" - "bowl in top drawer" - "black bowl" 12. Avoid: - fragmented phrases - fake combinations across unrelated spans - pronoun-centered phrases like "place it on" - low-information phrases 14
- too many static environment phrases - duplicated semantics across multiple keywords - more than 4 words in a keyword - less or more than 8 keywords 13. Do not explain anything. 14. Return valid JSON only. Example: Instruction: pick up the green sponge from the sink and wipe the wooden table near the window Output: { "keywords": [ "pick and wipe", "pick sponge from sink", "pick up sponge", "green sponge", "wipe wooden table", "wipe table near window", "table near window", "wooden table" ] } MUST FOLLOW: - Do NOT less or more than 8 keywords. - Do NOT use more than 4 words in a keyword. Must check the keywords strictly follow the rules above. """ A.3
Model Configuration Details
Simulation experiments. For the simulation experiments on LIBERO, LIBERO-Plus, and RoboTwin2.0, we choose the Key-Gram configuration according to the instruction length and linguistic complexity of these benchmarks. Specifically, we set the number of key-gram slots to S = 8, the number of hash heads per slot to H = 4, and the embedding width of each hash head to dh = 32. This gives a retrieved memory width of dm = SHdh = 1024 for each inserted layer. For each hash-head table, we use a row capacity of V = 8192. This capacity provides sufficient address space for more than four thousand object- and relation-level entities, reducing the probability of hash collisions and avoiding frequent updates of semantically unrelated entities to the same memory row. The final fusion stage uses a lightweight 1D convolution. To encourage interaction among all retrieved key-gram slots, we set the convolutional span to the slot budget, i.e., w = S = 8. Key-Gram is inserted into Transformer layers (1, 8, 13) under zero-based layer indexing, while the original VLA backbone and action expert are kept unchanged. Real-world experiments. For the real-world dual-arm experiments, we use a slightly different allocation because the real-world instructions are generally longer than those in simulation, while the number of visually instantiated assets and object-level semantic variants is smaller than in simulation benchmarks. We therefore increase the number of key-gram slots to S = 16 to better cover long-horizon and extensible-task instructions, but reduce the number of hash heads per slot to H = 2. The head embedding width remains dh = 32, resulting in the same retrieved memory width dm = SHdh = 1024 as in simulation. For consistency, each hash-head table also uses V = 8192 rows. This setting allocates more slots to instruction decomposition while keeping the total number of hash-head tables unchanged, which balances long-instruction coverage and memory efficiency. The fusion module again uses a lightweight 1D convolution, with the convolutional span set to the slot budget, i.e., w = S = 16, so that information from all key-gram slots can be jointly mixed before 15
Table 7: Key-Gram configuration for simulation experiments. The layer indices are zero-based. Configuration item
Value
Benchmarks Backbones Key-gram slots S Hash heads per slot H Head embedding width dh Retrieved memory width dm = SHdh Rows per hash-head table V Number of hash-head tables per layer S × H Embedding parameters per inserted layer SHV dh Convolution type Convolutional span w Inserted Transformer layers Layer indexing
LIBERO, LIBERO-Plus, RoboTwin2.0 π0 , π0.5 8 4 32 1024 8192 32 8.39M 1D convolution 8 (1, 8, 13) zero-based
residual injection. As in simulation, Key-Gram is inserted into Transformer layers (1, 8, 13) under zero-based layer indexing, and the backbone and action expert are kept unchanged. Table 8: Key-Gram configuration for real-world robot experiments. The layer indices are zero-based. Configuration item
Value
Tasks Backbones Key-gram slots S Hash heads per slot H Head embedding width dh Retrieved memory width dm = SHdh Rows per hash-head table V Number of hash-head tables per layer S × H Embedding parameters per inserted layer SHV dh Convolution type Convolutional span w Inserted Transformer layers Layer indexing
A.4
long-horizon tasks and extensible tasks π0 , π0.5 16 2 32 1024 8192 32 8.39M 1D convolution 16 (1, 8, 13) zero-based
Limitations and Future work
This work has some limitations. First, due to the high cost of large-scale simulation and real-world robot evaluation, we instantiate Key-Gram only on two open-source VLA backbones, π0 and π0.5 . Although the results are consistent across RoboTwin2.0, LIBERO, LIBERO-Plus, and real-world dual-arm tasks, they do not yet fully establish the generality of Key-Gram across a broader range of VLA architectures or world-model-based manipulation policies. Then, the current framework relies on an external parser to extract task-specific key-grams from language instructions. While this provides a controllable interface for memory retrieval, extraction quality may affect downstream performance, especially for ambiguous, long, or highly compositional instructions. Future work should evaluate Key-Gram on more diverse backbones, larger real-world task suites, and longer continual-learning settings, while exploring parser-free or jointly learned key-gram discovery. A.5
Computational Resources
For model training, we utilize a compute node equipped with 8 NVIDIA H200 GPUs for fullparameter training. Operating with a per-GPU batch size of 16 (yielding a total global batch size of 128), the VRAM consumption is approximately 135 GB per GPU. Under this hardware configuration, the training throughput reaches approximately 128 samples per second. For the complete RoboTwin2.0 dataset, training typically requires 1 to 1.5 days per epoch.
16