Skill Neologisms: Towards Skill-based Continual Learning
Antonin Berthon 1 Nicolas Astorga 1 Mihaela van der Schaar 1
arXiv:2605.04970v1 [cs.LG] 6 May 2026
Abstract
Task-centric dataset
LoRA
LLM
Mixed skill requirements per sample
Modern LLMs show mastery over an evergrowing range of skills, as well as the ability to compose them flexibly. However, extending model capabilities to new skills in a scalable manner is an open-problem: fine-tuning and parameter-efficient variants risk catastrophic forgetting, while context-based approaches have limited expressiveness and are constrained by the model’s effective context. We explore skill neologisms–i.e., soft tokens integrated in the model’s vocabulary and optimized to improve capabilities over a specific skill–as a way to selectively extend model capabilities to new skills without weight updates. We first observe that off-the-shelf pre-trained LLMs already demonstrate tokens associated with procedural knowledge. We then show that skill neologisms can be learned to improve model capabilities on specific skills while being composable with out-of-distribution skills, and that independently trained skill neologisms can be composed zero-shot. These results suggest that skill neologisms may provide a scalable path towards skill-based continual learning.
Train
⊕ Unique adapter for given task
Prompt Tuning Train
Skill-centric datasets All samples require skill + others
Skill Neologisms Train
LLM
Unique prefix for given task
✓ Property 1 No weight-updates
✦
LLM
One neologism per skill
Insert neologisms Train
All samples require skill + others
Zero-shot compositional abilities ✦
✦
✓ Property 2 Compositional transfer Skills
Text
✦
LLM
LoRA adapters
✦
✓ Property 3 Multi-skill composition Soft tokens
Skill
✦ neologism
Trainable
Fixed
Figure 1. Overview of Skill Neologisms.
et al., 2023). In-context learning has shown some success at skill composition in simple settings (Chen et al., 2023; Levy et al., 2023; Xu et al., 2024), but it does not adapt as well as PEFT methods (Liu et al., 2022a), and does not scale because of effective context limitations (Hsieh et al., 2024). Prompt tuning (Lester et al., 2021) can adapt models to new tasks by only learning soft tokens prepended to the prompt, and have been shown to rival full finetuning in some settings (Genewein et al., 2025). However, prefixes are typically task-specific instead of skill-specific, and learned prefix cannot be composed or adapted to new settings without retraining (Asai et al., 2022; Wang et al., 2023).
1. Introduction Recent works have established that pretrained LLMs develop mastery over various skills and the ability to combine them beyond the pretraining distribution (Arora & Goyal, 2023; Yu et al., 2024; Chen et al., 2023). As LLMs are used to tackle an ever-growing range of problems, the ability to continuously grow model capabilities to new skills in a controlled and scalable fashion is a promising research direction. Yet, existing approaches to extend model capabilities fall short of this objective (Table 1). Finetuning models on new datasets risks catastrophic forgetting (Kirkpatrick et al., 2017; Luo et al., 2025), where previously mastered capabilities might disappear and safety risks might be introduced (Qi
Central question Can the compositional abilities of LLMs be leveraged to learn new composable skills without weight updates?
1 University of Cambridge. Correspondence to: Antonin Berthon <[email protected]>.
We term this objective skill-based continual learning and distinguish the following required properties: (▷ P1) New skills can be learned without modifying model parameters;
Preprint.
1
Skill Neologisms: Towards Skill-based Continual Learning
2. Preliminaries
Table 1. Comparison of different approaches for skill-based continual learning. ‡ E.g., full finetuning and LoRA (Hu et al., 2022). § E.g., Prompt Tuning (Lester et al., 2021) and related methods. *Achievable with skill-centered training. Method Finetuning-based‡ Prefix-based§ Skill Neologisms
P1: No Weight Updates
P2: Composes w/ OOD Skills
P3: Multi-Skill Composition
✗ ✓ ✓
✗ ✓* ✓
✗ ✗ ✓
2.1. Skills and Composition in Large Language Models We build on the formalism introduced in Arora & Goyal (2023) in which skills refer to procedural knowledge—reusable capabilities arithmetic operations or logical reasoning—rather than factual knowledge. In this framework, any piece of text t is related to a set of skills S, and the understanding of text t requires mastery over all its underlying skills as well as their composition. Given a set of skills Σ, we denote by Ck (Σ) the set of text pieces that require a k-tuple of skills from Σ. By extension, Ck (S1 , ., Si , Σ) denotes text pieces that require at least skills S1 , ., Si , mixed with k − i other skills from Σ.
(▷ P2) Learned skills are composable with other existing skills, including ones out-of-distribution from the training set; (▷ P3) Multiple skills learned independently can be composed without joint training.
Closed-form assumption What does it mean to understand a text snippet t? A key assumption from (Arora & Goyal, 2023) is that the understanding of any piece of text can be tested via closed-form questions. This might be trivial if t relates to a closed form question (eg ”find the following number: 1,2,3,5,8,..”), or by generating a set of multiplechoice questions as described in (Arora & Goyal, 2023).
In this work, investigate whether skill neologisms might enable these properties (Figure 1). Inspired by neologisms proposed by Hewitt et al. (2025) for human-machine communication, skill neologisms aims to learn new vocabulary element that, when provided in the model’s context, improve the model capabilities on a specific skill. They rely on two key components:
Composition beyond training Modern LLMs demonstrate the ability to understand combinations of skills beyond their training distribution (Wei et al., 2022; He et al., 2024; Yu et al., 2024; Zhao et al., 2024). Theoretical analysis presented in Arora & Goyal (2023) links the emergence of skill composition ability to model scaling. Namely, scaling up model parameters by an order of magnitude leads to the same level of competence on 2k-tuples of skills as the competence on k-tuples of the original model.
• Skill-centered training. Training uses datasets where every sample requires the target skill, mixed with diverse skills already mastered by the model. Such datasets can be constructed in many settings, for example by leveraging the metacognitive capabilities of modern LLMs (Didolkar et al., 2024) (see Section 3.3). • Vocabulary-level integration. Individual skills are learned via soft tokens (skill tokens) integrated in the model vocabulary, optimized on skill-centric data while keeping model weights frozen (P1).
2.2. Soft Prompts and Prompt Tuning Soft tokens Soft tokens s = (s1 , ..., sl ) are sequences of continuous vectors of size dmodel (matching the model’s hidden dimension), that can be inserted in a model’s context after skipping the embedding matrix.
These two components encourage learning generally composable skill representations (P2), and enables zero-shot composition of independently learned skills (P3). Our main contributions are as follows:
Prompt tuning Prompt-tuning (Lester et al., 2021) is a parameter-efficient finetuning approach where trainable soft tokens s are introduced as a prefix to the model’s continuous representation of the input context. Only the soft tokens are optimized on a training set by back-propagating through the model while keeping the model’s parameters frozen.
• We propose skill neologisms as a path toward skill-based continual learning (§ 3), and motivate this approach via empirical evidence that pretrained LLMs naturally exhibit vocabulary elements that encapsulate procedural knowledge (§ 4).
Expressivity of Prompt Tuning Recent works (Petrov et al., 2024; Genewein et al., 2025) study the conditions under which methods like Prompt Tuning might or might not succeed at learning a new task. Informally, a necessary condition is that the new task is not too different from tasks within the model’s pretraining distribution, so that the model weights contain the necessary circuits to solve the new task.
• We demonstrate in controlled settings (§ 5.2) that skill neologisms compose with OOD skills unseen during training (P2) and enable zero-shot composition of independently learned skills (P3). • We provide ablation experiments (§ 5.3) analyzing how token capacity and composition complexity in the training set affect learning of composable skill representations. 2
Skill Neologisms: Towards Skill-based Continual Learning
2.3. Vocabulary Extensions via Neologisms
posable representations of specific skills from the model’s context.
Neologisms embedding learning (Hewitt et al., 2025) uses learnable soft tokens as new vocabulary elements in a model’s tokenizer and embedding matrix, that can then be used in prompts alongside text tokens. We denote a neologism of length l as soft tokens s = (s1 , ..., sl ), which extend the model’s embedding matrix to E ′ = E ∪ s ∈ Rdmodel ×(|V|+l) with columns (s1 , ..., sl ), and its vocabulary to l tokens: V ′ = V ∪ {⟨S1 ⟩, ..., ⟨Sl ⟩}.
3.2. Skill Neologisms: Overview Skill neologisms are soft tokens integrated in the model vocabulary and optimized such that providing them in the model’s context enhances the model’s capability for a specific skill. Figure 2 provides an overview of the different components required. We assume that a pretrained model M has mastered a set of skills Σ and has the ability to compose them (Figure 2A). Our aim is to learn a new skill S ∗ . First, a skill-centered dataset D is constructed for skill S ∗ , with samples that all require at least skill S ∗ , as well as a other skills from Σ (Figure 2B). A skill neologism is initialized and added to the model’s vocabulary and embeddings matrix. Then for each sample in D, the neologism is inserted in the prompt and trained on D while keeping the rest of the model parameters fixed (Figure 2C).
Like Prompt Tuning, the soft tokens of a neologism can be trained on samples that include the neologism tokens by back-propagating gradients through the frozen model. This is done via preference-based learning in Hewitt et al. (2025) but can be done similarly with supervised fine-tuning (SFT) or Reinforcement Learning Finetuning (RLFT).
3. Skill-based Continual Learning via Skill Neologisms 3.1. Skill-based Continual Learning: Problem Formulation
3.3. Skill-centered datasets
Given the composition capabilities of modern LLMs (Arora & Goyal, 2023; Yu et al., 2024; Zheng et al., 2024) as well as their in-context learning abilities (Wei et al., 2022), we investigate the following question: can LLMs learn new composable skills without weight updates? We term this objective skill-based continual learning, as it would allow models to acquire new composable skills without risk of catastrophic forgetting. For such an approach to be practical and scalable, it requires three key properties:
Most datasets used for model pretraining or finetuning are task-centered: different samples or snippets of text implicitly depends on various skills. In contrast, skill neologisms require training on a dataset where every sample requires at least the target skill, mixed with other skills mastered by the model (Figure 1). Definition 3.1 (Skill-centered dataset). For a skill S and set of skills Σ, an S-centered dataset is D(S, Σ) = {ti ∼ Cki (S, Σ)}i , where each text snippet ti requires skill S plus ki − 1 additional skills sampled from Σ, with ki drawn from {1, . . . , kmax } according to some distribution p. We extend this notation to (S1 , . . . , Sm )-centered datasets, where each snippet requires all of S1 , . . . , Sm plus up to kmax − m additional skills from Σ.
• Property 1 (No weight updates): New skills are learned without modifying model parameters, preventing any catastrophic forgetting. • Property 2 (Compositional transfer): Learned skills compose with the model’s existing skills in combinations not seen during training, including out-of-distribution skill combinations.
How to construct skill-centric datasets? Since the skills that underlie samples are usually implicit, it may not be immediately obvious how to construct datasets centered around a specific skill. However we note that this is possible in many practical settings. First, in structured or synthetic settings, the mapping between samples and skills is often explicit by construction. For example in the experiments presented Section 5, each sample (e.g. [ASC][ADD]4165=2567) maps naturally to the underlying skills (e.g. [ASC] and [ADD]). For more general settings, one can leverage the metacognitive abilities of strong LLMs to annotate samples with the implicit skills required (Didolkar et al., 2024), and then filter to examples that require at least the skill of interest. Finally, many datasets provide expertly curated multi-labels categorizing each data entry–such as in educational problem banks (Wang
• Property 3 (Multi-skill composition): Multiple independently learned skills can be composed together zero-shot, without joint training on their combination. Property 2 is necessary for the learned skill to be composable with skills held by the model beyond the training distribution, while Property 3 enables scalable continual learning where skills can be added incrementally and composed together even without joint training. We propose skill neologisms—soft tokens integrated in the model’s vocabulary—as one path towards achieving these properties. Our key hypothesis is that vocabulary-level interventions combined with skill-centered datasets can leverage the model’s existing compositional abilities to learn com3
Skill Neologisms: Towards Skill-based Continual Learning
A Initial Setting Pretrained model
B Skill-centered Dataset
C Skill Neologisms 1. Extend
Existing Skills + + +
New skill to learn:
+ +
2. Insert skill tokens
,
... Skills
Trainable
Text
Fixed
Skill
3. Train skill tokens only
Embedding:
+
LLM
with skill tokens
Vocabulary:
: Text samples requiring + with other skills from
Gradient flow
Figure 2. Overview of Skill Neologisms. (A) We consider pretrained model endowed with a set of implicit skills learned during pretraining. (B) A skill-centered dataset contains snippets of text that require at least the skill of interest, composed with pretraining skills. (C) Skill neologisms appends new token embeddings to the model’s vocabulary and embedding matrix, which are trained on the skill-centered dataset while keeping the model parameters frozen. (D) By leveraging the pretrained model’s compositional abilities, skill neologisms allow zero-shot composition with OOD skills, as well as composing independently learned skills.
et al., 2020; Liu et al., 2023), programming benchmarks (Li et al., 2023), or reasoning tasks (Yuan et al., 2025)– which can be used to filter data around specific skills.
Table 2. Examples of neologism insertion functions ϕs . In each case the tokens corresponding to the skill neologism are shown in ⟨.⟩ brackets.
3.4. Skill Neologisms Definition 3.2 (Skill neologism). Given a model M with parameters θLLM , a skill neologism for skill S is a set of learnable soft tokens (or skill tokens) θS ∈ Rdmodel ×l that minimizes some loss L over an S-centered dataset D(S, Σ):
Setting
Original text t
Modified text ϕS (t)
Word replacement (Hewitt et al., 2025) Word replacement (Section 5) Task instruction
”Ensure that the length of the response is at least 600 words.”
”⟨Ensurehw ⟩ that the length of the response is at least 600 words.”
”[ADD][SHIFT]7283=...”
”[ADD]⟨SHIFT⟩7283=...”
”Sort these numbers:”
”Sort these numbers using ⟨SORT⟩:”
Algorithm 1 Training Skill Neologisms Require: Pretrained model M with frozen parameters θLLM Require: Target skill S, set of pretrained skills Σ, skill-centered dataset D(S, Σ) Require: Skill neologism length l, insertion function ϕS Require: Learning rate η, number of epochs T 1: Initialize skill tokens θS ∈ Rdmodel ×l 2: Extend vocabulary: V ′ ← V ∪ {⟨S1 ⟩, . . . , ⟨Sl ⟩} 3: Extend embedding matrix: E ′ ← E ∪ θS 4: for epoch = 1 to T do 5: for each batch B ⊂ D(S, Σ) do 6: for each text sample t ∈ B do 7: t′ ← ϕS (t) # Insert skill tokens ⟨Si ⟩ into text 8: Compute loss: L ← CrossEntropy(M(θLLM , θS , t′ )) 9: Compute gradients: ∇θS L 10: end for 11: Update skill tokens: θS ← θS − η∇θS L 12: Keep model parameters θLLM unchanged 13: end for 14: end for 15: return Optimized skill neologism θS∗ , extended vocabulary V ′
θS∗ = argmin Et∼D(S,Σ) [L(M(θLLM , θS , ϕS (t)))] θS
where ϕS : Text → Text is an insertion function that inserts the skill neologism tokens into the text in a semantically appropriate way. The loss function L depends on the training paradigm: crossentropy loss for supervised fine-tuning (SFT), RL-style objectives (e.g., policy gradients) for reinforcement fine-tuning (RFT), or preference-based losses as in Hewitt et al. (2025). In our experiments, we use cross-entropy loss. The choice of insertion function ϕS depends on the nature of the skill and how it naturally appears in text. We illustrate this with several examples below.
competence from Arora & Goyal (2023) where a model’s competence τS on a skill S is its success rate on an Scentered dataset. We evaluate the neologism on two datasets: (i) in-distribution (ID) combinations involving skills Σtrain denoted as τSID ; (ii) out-of-distribution (OOD) combinations involving held-out skills Σtest , denoted as τSOOD .
Example of insertion functions Depending on the underlying skill and text snippet, the insertion function might be simply replacing a given word with the neologism s –as done in (Hewitt et al., 2025) by replacing Ensure by Ensurehw –, or introducing a short text such as ”Make sure to use ⟨S⟩” (see Table 2 for examples under different settings).
A successful neologism should achieve τSOOD ≈ τSID , indicating that the skill neologisms composes with novel skills zero-shot.
Training procedure. We outline the training procedure for skill neologisms in Algorithm 1. Evaluating compositional transfer. After training a skill neologism, we assess whether it has learned a general representation of skill S rather than only fitting the compositions between S and skills from Σtrain . We adopt the notion of
3.5. Comparison to Existing Approaches Common approaches to extend model capabilities, such as LoRA and Prompt Tuning-like methods, are typically 4
Accuracy
Skill Neologisms: Towards Skill-based Continual Learning
1.0 0.8 0.6 0.4 0.2 0.0
XOR
Qwen-2.5-7B
XNOR
Only Examples
Llama-3.1-8B
+ Text Description
Ministral-3-8B
+ Keyword
Phi-4
Figure 3. Accuracy on XOR and XNOR completion tasks across open-source models under different prompts. Only Examples provides three input–output examples before the query. + Text Description adds a natural-language description of the operation (e.g., ”output 1 iff the input bits differ” for XOR). + Keyword adds only the operation name (”XOR” or ”XNOR”). Results are averaged over N = 100 samples; error bars show standard error.
trained on task-centric datasets. As a result, they learn taskspecific patterns rather than generally composable skills, limiting out-of-distribution transferability (P2). Moreover, these approaches are structurally unable to achieve P3: independently trained adapters or prefixes cannot be combined without retraining on the target task.
description of the operation (e.g., “output 1 if and only if both input bits are different, and 0 otherwise” for XOR); and (3) Examples + keyword: only the keyword ”XOR” or ”XNOR”. For conditions (2) and (3), information is inserted before examples via: “Complete the following using the skill:description/keyword”. Each model is evaluated on N = 100 samples per setting.
Skill neologisms address this through two key components. First, skill-centered training with limited parameter capacity creates an inductive bias for learning generally composable skill representations (P2). Second, vocabulary-level integration leverages the model’s in-context compositional abilities, allowing multiple independently learned skills to be combined simply by inserting multiple skill tokens in the context (P3).
Results Results are shown in Figure 3. For XOR, providing the keyword substantially improves accuracy over both other conditions, suggesting the ”XOR” token has captured procedural knowledge through pretraining exposure, functioning as a genuine skill token. In contrast, for XNOR, neither keyword nor description improves accuracy beyond examples alone, indicating the “XNOR” token lacks sufficient training signal to encapsulate this operation. This demonstrates that skill tokens can emerge naturally when vocabulary tokens have sufficient exposure to skill-relevant contexts, and that they can encode procedural knowledge more efficiently than explicit descriptions.
4. Existence of Skill Tokens in Pretrained LLMs Before training skill neologisms in Section 5, we first illustrate that pretrained LLMs already exhibit analogous behavior—some vocabulary tokens are associated with specific procedural knowledge. During pretraining, certain tokens are encountered in contexts related to particular operations. For example, ”XOR” tokens will frequently appear in text discussing the corresponding logical operation, which analogous to a skill-centered dataset on the skill XOR. Consequently, these tokens might capture procedural knowledge for this operation. In contrast, less common tokens like ”XNOR” might not—according to Google NGram Viewer, ”XOR” appears approximately 15 times more frequently than ”XNOR” in text from the past decade. We test this hypothesis on various open-source LLMs below.
Takeaway: Pretrained LLMs exhibit tokens that encode procedural knowledge, motivating the use of vocabulary-level parameters to learn skill representations.
5. Experiments In this section, we evaluate skill neologisms on a controlled algorithmic skill composition task. We chose this setting because it provides explicit sample-skill definitions and unambiguous composition rules, unlike natural language tasks where skills are typically implicit. This enables us to construct exact ID/OOD splits over skills to cleanly measure whether the model learns general representations that compose with held-out skills (P2) and whether independently trained skills can be combined zero-shot (P3).
Setup To test this hypothesis, we evaluate various opensource models on binary operation tasks. Models must perform XOR or XNOR on 3-bit sequences with 3 in-context examples. We compare accuracy across three conditions: (1) Only examples; (2) Examples + description: a textual
5
Skill Neologisms: Towards Skill-based Continual Learning Table 4. Accuracy of Mpretrain over sequence lengths for singletask (C1 ), two-task (C2 ), and three-task (C3 ) compositions. * indicates lengths and combinations held-out during training. ID: in-distribution skill combinations, OOD: out-of-distribution skill combinations.
5.1. Setup Dataset We create a synthetic dataset based on operations over digits sequences. Each sample is of the form: ”[OP-1] . . . [OP-k]x=y”, where x is a random sequence of n digits, each OP-i is an operation and the output is the result of sequentially applying operations to x: y = (OP-k ◦ . . . ◦ OP-1)(x). Table 3 shows the different operations and example samples for n = 3.
Composition setting Sequence length 2 3 4 5* 6 7* 8 9*
Table 3. Digit-sequence transformation skills used in the synthetic experiments. Set
Σpretrain
Snew
Skill ASC DESC ADD SUB REV POL ID SHIFT INV-POL
Description Sort digits in ascending order Sort digits in descending order Add 1 to each digit Subtract 1 from each digit Reverse digit order Map odd (even) digits to 1 (0) Identity mapping Right-shift digits Map odd (even) digits to 0 (1)
Example (Seq. length: 3) [ASC]472 = 247 [DESC]472 = 742 [ADD]472 = 583 [SUB]472 = 361 [REV]472 = 274 [POL]472 = 010 [ID]472 = 472 [SHIFT]472 = 247 [INV-POL]472 = 101
C1 (Σpretrain )
C2 (Σpretrain )
ID
ID
C3 (Σpretrain ) ID
OOD*
100.0% 100.0% 99.1% 97.6% 95.6% 92.6% 92.6% 83.9%
100.0% 100.0% 100.0% 98.0% 94.0% 92.0% 90.0% 75.0%
100.0% 100.0% 98.0% 95.0% 95.0% 89.0% 79.0% 74.0%
97.0% 96.0% 97.0% 84.0% 89.0% 74.0% 74.0% 58.0%
with a prefix of length l = 20 using the same initialization; (3) LoRA (Hu et al., 2022) with rank r = 16. 5.2. Results
Base model We fine-tune Qwen2.5-0.5B (Qwen et al., 2025) on D(Σpretrain ) with up to 3-skill combinations, using digit sequences of lengths n ∈ [2, 9] \ {5, 7, 9}, holding out lengths 5, 7, and 9 for validation. To ensure the model learns to combine operations flexibly, we also hold out 25% of 3skill combinations. Training uses LoRA(Hu et al., 2022) in two phases: (i) 100k single-skill samples and (ii) 500k samples with k = {1, 2, 3} drawn uniformly. Table 4 shows accuracy across skill counts and sequence lengths (see Appendix A1 for per-operation details). The model achieves high accuracy on both in-distribution and held-out lengths for most skills, though REV shows 0% accuracy on heldout lengths, indicating overfitting—we therefore exclude it from Σheld-out in our compositional transfer tests below. The model shows high accuracy on out-of-distribution 3skill combinations, validating that it successfully learns to generalize to unseen skill combinations.
We now validate that skill neologisms satisfy P2 and P3 from Section 3.1. P1 (no weight updates) is satisfied by construction, as all model parameters are frozen when training skill neologisms. We evaluate P2 by testing whether learned skills compose with held-out skills not seen during training, and P3 by testing whether independently learned skill neologisms can be combined zero-shot. 5.2.1. P ROPERTY 2: C OMPOSITIONAL T RANSFER Figure 4 shows the accuracy of LoRA, Prompt Tuning, and Skill Neologisms on 2-combinations of Snew with either skills from Σtrain (in-distribution) or Sheld-out (outof-distribution). All three methods achieve near-perfect in-distribution accuracy. However, only Skill Neologisms consistently succeeds at composing Snew with Sheld-out . LoRA shows the poorest OOD generalization, suggesting it overfits the training distribution rather than learning a composable representation of Snew . Prompt Tuning performs intermediately; the gap with Skill Neologisms is notable given both optimize the same number of soft tokens. This suggests that semantically embedding the soft tokens inside the prompts may provide additional flexibility to learn composable skill representation. Accuracy on 3-combinations show similar patterns (Figure A2 in Appendix).
Learning new skills We then freeze the pre-trained model Mpretrain and aim to learn two new skills Σtest = {SHIFT, INV-POL}. For each skill Snew , we generate a dataset of 100k samples with 1-, 2-, and 3-combinations of Snew ∪ Σtrain . To test out-of-distribution generalization, we create multiple datasets by setting Σtrain = Σpretrain \Sheld-out where Sheld-out ∈ Σheld-out is a specific pretrained skill held-out during training. In all our experiments we set Σheld-out = Σpretrain \ {REV} as explained in the previous paragraph. This allows to test in-distribution on Σtrain and out-of-distribution on Sheld-out .
Takeaway: Skill Neologisms learn composable skill representations that successfully compose with OOD skills.
Models For each new skill Snew ∈ Σtest and held-out skill Sheld-out ∈ Σpretrain , we train three model variants on 100k samples from D(Snew , Σpretrain \ Sheld-out ) with up to kmax = 3 operations: (1) Skill Neologisms with length l = 20, initialized from the mean embedding of Σpretrain operation tokens; (2) Prompt Tuning (Lester et al., 2021)
5.2.2. P ROPERTY 3: M ULTI -S KILL C OMPOSITION We test whether the skill neologisms learned independently for SHIFT and INV-POL in § 5.2.1 can be combined zeroshot to handle compositions requiring both skills (Prop6
Skill Neologisms: Towards Skill-based Continual Learning
C 2 (Snew , Σ train )
1.0
C 2 (Snew , Sheld-out )
Accuracy
0.75
0.60
0.5
SHIFT
0.16
0.0 1.0 0.5 0.0
Snew
0.28
LoRA
Sheld-out :
PT
Skill Neologisms
LoRA
ADD
DESC
ASC
0.84
0.57
PT
INV-POL
Skill Neologisms POL
ID
SUB
Figure 4. Accuracy on 2-combinations of skills mixing Snew with Σtrain (in-distribution) or Sheld-out (out-of-distribution). Dotted lines show the average accuracy across all Sheld-out . PT: Prompt Tuning. Skill Neologisms
erty 3). This distinguishes skill neologisms from LoRA and Prompt Tuning-like approaches, which cannot be composed after independent training without retraining on the joint task. We compare against in-context learning (ICL)– a natural baseline for zero-shot composition–by providing Mpretrain with N ∈ {10, 20, 50, 100} examples from D(Snew , Σpretrain ) for Snew ∈ {SHIFT, INV-POL} (2N examples in total).
In-Context Learning
Accuracy
0.6 0.4 0.2 0.0
Figure 5 shows the average accuracy across different Sheld-out for Skill Neologisms and N for ICL, for different sequence lengths (increasing task difficulty). Traces for individual runs are shown Figure A3 in the Appendix. Skill neologisms significantly outperform ICL across all sequence lengths. This demonstrates that skill neologisms successfully capture reusable procedural knowledge that transfers zero-shot to new compositions, whereas ICL struggles to extract and combine the relevant patterns from examples alone.
2
3
4
5
6
Sequence Length ( harder)
7
8
Figure 5. Zero-shot composition of SHIFT and INV-POL. Skill Neologism: we compose the skill tokens learned independently for SHIFT and INV-POL for different a given Sheld-out , and plot the average accuracy (±std) across the 6 Sheld-out . In-context learning: we provide in-context N = {10, 20, 50, 100} examples sampled from D(Snew , Σpretrain ) for Snew ∈ {SHIFT, INV-POL} (2N examples in total), and plot the average accuracy(±std) across the 4 runs.
distribution, as we observed with LoRA in Figure 4. This suggests a trade-off on the length of the skill neologism: too few parameters may fail to learn the skill, while too many may reduce generalization to OOD compositions.
Takeaway: Independently learned skill neologisms can be successfully composed zero-shot.
Setup We train skill neologisms for SHIFT and INV-POL with Sheld-out = ADD, varying the neologism length from l = 1 (|θS | = 768 parameters) to l = 200 (|θS | = 153k parameters).
5.3. Insights and Ablation Experiments Having validated that skill neologisms satisfy Properties 2 and 3, we study and ablate different components to understand the mechanisms at play. We focus on three questions: (1) How does the capacity of skill tokens affect their ability to learn composable representations? (2) How does the diversity of skill combinations in training data impact generalization? (3) Is performance sensitive to initialization?
Results Figure 6 shows the accuracy on 2-combinations with Σtrain (ID) and Sheld-out (OOD) for varying skill lengths. The model gets near-perfect accuracy indistribution for l ≥ 5. However, the accuracy out-ofdistribution first increases with higher capacity, but then drops as l becomes too large (l > 20). This suggest that after a certain point, increased capacity for the skill tokens becomes detrimental to learning a generally composable representation of the skill.
5.3.1. S KILL N EOLOGISM L ENGTH Motivation We hypothesize that limited capacity of skill tokens provides an inductive bias to learn generally composable representations rather than overfitting to the training 7
Accuracy
1
Accuracy
Skill Neologisms: Towards Skill-based Continual Learning
1
C 2 (Snew , Σ train )
initialization against initialization from the average embedding of tokens in Σpretrain (see Appendix A.5 for detailed results across Sheld-out ). Initialization from pretrained skill embeddings shows marginally better performance (particularly for INV-POL on 2-skill compositions), but skill tokens trained from random initialization still show strong OOD composition abilities.
Skill Length
1 5 10 20 50 100 200
0
0
0.79
C 2 (Snew , Sheld-out )
0.89 0.94 0.92 0.89 0.88
0.74 0.09
0.02
SHIFT
0.45 0.48
Snew
0.10
INV-POL
0.00 0.00
Table 6. Effect of initialization. OOD accuracy(±std) on 2-skill and 3-skill combinations for random initialization versus initialization from average embeddings of pretrained skills Σpretrain . Results averaged across all held-out skills Sheld-out .
Figure 6. Effect of skill token length. Accuracy on 2-skill combinations with Sheld-out = ADD for varying skill token length l.
Takeaway: The limited capacity of skill neologisms acts as an inductive bias to learn more composable skill representations.
Snew
Init Method
C2 (Snew , Sheld-out )
C3 (Snew , Sheld-out , Σtrain )
INV-POL INV-POL
From Σpretrain Random
.85 ± .06 .63 ± .32
.65 ± .23 .58 ± .31
SHIFT SHIFT
From Σpretrain Random
.72 ± .18 .70 ± .19
.69 ± .14 .65 ± .14
5.3.2. C OMPOSITION C OMPLEXITY IN T RAINING S ET The complexity of skill combinations in the training set provides another source of inductive bias: exposing skill tokens to the target skill in more complex compositions (higher kmax ) may improve their ability to compose with held-out skills.
6. Discussion Related Work Our work relates to three main research directions (detailed comparison in Appendix B). First, prior research has investigated skills and compositional abilities in LLMs using in-context skill descriptions, synthetic skillrich data, or skill-targeted training. In contrast, we learn generally composable skill representations via soft tokens integrated into the model vocabulary. Second, while recent prefix-based adaptation methods improve transferability across tasks, they ultimately require training on the target task. We instead adopt a skill-centric perspective, focusing on out-of-distribution generalization and zero-shot composition of independently learned skills. Finally, meaningful soft tokens have been studied for visual concepts, tool representations, or prompt compression. To the best of our knowledge, our work is the first to propose learning composable soft tokens that encapsulate specific procedural knowledge.
Setup. We train neologisms for SHIFT and INV-POL for various Sheld-out , while varying the maximum number of compositions kmax ∈ {1, 2, 3} in the training set, keeping the total number of samples fixed at 100k. We compare the OOD accuracy on 2- and 3-compositions involving the held-out skill. Results. Table 5 shows the accuracy averaged across all Sheld-out (see Appendix A.4 for detailed results across Sheld-out ). Training on more compositions in the training data generally improves OOD generalization. In particular, 2-skill compositions benefit from having been trained on 3-skill composition data for INV-POL.
Limitations and Future Work Our work constitutes an initial proof-of-concept of skill neologisms as a path towards skill-based continual learning, focusing on a controlled experimental setting. Further work is needed to explore skill neologisms in more realistic settings. Key challenges include the construction and availability of diverse skill-centered datasets, as well as the optimization instability inherent to training soft tokens (see detailed discussion in Appendix C).
Table 5. Effect of composition complexity in the training set. OOD accuracy(±std) on 2-skill and 3-skill combinations when training with varying maximum composition complexity kmax . Results averaged across all held-out skills Sheld-out (see Appendix A.4 for a detailed breakdown across Sheld-out ). Snew
kmax
INV-POL
1 2 3 1 2 3
SHIFT
C2 (Snew , Sheld-out )
C3 (Snew , Sheld-out , Σtrain )
.36 ± .44 .60 ± .34 .85 ± .06 .46 ± .04 .72 ± .21 .72 ± .18
.05 ± .06 .61 ± .24 .65 ± .23 .55 ± .04 .61 ± .19 .69 ± .14
Conclusion We propose skill neologisms as a way to extend LLM capabilities to specific skills by optimizing vocabularyintegrated soft tokens on skill-centric data. By demonstrating compositional transfer to out-of-distribution skills and zero-shot combination of independently learned soft tokens, our findings confirm that skill neologisms are a promising direction for scalable skill-based continual learning.
5.3.3. I NITIALIZATION ROBUSTNESS Prompt Tuning-like methods are known to depend on initialization (Lester et al., 2021). Table 6 compares random 8
Skill Neologisms: Towards Skill-based Continual Learning
References
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. RULER: What’s the real context size of your long-context language models? In COLM, 2024.
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
Arora, S. and Goyal, A. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936, 2023.
Kaur, S., Park, S., Goyal, A., and Arora, S. Instruct-skillmix: A powerful pipeline for llm instruction tuning. In ICLR, 2025.
Asai, A., Hashimoto, T., Hajishirzi, H., Socher, R., and Xiong, C. Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. EMNLP, 2022.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
Chen, J., Pan, X., Yu, D., Song, K., Wang, X., Yu, D., and Chen, J. Skills-in-context prompting: Unlocking compositionality in large language models. arXiv preprint arXiv:2308.00304, 2023. Didolkar, A., Goyal, A., Ke, N. R., Guo, S., Valko, M., Lillicrap, T., Jimenez Rezende, D., Bengio, Y., Mozer, M. C., and Arora, S. Metacognitive capabilities of llms: An exploration in mathematical problem solving. NeurIPS, 2024.
Kuratov, Y., Arkhipov, M., Bulatov, A., and Burtsev, M. Cramming 1568 tokens into a single vector and back again: Exploring the limits of embedding space capacity. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 19323–19339, 2025.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. EMNLP, 2021.
Genewein, T., Li, K. W., Grau-Moya, J., Ruoss, A., Orseau, L., and Hutter, M. Understanding prompt tuning and in-context learning via meta-learning. arXiv preprint arXiv:2505.17010, 2025.
Levy, I., Bogin, B., and Berant, J. Diverse demonstrations improve in-context compositional generalization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1401–1422, 2023.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Li, R., Fu, J., Zhang, B.-W., Huang, T., Sun, Z., Lyu, C., Liu, G., Jin, Z., and Li, G. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852, 2023.
Hao, S., Liu, T., Wang, Z., and Hu, Z. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. NeurIPS, 2023.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 4582–4597, 2021.
He, T., Doshi, D., Das, A., and Gromov, A. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. NeurIPS, 2024.
Liu, A. H., Khandelwal, K., Subramanian, S., Jouault, V., Rastogi, A., Sadé, A., Jeffares, A., Jiang, A., Cahill, A., Gavaudan, A., et al. Ministral 3. arXiv preprint arXiv:2601.08584, 2026.
He, Y., Panigrahi, A., Lin, Y., and Arora, S. STAT: Skilltargeted adaptive training. In The 5th Workshop on Mathematical Reasoning and AI at NeurIPS 2025, 2025.
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. Few-shot parameter-efficient finetuning is better and cheaper than in-context learning. NeurIPS, 2022a.
Hewitt, J., Geirhos, R., and Kim, B. Position: We can’t understand ai using our existing vocabulary. In ICML, Position Paper Track, 2025. 9
Skill Neologisms: Towards Skill-based Continual Learning
Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z., Yang, Z., and Tang, J. P-tuning: Prompt tuning can be comparable to finetuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61–68, 2022b.
Wang, Z., Lamb, A., Saveliev, E., Cameron, P., Zaykov, Y., Hernández-Lobato, J. M., Turner, R. E., Baraniuk, R. G., Barton, C., Jones, S. P., et al. Instructions and guide for diagnostic questions: The neurips 2020 education challenge. arXiv preprint arXiv:2007.12061, 2020.
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J. Gpt understands, too. AI Open, 5:208–215, 2024. ISSN 2666-6510. doi: https://doi.org/10.1016/j. aiopen.2023.08.012.
Wang, Z., Panda, R., Karlinsky, L., Feris, R., Sun, H., and Kim, Y. Multitask prompt tuning enables parameterefficient transfer learning. In ICLR, 2023. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
Liu, Z., Liu, Q., Guo, T., Chen, J., Huang, S., Zhao, X., Tang, J., Luo, W., and Weng, J. Xes3g5m: A knowledge tracing benchmark dataset with auxiliary information. NeurIPS, 2023.
Xu, Z., Shi, Z., and Liang, Y. Do large language models have compositional ability? an investigation into limitations and scalability. In First Conference on Language Modeling, 2024. URL https://openreview.net/ forum?id=iI1CzEhEMU.
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing, 2025.
Yu, D., Kaur, S., Gupta, A., Brown-Cohen, J., Goyal, A., and Arora, S. Skill-mix: a flexible and expandable family of evaluations for ai models. In ICLR, 2024.
Mu, J., Li, X., and Goodman, N. Learning to compress prompts with gist tokens. NeurIPS, 2023. Petrov, A., Torr, P., and Bibi, A. When do prompting and prefix-tuning work? a theory of capabilities and limitations. In ICLR, 2024.
Yuan, J., Peng, T., Jiang, Y., Lu, Y., Zhang, R., Feng, K., Fu, C., Chen, T., Bai, L., Zhang, B., et al. Mme-reasoning: A comprehensive benchmark for logical reasoning in mllms. arXiv preprint arXiv:2505.21327, 2025.
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023.
Zhao, H., Kaur, S., Yu, D., Goyal, A., and Arora, S. Can models learn skill composition from examples? NeurIPS, 2024.
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report, 2025. URL https: //arxiv.org/abs/2412.15115.
Zheng, M., Chen, H., Guo, T., Zhu, C., Zheng, B., Xu, C., and Wang, Y. Enhancing large language models through adaptive tokenizers. NeurIPS, 2024.
Radevski, G., Gashteovski, K., Hong, G., Lawrence, C., and Glavaš, G. Compositional steering of large language models with steering tokens. arXiv preprint arXiv:2601.05062, 2026. Sastre, I. and Rosá, A. Memory tokens: Large language models can generate reversible sentence embeddings. arXiv preprint arXiv:2506.15001, 2025. Vu, T., Lester, B., Constant, N., Al-Rfou, R., and Cer, D. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 5039–5059, 2022. 10
Skill Neologisms: Towards Skill-based Continual Learning
A. Extended Results A.1. Model pre-training
Accuracy
Figure A1 shows the accuracy Mpretrain after pre-training (same as Table 4), across sequence lengths and operations. Sequence lengths {2, 3, 4, 6, 8} are in-distribution, while lengths {5, 7, 9} were held-out from pre-training data. The model successfully learns most operations over the training distribution and generalizes to unseen sequence lengths. The only exception is REV, which does not generalize to OOD sequence lengths.
ADD
1 0
ASC
DESC
ID
POL
REV
SUB
23456789 23456789 23456789 23456789 23456789 23456789 23456789
Sequence Length
Figure A1. Accuracy of Mpretrain sequence lengths for each pre-train operations. Sequence lengths {2, 3, 4, 6, 8} are in-distribution, while lengths {5, 7, 9} were held-out from pre-training data.
A.2. Out-of-distribution generalization Following the experimental setup from Section 5.2.1, Figure A2 show the ID and OOD accuracy on 3-compositions of skills. On OOD samples samples are drawn from C3 (Snew , Sheld-out , Σtrain ), where Snew and Sheld-out are always included and one operation from Σtrain is sampled, and the order of the three operations is randomly permuted. A.3. Multi-skill composition Figure A3 show detailed results from Section 5.2.2 with accuracy on individual pairs of neologisms (for a given Sheld-out ) for Skill Neologisms, and individual number of examples N for ICL. A.4. Effect of compositions in training set Table A1 shows the detailed accuracy across Sheld-out skills for the experiment presented Section 5.3.2. Table A1. Effect of number of compositions in training set
Acc on C2 (Snew , Sheld-out ) Snew
k-ops
INV-POL
1 2 3 1 2 3
SHIFT
Acc on C3 (Snew , Sheld-out , Σpretrain )
ASC
DESC
ADD
SUB
ID
POL
AVG
ASC
DESC
ADD
SUB
ID
POL
AVG
.15 .43 .73 .48 .40 .52
.06 .80 .80 .51 .50 .46
.00 .46 .88 .40 .95 .92
.00 .00 .86 .42 .92 .87
.99 .90 .89 .46 .84 .87
.95 .98 .91 .51 .71 .70
.36 .60 .85 .46 .72 .72
.07 .76 .83 .58 .69 .82
.03 .81 .78 .62 .76 .73
.00 .33 .21 .51 .38 .49
.00 .22 .47 .53 .37 .54
.02 .83 .81 .53 .90 .88
.18 .73 .82 .56 .58 .70
.05 .61 .65 .55 .61 .69
A.5. Effect of compositions in training set Table A2 shows the detailed accuracy across Sheld-out skills for the initialization ablation presented Section 5.3.3.
B. Extended Related Work Skills and compositional abilities in LLMs Recent works have proposed ways to extend model capabilities to new skills and compositions. Skill-in-Context (Chen et al., 2023) aims to elicit compositional abilities in LLMs by providing in-context a description of skills and step-by-step explanation on how to compose them. Zhao et al. (2024) show that training LLMs 11
Skill Neologisms: Towards Skill-based Continual Learning
C 3 (Snew , Σ train )
1.0
C 3 (Snew , Sheld-out )
Accuracy
0.70
0.70
0.5
0.32
0.0 1.0 0.5 0.0
Snew SHIFT
LoRA
Sheld-out :
PT
Skill Neologisms
LoRA
ADD
DESC
ASC
0.65
0.53
0.43
PT
Skill Neologisms POL
ID
INV-POL
SUB
Figure A2. Accuracy on 3-combinations of skills mixing Snew with Σtrain (in-distribution) or Sheld-out (out-of-distribution). Dotted lines show the average accuracy across all Sheld-out . PT: Prompt Tuning.
Skill Neologisms
In-Context Learning
Sheld-out
Accuracy
0.6 0.4
DESC ADD
# Examples
0.2 50 20 0.0
100 10
2
3
4
5
6
7
Sequence Length ( harder)
8
POL SUB ASC ID
Figure A3. Zero-shot composition of SHIFT and INV-POL. Skill Neologism: we compose the skill tokens learned independently for SHIFT and INV-POL for different a given Sheld-out (thin blue lines), and plot the average accuracy (±std) across the 6 Sheld-out (thick dashed blue line). In-context learning: we provide in-context N = {10, 20, 50, 100} examples sampled from D(Snew , Σpretrain ) for Snew ∈ {SHIFT, INV-POL} (2N examples in total), and plot individual results (thin orange lines) and the average (thick dashed orange line) across the 4 runs. Thin lines show the individual runs across Sheld-out and N .
on skill-rich synthetic datasets improve compositional abilities, even on held-out skills unseen during training. STAT (He et al., 2025) aims to improve model capabilities by uncovering specific skills lacking from the model, and targeting these skills via either reweighting or synthetic data augmentations. Didolkar et al. (2024) demonstrated that LLMs have the ability to describe skills required by a given task, while Kaur et al. (2025) leveraged such metacognition abilities of LLMs to create a skill-rich synthetic dataset for instruction-tuning. In contrast, we propose learning generally composable representation of skills via soft tokens, allowing composition with other skills thanks to the pre-trained model’s general compositional abilities. Prefix-Based Adaptations Prompt tuning (Lester et al., 2021) first introduced the paradigm of training soft tokens appended to the input prompt to adapt a frozen model to new tasks, which P-Tuning (Liu et al., 2024) extended by mixing soft prompts produced by a prompt encoder with discrete text tokens. Concurrently, prefix tuning (Li & Liang, 2021) proposed learning prefix key and value vectors at every layer of the model–yielding more expressive power than the input layer only–, which P-Tuning v2 (Liu et al., 2022b) extended to natural language understanding (NLU) settings. Several works have focused on enhancing the transferability of prompt tuning. SPoT (Vu et al., 2022) train prompts across diverse tasks to transferability to new ones; Multitask Prompt Tuning (MTP) (Wang et al., 2023) decomposes prompts between shared and task-specific components; ATTEMPT (Asai et al., 2022) combines prompts from different tasks using an attention mechanism. However, these methods still require training on the target task, unlike skill neologisms that can combine independently learned soft-prompts zero-shot. Meaningful Soft Tokens Another line of research has focused on learning soft tokens with specific, grounded meanings, 12
Skill Neologisms: Towards Skill-based Continual Learning Table A2. Effect of initialization. OOD accuracy on 2-skill and 3-skill combinations for random initialization versus initialization from average embeddings of pretrained skills Σpretrain .
Snew
Init Method
INV-POL
From Spretrain Random From Spretrain Random
SHIFT
ADD
ASC
DESC
Acc on C2 (Snew , Sheld-out ) ID POL SUB AVG
ADD
ASC
.88 .44 .92 .88
.73 .80 .52 .46
.80 .81 .46 .41
.89 .88 .87 .87
.21 .23 .49 .50
.83 .80 .82 .77
.91 .88 .70 .70
.86 .00 .87 .85
.85 .63 .72 .70
Acc on C3 (Snew , Sheld-out , Σpretrain ) DESC ID POL SUB AVG .78 .79 .73 .71
.81 .78 .88 .87
.82 .83 .70 .54
.47 .05 .54 .50
.65 .58 .69 .65
moving beyond their use as purely task-specific adapters. In the vision-language domain, Textual Inversion (Gal et al., 2023) learns a new pseudo-word in the embedding space of a frozen model to represent a novel visual concept, such as a specific object or artistic style. In function calling and tool use for LLMs, ToolkenGPT (Hao et al., 2023) represents tools via tokens integrated in the model vocabulary. In prompt compression, memory tokens (Sastre & Rosá, 2025; Kuratov et al., 2025) compress long sequences of text into a single reversible embedding, while gist tokens (Mu et al., 2023) replace prompts with gist tokens that preserve downstream model behavior. Recently, Radevski et al. (2026) proposed learning composable steering tokens for behavioral alignment. To the best of our knowledge, our work is the first to learn composable soft tokens that encapsulate specific procedural knowledge.
C. Extended Limitations Skill-centered dataset construction While we argue that skill-centered datasets can be identified or constructed in a variety of contexts, their availability for a given skill remains a key requirement for learning skill neologisms. Moreover, as our experiments suggest, the quality of the learned neologisms partly depends on the complexity of the data and on how the target skill is mixed with a diverse set of other skills during training. Assessing and ensuring such diversity may not be straightforward in all settings. Scope of applicability Our work constitutes a proof-of-concept of skill neologisms as a potential approach to skill-based continual learning. We therefore focus on a controlled experimental setting, as described at the end of Section 5.1. Further work is needed to investigate how skill neologisms could be deployed in more realistic scenarios. Soft token limitations Skill neologisms rely on optimizing soft tokens, in a manner similar to prompt tuning. As a result, they inherit several limitations commonly associated with prompt tuning, including sensitivity to initialization and to hyperparameters such as token length and learning rate. In addition, successful learning soft tokens requires that the target task remains reasonably close to the model’s pretraining distribution, as shown in Petrov et al. (2024) and Genewein et al. (2025). Computational cost Although skill tokens are substantially more parameter-efficient than standard fine-tuning methods, training them still requires backpropagation through the full model. This leads to computational costs that can be comparable to those of fine-tuning in practice. As a result, training skill neologisms for large-scale models (e.g., > 30B parameters) may remain challenging without access to substantial computational resources.
13
Skill Neologisms: Towards Skill-based Continual Learning
D. Experimental Details: Section 4 D.1. Experimental Setup Models Evaluated: • Qwen2.5-7B (Qwen et al., 2025) • Llama-3.1-8B (Grattafiori et al., 2024) • Ministral-3-8B-Base-2512 (Liu et al., 2026) • Phi-4 (Abdin et al., 2024) Tasks: Binary operations XOR and XNOR on 3-bit sequences. Dataset Configuration: • Test samples: 100 per task (XOR, XNOR) • Bit length: 3 • In-context examples: 3 examples per prompt • Example format: Each sample contains 3 input-output pairs followed by a query input Prompt Variations: Three prompt formulations were tested for each task: 1. Only Examples (Baseline): No additional context provided, only the 3 in-context example pairs 2. Examples + Keyword: A symbolic keyword prefix is added before the examples • XOR: “Complete the following using the skill: ‘XOR’ ” • XNOR: “Complete the following using the skill: ‘XNOR’ ” 3. Examples + Text Description: A natural language description is provided • XOR: “Complete the following using the skill: ‘output 1 iif both input bits are different, and 0 otherwise’ ” • XNOR: “Complete the following using the skill: ‘output 1 iif both input bits are the same, and 0 otherwise’ ” Example Prompt Structure: For the “Examples + Keyword” variant (XOR): Complete the following using the skill: ’XOR’ 101 011 = 110 100 110 = 010 011 001 = 010 111 010 = For the “Only Examples” variant: 101 011 = 110 100 110 = 010 011 001 = 010 111 010 = For the “Examples + Text Description” variant (XOR): 14
Skill Neologisms: Towards Skill-based Continual Learning
Complete the following using the skill: ’output 1 iif both input bits are different, and 0 otherwise’ 101 011 = 110 100 110 = 010 011 001 = 010 111 010 = D.2. Evaluation Details Inference Parameters: • Batch size: 16 • Generation method: Greedy decoding (deterministic) • Padding side: left • Models run in evaluation mode Metrics: • Exact Match Accuracy: Percentage of test samples where the model’s generated output exactly matches the ground truth q where p is accuracy and N = 100 • Standard Error: Computed assuming binomial distribution: SE = p(1−p) N
E. Experimental Details: Section 5 This appendix provides comprehensive details for all experiments presented in the main paper. E.1. Base Model Pretraining All experiments in Section 5 use a pretrained Qwen2.5-0.5B model trained on a digit-sequence transformation tasks. Table E1 summarizes the pretraining configuration. Table E1. Base model pretraining configuration. The model was trained in two phases: Phase 1 on single operations, Phase 2 on compositions of 1–3 operations. Parameter
Phase 1
Base Model PEFT Method Target Modules Training Samples Test Samples Operations per Sample Epochs Batch Size Learning Rate Warmup Steps Operations Sequence Lengths Held-out 3-op combinations
Phase 2
Qwen/Qwen2.5-0.5B LoRA (r=32, α=32) q, k, v, o, gate, up, down 100,000 500 1 3
500,000 500 1–3 3
64 2e-4 500
64 2e-4 500
[ASC], [DESC], [ADD], [SUB], [POL], [REV], [ID] 2, 3, 4, 6, 8 (held-out: 5, 7, 9) –
15
25%
Skill Neologisms: Towards Skill-based Continual Learning
E.2. Skill Neologisms Insertion function In our experiments Section 5, the insertion function ϕ simply swaps the tokens corresponding to the target skill (e.g. ”[SHIFT]”) with the skill tokens of length l in the prompt. E.3. Compositional Transfer Experiments (Figure 4) Table E2 summarizes the configuration for each method in Figure 4. Table E2. Configuration for compositional transfer experiments (Figure 4). All methods learn one of [SHIFT] and [INV-POL] and are evaluated on compositions with held-out pretrain operations. Parameter
Skill Neologisms
Prompt Tuning
LoRA
Trainable Structure Soft Tokens Length/Rank Trainable Params
Vocab. tokens 20 17,920
Prefix tokens 20 17,920
LoRA adapters r=16 ∼2.9M
Initialization
Mean of pretrain op. embeddings
–
Training Samples Validation Samples Test Samples Operations per Sample Held-out Skill Sequence Lengths
100,000 1,000 200 per Sequence Length and permutation 1–3 (requires Snew + 0–2 from Σtrain ) One operation per run (6 total scenarios) 2, 3, 4, 6, 8 (held-out: 5, 7, 9)
Epochs Learning Rate Batch Size Temperature at Inference
3 5e-3 32
Eval. Metrics
3 5e-3 32 0 (greedy)
3 1e-4 32
Acc. on C2 (Snew , Σtrain ) (ID) Acc. on C2 (Snew , Sheld-out ) (OOD)
Dataset Configuration: Training samples compose Snew with operations from Σtrain (6 of the 7 pretrain operations, with one held out). Training and validation data is distributed equally across operation counts (e.g., for max ops=3, each of 1-op, 2-op, and 3-op receives 100,000 ≈ 33,333 samples). 3 Test Dataset Generation: The test dataset evaluates all permutations of operation orderings to ensure the model learns composable skills rather than memorizing specific sequences. For each k ∈ {2, 3} operations: • Each sample requires exactly one Snew , one Sheld-out , and (k − 2) operations from Σtrain • The order of these 2 (resp. 3) operations is set by sampling one of the 2 (resp. 6) permutations of Snew , Sheld-out , and S ∈ Σtrain . • Ntest = 200 samples are generated for each sequence length and permutations, yielding 400 test samples per sequence length for k=2 and 1200 test samples per sequence length for k=3. Each method is trained on 6 configurations (one per held-out operation) for each of the 2 new skills, yielding 12 runs per method. E.4. Multi-Skill Composition Experiments (Figure 5) Table E3 presents the experimental setup for Figure 5. E.5. Ablation Studies E.5.1. E FFECT OF T RAINING C OMPOSITION C OMPLEXITY Table E4 shows how varying the maximum number of operations during training (max ops) affects generalization. Other parameters are the same as in Section under the ”Skill Neologisms” column. 16
Skill Neologisms: Towards Skill-based Continual Learning Table E3. Configuration for multi-skill composition experiments (Figure 5). Skill neologisms for [SHIFT] and [INV-POL] are learned independently, then composed zero-shot. Parameter
Value
Skill Neologisms Training Composition Method Evaluation
Two independently trained skills (config from Table E2) Insert both skill tokens into test prompts (no joint training) All 6 operations as Sheld-out , averaged per sequence length
In-Context Learning Baseline Examples per Skill N ∈ {10, 20, 50, 100} Total Examples 2N (N for each skill) Examples Pool Size 10,000 samples per skill Test Dataset Test Samples Sequence Lengths Operations per Sample Temperature
50 per sequence length 2–8 2 (both [SHIFT] and [INV-POL] required) 1
Table E4. Effect of training composition complexity. Each row shows results for a different max ops value during training. All configurations use skill length 20. max ops 1,2,3
Epochs
Training Samples
2
100,000
Evaluation: Each configuration is evaluated on both 2-operation and 3-operation compositions with held-out skills. The table in the paper reports mean accuracy across all 6 held-out operations for each Snew . E.5.2. E FFECT OF I NITIALIZATION M ETHOD Table E5 compares initialization strategies for skill token embeddings. Other parameters are the same as in Section under the ”Skill Neologisms” column. Table E5. Initialization method comparison. Both methods use skill length 20, learning rate 5e-3, and 2 epochs of training. Method From Pretrain Random
Description Mean of pretrain operation embeddings Random Gaussian initialization with σ = 0.2
Evaluation: Average accuracy on C2 and C3 compositions across all 6 held-out operations. E.5.3. E FFECT OF S KILL T OKEN L ENGTH (F IGURE 6) Figure 6 shows how skill token capacity affects learning and generalization. Parameters are the same as in Section under the ”Skill Neologisms” column, while only varying the skill token length l ∈ {1, 5, 10, 20, 50, 100, 200}. E.6. Summary of Key Hyperparameters Table E7 provides a unified view of all experimental configurations. E.7. Dataset and Evaluation Details Operations: All experiments use 7 pretrained operations on digit sequences: • [ASC]: Sort digits in ascending order • [DESC]: Sort digits in descending order 17
Skill Neologisms: Towards Skill-based Continual Learning Table E6. Configuration for length ablation experiments. Parameter
Value
Skills Evaluated Fixed Held-out Skill Training Samples max ops
[SHIFT], [INV-POL] [ADD] 100,000 2 (1 or 2 operations per sample)
Epochs Learning Rate Batch Size
1 5e-3 32
Metrics
ID: Acc. on C2 (Snew , Σtrain ) OOD: Acc. on C2 (Snew , ADD)
Table E7. Summary of key hyperparameters across all experiments. SN: Skill Neologisms, PT: Prompt Tuning. Experiment
Epochs
LR
BS
Samples
max ops
Length
Pretraining (Ph. 1) Pretraining (Ph. 2)
3 3
2e-4 2e-4
64 64
100K 500K
1 3
– –
SN (baseline) PT (baseline) LoRA
3 3 3
5e-3 5e-3 1e-4
32 32 32
100K 100K 100K
3 3 3
20 20 –
Variable k-ops Random Init Length Ablation
2 2 1
5e-3 5e-3 5e-3
32 32 32
100K 100K 100K
1–3 3 2
20 20 1–200
• [ADD]: Add 1 to each digit (mod 10) • [SUB]: Subtract 1 from each digit (mod 10) • [POLARITY]: Map odd digits to 1, even to 0 • [REVERSE]: Reverse digit order • [ID]: Identity (no transformation) Two new operations are learned in all main experiments: • [SHIFT]: Right-shift digits cyclically • [INV-POL]: Map odd digits to 0, even to 1 Sequence Lengths: • Training: 2, 3, 4, 6, 8 • Held-out: 5, 7, 9 Sample Format: Each sample follows the pattern [OP-1]...[OP-k]xxxx=yyyy, where xxxx is the input digit sequence and yyyy is the result of applying operations sequentially. Evaluation Metrics: • Exact Match Accuracy: The model must generate the complete correct output sequence. 18
Skill Neologisms: Towards Skill-based Continual Learning
E.8. Computational Resources Model: Qwen/Qwen2.5-0.5B • Embedding Dimension: 896 • Hidden Size: 896 • Layers: 24 • Attention Heads (Q / KV): 14 / 2 • Tie Embeddings: Yes Framework: • HuggingFace Transformers • PEFT library for LoRA • Custom implementation for Skill Neologisms and Prompt Tuning (same implementation for both, simply inserting soft tokens before every prompt for Prompt Tuning) • Weights & Biases for experiment tracking Hardware: Experiments were run on a NVIDIA RTX 6000 GPU (48GB VRAM).
19