CogScale: Scalable Benchmark for Sequence Processing
arXiv:2605.19758v1 [cs.AI] 19 May 2026
Yannis Bendi-Ouis Mnemosyne Inria Bordeaux, France [email protected]
Romain de Coudenhove ENS PSL École normale supérieure (Ulm) Paris, France [email protected]
Xavier Hinaut Mnemosyne Inria Bordeaux, France [email protected]
Abstract The ability to maintain and manipulate information over time is a fundamental aspect of living beings and Artificial Intelligence. While modern models have achieved remarkable success in tasks like natural language processing, evaluating the capacity of novel architectures to process sequential information remains computationally expensive and time-consuming. Testing a new architecture often requires scaling up to massive datasets and models, leading to vast computational costs and slow iteration cycles. In this paper, we propose CogScale, a benchmark of 14 scalable synthetic tasks designed to isolate and evaluate specific cognitive and memory abilities at different parametrizable scales. By providing a standardized, lightweight framework, CogScale allows researchers to rapidly validate architectural innovations before committing to large-scale training. To establish a solid baseline, we evaluate seven distinct architectures: Gated Recurrent Unit (GRU), Long Short-Term Memory (LSTM), xLSTM, Echo State Network (ESN), Mamba, Transformer Decoder, and Transformer Encoder-Decoder. These evaluations are conducted under strict parameter budgets (1k, 10k, and 100k) and across different difficulty levels and scales. Our results show that while classical RNNs and Echo State Networks excel at basic retention within strict parameter budgets, only attention mechanisms and modern state-space models consistently maintain high performance as reasoning complexity and task difficulty scale.
1
Introduction
The development of foundational models like Large Language Models (LLMs) requires architectures that can handle a diverse set of cognitive skills, ranging from information retention to complex reasoning. Historically, the evaluation of these capabilities has relied on massive Natural Language Processing (NLP) benchmarks, such as Lambada [1], WinoGrande [2], PiQA [3] or HellaSwag [4] and necessitate training on massive datasets like OpenWebText (40GB) [5] or The Pile (880GB) [6]. While these evaluations are needed to assess the overall performance of large language models, they only demonstrate meaningful distinctions when evaluated on models with massive parameter counts. This demands an exorbitant amount of computational resources and time, making such approaches highly prohibitive for early architectural exploration. This excludes the analysis of small models and strongly penalizes studies proposing novel architectures, as the training requirements Preprint - 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Track on Evaluations and Datasets.
make architectural iteration cycles slow and costly. Beyond the economic cost, the massive energy consumption required by current scaling laws poses a significant environmental challenge [7, 8], highlighting the need for more sustainable research methodologies [9]. Recent critiques of the “bigger-is-better” [10] paradigm also highlight how this scaling view of AI can reinforce unequal access to research by tying scientific progress to expensive computational infrastructure. It also leaves out researchers with no or limited computing resources and creates a consequent gap between academia (even more in low and middle-income countries) and large companies. Furthermore, existing datasets often lack scalability; for instance, the TSL benchmark [11] is constrained to 69 tasks with fixed sizes and difficulties. To resolve this issue, we introduce CogScale, a lightweight synthetic benchmark designed to serve as a cognitive "sanity check" before initiating any massive language model training. We argue that if a new architecture (aiming to scale on large datasets) is fundamentally incapable of resolving basic cognitive and sequential tasks, or fails to approach the baseline performance of a standard Transformer, it is both futile and environmentally costly to attempt to scale it on massive datasets of text. Beyond serving as a prerequisite for massive scaling, CogScale is equally valuable for developing small efficient models that must generalize from limited data [10]. CogScale provides a minimal evaluation framework, allowing researchers to rapidly validate their architecture for a fraction of the traditional cost. A major feature of CogScale is the scalability of its tasks. Unlike static benchmarks, CogScale allows researchers to modulate the difficulty and complexity of each task, enabling the assessment of various levels of expertise for the same cognitive skill. This scalability is required for verifying that a model’s performance improves effectively as its parameter count increases. Since more complex versions of a task demand greater representational capacity from the model and often require handling larger input and output dimensions, CogScale provides a precise diagnostic tool to ensure that scaling a model’s size genuinely translates to better cognitive capabilities. In this paper, our contributions are twofold. First, we introduce the CogScale framework, which includes 14 scalable synthetic tasks designed to evaluate specific cognitive abilities. Second, we provide a baseline evaluation of seven distinct architectures, including LSTM [12, 13], GRU [14], Transformer Encoder-Decoder (ED) [15] and Decoder-Only (DO) [16], Mamba [17], xLSTM [18] and an atypical but competitive dynamical system via Echo State Networks (ESN) [19, 20]. This experiment is conducted under a rigorous evaluation protocol under strict parameter budgets (1k, 10k, and 100k parameters). It allows us to highlight the strengths and weaknesses of each tested architecture and provides a methodology for ensuring fair comparisons in future architectural research.
2
Related Work
Many different benchmarks propose to isolate specific cognitive abilities without the computational cost of training on massive datasets. Notable works include the Long Range Arena (LRA) [21], designed to evaluate long context retrieval and spatial reasoning, bAbI [22], which introduces toy tasks for logical reasoning, and the Time Series Library [11] for continuous temporal dynamics. However, these existing datasets suffer from structural rigidity, consisting of fixed sequence lengths, static difficulty levels, and thus, a lack of scalability. CogScale distinguishes itself through its modularity. By offering configurable sequence lengths, vocabulary sizes, and difficulty levels, CogScale provides an adaptable evaluation framework that overcomes the rigidity of its predecessors, allowing architectures to be evaluated at different difficulty levels and scales. In order to establish a comparative baseline, a diverse set of sequential architectures has been evaluated. We include classical Recurrent Neural Networks (RNNs), specifically Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU). The field’s current standard, the Transformer, is also tested, allowing us to contrast the original Encoder-Decoder (ED) with the Decoder-Only (DO) architecture. Furthermore, we evaluate recent innovations in sequence processing, notably State Space Models with Mamba and advanced recurrent architectures such as xLSTM, which offer promising alternatives to standard attention mechanisms. Finally, we include the Echo State Network (ESN) from the Reservoir Computing paradigm [23, 24] into our evaluation. While ESNs represent an atypical baseline in contemporary deep learning due to their reliance on a fixed, randomly initialized recurrent reservoir rather than end-to-end backpropagation, they are strong contenders when number of parameters is limited. Including them provides a reference point, allowing us to assess whether the complex modern architectures genuinely yield superior temporal representations or if simple reservoir dynamics suffice for certain cognitive tasks, at a certain difficulty. 2
3
The CogScale Dataset
3.1
Synthetic Generation and Unified Evaluation
Because all tasks in CogScale are synthetic, data samples can be generated dynamically during the training process by setting a specific seed to ensure full reproducibility. This methodological choice presents several major advantages for architectural exploration: it requires zero disk storage, bypassing the bottleneck associated with loading massive datasets, and provides a theoretically infinite data distribution. By perpetually generating novel sequences, the framework effectively prevents models from simply memorizing, forcing them to learn the underlying generative rules of the tasks. Furthermore, CogScale uses a unified evaluation metric system. Depending on the task’s category, the benchmark evaluates the model on designated prediction timesteps using Mean Squared Error (MSE) for regression, error rate (1-Accuracy) for classification, and label-based error rate (1-Label Accuracy) for multi-label classification. All the source code required to generate the CogScale dataset is publicly available.1 3.2 Signal Processing and Forecasting 3.3
Sinus Forecasting t=1
t=2
t=3
t=4
t=5
t=6
t=7
Single Continuous Sinusoidal Signal
Input Sequence X (Continuous Signal)
Target Sequence Y (Forecast t+1)
0.0
0.6
1.0
0.6
0.0
-0.6
-1.0
0.6
1.0
0.6
0.0
-0.6
-1.0
-0.6
In this very simple task, the model is required to predict the future evolution of a sinusoidal signal. The difficulty is scaled by simultaneously extending the sequence length (e.g., from 200 to 2000) and the target forecast horizon (e.g., from 5 to 15).
Evaluation at every timestep
Figure 1: Illustration of Sinus Forecasting. 3.3.1
Chaotic Forecasting t=1
t=2
t=3
t=4
t=5
t=6
t=7
Chaotic Multi-dimensional Sequence
Input Sequence X (Lorenz 3D)
Target Sequence Y (Forecast t+1)
0.1
0.5
1.1
1.6
1.8
1.5
0.9
1.0
1.2
1.0
0.5
-0.1
-0.6
-0.9
1.1
0.9
0.4
-0.2
-0.8
-1.2
-1.4
0.5
1.1
1.6
1.8
1.5
0.9
0.2
1.2
1.0
0.5
-0.1
-0.6
-0.9
-1.0
0.9
0.4
-0.2
-0.8
-1.2
-1.4
-1.3
This task evaluates the model’s ability to model chaotic dynamics by predicting the future state of a three-dimensional chaotic system based on the Lorenz equations [25]. Similar to the sinusoidal task, the difficulty can be scaled by increasing both the sequence length provided to the model and the forecast horizon.
Evaluation
Figure 2: Illustration of Chaotic Forecasting. 3.4 3.4.1
Memory and Retention Discrete and Continuous Postcasting
Input Sequence
t=1
t=2
t=3
t=4
t=5
t=6
t=7
t=8
B
C
D
A
B
B
C
A
Input Sequence
X (Sequence)
t=1
t=2
t=3
t=4
t=5
t=6
t=7
t=8
0.5
-0.2
0.8
-0.4
0.1
0.9
-0.7
0.3
0.0
0.0
0.5
-0.2
0.8
-0.4
0.1
0.9
X (Continuous Signal)
Target Sequence Target Sequence
0
0
0
B
C
D
A
B
Y (Shifted Signal)
Y (Shifted Sequence)
Delay
Evaluation
Delay
Evaluation
Figure 3: Illustration of Discrete Postcasting. Figure 4: Illustration of Continuous Postcasting. Inspired from the Memory Capacity defined by Jaeger [19], postcasting is a pure temporal delay task. The model receives an input sequence and must reproduce it identically after a specified time shift. The scaling mechanism increases the sequence length and the retention delay (e.g., from 5 to 15 timesteps), while the discrete variant also scales the vocabulary size (e.g., from 3 to 8 symbols). 1 https://anonymous.4open.science/r/CogScale/
3
3.4.2
Simple Copy
t=1
Input Sequence
t=3
t=4
t=5
Memorize Sequence
t=2
Delay
Trigger
t=6
t=7
t=8
A
B
C
0
T
0
0
0
0
0
0
0
0
A
B
C
X (Sequence & Delay)
Target Sequence Y (Recall)
Evaluation (Copy)
Figure 5: Illustration of Simple Copy. 3.4.3
Selective Copy t=1
t=2
t=3
t=4
t=5
Selective Encoding
Input Sequence
A
B
C
t=6
t=7
t=8
0
0
A
C
Trigger
D
0
T
X (Sequence + Markers)
Target Sequence Y (Selected Only)
Evaluation
Figure 6: Illustration of Selective Copy. 3.4.4
t=2
t=3
t=4
t=5
t=6
Key-Value Pairs
Input Sequence X (Keys, Values, Query)
A
B
C
D
0
t=7
t=8
Trigger
Query
T
C
0
Target Sequence
D
Y (Recalled Value)
Evaluation
Figure 7: Illustration of Associative Recall.
3.5.1
Building upon the simple copy task and inspired from [17], this task introduces distraction elements. The model must memorize only a specific subset of marked tokens within a larger sequence and output only those targeted elements at the end. In addition to scaling the delay and vocabulary, the difficulty can be increased by scaling the number of target elements to retain (e.g., from 5 to 10 markers).
Associative Recall t=1
3.5
Inspired from [26], the simple copy task requires the model to read an entire sequence, hold it in memory during a silent delay period, and then reproduce the sequence in its entirety after a specific trigger token. Task difficulty is scaled up by increasing the sequence length, extending the waiting delay before the trigger appears, and expanding the vocabulary size.
Also inspired from [26], this task tests associative memory by presenting the model with a sequence of key-value pairs. At the conclusion of the sequence, a seen key is provided as a query, and the model must retrieve and predict the corresponding associated value. The scaling mechanism adjusts the vocabulary size, the total number of pairs to memorize (e.g., from 3 to 8), and the overall sequence length.
Pattern Recognition and Completion Discrete and Continuous Pattern Completion t=1
t=2
t=3
t=4
t=5
t=6
t=7
t=8
t=1
Symbolic Pattern
Input Sequence
A
?
C
A
B
?
A
Input Sequence
B
X (Masked)
Target Sequence
t=2
t=3
t=4
t=5
t=6
t=7
t=8
Continuous Pattern
0.8
-0.4
0.1
0.8
?
0.1
?
-0.4
0.8
-0.4
0.1
0.8
-0.4
0.1
0.8
-0.4
X (Masked Signal)
A
B
C
A
B
C
A
Target Sequence
B
Y (Completed)
Y (Completed Signal)
Evaluation
Evaluation
Evaluation
Evaluation
Figure 8: Illustration of D. Pattern Completion. Figure 9: Illustration of C. Pattern Completion. In these tasks, the model is exposed to a sequence with a periodic motif, but certain values within the pattern are masked. The objective is to identify the underlying pattern to successfully infer the missing components. The complexity scales by increasing the base length of the repeating motif (e.g., from 4 to 10), the total sequence length, and the vocabulary size for the discrete version.
4
3.5.2
Induction Heads
t=1
t=2
t=3
t=4
t=5
t=6
Sequence
Input Sequence X (Repeated Sequence)
A
B
t=7
t=8
Repetition
C
D
Target Sequence Y (Predict Next)
A
B
C
B
C
D
D
Evaluation
Figure 10: Illustration of Induction Heads. 3.6
Inspired from the in-context learning mechanisms observed in Transformer architectures [27], this task presents a sequence where the second half is an exact duplicate of the first half. The model must recognize this structure to predict the next token in the copied sequence. The difficulty can be increased by expanding the sequence length and increasing the vocabulary size.
Reasoning and Algorithmic Manipulation
3.6.1
Adding Problem
t=1
t=2
t=3
t=4
t=5
t=6
Sequence of Numbers (Two Marked)
Input Sequence
t=7
t=8
Trigger
4
7
2
8
5
1
T
0
0
0
0
0
0
0
0
12
X (Numbers + Markers)
Target Sequence Y (Sum)
Evaluation
Figure 11: Illustration of Adding Problem.
3.6.2
Inspired from [28], the model receives a sequence of random numbers as symbols along with positional markers. Once a trigger signal is received, it must compute and output the sum of only the marked numbers. Scaling increases the sequence length, making the localization of markers more challenging, and raises the maximum possible value of the numbers to be added (e.g., from 3 to 8), effectively increasing the vocabulary size.
Sorting Problem t=1
t=2
t=3
Unsorted Items
Input Sequence X (Symbol + Position)
Target Sequence Y (Sorted Symbols)
t=4
t=5
t=6
t=7
Trigger
3
1
2
B
C
A
T
0
0
0
0
0
0
0
C
A
B
Evaluation (Sorted Output)
The model is provided with a random sequence of symbols, each paired with a randomized target position. After a trigger signal, the model must output the entire sequence sorted into the correct positional order. Scaling this task involves increasing both the sequence length and the vocabulary size, which exponentially inflates the number of possible positional permutations.
Figure 12: Illustration of Sorting Problem.
3.6.3
Bracket Matching
This task evaluates hierarchical context maintenance. The sequence consists of opening and Input Sequence ( ( ) ( ) ) T 0 closing parentheses with random mutations, and the model must determine at the final timestep whether the entire string represents a valid hierTarget Sequence 0 0 0 0 0 0 0 1 archy. The scaling mechanism increases both the total sequence length and the maximum allowable depth of the bracket hierarchy (e.g., from 5 Figure 13: Illustration of Bracket Matching. to 10). t=1
t=2
t=3
t=4
t=5
t=6
t=7
t=8
Bracket Sequence (Valid)
X (Brackets)
Y (Validity)
Evaluation
5
3.6.4
Cross Situation
t=1
t=2
t=3
Situation 1
Input Sequence X (Sentence)
Target Sequence Y (Combined Roles)
t=4
t=5
Link
t=6
t=7
t=8
Defined in [29, 30] as a simplified natural language reasoning problem encoded in one-hot vectors, Cross Situation requires the model to read a sentence describing two crossed situations involving objects, colors, and positions. It must then infer the correct roles and attributes at the end of the sequence. The task features linguistic complexities such as synonyms (e.g., "center" and "middle" representing the same spatial label) and polysemous words (e.g., "orange" representing both a color and a fruit object). The difficulty scales by expanding the entities available, shifting from a restricted vocabulary (e.g., 2 objects, 2 colors, 2 positions) to a broader one (e.g., 8 objects, 8 colors, 8 positions).
Situation 2
a
blue
glass
and
a
red
cup
.
0
0
0
0
0
0
0
Y
Evaluation (Multi-Classes)
Figure 14: Illustration of Cross Situation.
4
Experimental Setup
To guarantee a fair comparison across fundamentally diverse architectures, we employ a strict parameter matching protocol. Rather than relying on arbitrary default hyperparameter configurations that might inadvertently favor one model over another, we utilize a binary search algorithm to dynamically adjust the hidden dimensions. This algorithm forces every architecture to adhere to predefined parameter budgets of 1k, 10k, and 100k trainable parameters. Furthermore, to ensure statistical robustness and mitigate the high variance often introduced by random weight initialization in small models, every experimental configuration is evaluated across 10 seeds, on two pre-configured difficulty levels: small (SM) and medium (MD) (refer to Appendix B for full tasks configurations). For all differentiable models (GRU, LSTM, xLSTM, Transformer Decoder-Only (DO) and Transformer Encoder-Decoder (ED)), training is conducted using PyTorch [31] with the Adam optimizer [32], utilizing a fixed batch size of 10 samples, a maximum number of 200 epochs and an early stopping of 10 epochs to prevent overfitting and ensure fair comparison across all architectures. We also manage sequence packing and padding, which is particularly crucial for architectures like xLSTM that typically operate on unpadded streams. Since deep learning models are sensitive to optimization hyperparameters at smaller scales, we perform a grid search over five distinct learning rates (10−2 , 3 × 10−3 , 10−3 , 3 × 10−4 , 10−4 ). Each learning rate is tested across the 10 seeds, resulting in 50 independent training runs per task, difficulty level, and parameter size combination. A model’s peak performance is highlighted through the best overall score, representing the single best result achieved across all 150 training runs (10 seeds x 5 learning rates x 3 model sizes). This metric is used in our Cognitive Radar (Figure 15). Furthermore, we report the mean and standard deviation of the test scores for the optimal configuration, determined by selecting the learning rate that yields the best average validation score across its 10 seeds. Full score descriptions for both measures are available in Appendix A. The Echo State Network (ESN) baseline provides a radically different computational paradigm, as it uses a fixed, randomized dynamical reservoir and does not rely on backpropagation. Consequently, standard deep learning training protocols are inapplicable. Instead, we implement a large, parallelized hyperparameter search protocol to optimize the reservoir’s dynamics. This search explores thousands of configurations across three critical hyperparameters: the leaking rate, the spectral radius, and the input scaling. For each reservoir configuration, the optimal Ridge for the linear readout layer is automatically selected on the validation set. All the source code required to reproduce all the experiments is publicly available.2
2 https://anonymous.4open.science/r/CogScale/
6
5
Results
5.1
Peak Cognitive Capabilities (The Cognitive Radar)
Figure 15: Cognitive Radar visualizing peak performance (Accuracy) for the seven baseline architectures across six selected tasks. Plots contrast architectural capabilities on small difficulty (left) and medium difficulty (right), based on the "best overall" scores (refer to Appendix A for full score description). Larger colored areas signify superior cognitive abilities, illustrating how Transformers and modern State Space Models maintain better performance under increased difficulty compared to classical recurrent and reservoir computing models. To evaluate the strengths of each architecture, we first analyze their peak cognitive capabilities using the "best overall" metric on the small (SM) and medium (MD) difficulty levels. By aggregating these optimal performances we construct a "Cognitive Radar" (Figure 15) that provides an immediate visual profile of each model. We can notably observe that attention architectures (Transformer models) demonstrate a superiority in reasoning and manipulation, achieving perfect scores (0.00 error) on tasks such as simple copy, adding problem, and induction heads. The modern State Space Model, Mamba, is highly competitive with Transformers (DO and EC) across these six discriminative tasks, yet it generally remains slightly behind. This difference in performance is more noticeable for medium tasks. For instance, while Transformers maintain perfect or near-perfect accuracy on simple copy and induction heads, Mamba experiences a noticeable degradation in performance. In contrast, the Echo State Network (ESN) shows a distinct duality. While the ESN performs remarkably well on the small difficulty level tasks that require memory retention, like on simple copy or induction heads, it struggles when confronted with tasks requiring reasoning, selection, and manipulation, such as the sorting problem and selective copy. 5.2
Impact of Task Difficulty (Small vs. Medium)
Transitioning from the small to the medium difficulty level involves extending sequence lengths, increasing delays, expanding vocabulary sizes and increasing the number of samples which is ideal to tests the scalability of architectures. This increase in complexity causes a performance drop for several baseline models. For example, on simple copy, the ESN’s error spikes drastically from 0.00 to 0.70, while the standard LSTM degrades from 0.57 to 0.84. A similar collapse is observed for the ESN on the adding problem, where its error jumps from 0.00 to 0.69. In contrast, modern architectures resist this difficulty scaling remarkably well. Transformers (both variants) and the xLSTM maintain near perfect error rates on the medium difficulty of the sorting problem and bracket matching tasks, demonstrating that their memory mechanisms are far more robust to increased difficulty than classical recurrent networks. On the other hand, Mamba really benefits from more samples, even if task difficulty is increased.
7
5.3
Scaling Behavior and Parameter Efficiency (1k to 100k)
Since we tested different model sizes (1k, 10k, and 100k parameters), we can observe the scaling behavior of each architecture on 10 training seeds. Generally, scaling from 1k to 10k parameter brings significant and consistent performance gains across the majority of deep learning models. However, scaling to 100k parameters does not always guarantee better results on these synthetic tasks. We observe that standard and advanced recurrent architectures (LSTM, GRU, and xLSTM) often have an optimal parameter threshold beyond which performance degrades. For example, the GRU’s performance on the small adding problem task falls from a 0.36 average error at 10k parameters to 0.55 at 100k parameters. On the other hand, attention architectures (Transformers) and State Space Models (Mamba) appear more resilient, stabilizing or continuing to improve their performance at the 100k scale without suffering from parameter inefficiency. Finally, it is important to note that the Echo State Network (ESN) was only evaluated for 1k and 10k parameter. Even so, we observe incredible error reductions when expanding the model size from 1k to 10k: the error rate drops from 0.47 to 0.02 on the small adding problem, from 0.57 to 0.10 on the medium discrete pattern completion, and from 0.63 to a perfect 0.00 on the medium discrete postcasting. Unfortunately, while backpropagation models naturally benefit from massive scaling, optimizing a 100k parameter ESN remains computationally expensive due to the hyperparameter search it requires, which prevents it from scaling efficiently.
6
Discussion
CogScale is designed to serve as a "sanity check" for new architecture elaboration. Our results show that the basic cognitive abilities, like retrieval, induction, and manipulation, are non-trivial prerequisites for any architecture aspiring to the status of a foundation model. Before spending thousands of GPU hours to train on massive datasets like The Pile or OpenWebText, an architecture must first prove its cognitive abilities at a small scale. We argue that a foundation model must, by definition, be "generalist". If an architecture, such as LSTM or ESN, fails to solve specific cognitive tasks like the sorting problem or selective copy, or fails to scale its performance across difficulty levels, it is highly unlikely that it will demonstrate emergent reasoning or robust generalization when scaled to millions, billions or even trillions of parameters. Among our evaluated baselines, only Transformers, Mamba, and to a significant extent xLSTM, emerge as legitimate candidates for massive NLP training, as they are the only models that maintain good performance across the full spectrum of CogScale tasks, at different scales. On the other hand, our results show very interesting performance for the ESN at a small scale. While the ESN fails to perform across complex tasks, its randomly initialized and non-trained reservoir actually outperforms deep learning models mostly on forecasting, continuous and memory task. However, as soon as a task requires reasoning, manipulation, or precise information selection, its performance is less competitive. This shows that the absence of backpropagation prevents the model from developing the complex internal representations necessary for complex symbolic operations, ultimately confining it to the role of a specialist rather than a generalist. The ability to identify such architectural limitations early in the development cycle is particularly important given the current race for better architectures. The reliance on large-scale training [33, 34] has led to a research paradigm that is both ecologically damaging [35, 10] and economically expensive for the majority of academic laboratories. CogScale proposes a more sustainable alternative by demonstrating that architectures can be effectively discriminated at a much smaller scale, specifically between 1k and 100k parameters. By providing a test protocol for new architectures, CogScale prevents the training of deficient models on massive supercomputers. Filtering out these architectures offers a substantial opportunity to reduce energy consumption and computational costs, while making architectural research more accessible to smaller teams that do not have the extensive resources of industrial AI labs. Beyond research and development, this focus on lower parameter budgets has immediate practical implications for real-world problems where massive foundation models are not suitable [10]. In industrial edge computing and healthcare devices, models are often constrained by memory and power limitations. Not every application requires a trillion parameter LLM; many require only the ability to perform specific reasoning and memory task with minimal overhead [36]. Our scaling analysis reveals that architectures like Transformers and Mamba achieve excellent performance even at the 10k parameter scale, and that simpler models like ESNs can outperform
8
them at 1k parameter scale. Consequently, CogScale provides a clear roadmap for engineers in these sectors, allowing them to select the most efficient architecture based on strict hardware constraints.
7
Conclusion
CogScale offers a lightweight alternative to the current race toward massive models, which has made the evaluation of new architectures increasingly slow and costly. This framework of 14 tasks serves as a "sanity check" to validate the basic cognitive abilities required for generalization. Our evaluation demonstrates that while models like the ESN provide impressive performance at the 1k parameter scale, only Transformers, Mamba, and xLSTM prove to be “generlist“ architectures, scaling effectively to complex compositional tasks that combine various rules and operations.. Although this synthetic benchmark should be complemented by evaluations on large-scale datasets such as OWT or The Pile, it successfully filters out deficient models at an early stage, helping researchers save valuable time and financial resources while avoiding unnecessary energy consumption.
Acknowledgments and Disclosure of Funding Experiments presented in this paper were performed using the PlaFRIM HPC cluster, supported by Inria Bordeaux. We would also like to thank Romain de Coudenhove for his key contribution during his internship. He developed the highly optimized framework that enabled the automated hyperparameter (HP) search for Reservoir Computing across all tasks, scales, and difficulty levels presented in this work.
9
References [1] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 1525–1534, 2016. [2] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [3] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020. [4] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800, 2019. [5] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019. [6] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [7] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 3645–3650, 2019. [8] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021. [9] Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54–63, 2020. [10] Gaël Varoquaux, Sasha Luccioni, and Meredith Whittaker. Hype, sustainability, and the price of the bigger-is-better paradigm in AI. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 61–75, 2025. [11] Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Yong Liu, Mingsheng Long, and Jianmin Wang. Deep time series models: A comprehensive survey and benchmark. 2024. [12] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. [13] Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with lstm. Neural computation, 12(10):2451–2471, 2000. [14] Kyunghyun Cho, Bart Van Merriënboer, Çağlar Gulçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder– decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1724–1734, 2014. [15] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [16] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. [17] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 10
[18] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. Advances in Neural Information Processing Systems, 37:107547– 107603, 2024. [19] Herbert Jaeger. The “echo state” approach to analysing and training recurrent neural networkswith an erratum note. Bonn, Germany: German national research center for information technology gmd technical report, 148(34):13, 2001. [20] Herbert Jaeger. Adaptive nonlinear system identification with echo state networks. Advances in neural information processing systems, 15, 2002. [21] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020. [22] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015. [23] Mantas Lukoševičius and Herbert Jaeger. Reservoir computing approaches to recurrent neural network training. Computer science review, 3(3):127–149, 2009. [24] Min Yan, Can Huang, Peter Bienstman, Peter Tino, Wei Lin, and Jie Sun. Emerging opportunities and challenges for the future of reservoir computing. Nature Communications, 15(1):2056, 2024. [25] Edward N Lorenz. Deterministic nonperiodic flow 1. In Universality in Chaos, 2nd edition, pages 367–378. Routledge, 2017. [26] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. [27] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022. [28] Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pages 1120–1128. PMLR, 2016. [29] Alexis Juven and Xavier Hinaut. Cross-situational learning with reservoir computing for language acquisition modelling. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2020. [30] Alexandre Variengien and Xavier Hinaut. A journey in ESN and LSTM visualisations on a language task. arXiv preprint arXiv:2012.01748, 2020. [31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. [32] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [33] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. [34] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 10, 2022.
11
[35] Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021. [36] Lihu Chen and Gaël Varoquaux. What is the role of small models in the llm era: A survey. arXiv preprint arXiv:2409.06857, 2024.
12
A
Detailed Experimental Results
A.1
Best Overall (lower is better)
Table 1: Best overall experimental results for small (SM) and medium (MD) tasks. Values represent the single best performance achieved by each model across all evaluated parameter counts (1k, 10k, 100k) and learning rates (150 runs). Bold text indicates the top performance for a given task difficulty. Tâches
ESN
GRU
LSTM
MAMBA
TRANS (DO)
TRANS (ED)
XLSTM
adding problem-SM adding problem-MD associative rec.-SM associative rec.-MD bracket matching-SM bracket matching-MD chaotic forecast-SM chaotic forecast-MD c. pattern completion-SM c. pattern completion-MD c. postcasting-SM c. postcasting-MD cross situation-SM cross situation-MD d. pattern completion-SM d. pattern completion-MD d. postcasting-SM d. postcasting-MD induction heads-SM induction heads-MD selective copy-SM selective copy-MD simple copy-SM simple copy-MD sinus forecast-SM sinus forecast-MD sorting problem-SM sorting problem-MD
0.00 0.69 0.44 0.76 0.13 0.13 0.00 0.00 0.02 0.06 0.00 0.00 0.00 0.00 0.06 0.09 0.00 0.00 0.01 0.64 0.41 0.70 0.00 0.70 0.00 0.00 0.43 0.71
0.00 0.00 0.35 0.88 0.18 0.01 0.00 0.00 0.01 0.01 0.00 0.00 0.03 0.00 0.06 0.18 0.00 0.00 0.35 0.47 0.22 0.23 0.49 0.64 0.01 0.01 0.42 0.17
0.02 0.00 0.57 0.91 0.21 0.00 0.01 0.00 0.01 0.01 0.00 0.00 0.05 0.00 0.05 0.11 0.00 0.00 0.37 0.77 0.52 0.86 0.57 0.84 0.01 0.00 0.49 0.56
0.01 0.00 0.59 0.91 0.29 0.02 0.00 0.00 0.01 0.02 0.00 0.02 0.05 0.00 0.06 0.14 0.00 0.08 0.48 0.12 0.21 0.00 0.42 0.13 0.01 0.00 0.51 0.00
0.00 0.00 0.36 0.67 0.19 0.04 0.04 0.08 0.03 0.01 0.00 0.00 0.03 0.00 0.07 0.07 0.00 0.00 0.00 0.00 0.05 0.00 0.00 0.00 0.02 0.07 0.00 0.00
0.00 0.00 0.39 0.78 0.27 0.01 0.04 0.07 0.06 0.00 0.00 0.00 0.04 0.00 0.36 0.00 0.00 0.00 0.00 0.00 0.17 0.01 0.00 0.00 0.02 0.07 0.00 0.00
0.00 0.00 0.40 0.85 0.06 0.00 0.01 0.00 0.01 0.01 0.00 0.00 0.03 0.00 0.06 0.18 0.00 0.00 0.39 0.25 0.34 0.23 0.42 0.51 0.01 0.01 0.05 0.06
13
A.2
Mean and Standard Deviation (lower is better)
Table 2: Mean and Standard Deviation results (first half) for small (SM) and medium (MD) tasks. Performance is evaluated across varying parameter count (1k, 10k, 100k). Values represent the mean and standard deviation (mean ± std) of the best Learning Rate among 5, on 10 seeds (each value represents 50 runs). Bold text indicates the best performance achieved for a given task configuration. N/D indicate the lack of results. Tâches adding problem-SM-1k adding problem-SM-10k adding problem-SM-100k adding problem-MD-1k adding problem-MD-10k adding problem-MD-100k associative rec.-SM-1k associative rec.-SM-10k associative rec.-SM-100k associative rec.-MD-1k associative rec.-MD-10k associative rec.-MD-100k bracket matching-SM-1k bracket matching-SM-10k bracket matching-SM-100k bracket matching-MD-1k bracket matching-MD-10k bracket matching-MD-100k chaotic forecast-SM-1k chaotic forecast-SM-10k chaotic forecast-SM-100k chaotic forecast-MD-1k chaotic forecast-MD-10k chaotic forecast-MD-100k c. pattern comp.-SM-1k c. pattern comp.-SM-10k c. pattern comp.-SM-100k c. pattern comp.-MD-1k c. pattern comp.-MD-10k c. pattern comp.-MD-100k c. postcasting-SM-1k c. postcasting-SM-10k c. postcasting-SM-100k c. postcasting-MD-1k c. postcasting-MD-10k c. postcasting-MD-100k cross situation-SM-1k cross situation-SM-10k cross situation-SM-100k cross situation-MD-1k cross situation-MD-10k cross situation-MD-100k
ESN
GRU
LSTM
MAMBA
TRANS. (DO)
TRANS. (ED)
XLSTM
0.47±0.09 0.02±0.03 N/D 0.85±0.02 0.71±0.02 N/D 0.57±0.07 0.52±0.04 N/D 0.85±0.01 0.78±0.01 N/D 0.25±0.11 0.20±0.06 N/D 0.20±0.02 0.16±0.02 N/D 0.00±0.00 0.00±0.00 N/D 0.00±0.00 0.00±0.00 N/D 0.03±0.00 0.04±0.01 N/D 0.08±0.00 0.06±0.00 N/D 0.00±0.00 0.00±0.00 N/D 0.07±0.02 0.00±0.00 N/D 0.01±0.01 0.01±0.01 N/D 0.13±0.01 0.00±0.00 N/D
0.08±0.05 0.13±0.13 0.16±0.10 0.01±0.01 0.01±0.00 0.01±0.00 0.66±0.03 0.58±0.11 0.57±0.11 0.93±0.01 0.93±0.01 0.90±0.01 0.30±0.09 0.35±0.03 0.39±0.11 0.06±0.09 0.14±0.13 0.11±0.11 0.04±0.02 0.02±0.01 0.02±0.02 0.02±0.03 0.01±0.02 0.00±0.01 0.01±0.00 0.01±0.00 0.01±0.00 0.01±0.00 0.01±0.00 0.01±0.00 0.00±0.00 0.00±0.00 0.00±0.00 0.00±0.01 0.00±0.00 0.00±0.00 0.23±0.22 0.07±0.02 0.04±0.01 0.07±0.01 0.00±0.00 0.00±0.00
0.50±0.31 0.36±0.33 0.55±0.29 0.02±0.01 0.01±0.01 0.02±0.01 0.66±0.04 0.65±0.04 0.65±0.07 0.93±0.01 0.93±0.00 0.93±0.01 0.33±0.06 0.36±0.06 0.38±0.05 0.08±0.11 0.07±0.11 0.20±0.16 0.04±0.01 0.02±0.01 0.03±0.01 0.01±0.02 0.02±0.01 0.02±0.01 0.02±0.01 0.01±0.00 0.01±0.00 0.02±0.00 0.01±0.00 0.01±0.00 0.00±0.00 0.00±0.00 0.00±0.00 0.01±0.01 0.00±0.00 0.00±0.00 0.29±0.22 0.08±0.03 0.08±0.01 0.10±0.01 0.00±0.00 0.00±0.00
0.63±0.21 0.63±0.20 0.61±0.23 0.88±0.01 0.23±0.37 0.23±0.38 0.65±0.04 0.66±0.04 0.66±0.04 0.93±0.01 0.93±0.01 0.93±0.01 0.39±0.05 0.38±0.04 0.38±0.04 0.27±0.11 0.19±0.11 0.19±0.13 0.04±0.02 0.01±0.01 0.01±0.01 0.00±0.00 0.00±0.00 0.00±0.00 0.04±0.03 0.02±0.00 0.02±0.00 0.04±0.02 0.02±0.00 0.02±0.00 0.00±0.00 0.00±0.00 0.00±0.00 0.15±0.04 0.06±0.03 0.09±0.04 0.55±0.21 0.16±0.06 0.14±0.04 0.56±0.18 0.00±0.00 0.00±0.00
0.56±0.24 0.15±0.22 0.04±0.03 0.70±0.29 0.01±0.01 0.01±0.01 0.64±0.04 0.59±0.12 0.55±0.09 0.92±0.01 0.86±0.03 0.84±0.07 0.45±0.08 0.46±0.05 0.45±0.11 0.42±0.10 0.23±0.14 0.11±0.07 0.11±0.04 0.07±0.01 0.06±0.02 0.11±0.02 0.16±0.04 0.10±0.02 0.07±0.02 0.07±0.01 0.07±0.01 0.07±0.02 0.01±0.00 0.01±0.00 0.04±0.09 0.00±0.00 0.00±0.00 0.00±0.00 0.00±0.00 0.00±0.00 0.61±0.17 0.09±0.03 0.06±0.01 0.52±0.27 0.01±0.00 0.00±0.00
0.69±0.06 0.42±0.27 0.22±0.32 0.88±0.01 0.54±0.44 0.20±0.36 0.65±0.05 0.61±0.07 0.56±0.10 0.93±0.01 0.90±0.03 0.90±0.06 0.48±0.06 0.46±0.06 0.43±0.07 0.47±0.05 0.37±0.14 0.14±0.14 0.08±0.01 0.07±0.01 0.05±0.01 0.10±0.00 0.11±0.02 0.09±0.01 0.08±0.01 0.07±0.01 0.07±0.01 0.08±0.00 0.07±0.02 0.03±0.04 0.15±0.08 0.00±0.00 0.00±0.00 0.10±0.08 0.00±0.00 0.00±0.00 0.70±0.00 0.22±0.25 0.13±0.20 0.80±0.00 0.04±0.01 0.00±0.00
0.32±0.37 0.07±0.06 0.06±0.07 0.44±0.45 0.00±0.00 0.00±0.00 0.64±0.05 0.56±0.05 0.52±0.07 0.92±0.01 0.91±0.01 0.88±0.01 0.38±0.06 0.28±0.10 0.32±0.10 0.17±0.13 0.03±0.01 0.02±0.01 0.06±0.02 0.05±0.02 0.02±0.01 0.03±0.02 0.01±0.01 0.00±0.00 0.03±0.01 0.02±0.00 0.02±0.01 0.05±0.00 0.03±0.01 0.02±0.00 0.06±0.01 0.00±0.00 0.00±0.00 0.20±0.00 0.07±0.03 0.00±0.00 0.50±0.22 0.12±0.04 0.05±0.01 0.57±0.20 0.00±0.00 0.00±0.00
14
Table 3: Complete experimental results (second half) for small (SM) and medium (MD) tasks. Performance is evaluated across varying parameter count (1k, 10k, 100k). Values represent the mean and standard deviation (mean ± std) of the best Learning Rate among 5, on 10 seeds (each value represents 50 runs). Bold text indicates the best performance achieved for a given task configuration. N/D indicate the lack of results. Tâches d. pattern comp.-SM-1k d. pattern comp.-SM-10k d. pattern comp.-SM-100k d. pattern comp.-MD-1k d. pattern comp.-MD-10k d. pattern comp.-MD-100k d. postcasting-SM-1k d. postcasting-SM-10k d. postcasting-SM-100k d. postcasting-MD-1k d. postcasting-MD-10k d. postcasting-MD-100k induction heads-SM-1k induction heads-SM-10k induction heads-SM-100k induction heads-MD-1k induction heads-MD-10k induction heads-MD-100k selective copy-SM-1k selective copy-SM-10k selective copy-SM-100k selective copy-MD-1k selective copy-MD-10k selective copy-MD-100k simple copy-SM-1k simple copy-SM-10k simple copy-SM-100k simple copy-MD-1k simple copy-MD-10k simple copy-MD-100k sinus forecast-SM-1k sinus forecast-SM-10k sinus forecast-SM-100k sinus forecast-MD-1k sinus forecast-MD-10k sinus forecast-MD-100k sorting problem-SM-1k sorting problem-SM-10k sorting problem-SM-100k sorting problem-MD-1k sorting problem-MD-10k sorting problem-MD-100k
ESN
GRU
LSTM
MAMBA
TRANS. (DO)
TRANS. (ED)
XLSTM
0.07±0.01 0.07±0.01 N/D 0.57±0.06 0.10±0.00 N/D 0.00±0.00 0.00±0.00 N/D 0.63±0.01 0.00±0.00 N/D 0.36±0.02 0.02±0.00 N/D 0.73±0.00 0.64±0.00 N/D 0.48±0.04 0.44±0.02 N/D 0.75±0.01 0.70±0.00 N/D 0.47±0.01 0.00±0.00 N/D 0.76±0.00 0.70±0.00 N/D 0.00±0.00 0.00±0.00 N/D 0.00±0.00 0.00±0.00 N/D 0.51±0.01 0.46±0.01 N/D 0.75±0.00 0.72±0.00 N/D
0.17±0.04 0.11±0.02 0.07±0.03 0.59±0.03 0.27±0.03 0.29±0.02 0.00±0.00 0.00±0.00 0.00±0.01 0.64±0.04 0.00±0.00 0.00±0.00 0.60±0.09 0.53±0.12 0.49±0.13 0.81±0.06 0.58±0.11 0.60±0.13 0.63±0.08 0.54±0.16 0.61±0.11 0.65±0.13 0.54±0.25 0.54±0.24 0.64±0.06 0.62±0.06 0.63±0.05 0.82±0.05 0.72±0.03 0.74±0.08 0.01±0.00 0.01±0.00 0.02±0.00 0.05±0.01 0.03±0.01 0.03±0.02 0.49±0.03 0.50±0.02 0.51±0.01 0.75±0.01 0.33±0.04 0.24±0.05
0.24±0.15 0.15±0.15 0.11±0.02 0.69±0.11 0.25±0.10 0.30±0.03 0.00±0.00 0.00±0.00 0.00±0.00 0.65±0.09 0.00±0.01 0.00±0.00 0.62±0.08 0.59±0.09 0.48±0.08 0.87±0.00 0.87±0.00 0.85±0.04 0.66±0.02 0.65±0.02 0.65±0.01 0.87±0.00 0.87±0.00 0.88±0.00 0.67±0.01 0.66±0.02 0.65±0.03 0.87±0.00 0.87±0.00 0.87±0.00 0.02±0.00 0.03±0.01 0.05±0.01 0.03±0.01 0.03±0.02 0.04±0.02 0.51±0.01 0.52±0.03 0.52±0.05 0.77±0.01 0.61±0.03 0.70±0.01
0.17±0.12 0.10±0.02 0.09±0.01 0.63±0.16 0.20±0.03 0.19±0.03 0.00±0.00 0.00±0.00 0.00±0.00 0.74±0.07 0.30±0.06 0.12±0.03 0.65±0.04 0.60±0.07 0.58±0.04 0.84±0.06 0.31±0.06 0.21±0.05 0.63±0.05 0.55±0.13 0.52±0.14 0.78±0.14 0.32±0.35 0.11±0.16 0.65±0.04 0.61±0.05 0.50±0.05 0.81±0.02 0.50±0.08 0.25±0.05 0.02±0.01 0.01±0.00 0.01±0.01 0.04±0.02 0.03±0.02 0.02±0.02 0.61±0.05 0.55±0.03 0.53±0.01 0.70±0.07 0.20±0.13 0.02±0.02
0.41±0.09 0.38±0.14 0.22±0.18 0.63±0.21 0.08±0.00 0.08±0.01 0.03±0.10 0.00±0.00 0.00±0.00 0.00±0.00 0.00±0.00 0.00±0.00 0.40±0.34 0.00±0.00 0.00±0.00 0.35±0.31 0.00±0.00 0.00±0.00 0.59±0.11 0.37±0.27 0.10±0.05 0.58±0.18 0.04±0.07 0.01±0.00 0.26±0.33 0.00±0.00 0.00±0.00 0.04±0.09 0.00±0.00 0.00±0.00 0.16±0.12 0.07±0.02 0.05±0.05 0.35±0.27 0.10±0.01 0.08±0.01 0.08±0.21 0.00±0.00 0.00±0.00 0.00±0.00 0.00±0.00 0.00±0.00
0.46±0.09 0.41±0.02 0.41±0.02 0.80±0.02 0.56±0.26 0.00±0.00 0.38±0.28 0.00±0.00 0.00±0.00 0.67±0.20 0.00±0.00 0.00±0.00 0.64±0.04 0.18±0.29 0.00±0.00 0.77±0.05 0.00±0.00 0.00±0.00 0.66±0.02 0.63±0.06 0.43±0.21 0.87±0.00 0.20±0.28 0.02±0.03 0.65±0.03 0.12±0.26 0.00±0.00 0.83±0.05 0.00±0.00 0.00±0.00 0.31±0.21 0.12±0.05 0.06±0.03 0.18±0.12 0.20±0.09 0.10±0.02 0.67±0.02 0.06±0.16 0.05±0.16 0.87±0.00 0.00±0.00 0.00±0.00
0.13±0.02 0.10±0.02 0.09±0.02 0.41±0.05 0.20±0.01 0.18±0.01 0.22±0.11 0.00±0.00 0.00±0.00 0.83±0.01 0.54±0.03 0.01±0.01 0.53±0.09 0.46±0.07 0.44±0.03 0.74±0.02 0.39±0.07 0.29±0.01 0.51±0.08 0.44±0.04 0.42±0.04 0.76±0.02 0.45±0.08 0.29±0.06 0.53±0.05 0.49±0.03 0.46±0.02 0.78±0.00 0.69±0.03 0.55±0.03 0.01±0.00 0.01±0.00 0.01±0.01 0.05±0.01 0.04±0.02 0.04±0.02 0.54±0.05 0.31±0.02 0.22±0.08 0.70±0.04 0.28±0.08 0.11±0.04
15
B
Task Configuration: Small & Medium
This section details the specific parameters used to generate the datasets for both the small (SM) and medium (MD) difficulty across all tasks. • Sinus Forecasting – Small: sequence_length=200, forecast_length=5, training_ratio=0.45, validation_ratio=0.1, testing_ratio=0.45 – Medium: sequence_length=2000, forecast_length=15, training_ratio=0.45, validation_ratio=0.1, testing_ratio=0.45 • Chaotic Forecasting – Small: sequence_length=200, forecast_length=5, training_ratio=0.45, validation_ratio=0.1, testing_ratio=0.45 – Medium: sequence_length=2000, forecast_length=15, training_ratio=0.45, validation_ratio=0.1, testing_ratio=0.45 • Discrete Postcasting – Small: n_train=100, n_valid=20, n_test=100, sequence_length=50, delay=5, n_symbols=3 – Medium: n_train=1000, n_valid=200, n_test=1000, sequence_length=100, delay=15, n_symbols=8 • Continuous Postcasting – Small: n_train=100, n_valid=20, n_test=100, sequence_length=50, delay=5 – Medium: n_train=1000, n_valid=200, n_test=1000, sequence_length=100, delay=15 • Discrete Pattern Completion – Small: n_train=100, n_valid=20, n_test=100, sequence_length=60, n_symbols=3, base_length=4, mask_ratio=0.2 – Medium: n_train=1000, n_valid=200, n_test=1000, sequence_length=150, n_symbols=8, base_length=10, mask_ratio=0.2 • Continuous Pattern Completion – Small: n_train=100, n_valid=20, n_test=100, sequence_length=60, base_length=4, mask_ratio=0.2 – Medium: n_train=1000, n_valid=200, n_test=1000, sequence_length=150, base_length=10, mask_ratio=0.2 • Bracket Matching – Small: n_train=100, n_valid=20, n_test=100, sequence_length=50, max_depth=5 – Medium: n_train=1000, n_valid=200, n_test=1000, sequence_length=100, max_depth=10
16
• Simple Copy – Small: n_train=100, n_valid=20, n_test=100, sequence_length=22, delay=5, n_symbols=3 – Medium: n_train=1000, n_valid=200, n_test=1000, sequence_length=50, delay=10, n_symbols=8 • Selective Copy – Small: n_train=100, n_valid=20, n_test=100, sequence_length=40, delay=5, n_markers=5, n_symbols=3 – Medium: n_train=1000, n_valid=200, n_test=1000, sequence_length=80, delay=10, n_markers=10, n_symbols=8 • Adding Problem – Small: n_train=100, n_valid=20, n_test=100, sequence_length=10, max_number=3 – Medium: n_train=1000, n_valid=200, n_test=1000, sequence_length=20, max_number=8 • Sorting Problem – Small: n_train=100, n_valid=20, n_test=100, sequence_length=10, n_symbols=3 – Medium: n_train=1000, n_valid=200, n_test=1000, sequence_length=20, n_symbols=8 • Cross Situation – Small: n_train=100, n_valid=20, n_test=100, objects=[’glass’, ’orange’], colors=[’blue’, ’orange’], positions=[’left’, ’right’] – Medium: n_train=1000, n_valid=200, n_test=1000, objects=[’glass’, ’orange’, ’cup’, ’bowl’], colors=[’blue’, ’orange’, ’green’, ’red’], positions=[’left’, ’right’, (’center’, ’middle’)] • Associative Recall – Small: n_train=100, n_valid=20, n_test=100, sequence_length=16, num_pairs=3, n_symbols=5 – Medium: n_train=1000, n_valid=200, n_test=1000, sequence_length=32, num_pairs=7, n_symbols=16 • Induction Heads – Small: n_train=100, n_valid=20, n_test=100, sequence_length=40, n_symbols=3 – Medium: n_train=1000, n_valid=200, n_test=1000, sequence_length=100, n_symbols=8
17