OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling Indraneil Paul1 , Falko Helm1 , Goran Glavaš2 , and Iryna Gurevych1

1 UKP Lab & Center for Applied Cybersecurity ATHENE, TU Darmstadt

arXiv:2608.05141v1 [cs.AI] 5 Aug 2026

2 CAIDAS, University of Würzburg

1

Abstract

1M

Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing longcontext corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependencyrich code contexts up to millions of tokens in length. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a ≈50B-token mixture containing ≈6.2B tokens of OctoLong code contexts, followed by ≈10B tokens of instruction tuning. Our training ablations and experimental evaluations against 18 strong long-context open-weight LMs show that supplanting just 12% of conventional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios.

256K 128K

32K

8K 4K 2K Jul 2023

Jan 2024

Jul 2024

Jan 2025

Jul 2025

Jan 2026

Jul 2026

Figure 1: Context lengths of open-weight LMs. The trend is exponential, doubling on average every 5 months.

attention (Yuan et al., 2025; Han et al., 2024; Beltagy et al., 2020) and linear attention (Hatamizadeh et al., 2026; Dao and Gu, 2024). In isolation, these approaches deteriorate the performance on longer contexts (Lu et al., 2024; Huang, 2024), causing LM developers to turn to orthogonal axes of improvement such as hardware-use improvements (Li et al., 2023a; Liu et al., 2023), inference-time context management (Li et al., 2024c; Zhang et al., 2023b), and better data engineering (Li et al., 2025b). Data Engineering for Context Extension. Data engineering for long-context training (Fu et al., 2024) revolves around engineering long text inputs with a high density of referrals (i.e., dependencies), spanning a range of distances. Existing approaches span the gamut of token-level dependency filtering (Deng et al., 2025; Rathi and Radford, 2026), curating multiple salient documents in context (Gao et al., 2025b; Jia et al., 2025; Li et al., 2024a), and referral-based packing (Zhuang et al., 2025; Staniszewski et al., 2025). Practically, however, the contexts engineered by such approaches often lack complex multi-hop dependencies (Li et al., 2024b) and mine irrelevant or contradictory information (Du et al., 2025b; Tang et al., 2025). In this work, we introduce cross-repository code dependency mining. This referral packing technique

Introduction

Emergent language model (LM) applications such as in-context learning (Bertsch et al., 2025), retrieval-augmented generation (Qian et al., 2025; Kang et al., 2025), and agentic tool-use (Merrill et al., 2026; Jimenez et al., 2024) demand longhorizon capabilities (Kwa et al., 2025), exponentially driving up the demand on the context lengths the LMs are expected to support (ref. Figure 1). Prior attempts to extend the context length mainly involve architectural interventions such as sparse 1

taps into openly available code data to unlock a virtually limitless source of multi-hop referral-dense contexts that transcend repository boundaries.

2024). Subsequent methods have since moved on to serializing code at the repository level (Ding et al., 2024; Deng et al., 2024). Serialized repositories have since become the anchor of mid-training (Liu et al., 2025a; Blakeney et al., 2024) and contextextension (Mo et al., 2025) corpora. However, most code is not self-contained at the repository level (Yu et al., 2024), which restricts models’ ability to reason about multi-repository projects (Chen et al., 2026). Attempts to train on cross-repository contexts are stymied by inadequate tooling (Usai et al., 2026) and have thus been limited to sourcing dependency library names (Liao et al., 2024). In Section 3, we detail a novel pipeline to source referred implementations beyond repository bounds.

Contributions. We address the abovementioned research gaps and make the following contributions: 1 We introduce OctoLong, a pipeline to generate cross-repository code contexts using an AST parser, an LSP server, and a package manager in tandem. Analyses of attention maps and model likelihoods show that our pipeline produces exceptionally dependency-rich long contexts (Section 3). 2 We collect ≈6.2B tokens of cross-repository long-context data with the OctoLong pipeline and combine it with other long-data sources to curate the ≈50B-token OctoLong-LCFT corpus. We then compile the ≈10B-token OctoLong-SFT mix of opensource instruction and agentic data (Section 4). 3 Using the above corpora, we subject Qwen3 dense base models to long-context fine-tuning (LCFT), quadrupling their supported context length to 128K, followed by supervised fine-tuning (SFT) to obtain the OctoLong-Instruct suite (Section 4). 4 We establish, via extensive experiments and ablations, that LCFT with our OctoLong data in the long-context data mix leads to substantial improvements not only in long-context code tasks but also in general-domain long-context applications including agentic use-cases, without penalty on short-context LM capabilities (Section 5).

2

The OctoLong Dataset

3

Sourcing long-context code for language modeling has traditionally focused on serializing individual code repositories in dependency traversal order (Lozhkov et al., 2024; Guo et al., 2024; Zhang et al., 2023a; Shrivastava et al., 2023; Sapronov and Glukhov, 2025). To test our hypotheses from Section 1, we develop the OctoLong pipeline to retrieve code dependencies across repository boundaries. Language Server Protocol. We leverage the Language Server Protocol (LSP),1 a JSON-RPC standard that provides a unified language- and clientagnostic wire interface to source code analyses by fronting language-specific compilers. Crucial for our purposes, mature language server implementations for popular programming languages can use the established conventions of detected package managers, in tandem with the abstract syntax tree (AST) information produced by the compiler, to map the location of imported classes and functions. Traditionally used by IDEs, the LSP ecosystem offers a robust, type- and scope-aware approach to retrieve dependency code unattainable by predecessors reliant on pure syntactic parsing.2

Related Work

We briefly outline two threads of relevant existing work: 1 cross-document context engineering, and 2 long-context code data mining. Cross-Document Context Engineering. Prior work towards engineering cross-document contexts have relied on hyperlinks (Zhuang et al., 2025; Yasunaga et al., 2022), book BISAC codes (Jia et al., 2026), event co-references (Caciularu et al., 2021), and repository-level code dependencies (Guo et al., 2024). These approaches are all either hard to scale or yield loosely related contexts. In contrast, we detail how to recursively mine referred class and function implementations beyond repository boundaries — an untapped source of tightly-coupled data whose acquisition is easy to scale.

OctoLong Data Acquisition. Figure 2 illustrates all stages of OctoLong data acquisition pipeline for long code contexts. We start by sourcing from SEART Hub (Dabic et al., 2024) reputable and actively maintained Python repositories3 , with at least 15 GitHub stars and more than 5 contributors. From these, we retain only those that (a) declare

Mining Long Self-Contained Code Data. Early file-level attempts to mine long code data found that long code files are usually low-quality configuration files and uninformative build artifacts (Singh et al.,

1

Language Server Protocol universal-ctags/ctags 3While our data collection is restricted to Python, the OctoLong pipeline is extensible to any programming language with a mature LSP backend and a package manager. 2

2

1

2

3

4

5

Seed Selection

Containerized Environment

Call-Site Indexing

Dependency Traversal

BFS Serialization

tree-sitter-query Repo's container

def fetch(url): requests.get()

deps pre-installed Seeds

Code-Quality Filter • avg line length • alphanum fraction • AST validity • AST depth Shortlist seed fns / classes

site-packages/ requests/ urllib3/ runnable, version-correct

LSP Server JEDI (in container) serves go-todefinition queries

indexed call site · app.py

(call function: (identifier)) (call (attribute)) (class instantiation)

Repo Image

urlparse(url)

Env

def fetch(url):

1

requests.get(url)

Sites

Indexed Call Sites

dependency · site-packages

• function calls • method calls (class) • object instantiations

def get(url, **kw):

Tree

Per-seed call-site index

2

urlparse(url): urlsplit()

3

get(url,**kw): request(...)

4 urlsplit(url): 5 request(m, u):

return request()

urlparse · requests.get

urlparse(url) requests.get(url)

goto-definition

recurse

app.py · fn

tree-sitter locates exact range

Breadth-first over dependency tree

Beyond repo → dependencies

Up to 128K Token Contexts

Depth ≤ 10

Figure 2: Overview of the OctoLong pipeline’s core stages. Utilizing AST information, a language server backend leverages package manager metadata to recursively retrieve dependency data and engineer very long code contexts.

dependencies and (b) are installable, resulting in 12K repositories. Each Python file from retained repositories becomes a candidate seed: we keep the 92K seeds that pass the OpenCoder’s default code-quality filtering pipeline (Huang et al., 2025). Subsequently, we provision each seed in a containerized sandbox with its repository and dependency environment and traverse the dependency graph via breadth-first search (BFS). We traverse up to a prespecified depth of 10 steps or when we collect 128K tokens of code (whichever is fulfilled first), prepending the code retrieved in each step to the context. Each hop consists of a tree-sitter4 query that extracts class and function call-sites, followed by a GoToDeclaration call5 per call-site using the jedi 6 language server; this is followed by another tree-sitter query that obtains the function or class boundaries and extracts the implementation from the target file. We make sure never to repeat classes and functions. We also discard contexts shallower than 5 levels as well as those with a hop success rate below 60%. Following this procedure, we end up with ≈6.2B code tokens.

context length distributions Table 1 shows that our OctoLong pipeline produces contexts that surpass those in any existing source by orders of magnitude. Further, we compare the density, uniformity, and salience of the dependencies within the longcontext data sources. This comparison is critical, as training on dependency-deprived long contexts offers limited performance gains over short-context language modeling (Chen et al., 2024; Deng et al., 2025), while substantially increasing training costs. We inspect all long-context corpora in Table 1 with a modified LongAttn (Wu et al., 2025b) score (the formula below). Specifically, we use the first attention layer of a Phi3 model (Microsoft, 2024)7 to calculate the mean attention mass 𝑀 of the last 2K query tokens against the most recent eighth, quarter, and half of the (𝑛 − 2)K token keys from the prefix. We average these as follows (denoted with (Dep. Mean in Table 1):

Comparison of Long Context Data Sources. Our goal is to test our hypothesis that cross-repository code contexts constitute a very good source of long and dependency-rich data. To this end, we additionally acquire preprocessed long-context corpora from a variety of traditional sources, including books, code repositories, citation graphs, and retrieved documents, and compare them against our data sourced via the OctoLong pipeline without token length restrictions, using BFS depth limits of 8, 10, and 15, respectively. A comparison of

We also track the mean attention mass fraction the final 2K tokens allot to the first four sink positions (Sink Prop. in Table 1). These metrics, in tandem, test for data-induced lost-middle patterns (Salvatore et al., 2025; Liu et al., 2024b) where recency (Baddeley and Hitch, 1993) and primacy (Asch, 1946) biases displayed by humans spill over into the data distribution. The LongAttn section of Table 1 demonstrates that OctoLong data exhibits a markedly lower sink concentration and notably lower incidence of lost-middle (i.e., higher Dep. Mean) patterns compared to other long-context data sources.

4

3

𝜇Dep. =

1 ∑︁ 6𝐾 𝑗=1

𝑛𝐾 ∑︁

(𝑛−2) ∑︁ 𝐾

𝑞=(𝑛−2) 𝐾+1

𝑖=𝑏 𝑗

where 𝑏 𝑗 = (𝑛 − 2)𝐾 − 𝑐 𝑗 + 1, 𝑐 𝑗 =

tree-sitter/tree-sitter

5While the LSP standard defines a GoToImplementation end-

point, which would significantly simplify our collection, this remains unsupported by most language servers. 6 pappasam/jedi-language-server

𝑀𝑖,𝑞

(𝑛 − 2)𝐾 , 𝑗 ∈ {1, 2, 3} 2𝑗

7 For context spans longer than 128K tokens, we compute dependency saliency metrics on the trailing 128K tokens.

3

Replication Source

Source

Dependency Score

Length Distribution Percentile

LongAttn

LongPPL

50th

90th

95th

99th

Dep. Mean

Sink Prop. (Pos 0–3)

% Key Tokens

Estimated Open Data

Books

RedPajama (Weber et al., 2024)

88,329

221,456

288,375

453,776

0.5521

1.6e-4

0.83

< 1T Tokens

Academic Papers

Common Pile (Kandpal et al., 2025)

16,565

40,387

54,660

93,114

0.5347

3.2e-4

1.21

< 500B Tokens

Event Co-Referenced Documents

MultiNews (Fabbri et al., 2019)

2,504

4,816

9,592

15,369

0.4422

1.1e-4

0.45

< 500B Tokens

Embedding Retrieval Augmented Documents

NextLong (Gao et al., 2025b)

147,269

235,444

282,611

313,076

0.5722

4.0e-5

2.12

Potentially Unlimited

Citation Retrieval Augmented Documents

MegaWika 2 (Barham et al., 2025)

4,012

7,388

16,656

24,879

0.4877

7.4e-4

0.26

< 5T Tokens

In-Repo Code

Stack V3 (Lozhkov et al., 2026)

4,879

21,868

49,559

207,681

0.5261

3.8e-4

1.55

< 10T Tokens

Cross-Repo Code – Depth Limit 8 – Depth Limit 10 – Depth Limit 15

OctoLong

142,046 201,477 372,559

298,467 480,116 808,196

403,450 907,474 1,747,283

487,595 1,582,313 9,779,895

0.6244 0.6113 0.6295

2.0e-5 2.0e-5 2.0e-5

6.06 5.99 6.02

Potentially Unlimited

Table 1: Comparison of the cross-repository contexts from the OctoLong pipeline (configured with varying depth limits) against existing long-context data sources. OctoLong flexibly sources long-context data that are (i) orders of magnitude longer and (ii) exhibit superior dependency density compared to existing long-context sources.

Diffuse dependencies by themselves do not imply salience, as they may be a byproduct of noisy (Leviathan et al., 2025; Bagla et al., 2021) or self-contradictory (Wu et al., 2026b) contexts. We thus add a modified LongPPL (Fang et al., 2025) diagnostic, measuring the proportion of contextsensitive key tokens among the final 2K tokens: these are the tokens for which perplexity (under the Phi3 model) is markedly reduced in the presence of the full context compared to just the trailing 2K tokens of the prefix (% Key Tokens in Table 1): 𝐹key =

1 2𝐾

𝑛𝐾 ∑︁

Data Source In-Repo & Code-Adjacent Data Academic Papers & PDF Data Math & Math-Adjacent Data OctoLong Data Books Data Educational Data Tech Content & Forum Data General Domain Data Agentic & Instruct-Like Data

ProLong

21.51 18.55 12.22 12.05 9.52 3.05 4.31 11.95 6.84

30.00 4.00 4.00 30.00 14.00 4.00 10.00 4.00

Table 2: Token prevalence by source category across the ≈50B-token long-context fine-tuning corpora, comparing the OctoLong-LCFT data mix against our control replication of ProLong (Gao et al., 2025c).

tokens. We acquire ≈6.2B tokens of OctoLong data as described in Section 3 and mix it with a variety of conventional short- and long-context mid-training data sources. Our auxiliary data mix spans the categories of the ProLong (Gao et al., 2025c) corpus. We set the length limit to 128K tokens, apply an English-only filter (Fedorova et al., 2026), and perform MinHash de-duplication (Broder, 1997). Table 2 compares the category composition of our LCFT corpus against a size-matched but proportionally faithful replication of the ProLong corpus.

1[log 𝑃 𝜃 (𝑥𝑖 | 𝒍 𝑖 ) − log 𝑃 𝜃 (𝑥𝑖 | 𝒔𝑖 ) > 2]

𝑖=(𝑛−2) 𝐾+1

where 𝒔𝑖 = 𝒙 (𝑛−4) 𝐾+1: 𝑖−1 , 𝒍 𝑖 = 𝒙 1: 𝑖−1 , 𝑖 ∈ {(𝑛 − 2)𝐾 + 1, . . . , 𝑛𝐾 }

The OctoLong’s cross-repository code contexts Table 1 contain 3–23 times more context-sensitive key tokens than other long-context corpora, including that consisting of individually serialized code repositories (In-Repo Code). These key tokens represent context-sensitive (Appicharla et al., 2026) and reasoning-intensive (Wang et al., 2025b) points in the data that make models more amenable to post-training (Shen et al., 2026).

4

Token Prevalence (%) OctoLong-LCFT

SFT Data. We compile a ≈10B-token SFT mix, with which we fine-tune all our models (incl. the ablation variants). Although some long-context benchmarks support base model evaluations (Hsieh et al., 2024; Yen et al., 2025; Kuratov et al., 2024), we choose to carry out SFT for practical reasons: most user- and industry-relevant long-context benchmarks demand instruction following (Wu et al., 2025c,a), structured outputs (Dou et al., 2026; An et al., 2024), and/or multi-turn interaction (Li et al., 2026). While earlier work suggested that a shortcontext SFT following LCFT suffices to post-train

OctoLong-Instruct Models

We train our OctoLong-Instruct models in two stages: 1 a context-extension/long-context finetuning (LCFT) stage, followed by 2 a supervised fine-tuning stage (SFT), i.e., instruction tuning. Mid-Training Data. In the interest of thoroughly validating our stated hypotheses from Section 1, we seek to pursue a realistically large (Ettinger et al., 2025; Bakouch et al., 2025) LCFT phase of ≈50B 4

Long-Context Fine-Tuning

Supervised Fine-Tuning

training. We use the best-fit-decreasing (Baker, 1985) scheme to pack training samples into 128K contexts for maximal efficiency. 9 We employ the AdamW (Loshchilov and Hutter, 2019) optimizer with a peak learning rate of 1e-5 along with the WSD scheduler (Hägele et al., 2024) for resumption friendliness and adopt a 5% warmup to ensure smooth continual training (Gupta et al., 2023). Table 5 details all training parameters of our LCFT and SFT training stages.

1.40 1.20 1.00 0.80 0.60

750

1500 0.6B

2250 1.7B

3000

300 4B

600 8B

900

1200

14B

Figure 3: Scaling trends of the training loss curves across the LCFT (yielding the OctoLong-Base models) and SFT phases (resulting in OctoLong-Instruct models).

solid long-context LMs (Gao et al., 2025c; Zhao et al., 2024a), more recent evidence points to dedicated long-sequence alignment as critical for unlocking competitive performance on generationand reasoning-heavy long-context tasks (Gao et al., 2025a; Bai et al., 2024; Wan et al., 2025; Li et al., 2025b; Yang et al., 2025b). Hence, our SFT mix covers a wide distribution of domains, context lengths, and interaction turns and is subjected to the same filtering pipeline as our LCFT data. We provide detailed descriptions and provenances of datasets in our final LCFT and SFT mixes in Appendix B.

5

Results and Discussion

We compare our OctoLong-Instruct suite of models against 18 open-weight baselines that support contexts of at least 64K tokens, across a range of long-context capability evaluations and shortcontext regression tests. We additionally perform three key ablations: 1 by replacing OctoLong data with the ProLong control corpus during LCFT (termed -Cross-Repo Code), 2 skipping modelmerging before the SFT stage (termed -Model Merging), and 3 skipping LCFT and directly performing fine-tuning on our SFT mix (termed -Context Extension). We skip benchmark subsets that average more than 128K tokens to ensure faithful findings. In all inference runs, we decode using nucleus sampling with a 𝑝 set to 0.95, a temperature of 0.6, and a budget of 2K tokens.

Training Configuration. All OctoLong models descend from Qwen3 (Team, 2025b) dense base models ranging from 600M to 14B parameters in size. Our LCFT phase adopts ABF RoPE scaling (Xiong et al., 2024), where we raise the base frequency from the Qwen3 default of 1M to 10M while expanding its context length from 32K to 128K tokens. We decide against using a multistage context-extension setup (Team, 2024; Zhu et al., 2025) because our exploratory runs indicated that a single-stage setup incurs no performance penalty. Although we attempt to mitigate performance regression on short inputs by mixing short- and long-context data in our mid-training mix (Zheng et al., 2025), short-input regressions can still persist (Dong et al., 2025). Hence, we follow industry practice and model-merge the pre- and post-LCFT checkpoints (Goddard, 2025; Bakouch et al., 2025) 8 , obtaining the OctoLong-Base suite. Subsequently, we undertake SFT with the modified RoPE frequency and context length to obtain the OctoLong-Instruct suite. Figure 3 shows the loss curves of the two training stages. All our training runs use a modified version of the Swift (Zhao et al., 2025) framework, leveraging the DeepSpeed-Ulysses (Jacobs et al., 2023) sequence parallelism and Liger kernel optimizations (Hsu et al., 2024) to facilitate efficient long sequence

RQ1: Does OctoLong improve long-context coding performance? In what constitutes the most “intrinsic” evaluation for OctoLong, we test whether training with crossrepository contexts improves models’ recall and state-tracking abilities on long code inputs where relevant information occurs over a range of span sizes and dispersion settings (Goldman et al., 2024). LooGLE V2: We leverage the Code subset of LooGLE V2 (He et al., 2025). This contains the MCQ Call Graph task that tests high-dispersion callstack dependency tracking between two call sites at the repository level, and the Version Control string similarity task that asks models to detect multi-file changes between two repository snapshots. LongCodeQa: We also test models’ pure recall abilities in repository-level contexts using the 64K and 128K subsets of the MCQ tasks sourced from LongCodeQA (Rando et al., 2025). RepoQa: Pure-recall tests can be compromised by models using surface-level matching (Modar-

8We perform a 1:9 linear merge between the respective Qwen3 base and our post-LCFT checkpoints.

9While our choice of packing algorithm is greedy, empirically we observe a packing efficiency over 99.87%.

5

XS

OctoLong-0.6B-Instruct

0.6B

128K

6.92

19.34

6.09

22.78

9.21

5.00

S

meta-llama/Llama-3.2-1B-Instruct 01-ai/Yi-Coder-1.5B-Chat OctoLong-1.7B-Instruct ibm-granite/granite-3.1-2b-instruct

1B 1.5B 1.7B 2B

128K 128K 128K 128K

4.82 3.32 15.08 15.15

15.94 2.41 44.83 36.57

0.66 0.83 25.77 24.89

21.14 16.48 26.51 24.41

2.50 0.55 19.49 13.57

7.00 1.00 8.00 6.00

meta-llama/Llama-3.2-3B-Instruct Qwen/Qwen3-4B-Instruct-2507 OctoLong-4B-Instruct arcee-ai/AFM-4.5B

3B 4B 4B 4.5B

128K 256K 128K 64K

23.90 28.52 29.77 16.15

18.84 63.65 64.05 61.00

6.40 57.27 58.00 6.96

22.53 29.89 28.20 22.52

15.51 38.66 36.93 5.91

7.00 10.00 10.00 7.00

Qwen/Qwen2.5-7B-Instruct-1M aws-prototyping/MegaBeam-Mistral-7B-512k internlm/internlm2_5-7b-chat-1m OctoLong-8B-Instruct meta-llama/Llama-3.1-8B-Instruct nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct princeton-nlp/Llama-3-8B-ProLong-512k-Instruct ibm-granite/granite-3.1-8b-instruct gradientai/Llama-3-8B-Instruct-262k mistralai/Ministral-8B-Instruct-2410 zai-org/glm-4-9b-chat-1m 01-ai/Yi-Coder-9B-Chat

7B 7B 7B 8B 8B 8B 8B 8B 8B 8B 9B 9B

1024K 512K 1024K 128K 128K 1024K 512K 128K 256K 128K 1024K 128K

26.99 18.09 13.02 29.76 23.41 21.38 18.92 22.21 10.97 18.38 22.65 17.14

60.99 40.17 39.93 65.01 51.80 35.41 25.94 31.60 7.05 51.94 61.98 6.71

43.45 25.94 37.87 64.63 34.88 27.08 35.61 38.02 30.87 21.82 59.15 12.80

28.22 23.56 25.30 39.17 28.14 24.20 24.73 22.47 33.35 25.56 28.53 21.36

35.27 0.00 27.50 40.91 30.58 4.17 41.69 28.37 32.67 13.11 2.11 1.89

13.00 6.00 6.00 13.00 13.00 10.00 6.00 7.00 2.00 4.00 12.00 5.00

OctoLong-14B-Instruct Qwen/Qwen2.5-14B-Instruct-1M

14B 14B

128K 1024K

39.88 37.16

77.98 72.07

76.01 75.02

41.36 31.35

44.79 36.13

21.00 23.00

OctoLong-8B-Instruct - Cross-Repo Code

8B

128K

26.38

62.17

61.58

37.03

36.88

11.00

-3.38

-2.84

-3.05

-2.14

-4.03

-2.00

OctoLong-8B-Instruct - Model Merging

8B

128K

28.13

64.25

63.73

38.61

39.20

12.00

-1.63

-0.76

-0.90

-0.56

-1.71

-1.00

Abl.

Model

M

Preference/RL-Tuned

Coding Long-Context (RQ1) General Long-Context (RQ2) Max Size Length LooGLE LongCode RepoQA LongBench OpenAI AA-LCR V2 Code QA V2 MRCR

L

Context-Extended-After-Post-Training

XL

Code-Specialised

Table 3: Comparison of benchmark-level averages between extant open-weight long-context LMs and the OctoLong-Instruct suite on our coding (RQ1: ref. RQ1) and general-domain (RQ2: ref. RQ2) long-context evaluation suites, followed by OctoLong-8B-Instruct training ablations. Per-benchmark scores are the mean over the constituent splits. Detailed per-split scores are reported in Tables 6 and 7.

ressi et al., 2025). Hence, we audit models’ ability to recall needle function implementations from repository-level contexts based on natural language description queries by evaluating them on the 64K and 128K subsets of RepoQA (Liu et al., 2024a). We use a BLEU score of 0.8 on recalled implementations as our acceptance threshold.

bilities (An et al., 2025). Finally, we see signs that the performance-enhancing value of OctoLong data transfers beyond its domain. Specifically, despite restricting our data collection to Python contexts, we observe strong averages on the RepoQA task, which covers five programming languages.

RQ2: Does OctoLong enhance long-context performance in the general domain?

Results. Table 3 displays consistently strong performance of the OctoLong-Instruct suite in longcontext code state-tracking (Call Graph) and retrieval (Version Control and LongCodeQA) scenarios, including ones with low lexical overlap (RepoQA). This pattern holds across size classes and sequence lengths (ref. Table 6). Comparing our 8B variants trained with (OctoLong-8B-Instruct) and without OctoLong data (-Cross-Repo Code), we observe a substantial performance contribution of the dependency-rich cross-repository code data. We observe especially outsized gains from the OctoLong cross-repository contexts with high dependency density and range on the Call Graph task (ref. Table 6). Poor baseline scores on this task support prior findings that high-dispersion long contexts constitute particularly challenging scenarios for LMs (Tian et al., 2025). Our models show minimal performance regression between 64K and 128K settings (ref. Table 6), which is rare for models at the edge of their supported context capa-

To test the transfer effects of our OctoLong recipe beyond code, we run a variety of benchmarks that test in-context learning, retrieval, state tracking, and synthesis in the “general domain”. LongBench V2: We evaluate focused and dispersed general-domain retrieval using the singledocument- and multi-document-QA MCQ subsets of LongBench V2 (Bai et al., 2025a). We also test the Long ICL MCQ subset’s performance as a measure of in-context learning on long inputs. Additionally, we stress-test the models’ conversational state-tracking capabilities using the Long Dialog MCQ subset. We benchmark on the 32K–128K (medium) length portions of all subsets. OpenAi Mrcr: We leverage the 64K and 128K capped-length subsets of the 2-Needle multi-round co-reference resolution task published by OpenAI MRCR (OpenAI, 2025). This string similarity task requires models to recall the hashed answer contents 6

𝜏 2 -Bench: We further report pass@1 scores on 𝜏 2 -Bench (Barrès et al., 2025), a challenging tool-calling benchmark consisting of underspecified queries that supply numerous domain-specific tools and scores based on state equivalence in an environment-level database. Along with multi-turn capability, the benchmark demands long-context multi-step support with several simulated-user turns per trajectory. 10 Additionally, the evaluation also features dual-control scenarios where the simulated user makes concurrent changes to the shared environment state, thus constituting a realistic proxy for long-horizon user-LM cooperation.

for a query about a multi-turn chat. It represents a realistic evaluation of models’ recall abilities in high-dispersion settings with multiple distractors. Aa-Lcr: Long-context evaluations largely center around direct and indirect retrieval, which, however challenging, can hide large regressions in models’ reasoning abilities in free-form generation (Ling et al., 2025; Xiao et al., 2026). To head off this possibility, we evaluate on AA-LCR (Team, 2025a), an industry-standard test requiring models to perform multi-hop reasoning and inference on documents from a variety of domains in-context. Scoring relies on pass@1 answer matching using a Qwen3-235B-A22B-Instruct-2507 judge model.

Results. Table 4 shows the competitive to outstanding performance of the OctoLong-Instruct suite across size classes, often against baselines that have undergone extensive dedicated post-training for multi-turn interaction (Yang et al., 2025a; Young et al., 2024; Wu and Song, 2025). Crucially, we observe the greatest performance contribution from our cross-repository OctoLong data here on the agentic tool-use tasks (i.e., we observe the largest performance drops for the -Cross-Repo Code variant). This suggests that our context-extension recipe avoids long-form generation pathologies common for many long-context LMs (Du et al., 2025b).

Results. Table 3 summarizes the results that answer our RQ2. Our LCFT data mixture brings strong results across the board on generaldomain long-context tasks. In particular, the OctoLong-Instruct suite compares favorably against the -Cross-Repo Code ablation, highlighting how training on cross-repository contexts transfers positively to the general domain. Our OctoLong-Instruct-4B fares competitively against Qwen3-4B-Instruct-2507, a strong context-extended and RL-post-trained Qwen3 model, while dominating all other size classes. Encouragingly, our recipe puts in a strong showing in the industry-relevant MRCR and AA-LCR benchmarks, suggesting that the dependency-rich data we train on produces tangible gains in the most challenging, distractor-laden, high-dispersion, and reasoning-intensive settings. We also note the nonnegligible contribution of model merging (-Model Merging performance drops) in the general domain.

RQ4: Does OctoLong preserve short-context coding and API-use abilities? Finally, we resort to short-context executionvalidated code generation tests to profile the effect of training long cross-repository data on models’ performance on short-context coding tasks. On all benchmarks, we report models’ pass@{1,5,10,25} on a sampling group size 𝑁=50. LiveCodeBench V6: We benchmark algorithmic coding capabilities by sourcing code judge problems from LiveCodeBench V6 (Jain et al., 2025).11 BigCodeBench Hard: The OctoLong pipeline often sources function and class call sites with their reference implementation together in the same context. Prior work shows that this improves API usage (Liu et al., 2026a). We test this directly via BigCodeBench Instruct Hard (Zhuo et al., 2025).

RQ3: Do OctoLong gains extend downstream to agentic use-cases? Models’ capabilities in multi-turn long-horizon interactions with their environment (Jha et al., 2025; Rank et al., 2026; Patwardhan et al., 2025) arguably represent the primary value proposition behind the recent trend towards increasing context lengths (Kwa et al., 2025). Hence, we directly evaluate how our LCFT recipe fares post-SFT on downstream agentic tasks demanding tool-use across a range of turn budgets and difficulty levels. Bfcl V3: We evaluate models on the live, nonlive, and multi-turn splits of BFCL V3 (Patil et al., 2025). The benchmark stress-tests models in singleand multi-turn tool calling while also evaluating resilience against common pathological scenarios such as missing arguments and shifting live APIs.

Results. Table 4 shows that OctoLong-Instruct models yield strong results on short-length code generation across all size classes. Our context extension largely retains short-length coding abilities (ref. 10We use OpenAI gpt-5.1 as the user simulator. 11 To minimize the probability of our auxiliary data being contaminated, we only test on the new code contest problems released in the V6 split (Dec 2024–April 2025), rather than the cumulative setup.

7

XS

OctoLong-0.6B-Instruct

0.6B

128K 40.45

15.89

4.55

1.60

S

meta-llama/Llama-3.2-1B-Instruct 01-ai/Yi-Coder-1.5B-Chat OctoLong-1.7B-Instruct ibm-granite/granite-3.1-2b-instruct

1B 1.5B 1.7B 2B

128K 128K 128K 128K

19.74 14.89 48.18 41.78

13.63 17.33 17.03 17.94

2.45 8.91 10.79 9.57

0.80 5.11 4.90 6.21

meta-llama/Llama-3.2-3B-Instruct Qwen/Qwen3-4B-Instruct-2507 OctoLong-4B-Instruct arcee-ai/AFM-4.5B

3B 4B 4B 4.5B

128K 256K 128K 64K

24.26 57.92 58.37 43.29

14.30 31.18 25.15 20.15

8.33 17.86 17.44 5.59

5.02 19.87 16.31 2.14

Qwen/Qwen2.5-7B-Instruct-1M aws-prototyping/MegaBeam-Mistral-7B-512k internlm/internlm2_5-7b-chat-1m OctoLong-8B-Instruct meta-llama/Llama-3.1-8B-Instruct nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct princeton-nlp/Llama-3-8B-ProLong-512k-Instruct ibm-granite/granite-3.1-8b-instruct gradientai/Llama-3-8B-Instruct-262k mistralai/Ministral-8B-Instruct-2410 zai-org/glm-4-9b-chat-1m 01-ai/Yi-Coder-9B-Chat

7B 7B 7B 8B 8B 8B 8B 8B 8B 8B 9B 9B

1024K 512K 1024K 128K 128K 1024K 512K 128K 256K 128K 1024K 128K

51.21 21.29 28.32 61.65 31.28 27.06 39.00 49.24 19.92 10.34 33.79 49.98

23.06 0.00 22.43 27.35 16.97 16.90 21.08 21.02 19.12 8.98 21.60 20.73

16.45 2.77 2.82 20.82 11.83 14.34 5.12 12.87 3.42 13.23 13.62 17.45

13.11 0.52 4.23 19.44 11.21 10.35 7.21 13.12 3.68 14.21 11.33 13.04

OctoLong-14B-Instruct Qwen/Qwen2.5-14B-Instruct-1M

14B 14B

128K 67.31 1024K 55.59

34.42 26.02

22.43 20.89

21.52 19.61

OctoLong-8B-Instruct - Cross-Repo Code

8B

22.38

19.67

17.56

-4.97

-1.15

-1.88

OctoLong-8B-Instruct - Model Merging

8B

57.74 128K -3.91 61.35 128K -0.30

26.91

17.75

18.04

-0.44

-3.07

-1.40

OctoLong-8B-Instruct - Context Extension

8B

32K

-

-

20.49

19.03

-0.33

-0.41

Qwen3-8B

8B

32K

-

-

23.18

20.73

+2.36

+1.29

Abl.

Model

M

Preference/RL-Tuned

Agentic Tool Use (RQ3) Short-Context Coding (RQ4) Max Size Length BFCL 𝜏2 LiveCode BigCodeBench V3 Bench Bench V6 Instruct Hard

L

Context-Extended-After-Post-Training

XL

Code-Specialised

Table 4: Comparison of benchmark-level summaries between extant open-weight long-context LMs and the OctoLong-Instruct suite on our agentic tool-use (RQ3: ref. RQ3) and short-context coding (RQ4: ref. RQ4) evaluation suites, followed by OctoLong-8B-Instruct training ablations. Agentic tool-use evaluations report the mean over their constituent settings, while short-context coding evaluations report the Pass@1 score. Detailed per-split scores for RQ3 are reported in Table 8 and higher pass-rate scores for RQ4 are reported in Table 9.

-Context Extension ablation for the 8B model). Our models surpass several baselines that have been post-trained on code generation while also being competitive with the post-trained Qwen3-8B model on lower sampling budgets and comfortably surpassing it on higher budgets (ref. Table 9). This discrepancy likely stems from distribution sharpening during RL post-training (Kirk et al., 2024; Fan et al., 2026). The -Cross-Repo Code ablation results also isolate the specific benefits of OctoLong data, especially for the API use tasks. The -Model Merging ablation renders merging with a pre-LCFT checkpoint effective for retaining short-context performance. Finally, the comparative results of Llama 3.1, ProLong (Gao et al., 2025c), and UltraLong (Xu et al., 2026) show that the common practice of context extension on an already post-trained model is suboptimal and can lead to regression in short-context performance.

6

queries in tandem with a language server backend inside a dependency-provisioned containerized environment to source referenced class and function implementations when feasible. We use this pipeline to curate ≈6.2B tokens worth of highquality dependency-rich contexts capped at 128K tokens in length. Subsequently, we mix our crossrepository contexts with data acquired from conventional long-context sources to compile the ≈50Btoken OctoLong-LCFT corpus. This corpus, in conjunction with OctoLong-SFT, our ≈10B-token curated mix of agentic and instruction-tuning data, enables us to develop the OctoLong-Instruct suite of 128K-context models (derived from Qwen3), with between 600M and 14B parameters. Extensive experiments and ablations demonstrate that context extension with dependency-rich code data improves context retrieval, state tracking, and repository-level code understanding while unlocking superior agentic capabilities and API usage. We hope our work helps models support larger context lengths while minimizing context rot (Hong et al., 2025), enabling them to tackle emerging challenges like test-time training (Tandon et al., 2025) and recursive selfimprovement (Yin et al., 2025).

Conclusion

In this work, we present OctoLong, an effort to engineer dependency-dense long-context corpora via recursive retrieval of code across repository boundaries. We detail a pipeline that runs AST 8

puting Center supported by GCS through funding by the German Federal Ministry of Research, Technology and Space and the Ministry of Culture and Science of the State of North Rhine-Westphalia.

Limitations We simplify our training runs by omitting lowresource language data, limiting model sizes to 14B parameters and restricting context lengths to 128K tokens. Each of these represents a non-trivial departure from the contemporary mode of operation of most frontier LMs. However, there exist important signs that OctoLong should transfer well along these axes. First, prior methods for engineering long contexts also omit multilingual long-context data (Gao et al., 2025c; Fu et al., 2024) owing to its rarity (de Vries, 2023) but generalize well to such settings nonetheless (Hengle et al., 2025; Kim et al., 2025). Secondly, our experiments in Section 5 detail how the OctoLong-Instruct models get increasingly adept at longer contexts with size, following trends described in existing literature (Liu et al., 2024b,c). Thirdly, our analyses in Section 3 demonstrate how OctoLong can source code contexts of millions of tokens without sacrificing dependency density or salience. Finally, we note that the recursive dependency retrieval in OctoLong is limited to within-language code where the language server backend can readily leverage package manager conventions to locate implementations. Our pipeline likely does not account for important dependencies in complex codebases that contain subroutines compiled in a different language that the calling code interacts with using an application binary interface (ABI) or a foreign function interface (FFI). Handling such cases demands polyglot static analysis and control flow tracking, which is an area of active research (Houdaille et al., 2023; Rask et al., 2021).

References Tushar Aggarwal, Swayam Singh, Abhijeet Awasthi, Aditya Kanade, and Nagarajan Natarajan. 2025. Nextcoder: Robust adaptation of code lms to diverse code edits. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net. Syeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2025. MIND: math informed synthetic dialogues for pretraining llms. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024. L-eval: Instituting standardized evaluation for long context language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 14388–14411. Association for Computational Linguistics. Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. 2025. Why does the effective context length of llms fall short? In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Ramakrishna Appicharla, Baban Gain, Santanu Pal, and Asif Ekbal. 2026. Which tokens need context? A reference-based analysis of translation responsibility using fertility and entropy. CoRR, abs/2606.29489.

Acknowledgments We gratefully acknowledge support from the hessian.AI Service Center (funded by the German Federal Ministry of Research, Technology, and Space, BMFTR, grant no. 16IS22091) and the hessian.AI Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant no. S-DIW04/0013/003). Additionally, our work is supported by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE. We also thank the Gauss Center for Supercomputing e.V. (GCS) for providing computing time on the Supercomputer JUWELS at the Jülich Supercom-

Solomon E. Asch. 1946. Forming impressions of personality. Journal of abnormal psychology, 41:258–90. Alan D. Baddeley and Graham J. Hitch. 1993. The recency effect: Implicit learning with explicit retrieval? Memory & Cognition, 21:146–155. Kartikay Bagla, Ankit Kumar, Shivam Gupta, and Anuj Gupta. 2021. Noisy text data: Achilles’ heel of popular transformer-based NLP models. CoRR, abs/2110.03353. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. Longalign: A recipe for long context alignment of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, volume

9

EMNLP 2024 of Findings of ACL, pages 1376–1395. Association for Computational Linguistics.

Andrei Z. Broder. 1997. On the resemblance and containment of documents. In Compression and Complexity of SEQUENCES 1997, Positano, Amalfitan Coast, Salerno, Italy, June 11-13, 1997, Proceedings, pages 21–29. IEEE.

Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025a. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 3639–3664. Association for Computational Linguistics.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025b. Longwriter: Unleashing 10, 000+ word generation from long context llms. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E. Peters, Arie Cattan, and Ido Dagan. 2021. CDLM: crossdocument language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, volume EMNLP 2021 of Findings of ACL, pages 2648–2662. Association for Computational Linguistics.

Brenda S. Baker. 1985. A new proof for the first-fit decreasing bin-packing algorithm. J. Algorithms, 6(1):49–70. Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Patiño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, and 4 others. 2025. SmolLM3: smol, multilingual, long-context reasoner. https://huggingface.co/blog/smollm3.

Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, Noah Wang, Boyang Wang, Xianjie Wu, Bing Wang, Tongliang Li, Liqun Yang, Sufeng Duan, Zhaoxiang Zhang, and Zhoujun Li. 2025. Mceval: Massively multilingual code evaluation. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Samuel Barham, Chandler May, and Benjamin Van Durme. 2025. Megawika 2: A more comprehensive multilingual collection of articles and their sources. CoRR, abs/2508.03828.

Guoxin Chen, Fanzhe Meng, Jiale Zhao, Minghao Li, Daixuan Cheng, Huatong Song, Jie Chen, Yuzhi Lin, Hui Chen, Xin Zhao, Ruihua Song, Chang Liu, Cheng Chen, Kai Jia, and Ji-Rong Wen. 2026. Beyondswe: Can current code agent survive beyond single-repo bug fixing? CoRR, abs/2603.03194.

Victor Barrès, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. 𝜏 2 -bench: Evaluating conversational agents in a dual-control environment. CoRR, abs/2506.07982.

Longze Chen, Ziqiang Liu, Wanwei He, Yinhe Zheng, Hao Sun, Yunshui Li, Run Luo, and Min Yang. 2024. Long context is not long at all: A prospector of long-dependency data for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 8222–8234. Association for Computational Linguistics.

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150. Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. 2025. In-context learning with long-context models: An in-depth exploration. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 12119–12149. Association for Computational Linguistics.

Zhi Chen, Qiguang Chen, Libo Qin, Qipeng Guo, Haijun Lv, Yicheng Zou, Hang Yan, Kai Chen, and Dahua Lin. 2025. What are the essential factors in crafting effective long context multi-hop instruction datasets? insights and best practices. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 27129–27151. Association for Computational Linguistics.

Cody Blakeney, Mansheej Paul, Brett W. Larsen, Sean Owen, and Jonathan Frankle. 2024. Does your data spark joy? performance gains from domain upsampling at the end of training. CoRR, abs/2406.03476.

10

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 615–621. Association for Computational Linguistics.

Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 10687–10707. Association for Computational Linguistics. Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, Huaibing Xie, Jianglu Hu, Shaolei Wang, Weichao Wang, Yanling Xiao, Yiting Liu, Zenan Xu, Zhen Guo, Pluto Zhou, and 8 others. 2026. Cl-bench: A benchmark for context learning. CoRR, abs/2602.03587.

Ozren Dabic, Rosalia Tufano, and Gabriele Bavota. 2024. SEART data hub: Streamlining large-scale source code mining and pre-processing. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2024, Flagstaff, AZ, USA, October 6-11, 2024, pages 888–892. IEEE.

Wei Du, Shubham Toshniwal, Branislav Kisacanin, Sadegh Mahdavi, Ivan Moshkov, George Armstrong, Stephen Ge, Edgar Minasyan, Feng Chen, and Igor Gitman. 2025a. Nemotron-math: Efficient longcontext distillation of mathematical reasoning from multi-mode supervision. CoRR, abs/2512.15489.

Tri Dao and Albert Gu. 2024. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pages 10041–10071. PMLR / OpenReview.net.

Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A. Huerta, and Hao Peng. 2025b. Context length alone hurts LLM performance despite perfect retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 23281–23298. Association for Computational Linguistics.

Harm de Vries. 2023. In the long (context) run. Haoran Deng, Yingyu Lin, Zhenghao Lin, Xiao Liu, Yizhou Sun, Yi-An Ma, and Yeyun Gong. 2025. Beyond length: Quantifying long-range information for long-context LLM pretraining data. CoRR, abs/2510.25804.

Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. 2024. What’s in my big data? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Ken Deng, Jiaheng Liu, He Zhu, Congnan Liu, Jingxin Li, Jiakai Wang, Peng Zhao, Chenchen Zhang, Yanan Wu, Xueqiao Yin, Yuanxing Zhang, Wenbo Su, Bangyu Xiang, Tiezheng Ge, and Bo Zheng. 2024. R2c2-coder: Enhancing and benchmarking realworld repository-level code completion abilities of code large language models. CoRR, abs/2406.01359.

Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee F. Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, and 48 others. 2025. Olmo 3. CoRR, abs/2512.13961.

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 3029–3051. Association for Computational Linguistics.

Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 1074–1084. Association for Computational Linguistics.

Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2024. Cocomic: Code completion by jointly modeling infile and cross-file context. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 3433–3445. ELRA and ICCL.

Mingyuan Fan, Weiguang Han, Daixin Wang, Cen Chen, Zhiqiang Zhang, and Jun Zhou. 2026. When sharpening becomes collapse: Sampling bias and semantic coupling in RL with verifiable rewards. CoRR, abs/2601.15609.

Zican Dong, Junyi Li, Jinhao Jiang, Mingyu Xu, Xin Zhao, Bingning Wang, and Weipeng Chen. 2025. Longred: Mitigating short-text degradation of longcontext large language models via restoration distillation. In Proceedings of the 63rd Annual Meeting of the

Run-Ze Fan, Zengzhi Wang, and Pengfei Liu. 2025. Megascience: Pushing the frontiers of posttraining datasets for science reasoning. CoRR, abs/2507.16812.

11

Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. 2025. What is wrong with perplexity for long-context language modeling? In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

genuinely difficult long context NLP. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 16576–16586. Association for Computational Linguistics. Etash Kumar Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, and 31 others. 2025. Openthoughts: Data recipes for reasoning models. CoRR, abs/2506.04178.

Mariia Fedorova, Nikolay Arefyev, Maja Buljan, Jindrich Helcl, Stephan Oepen, Egil Rønningstad, and Yves Scherrer. 2026. Openlid-v3: Improving the precision of closely related language identification - an experience report. In Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects, VarDial@EACL 2026, Rabat, Morocco, March 29, 2026, pages 275–292. Association for Computational Linguistics.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. Deepseek-coder: When the large language model meets programming - the rise of code intelligence. CoRR, abs/2401.14196.

Robert Friel, Masha Belyi, and Atindriyo Sanyal. 2024. Ragbench: Explainable benchmark for retrieval-augmented generation systems. CoRR, abs/2407.11005.

Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. 2023. Continual pretraining of large language models: How to (re)warm your model? CoRR, abs/2308.04014.

Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. Data engineering for scaling language models to 128k context. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pages 14125–14134. PMLR / OpenReview.net.

Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro von Werra, and Martin Jaggi. 2024. Scaling laws and compute-optimal training beyond fixed training durations. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Chaochen Gao, Xing Wu, Zijia Lin, Debing Zhang, and Songlin Hu. 2025a. Longmagpie: A self-synthesis method for generating large-scale long-context instructions. CoRR, abs/2505.17134. Chaochen Gao, Xing Wu, Zijia Lin, Debing Zhang, and Songlin Hu. 2025b. Nextlong: Toward effective long-context training without long documents. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net.

Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. Lminfinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 3991–4008. Association for Computational Linguistics.

Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2025c. How to train long-context language models (effectively). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 7376– 7399. Association for Computational Linguistics.

Bingguang Hao, Zengzhuang Xu, Yuntao Wen, Xinyi Xu, Yang Liu, Tong Zhao, Maolin Wang, Long Chen, Dong Wang, Yicheng Chen, Cunyin Peng, Xiangyu Zhao, Chenyi Zhuang, and Ji Zhang. 2026. From failure to mastery: Generating hard samples for tooluse agents. CoRR, abs/2601.01498.

Charles Goddard. 2025. Extending afm-4.5b to 64k context length. Technical report, Arcee AI.

Ali Hatamizadeh, Yejin Choi, and Jan Kautz. 2026. Gated deltanet-2: Decoupling erase and write in linear attention. CoRR, abs/2605.22791.

Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F. Shen, Ilias Leontiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping, and Chenxi Whitehouse. 2025. Training AI co-scientists using rubric rewards. CoRR, abs/2512.23707.

Ziyuan He, Yuxuan Wang, Jiaqi Li, Kexin Liang, and Muhan Zhang. 2025. Loogle v2: Are llms ready for real-world long dependency challenges? CoRR, abs/2510.22548.

Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. 2024. Is it really long context if all you need is retrieval? towards

Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. 2025. Multilingual needle in a haystack:

12

Investigating long-context behavior of multilingual large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 5165–5180. Association for Computational Linguistics.

Saurabh Jha, Rohan R. Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, Noah Zheutlin, Saki Takano, Divya Pathak, Felix George, Xinbo Wu, Bekir O. Turkkan, Gerard Vanloo, Michael Nidd, Ting Dai, and 21 others. 2025. Itbench: Evaluating AI agents across diverse realworld IT automation tasks. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net.

Kelly Hong, Anton Troynikov, and Jeff Huber. 2025. Context rot: How increasing input tokens impacts llm performance. Technical report, Chroma. Philémon Houdaille, Djamel Eddine Khelladi, Romain Briend, Robbert Jongeling, and Benoît Combemale. 2023. Polyglot AST: towards enabling polyglot code analysis. In 27th International Conference on Engineering of Complex Computer Systems, ICECCS 2023, Toulouse, France, June 14-16, 2023, pages 116–125. IEEE.

Junlong Jia, Ziyang Chen, Xing Wu, Chaochen Gao, Zijia Lin, Debing Zhang, Songlin Hu, and Binghui Guo. 2025. Entropylong: Effective long-context training via predictive uncertainty. CoRR, abs/2510.02330. Junlong Jia, Xing Wu, Chaochen Gao, Ziyang Chen, Zijia Lin, Zhongzhi Li, Weinong Wang, Haotian Xu, Donghui Jin, Debing Zhang, and Binghui Guo. 2026. Litelong: Resource-efficient long-context data synthesis for llms. In Fortieth AAAI Conference on Artificial Intelligence, Thirty-Eighth Conference on Innovative Applications of Artificial Intelligence, Sixteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2026, Singapore, January 20-27, 2026, pages 31274–31282. AAAI Press.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: what’s the real context size of your long-context language models? CoRR, abs/2404.06654. Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. 2024. Liger kernel: Efficient triton kernels for LLM training. CoRR, abs/2410.10989.

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Jerry Huang. 2024. How well can a long sequence model model long sequences? comparing architectural inductive biases on long-context abilities. CoRR, abs/2407.08112. Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Xianzhen Luo, Qiufeng Wang, YuanTao Fan, Qingfu Zhu, Zhaoxiang Zhang, Yang Gao, Jie Fu, and 7 others. 2025. Opencoder: The open cookbook for top-tier code large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 33167–33193. Association for Computational Linguistics.

Nikhil Kandpal, Brian Lester, Colin Raffel, Sebastian Majstorovic, Stella Biderman, Baber Abbasi, Luca Soldaini, Enrico Shippole, A. Feder Cooper, Aviya Skowron, John Kirchenbauer, Shayne Longpre, Lintang Sutawika, Alon Albalak, Zhenlin Xu, Guilherme Penedo, Loubna Ben Allal, Elie Bakouch, John David Pressman, and 8 others. 2025. The common pile v0.1: An 8tb dataset of public domain and openly licensed text. CoRR, abs/2506.05209. Jikun Kang, Wenqi Wu, Filippos Christianos, Alex J. Chan, Fraser Greenlee, George Thomas, Marvin Purtorab, and Andy Toulis. 2025. LM2: large memory models. CoRR, abs/2502.06049.

Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. CoRR, abs/2309.14509.

Yekyung Kim, Jenna Russell, Marzena Karpinska, and Mohit Iyyer. 2025. One ruler to measure them all: Benchmarking multilingual long-context language models. CoRR, abs/2503.01996.

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando SolarLezama, Koushik Sen, and Ion Stoica. 2025. Livecodebench: Holistic and contamination-free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. Understanding the effects of RLHF on LLM generalisation and diversity. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

13

Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. 2023. The stack: 3 TB of permissively licensed source code. Trans. Mach. Learn. Res., 2023.

Han Li, Letian Zhu, Bohan Zhang, Rili Feng, Jiaming Wang, Yue Pan, Earl T. Barr, Federica Sarro, Zhaoyang Chu, and He Ye. 2026. Contextbench: A benchmark for context retrieval in coding agents. CoRR, abs/2602.05892. Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Hong Chen, and Cuiping Li. 2025a. Omnisql: Synthesizing highquality text-to-sql data at scale. Proc. VLDB Endow., 18(11):4695–4709.

Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. BOOKSUM: A collection of datasets for long-form narrative summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, volume EMNLP 2022 of Findings of ACL, pages 6536–6558. Association for Computational Linguistics.

Jiaxi Li, Xingxing Zhang, Xun Wang, Xiaolong Huang, Li Dong, Liang Wang, Si-Qing Chen, Wei Lu, and Furu Wei. 2025b. Wildlong: Synthesizing realistic long-context instruction data at scale. CoRR, abs/2502.16684.

Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Y. Sorokin, and Mikhail Burtsev. 2024. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. 2025c. Codeio: Condensing reasoning patterns via code input-output prediction. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net. Siheng Li, Cheng Yang, Zesen Cheng, Lemao Liu, Mo Yu, Yujiu Yang, and Wai Lam. 2024a. Large language models can self-improve in long-context reasoning. CoRR, abs/2411.08147.

Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Neev Parikh, and 6 others. 2025. Measuring AI ability to complete long software tasks. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.

Yanyang Li, Shuo Liang, Michael R. Lyu, and Liwei Wang. 2024b. Making long-context language models better multi-hop reasoners. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 2462–2475. Association for Computational Linguistics.

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, and 4 others. 2024. Tülu 3: Pushing frontiers in open language model post-training. CoRR, abs/2411.15124.

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024c. Snapkv: LLM knows what you are looking for before generation. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2025. Selective attention improves transformer. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Dianshu Liao, Shidong Pan, Xiaoyu Sun, Xiaoxue Ren, Qing Huang, Zhenchang Xing, Huan Jin, and Qinying Li. 2024. $\mathbf{Aˆ{3}}$a3-codgen: A repository-level code generation framework for code reuse with local-aware, global-aware, and thirdparty-library-aware. IEEE Trans. Software Eng., 50(12):3369–3384.

Dacheng Li, Rulin Shao, Anze Xie, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023a. Lightseq: Sequence level parallelism for distributed training of long context transformers. CoRR, abs/2310.03294.

Yong Lin, Shange Tang, Bohan Lyu, Ziran Yang, JuiHui Chung, Haoyu Zhao, Lai Jiang, Yihan Geng, Jiawei Ge, Jingruo Sun, Jiayun Wu, Jiri Gesi, Ximing Lu, David Acuna, Kaiyu Yang, Hongzhou Lin, Yejin Choi, Danqi Chen, Sanjeev Arora, and Chi Jin. 2025. Goedel-prover-v2: Scaling formal theorem proving with scaffolded data synthesis and self-correction. CoRR, abs/2508.03613.

Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023b. CAMEL: communicative agents for "mind" exploration of large scale language model society. CoRR, abs/2303.17760.

14

Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S. Weld. 2020. S2ORC: the semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4969–4983. Association for Computational Linguistics.

Zhan Ling, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. 2025. Longreason: A synthetic longcontext reasoning benchmark via context expansion. CoRR, abs/2501.15089. Chen Liu, Qingyuan Liang, Hanwen Zhang, Zeyu Sun, Yakun Zhang, and Lu Zhang. 2026a. Contextualized code pretraining for code generation. CoRR, abs/2605.17957.

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.

Emmy Liu, Graham Neubig, and Chenyan Xiong. 2025a. Midtraining bridges pretraining and posttraining distributions. CoRR, abs/2510.14865. Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring attention with blockwise transformers for near-infinite context. CoRR, abs/2310.01889.

Lefteris Loukas, Manos Fergadiotis, Ion Androutsopoulos, and Prodromos Malakasiotis. 2021. EDGARCORPUS: billions of tokens make the world go round. CoRR, abs/2109.14394.

Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang. 2024a. Repoqa: Evaluating long context code understanding. CoRR, abs/2406.06025.

Anton Lozhkov, Hugo Larcher, Mathieu Morlon, Loubna Ben Allal, and Leandro von Werra. 2026. The stack v3: The largest open code dataset. Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, and 38 others. 2024. Starcoder 2 and the stack v2: The next generation. CoRR, abs/2402.19173.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024b. Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguistics, 12:157–173. Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. 2026b. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and LLM alignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, pages 17417–17437. Association for Computational Linguistics.

Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M. Rush. 2024. A controlled study on long context extension and generalization in llms. CoRR, abs/2409.12181. Rabeeh Karimi Mahabadi, Sanjeev Satheesh, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2025. Nemotron-cc-math: A 133 billion-token-scale high quality math pretraining dataset. CoRR, abs/2508.15096.

Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong Wang, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang, Yong Liu, Yasheng Wang, and 8 others. 2025b. Toolace: Winning the points of LLM function calling. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Mike A. Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, Estefany Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis-Mastromichalakis, and 66 others. 2026. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. CoRR, abs/2601.11868.

Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. 2024c. Scaling laws of rope-based extrapolation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Microsoft. 2024. Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404.14219.

Yifei Liu, Li Lyna Zhang, Yi Zhu, Bingcheng Dong, Xudong Zhou, Ning Shang, Fan Yang, Cheng Li, and Mao Yang. 2025c. rstar-coder: Scaling competitive code reasoning with a large-scale verified dataset. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.

Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andrés Codas, Yadong Lu, Weige Chen, Olga Vrousgos, Corby Rosset, Fillipe Silva, Hamed Khanpour, Yash Lara, and Ahmed Awadallah. 2024. Agentinstruct: Toward generative teaching with agentic flows. CoRR, abs/2407.03502.

15

Kaixiang Mo, Yuxin Shi, Weiwei Weng, Zhiqiang Zhou, Shuman Liu, Haibo Zhang, and Anxiang Zeng. 2025. Mid-training of large language models: A survey. CoRR, abs/2510.06826.

in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.

Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schütze. 2025. Nolima: Longcontext evaluation beyond literal matching. In Fortysecond International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net.

Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Defu Lian, Zhicheng Dou, and Tiejun Huang. 2025. Memorag: Boosting long context processing with global memory-enhanced retrieval augmentation. In Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 20252 May 2025, pages 2366–2377. ACM.

OpenAI. 2025. Openai-mrcr: Multi-round co-reference resolution benchmark. Open-source long-context multiple needle-in-a-haystack evaluation dataset.

Yiwei Qin, Zhen Huang, Tiantian Mi, Weiye Si, Chenyang Zhou, Qipeng Guo, Siyuan Feng, and Pengfei Liu. 2026. Data darwinism part I: unlocking the value of scientific data for pre-training. CoRR, abs/2602.07824.

Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The berkeley function calling leaderboard (BFCL): from tool use to agentic evaluation of large language models. In Fortysecond International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net.

Stefano Rando, Luca Romani, Alessio Sampieri, Yuta Kyuragi, Luca Franco, Fabio Galasso, Tatsunori Hashimoto, and John Yang. 2025. Longcodebench: Evaluating coding llms at 1m context windows. CoRR, abs/2505.07897. Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg, Karina Nguyen, Matthias Bethge, and Maksym Andriushchenko. 2026. Posttrainbench: Can LLM agents automate LLM post-training? CoRR, abs/2603.08640.

Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, and Jerry Tworek. 2025. Gdpval: Evaluating AI model performance on real-world economically valuable tasks. CoRR, abs/2510.04374.

Jonas Kjær Rask, Frederik Palludan Madsen, Nick Battle, Hugo Daniel Macedo, and Peter Gorm Larsen. 2021. The specification language server protocol: A proposal for standardised LSP extensions. In Proceedings of the 6th Workshop on Formal Integrated Development Environment, F-IDE@NFM 2021, Held online, 24-25th May 2021, volume 338 of EPTCS, pages 3–18.

Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Neil Rathi and Alec Radford. 2026. Shaping capabilities with token-level data filtering. CoRR, abs/2601.21571. Hayley Ross, Ameya Sunil Mahabaleshwarkar, and Yoshi Suhara. 2025. When2call: When (not) to call tools. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 3391–3409. Association for Computational Linguistics.

Xiaoxuan Peng, Kaiqi Zhang, Xinyu Lu, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, and Le Sun. 2026. Litecoder-terminal: Scaling long-horizon terminal environments for learning language agents. CoRR, abs/2605.29559. Renjie Pi, Grace Lam, Mohammad Shoeybi, Pooya Jannaty, Bryan Catanzaro, and Wei Ping. 2026. On data engineering for scaling LLM terminal capabilities. CoRR, abs/2602.21193.

Nikolaus Salvatore, Hao Wang, and Qiong Zhang. 2025. Lost in the middle: An emergent property from information retrieval demands in llms. CoRR, abs/2510.10276.

Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Manoj Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Weiran Yao, Huan Wang, Silvio Savarese, and Caiming Xiong. 2025. Apigenmt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay. In Advances

Maksim Sapronov and Evgeniy Glukhov. 2025. On pretraining for project-level code completion. CoRR, abs/2510.13697.

16

Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah Goldblum, Matus Telgarsky, and Pavel Izmailov. 2026. Understanding reasoning from pretraining to post-training.

In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 August 1, 2025, volume ACL 2025 of Findings of ACL, pages 521–533. Association for Computational Linguistics.

Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, and Torsten Scholak. 2023. Repofusion: Training code models to understand your repository. CoRR, abs/2306.10998.

Niko Usai, Dario Montagnini, Kristian Ilianov Iliev, and Raffaele Camanzo. 2026. Logiclens: Leveraging semantic code graph to explore multi repository large systems. CoRR, abs/2601.10773.

Aaditya K. Singh, Yu Yang, Kushal Tirumala, Mostafa Elhoushi, and Ari S. Morcos. 2024. Brevity is the soul of wit: Pruning long files for code generation. CoRR, abs/2407.00434.

Fanqi Wan, Weizhou Shen, Shengyi Liao, Yingcheng Shi, Chenliang Li, Ziyi Yang, Ji Zhang, Fei Huang, Jingren Zhou, and Ming Yan. 2025. Qwenlong-l1: Towards long-context large reasoning models with reinforcement learning. CoRR, abs/2505.17667.

Konrad Staniszewski, Szymon Tworkowski, Sebastian Jaszczur, Yu Zhao, Henryk Michalewski, Lukasz Kucinski, and Piotr Milos. 2025. Structured packing in LLM training improves long context utilization. In Thirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2025, Philadelphia, PA, USA, February 25 - March 4, 2025, pages 25201–25209. AAAI Press.

Boxin Wang, Chankyu Lee, Nayeon Lee, Sheng-Chieh Lin, Wenliang Dai, Yang Chen, Yangyi Chen, Zhuolin Yang, Zihan Liu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025a. Nemotron-cascade: Scaling cascaded reinforcement learning for generalpurpose reasoning models. CoRR, abs/2512.13607. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. 2025b. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.

Arnuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, Jed McCaleb, Yejin Choi, and Yu Sun. 2025. End-to-end test-time training for long context. CoRR, abs/2512.23675. Zecheng Tang, Baibei Ji, Juntao Li, Lijun Wu, Haijia Gui, and Min Zhang. 2025. Revisiting long-context modeling from context denoising perspective. CoRR, abs/2510.05862. Artificial Analysis Team. 2025a. Artificial analysis long context reasoning benchmark(lcr).

Yejie Wang, Keqing He, Dayuan Fu, Zhuoma Gongque, Heyang Xu, Yanxu Chen, Zhexu Wang, Yujia Fu, Guanting Dong, Muxi Diao, Jingang Wang, Mengdi Zhang, Xunliang Cai, and Weiran Xu. 2024. How do your code llms perform? empowering code instruction tuning with high-quality data. CoRR, abs/2409.03810.

Llama Team. 2024. The llama 3 herd of models. CoRR, abs/2407.21783. MiroMind Team, Song Bai, Lidong Bing, Carson Chen, Guanzheng Chen, Yuntao Chen, Zhe Chen, Ziyi Chen, Jifeng Dai, Xuan Dong, Wenhan Dou, Yue Deng, Yunjie Fu, Junqi Ge, Chenxia Han, Tammy Huang, Zhenhang Huang, Jerry Jiao, Shilei Jiang, and 36 others. 2025. Mirothinker: Pushing the performance boundaries of open-source research agents via model, context, and interactive scaling. CoRR, abs/2511.11793.

Yubo Wang, Xiang Yue, and Wenhu Chen. 2025c. Critique fine-tuning: Learning to critique is more effective than learning to imitate. CoRR, abs/2501.17703. Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. 2025d. Octothinker: Mid-training incentivizes reinforcement learning scaling. CoRR, abs/2506.20512.

Qwen Team. 2025b. Qwen3 technical report. CoRR, abs/2505.09388.

Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. 2024. Redpajama: an open dataset for training large language models. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Ryan Teknium, Jeffrey Quesnelle, and Chen Guang. 2024. Hermes 3 technical report. CoRR, abs/2408.11857. Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Huadong Wang, and Xiaojiang Liu. 2025. Distance between relevant information pieces causes bias in long-context llms.

17

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pages 52632–52657. PMLR / OpenReview.net.

2026. A decomposition perspective to long-context reasoning for llms. CoRR, abs/2604.07981. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, and 2 others. 2024. Effective long-context scaling of foundation models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 4643–4663. Association for Computational Linguistics.

Chen Wu and Yin Song. 2025. Scaling context, not parameters: Training a compact 7b language model for efficient long-context processing. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 61–68. Association for Computational Linguistics. Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. 2025a. Longmemeval: Benchmarking chat assistants on long-term interactive memory. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, and Bryan Catanzaro. 2026. From 128k to 4m: Efficient training of ultralong context large language models. In Findings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2-7, 2026, pages 13122–13133. Association for Computational Linguistics.

Longyun Wu, Dawei Zhu, Guangxiang Zhao, Zhuocheng Yu, Junfeng Ran, Xiangyu Wong, Lin Sun, and Sujian Li. 2025b. Longattn: Selecting long-context training data via token-level attention. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Findings of ACL, pages 19367–19380. Association for Computational Linguistics.

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2025a. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Siwei Wu, Yizhi Li, Yuyang Song, Wei Zhang, Yang Wang, Riza Batista-Navarro, Xian Yang, Mingjie Tang, Bryan Dai, Jian Yang, and Chenghua Lin. 2026a. Large-scale terminal agentic trajectory generation from dockerized environments. CoRR, abs/2602.01244.

Zhangchen Xu, Adriana Meza Soria, Shawn Tan, Anurag Roy, Ashish Sunil Agrawal, Radha Poovendran, and Rameswar Panda. 2025b. TOUCAN: synthesizing 1.5m tool-agentic data from real-world MCP environments. CoRR, abs/2510.01179.

Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Xiangju Li, Junmin Zhu, and Wei Zhang. 2025c. Lifbench: Evaluating the instruction following performance and stability of large language models in long-context scenarios. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 16445–16468. Association for Computational Linguistics.

An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, and 9 others. 2025a. Qwen2.5-1m technical report. CoRR, abs/2501.15383. Cehao Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Shengjie Ma, Aofan Liu, Hui Xiong, and Jian Guo. 2025b. Longfaith: Enhancing long-context reasoning in llms with faithful synthetic data. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Findings of ACL, pages 3236–3256. Association for Computational Linguistics.

Yuhe Wu, Guangyu Wang, Yuran Chen, Jiatong Zhang, Yutong Zhang, Yujie Chen, Jiaming Shang, Guang Zhang, and Zhuang Liu. 2026b. PRISM: probing reasoning, instruction, and source memory in LLM hallucinations. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, pages 33619–33656. Association for Computational Linguistics.

Chen Yang, Ran Le, Yun Xing, Zhenwei An, Zongchao Chen, Wayne Xin Zhao, Yang Song, and Tao Zhang. 2025c. Toolmind technical report: A largescale, reasoning-enhanced tool-use dataset. CoRR, abs/2511.15718.

Yanling Xiao, Huaibing Xie, Guoliang Zhao, Shihan Dou, Shaolei Wang, Yiting Liu, Nantao Zheng, Cheng Zhang, Pluto Zhou, Zhisong Zhang, and Lemao Liu.

Chen Yang, Guangyue Peng, Jiaying Zhu, Ran Le, Ruixiang Feng, Tao Zhang, Xiyun Xu, Yang Song, Yiming Jia, Yuntao Wen, Yunzhi Xu, Zekai Wang,

18

Zhenwei An, Zhicong Sun, and Zongchao Chen. 2026. Nanbeige4.1-3b: A small general model that reasons, aligns, and acts. CoRR, abs/2602.13367.

August 1, 2025, pages 23078–23097. Association for Computational Linguistics. Ji Zeng, Dayuan Fu, Tiantian Mi, Yumin Zhuang, Yaxing Huang, Xuefeng Li, Lyumanshan Ye, Muhang Xie, Qishuo Hua, Zhen Huang, Mohan Jiang, Hanning Wang, Jifan Lin, Yang Xiao, Jie Sun, Yunze Wu, and Pengfei Liu. 2026. davinci-dev: Agentnative mid-training for software engineering. CoRR, abs/2601.18418.

John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023a. Repocoder: Repositorylevel code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 2471–2484. Association for Computational Linguistics.

Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. Linkbert: Pretraining language models with document links. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8003–8016. Association for Computational Linguistics. Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2025. HELMET: how to evaluate long-context models effectively and thoroughly. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, Xingtai Lv, Jinfang Hu, Zhiyuan Liu, and Bowen Zhou. 2024. Ultramedical: Building specialized generalists in biomedicine. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Xunjian Yin, Xinyi Wang, Liangming Pan, Li Lin, Xiaojun Wan, and William Yang Wang. 2025. Gödel agent: A self-referential agent framework for recursively self-improvement. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 27890–27913. Association for Computational Linguistics.

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. 2023b. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, and 11 others. 2024. Yi: Open foundation models by 01.ai. CoRR, abs/2403.04652.

Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang, Peng Cheng, Lijie Wang, Chenxia Li, Xuejie Wu, Bo Zhu, Yimeng Gan, Rui Hu, Shuicheng Yan, Han Fang, and Yahui Zhou. 2024a. Longskywork: A training recipe for efficiently extending context length in large language models. CoRR, abs/2406.00605.

Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pages 37:1–37:12. ACM.

Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024b. Wildchat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025. Native sparse attention: Hardwarealigned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 -

Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. 2025. SWIFT: A scalable lightweight infrastructure for fine-tuning. In Thirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial

19

Intelligence, Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2025, Philadelphia, PA, USA, February 25 - March 4, 2025, pages 29733–29735. AAAI Press.

A

Architecture & Training Details Attribute

Value Common Attributes

Backbone Family Backbone Type Backbone Sizes Tuning Strategy Training Sequence Length Model Datatype Flash Attention Variant FSDP Variant Sequence Parallelism Mode Sequence Parallelism Degree

Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. Opencodeinterpreter: Integrating code generation with execution and refinement. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, volume ACL 2024 of Findings of ACL, pages 12834–12859. Association for Computational Linguistics.

Liger Kernels Gradient Checkpointing Optimizer Type Beta Epsilon Weight Decay Gradient Clipping Scheduler Type Scheduler Warmup Proportion Peak Learning Rate Terminal Learning Rate Ratio Per-Device Batch Size Training Epochs Random Seed

Yingming Zheng, Hanqi Li, Kai Yu, and Lu Chen. 2025. When long helps short: How context length in supervised fine-tuning affects behavior of large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 10293–10308. Association for Computational Linguistics. Tongyao Zhu, Qian Liu, Haonan Wang, Shiqi Chen, Xiangming Gu, Tianyu Pang, and Min-Yen Kan. 2025. Skyladder: Better and faster pretraining via context window scheduling. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.

Qwen/Qwen3 Dense 0.6B, 1.7B, 4B, 8B, 14B Full Fine-Tuning 131,072 Tokens bfloat16 Version 2 Version 2 DeepSpeed-Ulysses 0.6B: 2 1.7B: 2 4B: 2 8B: 4 14B: 4 True True AdamW-Fused {0.9, 0.95} 1e-8 0.1 1.0 Warmup-Stable-Decay 0.05 1e-5 0.02 1 1.0 77

LCFT-Specific Attributes (OctoLong-Base) Training Corpus Training Tokens Optimization Steps Scheduler Decay Steps Global Batch Size Training GPU Count

OctoLong-LCFT 51.5B 3072 0 16.8M Tokens 0.6B: 32 1.7B: 64 4B: 64 8B: 128 14B: 128

SFT-Specific Attributes (OctoLong-Instruct) Initialization Training Corpus Training Tokens Optimization Steps Scheduler Decay Steps Global Batch Size Training GPU Count

Yonghao Zhuang, Lanxiang Hu, Longfei Yun, Souvik Kundu, Zhengzhong Liu, Eric P. Xing, and Hao Zhang. 2025. Scaling long context training data by long-distance referrals. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, and 2 others. 2025. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

OctoLong-*-Base OctoLong-SFT 10.1B 1202 75 8.4M Tokens 0.6B: 16 1.7B: 32 4B: 64 8B: 64 14B: 128

Table 5: Architectural heritage and training attributes of the OctoLong suite of models across the LCFT and SFT stages. The training data curation is outlined in Section 4 and its composition is detailed in Appendix B.

B

Data Composition & Processing

All constituent datasets in both our LCFT and SFT mixtures are subjected to a common cleaning and decontamination pipeline. Specifically, we first enforce an English-only language filter using the OpenLID (Fedorova et al., 2026) classifier with a confidence threshold of 0.9, followed by a MinHash de-duplication (Broder, 1997) filter configured with a shingle size of 20 and a Jaccard similarity threshold of 0.7. Next, following prior work (Brown et al., 2020; Elazar et al., 2024), we decontaminate against all of our downstream evaluations by discarding any sample whose text registers a 13-gram 20

Scientific-domain source code harvested from GitHub across 178 research topics (e.g., bioinformatics, numerical methods).

overlap with a test-set instance. We serialize every function-call interaction in the SFT data into the Hermes (Teknium et al., 2024) format to ensure smooth operation with the Qwen3 chat templates. Additionally, for any dataset that ships intermediate chain-of-thought or reasoning traces, we retain only the final response and discard the thinking blocks. B.1

5

TheStackV2

61111 Samples

bigcode/the-stack-v2-dedup

Deduplicated repository-level code contexts from the Software Heritage archive, released as part of StarCoder2 (Lozhkov et al., 2024).

LCFT Data Mixture

Our LCFT corpus comprises ≈50B tokens spanning the nine source categories outlined in Table 2. We detail the data sources that feed into the categorylevel document pools, which are used to create the OctoLong-LCFT and our replication of the ProLong (Gao et al., 2025c) corpus. We also state the sample counts that each source contributes to the final OctoLong-LCFT dataset.

6

TheStack-Issues

54985 Samples

bigcode/the-stack-github-issues

GitHub issue and pull-request conversations from The Stack (Kocetkov et al., 2023), providing diverse code-grounded text data. Books Data

OctoLong Data 1 1

OctoLong

57993 Samples

RedPajama-Books

61206 Samples

togethercomputer/RedPajama-Data-1T

Our own cross-repository code contexts, averaging ≈107K tokens in length, mined and assembled as described in Section 3 containing ≈6.2B tokens of dependency-rich long-range code data that anchor our LCFT mixture.

The books subset of the RedPajama (Weber et al., 2024) corpus, offering full-length narrative and non-fiction prose as naturally long-form text. 2

BookSum

2011 Samples

ubaada/booksum-complete-cleaned

In-Repo & Code-Adjacent Data 1

OpenCoder-Annealing

Book- and chapter-level summarization pairs from BookSum (Kryscinski et al., 2022), which couple long source documents with abstractive summaries to exercise long-range aggregation.

2830910 Samples

OpenCoder-LLM/opc-annealing-corpus

The algorithmic_corpus subset of the OpenCoder (Huang et al., 2025) mid-training corpus, comprising file-level code data filtered from The Stack v2 (Lozhkov et al., 2024) comprised of dense, self-contained procedural code. 2

CommonPile-GitHub

Math & Math-Adjacent Data 1

The CraneMath subset of the Olmo 3 (Ettinger et al., 2025) Dolmino mixture, comprising synthetically generated mathematical text.

2288814 Samples

GitHub issue reports and pull-request discussions from Common Pile (Kandpal et al., 2025). CraneCode

2

1880920 Samples

657135 Samples

Synthetic mathematical dialogues generated in seven conversational styles via the MIND (Akter et al., 2025) recipe, providing structured, mathfocused multi-turn text.

The CraneCode subset of the Olmo 3 (Ettinger et al., 2025) Dolmino corpus, consisting of synthetically generated, self-contained code samples that broaden the stylistic and topical coverage of our code data. SciCode-Domain

Nemotron-MIND nvidia/Nemotron-MIND

allenai/dolma3_dolmino_mix-100B-1125

4

1421783 Samples

allenai/dolma3_dolmino_mix-100B-1125

common-pile/github_archive_filtered

3

CraneMath

3

NuminaMath-CoT

433437 Samples

AI-MO/NuminaMath-CoT

Competition and olympiad mathematics problems paired with step-by-step solutions from the NuminaMath collection.

181596 Samples

SciCodePile/SciCode-Domain-Code

21

4

Nemotron-CC-Math

250470 Samples

3

nvidia/Nemotron-CC-Math-v1

arXiv papers paired with their abstracts repurposed as summaries (Cohan et al., 2018). 4

MegaMath-Web-Pro

204531 Samples

Tech Content & Forum Data

99361 Samples

nvidia/Nemotron-Math-Proofs-v2

1

Formal and natural-language mathematical proofs with verification and meta-verification annotations (Du et al., 2025a). We strip the intermediate proof-search and verification reasoning, retaining the final proof text.

General Domain Data

196874 Samples 1

HuggingFaceTB/issues-kaggle-notebooks

127500 Samples

Digitized United States court opinions from the Caselaw Access Project, supplying long-form legal reasoning and citation-dense prose.

984805 Samples

HuggingFaceFW/finepdfs-edu

The education-filtered subset of FinePDFs, comprising long-form structurally rich documents extracted from openly available PDFs. peS2o

Caselaw-Access free-law/Caselaw_Access_Project

Academic Papers & PDF Data

2

361292 Samples

Full-text United States patent documents sourced from PatentsView, contributing long, formally structured legal and technical writing. 2

FinePDFs-Edu

US-Patents allenai/us-patents

Structured Kaggle data-science notebooks released with StarCoder2 (Lozhkov et al., 2024), interleaving code, prose explanation, and outputs into pedagogically structured long documents.

1

765921 Samples

Long-form programming tutorials and howto guides, interleaving explanatory prose with code, as a source of instructional technical content.

728882 Samples

The educational-heavy subset of FineWeb (Penedo et al., 2024), filtered for high-signal instructional web text. Kaggle-Notebooks

Coding-Tutorials bigcode/coding_tutorials

HuggingFaceFW/fineweb-edu

2

2000000 Samples

StackExchange question-and-answer threads rendered to clean Markdown by the Marin community, covering technical forum discussion. 2

FineWeb-Edu

Marin-StackExchange

marin-community/stackexchange-markdown

Educational Data 1

79513 Samples

A quality-graded scientific document corpus, released with the Data Darwinism (Qin et al., 2026) study, contributing curated technical content and scientific prose.

A high-quality, refined math web corpus released with OctoThinker (Wang et al., 2025d). We use it as a source of naturally occurring (rather than synthetic) mathematical prose. Nemotron-Math-Proofs

Darwin-Science GAIR/Darwin-Science

OctoThinker/MegaMath-Web-Pro-Max

6

90000 Samples

ccdv/arxiv-summarization

The highest quality subset of Nemotron-CCMath (Mahabadi et al., 2025), comprised of math-heavy text from Common Crawl. 5

arXiv-Summarization

3

SEC-Filings

25669 Samples

PleIAs/SEC

Form 10-K corporate filings (Loukas et al., 2021), providing long, highly-structured financial and regulatory documents.

416910 Samples

allenai/peS2o

Full-text long-form research papers derived from the S2ORC (Lo et al., 2020) corpus.

Agentic & Instruct-Like Data 22

1

Magicoder-Evol-Instruct

910234 Samples

2

ise-uiuc/Magicoder-Evol-Instruct-110K

The tool_calling and interactive_agent subsets of the Nemotron function-calling corpus, covering single-shot function calling and multi-turn interactive tool use, respectively. 3

Synthetic-Text2SQL

340000 Samples

3

SWE-Zero

299147 Samples 4

120000 Samples 5

Single-turn request-response pairs obtained via the AgentInstruct (Mitra et al., 2024) pipeline. daVinci-Dev

14156 Samples

6

callumpurcell/SynthQA

TerminalTraj

6987 Samples 7

Terminal-based agentic trajectories from TerminalTraj (Wu et al., 2026a), in which an agent issues shell commands to solve tasks.

184827 Samples

Large-scale multi-round instructional dialogues from UltraChat (Ding et al., 2023), supplying diverse open-domain conversational data.

Supervised Fine-Tuning (SFT) Data Mixture

MegaScience

UltraChat openbmb/UltraChat

8

Our SFT corpus comprises ≈10B tokens spanning general instruction following, mathematics, code, tool use, and long-context interaction. The constituent datasets and their sample counts follow: 1

190982 Samples

The general subset of the NemotronCascade (Wang et al., 2025a), covering broad general-purpose instruction following.

m-a-p/TerminalTraj

B.2

Nemotron-Cascade

nvidia/Nemotron-Cascade-SFT-Stage-1

Synthetic extractive question-answering examples with explanations. 7

226597 Samples

Three code instruction subsets from OpenCoder (Huang et al., 2025): largescale_diverse_instruct and realuser_instruct from stage 1, together with package_instruct from stage 2, spanning diverse synthetic instructions, user-derived queries, and library-usage tasks.

39130 Samples

The llm_enhanced_prs subset of daVinciDev (Zeng et al., 2026), comprising pullrequest-derived software-development trajectories. SynthQA

OpenCoder-SFT OpenCoder-LLM/opc-sft-stage1 OpenCoder-LLM/opc-sft-stage2

GAIR/daVinci-Dev

6

259595 Samples

Tool-integrated-reasoning mathematics data in which solutions invoke a Python interpreter.

microsoft/orca-agentinstruct-1M-v1

5

NuminaMath-TIR AI-MO/NuminaMath-TIR

Execution-verified software-engineering agent trajectories over real repositories. Orca-AgentInstruct

283328 Samples

The multi- and single-document long-context subsets of LongMagpie (Gao et al., 2025a), with synthetic long question-answering tuples.

pairs

AlienKevin/SWE-ZERO-12M-trajectories

4

LongMagpie

caskcsg/LongMagpie_multidoc_longcontext_dataset caskcsg/LongMagpie_singledoc_longcontext_dataset

gretelai/synthetic_text_to_sql

Synthetic natural-language-to-SQL grounded in diverse database schemas.

290815 Samples

nvidia/Nemotron-SFT-Agentic-v2

Evolved code instruction-response pairs, decontaminated from the evol-instruct (Yang et al., 2024) lineage and released with Magicoder (Wei et al., 2024), used to inject instruction-following behavior during LCFT. 2

Nemotron-Agentic

SynSQL

179503 Samples

seeklhy/SynSQL-2.5M

Synthetic text-to-SQL examples with explanations from OmniSQL (Li et al., 2025a).

315342 Samples

9

Dolci-Tool

178905 Samples

allenai/Dolci-Instruct-SFT-Tool-Use

MegaScience/MegaScience

Large-scale scientific reasoning instruction data from MegaScience (Fan et al., 2025).

Tool-use instruction data from the Olmo 3 (Ettinger et al., 2025) post-training suite. 23

10 MiroVerse

19 Orca-AgentInstruct-MT

116597 Samples

microsoft/orca-agentinstruct-1M-v1

miromind-ai/MiroVerse-v0.1

The multi-turn follow_up split of the AgentInstruct (Mitra et al., 2024) data, providing extended multi-turn agentic instruction flows.

Deep-research agentic trajectories from MiroThinker (Team et al., 2025). 11 UltraMedical

104085 Samples 20 WebInstruct-CFT

TsinghuaC3I/UltraMedical

Critique fine-tuning data from WebInstructCFT (Wang et al., 2025c), which pairs responses with model-generated reflective critiques.

89409 Samples 21 NextCoder

Agent-Ark/Toucan-1.5M

40448 Samples

microsoft/NextCoderDataset-Conversational

The SFT split of Toucan (Xu et al., 2025b), comprising agentic trajectories on real MCPs. 13 WildChat

49247 Samples

TIGER-Lab/WebInstruct-CFT

Biomedical instruction data from UltraMedical (Zhang et al., 2024), covering clinical and life-sciences question answering. 12 Toucan

51753 Samples

The conversational split of NextCoder (Aggarwal et al., 2025), covering multi-turn codeediting.

85766 Samples

allenai/WildChat-4.8M 22 OpenRubrics

Real user–assistant conversation logs from WildChat (Zhao et al., 2024b), supplying naturallyoccurring instruction distributions. 14 rStar-Coder

36445 Samples

OpenRubrics/OpenRubric-v2

Rubric-conditioned instruction data from OpenRubrics (Liu et al., 2026b), pairing prompts with synthetic scoring rubrics and aligned answers.

84369 Samples

microsoft/rStar-Coder 23 OpenThoughts

Competitive-programming problems with solution from rStar-Coder (Liu et al., 2025c). 15 Magicoder-OSS-Instruct

bethgelab/CuratedThoughts

The OpenThoughts-114k-Math-default subset of the CuratedThoughts filtering of OpenThoughts (Guha et al., 2025).

61552 Samples

ise-uiuc/Magicoder-OSS-Instruct-75K

Open-source-derived code instruction pairs sourced via OSS-Instruct (Wei et al., 2024). 16 Code-Feedback

24 Jupyter-Agents

60186 Samples

The non_thinking subset of the Jupyter Agents data, comprising notebook-based codeexecution agent trajectories.

Multi-turn code-execution-feedback dialogues from OpenCodeInterpreter (Zheng et al., 2024).

25 Tulu3-Personas-IF

25056 Samples

allenai/tulu-3-sft-personas-instruction-following

57389 Samples

Persona-grounded instruction-following data from TÜLU 3 (Lambert et al., 2024), targeting precise constraint satisfaction.

Magpie-Align/Magpie-Qwen2.5-Coder-Pro-300K-v0.1

Synthetic coding instructions self-generated via the Magpie (Xu et al., 2025a) prompting method, seeded from a code-specialized model. 18 ToolMind

34812 Samples

jupyter-agent/jupyter-agent-dataset

m-a-p/Code-Feedback

17 Magpie-Coder

36259 Samples

26 CAMEL-Science

24336 Samples

camel-ai/physics camel-ai/chemistry camel-ai/biology

56963 Samples

Nanbeige/ToolMind

Role-play-derived scientific instruction data from CAMEL (Li et al., 2023b), drawn from its physics, chemistry, and biology domains.

The graph_syn_datasets subset of ToolMind (Yang et al., 2025c), comprising functiongraph-derived synthetic tool-use data. 24

27 McEval-Instruct

36 LongAlign

22310 Samples

Multilingual-Multimodal-NLP/McEval-Instruct

zai-org/LongAlign-10k

Multilingual code instruction data released with McEval (Chai et al., 2025), spanning a broad set of programming languages. 28 XCoder

Long-context alignment instructions from LongAlign (Bai et al., 2024), designed to align models on tasks with long inputs. 37 LiteCoder-Terminal

20811 Samples

banksy235/XCoder-80K

5407 Samples

Lite-Coder/LiteCoder-Terminal-SFT

A quality- and diversity-selected code instruction set from XCoder (Wang et al., 2024). 29 Research-Plan-Gen

5878 Samples

Long-horizon terminal agent trajectories from LiteCoder-Terminal (Peng et al., 2026).

19389 Samples

38 When2Call

facebook/research-plan-gen

4528 Samples

nvidia/When2Call

Research-planning instruction data released alongside work on rubric-rewarded AI coscientists (Goel et al., 2025), pairing research goals with structured plans.

Irrelevant tool-calling abstention training data from When2Call (Ross et al., 2025). 39 ToolMind-Web

3392 Samples

Nanbeige/ToolMind-Web-QA 30 Mini-SWE

17715 Samples

Web-search tool-use question-answering data released with Nanbeige4.1 (Yang et al., 2026).

ricdomolm/mini-coder-trajs-400k

Execution verified mini-swe-agent (Yang et al., 2024) rollouts. 31 RAGBench

40 LongWriter zai-org/LongWriter-6k

17084 Samples

Ultra-long-form generation instructions from LongWriter (Bai et al., 2025b), targeting coherent outputs exceeding 10K words.

galileo-ai/ragbench

Examples from RAGBench (Friel et al., 2024), pairing retrieved context with grounded answers to reinforce long-context grounding. 32 Nemotron-IF

41 ToolACE

12002 Samples

Function-calling data from ToolACE (Liu et al., 2025b), emphasizing diverse API invocation.

Chat-formatted instruction-following data emphasizing precise constraint adherence.

42 APIGen-MT

Multi-turn agent–human interaction data synthesized via APIGen-MT (Prabhakar et al., 2025).

The skill_based_medium subset of the Nemotron Terminal corpus (Pi et al., 2026), comprising terminal-agent trajectories.

43 HardGen

188 Samples

Bingguang/HardGen

Hard tool-use samples generated for failuredriven learning of tool agents (Hao et al., 2026).

9425 Samples

donmaclean/LongMIT-128K

Long-context multi-hop instruction data (up to 128K tokens) constructed via the best-practice recipe of Chen et al. (2025). 35 CodeIO-PyEdu

410 Samples

Salesforce/APIGen-MT-5k

10740 Samples

nvidia/Nemotron-Terminal-Corpus

34 LongMIT

2094 Samples

Team-ACE/ToolACE

nvidia/Nemotron-Instruction-Following-Chat-v1

33 Nemotron-Terminal

2304 Samples

44 Goedel-Prover

179 Samples

Goedel-LM/SFT_dataset_v2

Formal theorem-proving data from GoedelProver-V2 (Lin et al., 2025).

9324 Samples

hkust-nlp/CodeIO-PyEdu-Reasoning

Data from CodeIO (Li et al., 2025c), which gleans reasoning patterns from single- and multi-turn code input-output prediction tasks. 25

C

Detailed Results Code-Specialised

Context-Extended-After-Post-Training

Preference/RL-Tuned

LooGLE V2 Code

Max Size Length Call Version Graph Control

LongCodeQA

RepoQA

64K

64K

128K

XS

0.6B

128K 6.11

7.73

17.91 20.76 7.11

5.07

S

meta-llama/Llama-3.2-1B-Instruct 01-ai/Yi-Coder-1.5B-Chat OctoLong-1.7B-Instruct ibm-granite/granite-3.1-2b-instruct

1B 1.5B 1.7B 2B

128K 128K 128K 128K

0.85 2.55 14.49 18.78

8.78 4.09 15.67 11.52

22.34 2.86 48.79 39.51

0.43 0.71 24.16 18.56

M

meta-llama/Llama-3.2-3B-Instruct Qwen/Qwen3-4B-Instruct-2507 OctoLong-4B-Instruct arcee-ai/AFM-4.5B

3B 4B 4B 4.5B

128K 256K 128K 64K

28.36 27.87 38.55 24.35

19.44 29.16 20.98 7.94

23.47 14.20 7.57 5.22 63.37 63.92 60.21 54.33 68.76 59.34 63.16 52.84 61.00 6.96 -

Qwen/Qwen2.5-7B-Instruct-1M aws-prototyping/MegaBeam-Mistral-7B-512k internlm/internlm2_5-7b-chat-1m OctoLong-8B-Instruct meta-llama/Llama-3.1-8B-Instruct nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct princeton-nlp/Llama-3-8B-ProLong-512k-Instruct ibm-granite/granite-3.1-8b-instruct gradientai/Llama-3-8B-Instruct-262k mistralai/Ministral-8B-Instruct-2410 zai-org/glm-4-9b-chat-1m 01-ai/Yi-Coder-9B-Chat

7B 7B 7B 8B 8B 8B 8B 8B 8B 8B 9B 9B

1024K 512K 1024K 128K 128K 1024K 512K 128K 256K 128K 1024K 128K

34.31 26.10 8.76 33.17 22.77 23.08 21.26 27.54 6.39 17.23 23.56 26.39

19.66 10.08 17.27 26.35 24.04 19.67 16.58 16.88 15.55 19.52 21.74 7.88

58.43 38.19 39.64 65.59 55.71 36.55 27.18 33.53 9.23 55.85 66.01 9.77

OctoLong-14B-Instruct Qwen/Qwen2.5-14B-Instruct-1M

14B 14B

128K 45.89 1024K 43.26

33.87 31.06

78.55 77.41 80.58 71.43 73.21 70.92 76.86 73.18

OctoLong-8B-Instruct - Cross-Repo Code

8B

28.78 128K -4.39 30.55 128K -2.62

23.98

64.11 60.23 64.24 58.91

Abl.

OctoLong-0.6B-Instruct

L

128K

XL

Model

OctoLong-8B-Instruct - Model Merging

8B

9.53 1.96 40.87 33.62

63.55 42.15 40.21 64.43 47.89 34.26 24.69 29.67 4.87 48.03 57.95 3.65

-4.20

0.88 0.95 27.38 31.21

45.28 31.11 40.05 67.82 37.56 31.67 38.57 44.16 31.62 27.55 61.56 16.74

41.62 20.77 35.68 61.44 32.19 22.48 32.65 31.88 30.11 16.09 56.73 8.86

-2.37

-1.48

25.70

64.60 63.89 66.39 61.07

-0.65

-0.99

-0.54

-3.58 -1.43

-2.53 -0.37

Table 6: Detailed per-setting scores on the coding long-context evaluation suite (RQ1). LooGLE V2 Code (He et al., 2025) is split into its call-graph and version-control subsets, while LongCodeQA (Rando et al., 2025) and RepoQA (Liu et al., 2024a) are reported at their 64K and 128K context settings. We note the strong performance of the OctoLong-Instruct suite across the retrieval- and state-tracking-heavy multi-file and repository-level coding tasks. RQ1 discusses the results in detail. AALCR

XS

OctoLong-0.6B-Instruct

0.6B

128K 14.77 26.54 22.11

27.71

12.81 5.60

5.00

S

meta-llama/Llama-3.2-1B-Instruct 01-ai/Yi-Coder-1.5B-Chat OctoLong-1.7B-Instruct ibm-granite/granite-3.1-2b-instruct

1B 1.5B 1.7B 2B

128K 128K 128K 128K

26.44 18.56 25.06 24.33

12.91 15.57 27.38 22.10

20.33 11.23 17.21 24.95

24.89 20.55 36.38 26.24

3.10 1.10 22.76 18.25

7.00 1.00 8.00 6.00

meta-llama/Llama-3.2-3B-Instruct Qwen/Qwen3-4B-Instruct-2507 OctoLong-4B-Instruct arcee-ai/AFM-4.5B

3B 4B 4B 4.5B

128K 256K 128K 64K

19.45 26.78 23.44 17.44

20.82 27.55 24.87 25.56

25.11 21.90 23.14 20.85

24.74 43.33 41.33 26.22

22.07 8.94 7.00 39.76 37.55 10.00 38.41 35.44 10.00 5.91 7.00

Qwen/Qwen2.5-7B-Instruct-1M aws-prototyping/MegaBeam-Mistral-7B-512k internlm/internlm2_5-7b-chat-1m OctoLong-8B-Instruct meta-llama/Llama-3.1-8B-Instruct nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct princeton-nlp/Llama-3-8B-ProLong-512k-Instruct ibm-granite/granite-3.1-8b-instruct gradientai/Llama-3-8B-Instruct-262k mistralai/Ministral-8B-Instruct-2410 zai-org/glm-4-9b-chat-1m 01-ai/Yi-Coder-9B-Chat

7B 7B 7B 8B 8B 8B 8B 8B 8B 8B 9B 9B

1024K 512K 1024K 128K 128K 1024K 512K 128K 256K 128K 1024K 128K

33.24 20.80 19.25 28.44 32.57 28.44 29.43 28.34 27.31 31.33 26.00 19.44

29.06 25.11 25.46 36.12 25.72 20.57 26.75 29.55 32.59 26.48 25.19 21.52

28.44 21.89 15.07 34.89 32.60 26.45 26.77 16.10 20.26 22.74 11.37 18.26

22.12 26.43 41.41 57.22 21.67 21.33 15.97 15.88 53.23 21.67 51.54 26.22

36.42 0.00 30.46 42.90 35.67 4.90 43.33 29.32 41.78 21.67 4.22 2.55

OctoLong-14B-Instruct Qwen/Qwen2.5-14B-Instruct-1M

14B 14B

128K 37.23 35.12 30.45 1024K 29.86 30.77 33.10

62.65 31.67

46.91 42.67 21.00 37.36 34.89 23.00

OctoLong-8B-Instruct - Cross-Repo Code

8B

128K

27.76 35.68 31.77

52.89

39.44 34.31 11.00

Abl.

Model

M

Preference/RL-Tuned

LongBench V2 (Medium) OpenAI MRCR Max Size Length Single Multi Long Long 32K– 64K– Doc Doc ICL Dialogue 64K 128K

L

Context-Extended-After-Post-Training

XL

Code-Specialised

OctoLong-8B-Instruct - Model Merging

8B

128K

-0.68

-0.44

1.90 0.00 16.22 8.89

34.11 0.00 24.54 38.91 25.49 3.44 40.04 27.41 23.56 4.55 0.00 1.22

-3.12

-4.33

-3.46

28.52 35.97 34.06

55.88

41.58 36.82 12.00

+0.08

-1.34

-1.32

-0.15

-0.83

-4.60

13.00 6.00 6.00 13.00 13.00 10.00 6.00 7.00 2.00 4.00 12.00 5.00

-2.09

-2.00 -1.00

Table 7: Detailed per-setting scores on the general-domain long-context evaluation suite (RQ2). LongBench V2 Medium (Bai et al., 2025a) is split into its single-document, multi-document, long in-context learning, and longdialogue subsets; OpenAI MRCR (OpenAI, 2025) is reported at its 32K–64K and 64K–128K depth bins; AA-LCR (Team, 2025a) denotes the Artificial Analysis long-context reasoning score. The consistently strong performance of the OctoLong-Instruct suite across a range of realistic and industry-standard long-context benchmarks is noteworthy. A detailed discussion of results resides in RQ2.

26

Code-Specialised

Context-Extended-After-Post-Training Max Size Length NonLive

𝜏 2 -Bench

XS

0.6B

128K 63.68 53.45

4.23

32.11 7.64

7.91

S

meta-llama/Llama-3.2-1B-Instruct 01-ai/Yi-Coder-1.5B-Chat OctoLong-1.7B-Instruct ibm-granite/granite-3.1-2b-instruct

1B 1.5B 1.7B 2B

128K 128K 128K 128K

36.76 25.21 74.54 67.87

22.45 19.47 63.22 55.32

0.00 0.00 6.79 2.14

30.25 43.52 26.89 38.61

2.79 4.67 11.44 6.44

7.86 3.81 12.75 8.77

meta-llama/Llama-3.2-3B-Instruct Qwen/Qwen3-4B-Instruct-2507 OctoLong-4B-Instruct arcee-ai/AFM-4.5B

3B 4B 4B 4.5B

128K 256K 128K 64K

38.54 81.34 81.56 72.32

29.68 4.56 28.45 8.33 70.79 21.64 34.98 42.88 68.87 24.67 33.86 25.87 54.12 3.43 43.36 7.88

6.13 15.67 15.73 9.21

Qwen/Qwen2.5-7B-Instruct-1M aws-prototyping/MegaBeam-Mistral-7B-512k internlm/internlm2_5-7b-chat-1m OctoLong-8B-Instruct meta-llama/Llama-3.1-8B-Instruct nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct princeton-nlp/Llama-3-8B-ProLong-512k-Instruct ibm-granite/granite-3.1-8b-instruct gradientai/Llama-3-8B-Instruct-262k mistralai/Ministral-8B-Instruct-2410 zai-org/glm-4-9b-chat-1m 01-ai/Yi-Coder-9B-Chat

7B 7B 7B 8B 8B 8B 8B 8B 8B 8B 9B 9B

1024K 512K 1024K 128K 128K 1024K 512K 128K 256K 128K 1024K 128K

71.21 35.41 42.65 82.45 42.78 41.23 60.11 80.04 25.11 14.34 54.87 70.63

66.44 27.62 40.76 72.98 40.71 38.27 53.46 59.78 31.58 16.68 43.87 68.78

16.95 0.00 11.52 16.58 8.44 4.93 11.68 10.11 9.23 4.22 11.34 10.43

OctoLong-14B-Instruct Qwen/Qwen2.5-14B-Instruct-1M

14B 14B

128K 84.39 75.67 41.87 45.43 43.87 13.95 1024K 72.13 69.21 25.43 27.43 30.13 20.51

OctoLong-8B-Instruct - Cross-Repo Code

8B

OctoLong-8B-Instruct - Model Merging

8B

Abl.

OctoLong-0.6B-Instruct

M

Re- Teletail com

L

Live Multi- AirTurn line

XL

Model

Preference/RL-Tuned BFCL V3

79.43 128K -3.02 83.08 128K +0.63

15.98 0.84 1.54 29.53 10.34 1.68 3.44 7.89 3.08 0.00 2.62 10.52

29.45 0.00 48.32 30.61 33.12 40.41 41.21 46.58 41.11 20.43 46.55 45.66

22.77 0.00 7.44 34.85 9.34 5.36 10.35 6.38 7.02 2.29 6.91 6.11

68.11 25.69 27.45 25.46 14.24 -4.87

-3.84

-3.16

-9.39

-2.34

72.76 28.22 29.85 35.08 15.79 -0.22

-1.31

-0.76

+0.23

-0.79

Table 8: Detailed per-setting scores on the agentic tool-use evaluation suite (RQ3). BFCL V3 (Patil et al., 2025) is split into its non-live, live, and multi-turn subsets, while 𝜏 2 -Bench (Barrès et al., 2025) reports its airline, retail, and telecom domains. Despite comparing against several preference-and RL-tuned baselines, we observe consistently strong performance from the OctoLong-Instruct suite on short- and longer-horizon generative tool-calling tasks. RQ3 expounds on the results in detail. Code-Specialised

Context-Extended-After-Post-Training

Preference/RL-Tuned

Pass @5

Pass @1

Pass @5

Pass @10

XS

OctoLong-0.6B-Instruct

0.6B

128K 4.55

9.24 11.26 13.86 1.60

5.10

7.92 14.80

S

meta-llama/Llama-3.2-1B-Instruct 01-ai/Yi-Coder-1.5B-Chat OctoLong-1.7B-Instruct ibm-granite/granite-3.1-2b-instruct

1B 1.5B 1.7B 2B

128K 128K 128K 128K

2.45 8.91 10.79 9.57

6.35 15.61 18.34 14.36

7.87 17.34 20.94 17.43

10.23 20.87 24.53 19.78

0.80 1.83 3.21 6.24 5.11 15.99 20.88 27.45 4.90 16.83 23.61 32.34 6.21 13.24 18.06 23.96

M

meta-llama/Llama-3.2-3B-Instruct Qwen/Qwen3-4B-Instruct-2507 OctoLong-4B-Instruct arcee-ai/AFM-4.5B

3B 4B 4B 4.5B

128K 256K 128K 64K

8.33 17.86 17.44 5.59

13.78 29.44 27.91 8.34

16.96 33.56 31.34 10.23

18.77 36.76 34.91 11.65

5.02 19.87 16.31 2.14

14.17 28.12 31.70 6.35

20.09 30.76 39.59 9.98

27.62 33.46 44.82 15.43

L

BigCodeBench Ins. Hard

Max Size Length Pass @1

Qwen/Qwen2.5-7B-Instruct-1M aws-prototyping/MegaBeam-Mistral-7B-512k internlm/internlm2_5-7b-chat-1m OctoLong-8B-Instruct meta-llama/Llama-3.1-8B-Instruct nvidia/Llama-3.1-Nemotron-8B-UltraLong-1M-Instruct princeton-nlp/Llama-3-8B-ProLong-512k-Instruct ibm-granite/granite-3.1-8b-instruct gradientai/Llama-3-8B-Instruct-262k mistralai/Ministral-8B-Instruct-2410 zai-org/glm-4-9b-chat-1m 01-ai/Yi-Coder-9B-Chat

7B 7B 7B 8B 8B 8B 8B 8B 8B 8B 9B 9B

1024K 512K 1024K 128K 128K 1024K 512K 128K 256K 128K 1024K 128K

16.45 2.77 2.82 20.82 11.83 14.34 5.12 12.87 3.42 13.23 13.62 17.45

23.19 5.84 4.95 28.78 19.41 21.76 9.45 17.95 7.54 19.56 19.54 24.66

25.34 6.87 6.54 33.86 21.96 24.35 11.49 21.45 9.67 22.54 22.04 29.45

28.74 9.41 10.23 37.24 25.56 26.75 14.29 25.44 13.23 26.78 25.47 33.43

13.11 0.52 4.23 19.44 11.21 10.35 7.21 13.12 3.68 14.21 11.33 13.04

28.23 0.96 14.34 37.16 27.43 24.32 18.34 26.45 12.76 27.46 27.22 26.94

33.32 1.56 18.83 42.84 33.09 30.97 23.32 32.87 18.65 32.89 33.01 33.81

38.87 3.12 24.97 49.23 39.87 37.66 28.76 38.45 26.45 37.71 39.54 40.44

XL

LiveCodeBench V6

OctoLong-14B-Instruct Qwen/Qwen2.5-14B-Instruct-1M

14B 14B

128K 22.43 32.77 35.86 41.77 21.52 38.43 44.19 51.26 1024K 20.89 29.23 33.42 35.68 19.61 33.67 37.65 43.54

OctoLong-8B-Instruct - Cross-Repo Code

8B

OctoLong-8B-Instruct - Model Merging

8B

OctoLong-8B-Instruct - Context Extension

8B

Qwen3-8B

8B

Abl.

Model

Pass @10

Pass @25

Pass @25

19.67 128K -1.15 17.75 128K -3.07

26.57 32.24 35.91 17.56 35.04 39.45 43.56

20.49 32K -0.33 23.18 32K +2.36

28.91 33.44 36.95 19.03 36.55 41.24 47.98

-2.21

-1.62

-1.33

-1.88

-2.12

-3.39

-5.67

24.56 31.45 34.59 18.04 36.35 40.68 46.05 -4.22 +0.13

-2.41 -0.42

-2.65 -0.29

-1.40 -0.41

-0.81 -0.61

-2.16 -1.60

-3.18 -1.25

29.31 33.06 34.91 20.73 34.46 37.55 40.11 +0.53

-0.80

-2.33

+1.29

-2.70

-5.29

-9.12

Table 9: Detailed per-setting scores on the short-context code-generation evaluation suite (RQ4). LiveCodeBench V6 (Jain et al., 2025) and BigCodeBench Instruct Hard (Zhuo et al., 2025) are reported at their Pass@{1,5,10,25} sampling budgets. We observe no regressions of the OctoLong-Instruct suite on short-context coding tasks along with important gains on API usage, particularly at higher sampling budgets. RQ4 discusses the results in detail.

27

Related documents

• MRCoder: An Efficient Context Selecting Approach for Repository-Level Code Generation#411106
• RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models#411120
• CoCoMUT: A Tool for Code-Context Mining and Automated Dataset Generation#324954
• Effective and Efficient Context Retrieval via Partial Dependency Graph for Repository-Level Code Generation#424041
• Probe-and-Refine Tuning of Repository Guidance for Coding Agents#290565