1
TeleResilienceBench: Quantifying Resilience for LLM Reasoning in Telecommunications
arXiv:2605.09929v1 [cs.LG] 11 May 2026
Pranshav Gajjar1 , Emmanuel Ojo1 , and Vijay K Shah1 1 NextG Wireless Lab, North Carolina State University
Abstract—Deploying large language models in telecommunications requires more than task accuracy. In realistic workflows, a model may inherit partially completed reasoning from a prior step, an upstream agent, or its own earlier generation, and must continue that reasoning even when it is already going wrong. We introduce TeleResilienceBench, a benchmark that quantifies this capability, which we term reasoning resilience, across seven telecom sub-domains drawn from the GSMA OpenTelco LLM suite. Instances are constructed by collecting failures from a weak generator model, truncating the flawed reasoning trace at its midpoint, and asking a target model to continue and correct it. We propose the Correct Flip Rate (CFR) as a direct measure of successful recovery and evaluate eight models spanning the Qwen3.5, Gemma4, and Nemotron-3 families. Our results show that even the strongest model achieves a macroaverage CFR of only 29.1%, and scale does not reliably improve resilience within families. Nemotron-3-nano 4b outperforms all Qwen3.5 variants including the 27b model and leads the auxiliary TeleMath numerical evaluation at 23.4% CR%, offering the best resilience-to-cost ratio in the set. A difficulty-stratified analysis further reveals that existing telecom benchmark difficulty labels reflect factual specificity rather than reasoning depth, suggesting that current evaluations measure knowledge coverage more than reasoning ability.
I. I NTRODUCTION Large language models (LLMs) are rapidly shifting from general-purpose assistants to domain-specific components for technical decision support. In telecommunications, this transition is being actively driven by the AI-RAN paradigm, championed by industry leaders including NVIDIA and Nokia, which envisions LLMs as native operational components embedded directly within radio access network infrastructure [1, 2]. Rather than sitting at the periphery as query interfaces, models in these settings are expected to parse 3GPP specifications, generate network configuration intents, diagnose faults from telemetry logs, and support real-time network management decisions [3, 4]. Broader benchmarking efforts across telecom knowledge assessment, table interpretation, root-cause analysis, software-stack reasoning, and AI-native 6G tasks reflect how seriously the community is taking this shift [5, 6, 7, 8, 9, 10]. However, progress on task accuracy alone does not resolve a more basic question of deployment readiness: how reliable is a model once its ongoing reasoning has already gone wrong? This question matters because modern LLM performance is often improved by eliciting explicit intermediate reasoning. Chain-of-thought prompting can substantially improve performance on difficult reasoning tasks by encouraging models to produce step-by-step rationales before answering [11]. Yet
subsequent work has shown that these rationales are not necessarily faithful to the model’s actual decision process. Models can generate plausible explanations for incorrect answers, and explicit reasoning may therefore increase apparent transparency without guaranteeing correctness [12]. More broadly, intrinsic self-correction remains unreliable in reasoning settings: although iterative refinement methods can improve output quality under suitable feedback and prompting regimes, prior evidence shows that LLMs often fail to repair their own reasoning when external guidance is weak or absent [13, 14, 15]. In short, the existence of a reasoning trace does not imply that the model can recognize and recover from its own mistakes. The telecommunications domain makes this limitation particularly consequential. Telecom tasks rarely reduce to surfacelevel recall. Instead, they require combining specialized domain knowledge with procedural reasoning over standards, tables, logs, codebases, and troubleshooting flows. Existing telecom benchmarks have been valuable for measuring clean-input performance across these dimensions [5, 6, 7, 8, 10]. However, most of these evaluations begin from a clean state. That is, the model is given a question or scenario and is judged by the final answer, resolution, or action sequence it produces. Such protocols reveal whether a model can solve a telecom problem from scratch, but they do not isolate a complementary capability that is critical in realistic workflows, namely, whether the model can recover after inheriting an already flawed line of reasoning. We refer to this capability as reasoning resilience: the ability of an LLM to detect that an intermediate reasoning trajectory is defective, revise it, and still arrive at the correct conclusion. Reasoning resilience is distinct from standard answer accuracy. A model may answer correctly when solving a problem from scratch yet remain fragile when conditioned on an erroneous partial derivation. Conversely, a model that can reliably repair flawed reasoning is more suitable for multi-stage settings in which it must interpret prior analyst notes, partially completed traces, or its own earlier generations. Measuring this property requires an evaluation protocol that does not merely test whether a model can answer telecom questions, but whether it can course-correct under realistic domain-specific error conditions. In this paper, we introduce TeleResilienceBench1 , a benchmark designed to quantify reasoning resilience for telecom LLMs. Our core idea is simple: we first use a small generator 1 Dataset: https://huggingface.co/datasets/prnshv/TeleResilienceBench Code: https://github.com/prnshv/TeleResilienceBench/tree/main
2
model to produce reasoning traces on telecom tasks, retain the the value of generating or aggregating step-by-step rationales instances in which that generator reaches an incorrect answer, for stronger final-answer accuracy [11, 17, 18]. At the same and then truncate the flawed trace partway through. A target time, such rationales are not necessarily faithful as models can model is subsequently prompted with the original problem, its produce plausible but incorrect reasoning traces, motivating candidate answers, and the interrupted erroneous rationale, and evaluation beyond final-answer accuracy alone [12]. This must complete the reasoning to produce the correct final answer. has led to growing interest in process-level evaluation and This setup turns resilience into a measurable property. Rather supervision. REVEAL studies verification at the level of than rewarding only fresh problem-solving, it evaluates whether individual reasoning steps, while Let’s Verify Step by Step a model can resist being anchored to a domain-grounded and Math-Shepherd show that supervising intermediate steps but incorrect reasoning path. Because the corrupted traces can improve reasoning reliability [19, 20, 21]. arise from real telecom questions rather than synthetic logic Self-correction under flawed reasoning. A related line of puzzles, the benchmark captures failure modes tied to the work asks whether models can repair their own mistakes after vocabulary, ambiguity, and procedural structure of technical a flawed reasoning path has already begun. Iterative refinement telecom reasoning. The main contributions of this work are: frameworks such as Self-Refine and Reflexion show that models • We introduce TeleResilienceBench, a new bench- can improve outputs when given critique or feedback [14, 15], mark for evaluating reasoning resilience across seven yet intrinsic self-correction remains unreliable when strong discrete-choice telecom sub-domains spanning standards- external guidance is absent [13]. Most closely related to our centric knowledge, troubleshooting, and software-oriented setting, recent work shows that early errors in a reasoning tasks, complemented by an auxiliary open-ended mathe- chain can strongly anchor final predictions, highlighting the importance of evaluating recovery from corrupted intermediate matical reasoning subset evaluated separately. reasoning rather than only final-answer accuracy [22]. • We benchmark multiple open-weight model families across a controlled range of small and medium scales, enabling Our benchmark sits at the intersection of these literatures. a comparative analysis of how resilience varies across Prior telecom benchmarks measure whether a model can architectures and parameter budgets. solve domain tasks from scratch, while prior self-correction • We show that successful correction is not reducible to studies are largely developed on general-domain reasoning simply producing longer outputs or using more memory, tasks. TeleResilienceBench instead evaluates whether a and that existing telecom benchmark difficulty labels telecom model can course-correct after inheriting a domainappear to reflect factual specificity more than reasoning grounded but incorrect partial trace. In this sense, our focus depth. is not standard telecom QA, nor generic reasoning robustness Together, these contributions position in isolation, but reasoning resilience under realistic telecomTeleResilienceBench to evaluate a crucial capability specific error conditions. missing from standard telecom benchmarks: the ability to recover from a compromised reasoning state, rather than simply generating answers from a clean starting point. III. T ELE R ESILIENCE B ENCH II. R ELATED W ORK LLMs for telecommunications. Recent work has established telecommunications as a distinct evaluation domain for LLMs. TeleQnA introduced large-scale telecom question answering over standards and research documents, showing that generalpurpose models remain brittle on technical telecom knowledge [5]. Subsequent efforts expanded this direction along more specialized axes. TSpec-LLM focused on 3GPP document understanding and retrieval augmentation [16]; ORAN-Bench targeted Open RAN specifications [6]; TeleTables isolated reasoning over tabular content in 3GPP standards [7]; and TeleLogs studied root-cause analysis and structured troubleshooting in 5G settings [8]. Collectively, these benchmarks show that telecom tasks require more than generic language competence. However, they evaluate models from a clean initial context and therefore measure task accuracy, not whether a model can recover once the reasoning trajectory is already flawed. Chain-of-thought reasoning and process-level reliability. Beyond telecom, prior work shows that explicit intermediate reasoning can improve performance on difficult tasks. Chainof-thought prompting, self-consistency, and STaR demonstrate
We construct TeleResilienceBench from seven classbased subsets of the GSMA/ot-lite suite: TeleQnA, TeleTables, TeleLogs, 3GPP_TSG, ORANBench, srsRANBench, and SixG_Bench. These subsets collectively cover standardsoriented question answering, table interpretation, root-cause reasoning, standards classification, O-RAN knowledge, softwarestack understanding, and AI-native 6G analysis [5, 6, 7, 8, 9]. A high-level overview for the same is available in Figure 1, with more details regarding the evaluation criteria in Section IV-A. To generate resilience instances, we use qwen3.5:2b as a weak base generator. For each input problem, the generator produces a reasoning trace and a final answer. We retain only cases in which the final answer is parseable but incorrect. This filtering step is essential as it transforms an ordinary telecom benchmark example into an error-recovery instance by ensuring that the continuation model is exposed to a realistic but flawed reasoning trajectory rather than a clean prompt. For each retained example, we remove the generator’s final answer and truncate the remaining reasoning trace at the token-level midpoint using the cl100k_base tokeniser, snapping the cut point backward to the nearest sentence boundary to preserve
3
Fig. 1. Overview of TeleResilienceBench. A target model then receives the original question, answer options, and inherited flawed trace, and is evaluated on whether it recovers the correct answer (CFR), repeats the original wrong answer (NFR), or flips to another wrong answer (WFR), along with efficiency measures such as output tokens and peak VRAM.
syntactic integrity. Let ri denote the full reasoning trace for instance i. The benchmark stores the truncated prefix r̃i = HalfTrace(ri ),
(1)
where HalfTrace(·) returns approximately the first half of the trace while preserving sentence integrity. The truncation is chosen to preserve enough of the flawed rationale to bias the continuation model, while withholding the later portion of the derivation so that recovery remains nontrivial. The resulting benchmark therefore evaluates whether a model can detect that the inherited reasoning is misaligned and revise course before committing to a final answer. A. Benchmark Structure and Prompt Formulation
7.7% 9.3% 11.9%
Total
n = 818 43.9%
11.0% 10.5%
TeleQnA TeleLogs 3GPP_TSG SixG_Bench TeleTables ORANBench srsRANBench
Fig. 2. Sub-benchmark distribution
At evaluation time, the target model receives three elements: the original telecom question, the candidate answer options, and the truncated flawed reasoning trace produced by the base model. The tuple formally defines each benchmark instance (q, O, y ∗ , y orig , r̃),
5.7%
(2)
where q is the question text, O is the ordered list of candidate options, y ∗ is the ground-truth answer, y orig is the incorrect answer produced by the base generator, and r̃ is the flawed half-trace. The target model is asked to continue the reasoning process and return a single final option. This continuation-based formulation isolates resilience from standard task accuracy: the model is not solving from a clean state, but reasoning under a corrupted intermediate context. The exact continuation prompt template used for the benchmark is provided in Appendix A.
The benchmark is intentionally asymmetric. The target model does not observe y orig explicitly; it only sees the partial rationale r̃ that led toward it. As a result, success requires more than simply disagreeing with a prior prediction. The model must infer from the inherited trace itself that the reasoning path is defective and must be corrected. B. Benchmark Statistics The resulting benchmark contains 818 discrete-choice instances spanning all seven subsets. The distribution is intentionally heterogeneous, with TeleQnA contributing the largest share and the remaining subsets providing complementary coverage over tabular reasoning, troubleshooting, standards-
4
centric classification, O-RAN-specific knowledge, softwarestack reasoning, and 6G-oriented telecom analysis. Standardsrecall errors differ structurally from table-reasoning and rootcause diagnosis errors; treating them as a single evaluation surface would mask meaningful variation in how different architectures fail and recover. Where available, source-level difficulty annotations are preserved, enabling downstream analysis of how recovery behavior varies with problem difficulty. In this way, TeleResilienceBench supports not only aggregate comparison across models, but also finer-grained analysis of which telecom sub-domains are most vulnerable to error propagation. Furthermore, samples for every sub-benchmark are available in Appendix B and C. IV. E XPERIMENTAL S ETUP Both the benchmark construction and evaluation were performed with an Intel Core i9 14900KF CPU and 62 GiB of system RAM. GPU acceleration is provided by an NVIDIA GeForce RTX 4090 GPU featuring 24 GB of GDDR6X memory. To have a robust evaluation, we use every single model natively supported by Ollama [23] that can be accommodated within our compute setup and has been released recently with state-ofthe-art performance on general-purpose reasoning benchmarks. Selected models encompass the Nemotron3-nano [24] series (4b), the Qwen3.5 [25] series (4b, 9b, and 27b), and the Gemma4 [26] series (e2b, e4b, 26b, and 31b). We subject all the models to the prompt that is explained in Appendix A. A. Metrics To quantitatively assess the resilience of each model when confronted with flawed reasoning traces, we define three primary metrics based on the final answer selected by the target model. First is the Correct Flip Rate (CFR), formally defined as P|Df lawed | 1 CF R = |Df lawed ⊮ (M (qi ⊕ oi ⊕ ti ) = yi∗ )×100 i=1 | where Df lawed represents the set of benchmark instances where the base generator initially failed, ⊮(·) denotes the indicator function for a successful correction, M is the target model, qi ⊕ oi ⊕ ti represents the concatenated sequence of the question, options, and flawed half trace, and yi∗ is the ground truth correct answer. Conversely, the No Flip Rate (NFR) quantifies the frequency with which the model continues the flawed logic presented in the prompt and ultimately selects the original incorrect answer chosen by the generator. Let yiorig denote this initial flawed selection. The NFR is calculatedas: P|Df lawed | 1 N F R = |Df lawed ⊮ M (qi ⊕ oi ⊕ ti ) = yiorig × i=1 | 100 Furthermore, the Wrong Flip Rate (WFR) captures instances where the model successfully recognizes an error and deviates from the generator’s path, yet ultimately selects a different incorrect option rather than the ground truth. This is mathematically formulated as: W FR = P|Df lawed | 1 ⊮ M (qi ⊕ oi ⊕ ti ) ∈ / {yi∗ , yiorig } ×100 i=1 |Df lawed | In addition to accuracy-based metrics, we evaluate computational efficiency to understand the practical deployment costs. Average Token Consumption measures the verbosity and output tokens, and VRAM usage tracks the peak GPU
memory consumption during inference. This metric is critical for assessing the feasibility of deploying these models in edge telecommunications environments where hardware resources are heavily constrained. V. R ESULTS AND A NALYSIS We evaluate reasoning resilience across the seven discretechoice subsets and then examine the auxiliary TeleMath results and difficulty-stratified analyses. A. Reasoning Resilience (Flip Rates) Table I reports CFR, NFR, and WFR for all eight evaluated models across the seven discrete-choice sub-benchmarks. The overarching picture is one of low resilience. Even the strongest model, Gemma4-31b, achieves a macro-average CFR of only 29.1%, meaning that across roughly seven out of ten instances where the base generator failed, the continuation model either perpetuates the flawed conclusion or recognises an error but selects a different incorrect option. The WFR is the dominant outcome for the large majority of model and subset combinations. It basically shows that models are generally sensitive to the presence of an erroneous reasoning thread and do not simply continue it passively. Yet, they are frequently unable to identify the correct answer once they depart from the inherited trajectory. The core difficulty captured by TeleResilienceBench is therefore not that models are blind to errors in a flawed trace; it is that detecting the flaw is considerably easier than recovering the correct conclusion from that same trace. Within the Qwen3.5 family, scaling from 4b to 27b yields essentially no improvement in macro-average CFR: 16.9% → 17.4% → 17.6%, a range of less than one percentage point across a 6.75× increase in parameter count. This near-flat profile holds across most sub-benchmarks. On 3GPP_TSG specifically, the 27b model (4.4%) actually underperforms the 9b model (5.6%), indicating that additional scale can, in some cases make a model marginally more susceptible to the flawed prior rather than more capable of overriding it. The Gemma4 family presents a different picture. Scaling from e2b to e4b produces a small decrease in macro CFR (24.5% → 23.7%), while the transition to 26b and then 31b yields a steady improvement (27.2% and 29.1% respectively). This non-monotonic trajectory suggests that intermediate-scale models in this family have not fully consolidated the reasoning strategies required for error recovery, whereas the larger variants develop a more consistent ability to detect and redirect defective derivations. One of the sharper findings concerns Nemotron-3-nano 4b. It achieves a macro-average CFR of 27.5%, outperforming every Qwen3.5 variant, including the 27b model by more than ten percentage points, and sitting essentially level with Gemma4-26b (27.2%) at a fraction of the parameter count. A 4b model from one family substantially outperforming a 27b model from another on this capability is not something standard accuracy benchmarks would have predicted. Nemotron3-nano 4b is particularly strong on ORANBench (39.7%) and srsRANBench (46.8%), where it leads all tested models.
5
TABLE I R EASONING R ESILIENCE ACROSS 7 DISCRETE - CHOICE GSMA SUB - BENCHMARKS . W E REPORT THE C ORRECT F LIP R ATE (CFR), N O F LIP (NF), AND W RONG F LIP (WF) PERCENTAGES FOR EACH MODEL . M ACRO AVERAGE IS THE UNWEIGHTED MEAN OF THE SEVEN SUBSET PERCENTAGES ( EACH SUBSET WEIGHTED EQUALLY, NOT BY SAMPLE COUNT ).
Family
TeleQnA
Size
TeleTables
TeleLogs
3GPP_TSG
CFR
NFR
WFR
CFR
NFR
WFR
CFR
NFR
WFR
CFR
NFR
WFR
Qwen3.5
4b 9b 27b
23.7 21.2 23.4
40.1 37.6 36.5
35.7 39.0 39.0
13.2 9.2 14.5
28.9 39.5 34.2
55.3 47.4 48.7
4.7 14.0 17.4
22.1 25.6 17.4
73.3 60.5 65.1
3.3 5.6 4.4
28.9 16.7 16.7
67.8 76.7 78.9
Gemma4
e2b e4b 26b 31b
32.6 26.7 30.1 32.9
40.1 33.1 37.9 40.4
27.3 38.2 30.9 26.5
15.8 21.1 18.4 18.4
21.1 23.7 34.2 27.6
52.6 52.6 46.1 52.6
10.5 14.0 15.1 19.8
9.3 19.8 14.0 10.5
51.2 64.0 69.8 69.8
5.6 5.6 16.7 16.7
6.7 18.9 11.1 10.0
74.4 70.0 60.0 70.0
Nemotron-3
4b
34.0
37.6
27.9
26.3
26.3
47.4
5.8
15.1
75.6
11.1
15.6
67.8
Family
Size
ORANBench
srsRANBench
Macro Average
SixG_Bench
CFR
NFR
WFR
CFR
NFR
WFR
CFR
NFR
WFR
CFR
NFR
WFR
Qwen3.5
4b 9b 27b
28.6 23.8 23.8
39.7 31.7 27.0
30.2 41.3 49.2
17.0 23.4 19.1
34.0 36.2 44.7
46.8 38.3 34.0
27.8 24.7 20.6
25.8 28.9 33.0
46.4 46.4 45.4
16.9 17.4 17.6
31.4 30.9 29.9
50.8 49.9 51.5
Gemma4
e2b e4b 26b 31b
36.5 38.1 33.3 31.7
31.7 28.6 33.3 33.3
31.7 33.3 33.3 33.3
36.2 25.5 44.7 44.7
38.3 25.5 27.7 36.2
25.5 42.6 25.5 19.1
34.0 35.1 32.0 39.2
27.8 20.6 27.8 21.6
36.1 43.3 38.1 39.2
24.5 23.7 27.2 29.1
25.0 24.3 26.6 25.7
42.7 49.1 43.4 44.4
Nemotron-3
4b
39.7
41.3
19.0
46.8
31.9
21.3
28.9
24.7
45.4
27.5
27.5
43.5
The gap between sub-benchmarks is also considerable. On srsRANBench the best model achieves 46.8% CFR, while on 3GPP_TSG the best model reaches only 16.7%. TeleLogs is the one sub-benchmark where Qwen3.5 scaling matters at all: CFR rises from 4.7% at 4b to 14.0% at 9b and 17.4% at 27b, a meaningful improvement, though still modest in absolute terms. On TeleTables, Nemotron-3-nano 4b (26.3%) leads the next best model, Gemma4-31b at 18.4%, by nearly eight percentage points. Representative qualitative examples of successful recoveries across sub-benchmarks are provided in Appendix B.
TeleQnA SixG Bench
40
28.1% 30.3%
30 20
10
srsRAN Bench
32.2%
TeleTables
0
8.6%
17.1% 12.6%
Tele Logs
B. Sub-Benchmark Difficulty Map Reasoning resilience is not uniform across telecom sub31.9% domains. The seven sub-benchmarks separate into two tiers, as shown in Figure 3. plots the mean Correct Flip Rate (CFR) for ORANBench 3GPP TSG each sub-benchmark, averaged across all eight evaluated models. The results show substantial variation across telecom domains. The highest mean CFRs are observed on srsRANBench (32.2%), Fig. 3. Correct Flip Rate (CFR) comparison across different benchmarks. ORANBench (31.9%), and SixG_Bench (30.3%), followed by TeleQnA at 28.1%. In contrast, TeleTables (17.1%), TeleLogs (12.6%), and especially 3GPP_TSG (8.6%) form a clearly mentally a recall problem grounded in organisational structure lower-resilience group. rather than a problem amenable to step-by-step derivation. The worst-performing sub-benchmark, 3GPP_TSG, is also When the generator assigns the excerpt to the wrong working the most structurally distinct from a reasoning perspective. The group, no amount of continued reasoning can recover the task requires identifying which 3GPP Technical Specification correct answer unless the continuation model independently Group produced a given document excerpt, which is funda- recalls it from training data. The low and relatively uniform
6
Qwen 3.5
Gemma 4
30
CFR (%)
25
e2b
20
CR%
31B 26B e4b
4B
Qwen 3.5
Nemotron-3
27B 9B 4B 200 250 300 350 400 450 500
Mean Output Tokens
25 20 15 10 5 0
Gemma 4
Nemotron-3
4B 9B 27B e2b e4b 26B 31B 4B
Fig. 4. CFR vs. Mean Output Tokens for various models.
Qwen 3.5
CFR (%)
30
Nemotron-3
31B
4B e2b
25 20
Gemma 4
26B 27B
4B 9B 20
e4b
40
Fig. 6. Correction Rate (CR%) for Qwen 3.5, Gemma 4, and Nemotron-3 models on TeleMath.
60
VRAM usage (%)
80
improving CFR, with the 31b variant at the top on both dimensions. For Qwen3.5, the family spans a wide range of VRAM but remains confined to a narrow low-CFR band throughout, showing that memory footprint is not a reliable proxy for resilience across architectures. The standout case is Nemotron-3-nano 4b, which achieves 27.5% macro-average CFR at substantially lower VRAM than Gemma4-26b and Gemma4-31b, the two models it most closely matches in resilience. For edge-telecom deployments where GPU memory is a hard constraint, it represents the most favourable resilienceto-cost trade-off in this evaluation.
Fig. 5. CFR vs. VRAM usage for various models.
D. TeleMath: Continuous Evaluation CFR across all models on this subset, ranging from 3.3% to 16.7%, is consistent with this interpretation. The three higher-resilience sub-benchmarks share a more compositional structure, drawing on interface specifications, protocol relationships, and network-level design principles that a continuation model can approach through inference even when the inherited trace contains an incorrect intermediate claim. TeleLogs sits in the lower tier despite involving causal reasoning over structured diagnostic data, because errors introduced early in the generator’s troubleshooting chain tend to determine the final diagnosis rather than be identified and abandoned mid-trace. C. Computational Efficiency Resilience does not come from simply generating more output. Across all model families, there is no consistent positive relationship between token count and CFR (Figure 4). The Qwen3.5 27b model generates substantially more tokens than the 4b model yet their macro-average CFRs are within one percentage point of each other. The Gemma4-31b achieves the highest CFR in the set, but the Gemma4-e2b reaches 24.5% at markedly fewer tokens, while Qwen3.5 models at comparable output lengths plateau near 17%. Verbosity is neither necessary nor sufficient for successful error recovery. The VRAM plots gives a more defined description within families (Figure 5). For Gemma4, increasing model size tracks
TeleMath is evaluated separately from the main flip-rate analysis because its questions require open-ended numerical answers rather than a selection from a fixed option set. A Wrong Flip has no consistent meaning in this setting, so performance is reported as a binary Correction Rate (CR%), measuring the fraction of the 77 TeleMath instances where the continuation model recovers the correct numerical answer within a tolerance of 10−3 relative error or 10−5 absolute error. Figure 6 shows that CR% values are substantially lower than the discrete-choice CFRs across all models, reflecting the harder requirement of committing to an exact numerical value rather than selecting from a small option set. The Qwen3.5 family clusters between 4% and 5.3% with no meaningful improvement across scale, the 27b model matching the 4b at 4%. The Gemma4 family is non-monotonic: e2b reaches 11.7%, e4b drops to 7.9%, and the family recovers to 10.5% at 26b and 19.7% at 31b. Nemotron-3-nano 4b leads the entire set at 23.4%, outperforming Gemma4-31b by nearly four percentage points despite being the smallest model evaluated, a result that is consistent with its strong performance on the discrete-choice sub-benchmarks. E. Correlation Between Sample Difficulty and Resilience We examine whether the source-level difficulty of a question predicts how easily a continuation model can recover from a flawed trace on that question. This analysis is restricted to
7
ORANBench
30
15
20
10
Benchmarks where harder instances require longer inference chains with more opportunities for compounding errors are what the community needs to distinguish these contributions. The difficulty-resilience result here suggests that even within TeleResilienceBench, the inherited difficulty tiers do not yet capture reasoning complexity in the intended sense. Building difficulty measures grounded in inference depth rather than factual specificity remains an open problem worth addressing directly.
10
5
VI. L IMITATIONS
[17-48]
50 40
CFR (%)
TeleTables
0
[16-47]
28.9
35.9 25
[24-38]
30.4
[10-28]
18.3
[5-24]
15.9
20
0
While TeleResilienceBench provides a novel mechanism for evaluating reasoning resilience in telecommunication LLMs, our methodology is subject to certain constraints. Fig. 7. Correct Flip Rate stratified by source-task difficulty for ORANBench Generally, a robust evaluation benchmark requires some (Easy/Medium/Hard) and TeleTables (Easy/Hard). Bar heights denote mean form of direct human-based correction or evaluation of the CFR across all evaluated models; error bars represent one standard deviation; generated questions to ensure strict correctness and prevent bracketed values indicate the observed minimum and maximum CFR. the inclusion of ambiguous or flawed options. However, for TeleResilienceBench, this manual verification step is ORANBench and TeleTables, the only two sub-benchmarks in not needed. Our pipeline derives its underlying questions the main release that carry per-instance difficulty annotations. and candidate options directly from the class-based subsets SixG_Bench is excluded because all of its instances fall of the GSMA’s ot-suite, which is widely recognized into a single difficulty tier (very hard), leaving no withinand rigorously validated as the industry-standard evaluation subset variation to examine. Figure 7 reports CFR stratified by benchmark for telecommunications. We implicitly trust the difficulty tier for both sub-benchmarks. Error bars represent structural integrity, questions, and candidate options that one standard deviation, and bracket annotations indicate the constitute our dataset. minimum and maximum CFR observed across models at each tier. VII. C ONCLUSION AND F UTURE W ORK For ORANBench, the mean CFR across Easy, Medium, and Hard questions is 28.9% (range: 16–47%), 30.4% (range: 24– We introduced TeleResilienceBench, a benchmark 38%), and 35.9% (range: 17–48%), respectively. CFR increases designed to measure a capability that standard telecom evaluslightly as difficulty increases, which is the opposite of what ations leave untested: whether a model can recover from an one would intuitively expect. The direction is weak and the already-flawed reasoning trajectory rather than simply solve a overlapping ranges mean no strong conclusion can be drawn, problem from a clean starting point. The benchmark comprises but the pattern is clearly not consistent with the idea that harder 818 discrete-choice instances drawn from seven telecom subquestions produce harder-to-correct traces. For TeleTables, domains, constructed by running qwen3.5:2b as a weak the direction reverses, with Easy questions yielding a mean generator, retaining its failures, and truncating the resulting CFR of 18.3% (range: 10–28%) and Hard questions yielding erroneous traces at the midpoint before presenting them to target 15.9% (range: 5–24%). Again, the gap is narrow relative to the models as a continuation task. The results across eight evaluated within-tier variance, and neither sub-benchmark tells a coherent models are sobering. Even the strongest model, Gemma4-31b, difficulty-resilience story. The absence of a consistent trend achieves a macro-average Correct Flip Rate of 29.1%, and the is, in fact, the most useful finding here. If difficulty labels Wrong Flip Rate is the dominant outcome across most model reflected the depth of reasoning required, harder instances and sub-benchmark combinations. Models are generally able to should produce more entrenched, flawed traces and therefore sense that something in the inherited trace is off, but translating lower recovery rates. The flat or inverted profiles suggest that into the correct answer is a considerably harder step. The instead that the difficulty annotations in these sub-benchmarks Qwen3.5 family shows virtually no improvement in resilience index something closer to recall difficulty, specifically the as scale increases from 4b to 27b, while Nemotron-3-nano 4b rarity or specificity of the technical fact required, rather than matches Gemma4-26b on macro-average CFR at a fraction reasoning difficulty in the sense of compositional inference of the memory cost, making it the most practical option for depth. A question labelled Hard may simply reference an edge-constrained deployments. The sub-benchmark analysis obscure O-RAN interface parameter; if the continuation model reveals that 3GPP_TSG is largely impervious to recovery across happens to recall that parameter, the recovery is no harder all models, a finding we attribute to the recall-centric nature than for an Easy question. Conversely, an Easy question may of that task rather than any failure of reasoning per se. The still defeat the continuation model if the required fact is not difficulty-stratification analysis further shows that the existing strongly encoded, regardless of how the source benchmark has difficulty labels in ORANBench and TeleTables do not track labelled it. Progress on difficulty-stratified telecom benchmarks reasoning complexity in any meaningful way, pointing to a likely reflects improvements in factual coverage more than broader limitation in how telecom benchmarks currently define improvements in reasoning, and the two are not the same thing. and measure hardness.
Easy Medium Hard
Easy
Hard
8
For future work, we want to extend the evaluation to larger frontier models, including those with explicit extendedthinking capabilities, to determine whether the resilience ceiling observed here is a property of the small-to-medium scale regime or persists at greater capacity. Second, the consistently low CFR across all tested architectures suggests that targeted fine-tuning on the error-recovery signal could be a productive intervention; constructing training data specifically from flawedtrace continuation tasks is a natural next step. Finally, the difficulty analysis points to the need for telecom benchmarks where harder questions are harder because they require longer chains of inference, not because they reference more obscure facts. Developing that kind of difficulty taxonomy is arguably a prerequisite for measuring real progress in LLM reasoning for this domain. R EFERENCES [1] NVIDIA Corporation, “NVIDIA and Nokia to pioneer the AI platform for 6G: Powering America’s return to telecommunications leadership.” NVIDIA Newsroom, 2025. [2] NVIDIA Corporation, “NVIDIA and partners show that software-defined AI-RAN is the next wireless generation.” NVIDIA Blog, Mar. 2026. [3] M. Polese, N. Mohamadi, S. D’Oro, L. Bonati, and T. Melodia, “Beyond connectivity: An open architecture for ai-ran convergence in 6g,” 2025. [4] S. Salmi, M. A. Ouameur, M. Bagaa, G. C. Alexandropoulos, A. Tahenni, D. Massicotte, and A. Ksentini, “Ai-native o-ran architectures for 6g: Towards real-time adaptation, conflict resolution, and efficient resource management,” IEEE Transactions on Network and Service Management, pp. 1–1, 2026. [5] A. Maatouk, F. Ayed, N. Piovesan, A. D. Domenico, M. Debbah, and Z.-Q. Luo, “Teleqna: A benchmark dataset to assess large language models telecommunications knowledge,” IEEE Network, vol. 40, no. 2, pp. 253– 260, 2026. [6] P. Gajjar and V. K. Shah, “Oran-bench-13k: An open source benchmark for assessing llms in open radio access networks,” in 2025 IEEE 22nd Consumer Communications & Networking Conference (CCNC), pp. 1–4, 2025. [7] A. Ezzakri, N. Piovesan, M. Sana, A. D. Domenico, F. Ayed, and H. Zhang, “Teletables: A benchmark for large language models in telecom table interpretation,” 2025. [8] M. Sana, N. Piovesan, A. D. Domenico, Y. Kang, H. Zhang, M. Debbah, and F. Ayed, “Reasoning language models for root cause analysis in 5g wireless networks,” 2025. [9] M. A. Ferrag, A. Lakas, and M. Debbah, “6g-bench: An open benchmark for semantic communication and network-level reasoning with foundation models in ainative 6g networks,” IEEE Open Journal of the Communications Society, vol. 7, pp. 3305–3330, 2026. [10] L. Bariah, B. Mefgouda, F. Tavakkoli, E. Molero, L. Powell, and M. Debbah, “Telcoagent-bench: A multilingual benchmark for telecom ai agents,” Mar. 2026.
[11] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023. [12] M. Turpin, J. Michael, E. Perez, and S. Bowman, “Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting,” in Advances in Neural Information Processing Systems (A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, eds.), vol. 36, pp. 74952–74965, Curran Associates, Inc., 2023. [13] J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou, “Large language models cannot self-correct reasoning yet,” 2024. [14] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, “Self-refine: iterative refinement with self-feedback,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, (Red Hook, NY, USA), Curran Associates Inc., 2023. [15] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: language agents with verbal reinforcement learning,” in Advances in Neural Information Processing Systems (A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, eds.), vol. 36, pp. 8634–8652, Curran Associates, Inc., 2023. [16] R. Nikbakht, M. Benzaghta, and G. Geraci, “Tspec-llm: An open-source dataset for llm understanding of 3gpp specifications,” 2024. [17] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” 2023. [18] E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman, “Star: self-taught reasoner bootstrapping reasoning with reasoning,” NIPS ’22, (Red Hook, NY, USA), Curran Associates Inc., 2022. [19] A. Jacovi, Y. Bitton, B. Bohnet, J. Herzig, O. Honovich, M. Tseng, M. Collins, R. Aharoni, and M. Geva, “A chain-of-thought is as strong as its weakest link: A benchmark for verifiers of reasoning chains,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (L.-W. Ku, A. Martins, and V. Srikumar, eds.), (Bangkok, Thailand), pp. 4615–4634, Association for Computational Linguistics, Aug. 2024. [20] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” 2023. [21] P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui, “Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (L.-W. Ku, A. Martins, and V. Srikumar, eds.), (Bangkok, Thailand), pp. 9426–9439, Association for Computational Linguistics, Aug. 2024.
9
[22] B. Liao, X. Chen, S. Rajaee, Y. Xu, C. Herold, A. Søgaard, S. Kashirsky, S. Poulos, S. Mor, S. Ramasamy, S. Acharya, M. de Rijke, and C. Monz, “Lost at the beginning of S. Ghosh, S. T. Sreenivas, S. Thomas, S. Fan, S. Gopal, reasoning,” 2025. S. Prabhumoye, S. Pachori, S. Toshniwal, S. Ding, S. Singh, S. Sun, S. Ithape, S. Majumdar, S. Singhal, [23] “Ollama.” https://ollama.com/. [Online; accessed 202605-08]. S. Alborghetti, S. Ge, S. D. Devare, S. K. Barua, S. Panguluri, S. Gupta, S. Priyadarshi, S. N. Akter, T. Bui, [24] NVIDIA, :, A. Blakeman, A. Grattafiori, A. Basant, A. Gupta, A. Khattar, A. Renduchintala, A. Vavre, T.-D. Ene, T. Kong, T. Do, T. Blankevoort, T. Balough, A. Shukla, A. Bercovich, A. Ficek, A. Shaposhnikov, T. Asida, T. B. Natan, T. Konuk, T. Vashishth, U. Karpas, A. Kondratenko, A. Bukharin, A. Milesi, A. Taghibakhshi, U. De, V. Noorozi, V. Noroozi, V. Srinivasan, V. Elango, A. Liu, A. Barton, A. S. Mahabaleshwarkar, A. Klein, V. Korthikanti, V. Kurin, V. Lavrukhin, W. Jiang, W. U. A. Zuker, A. Geifman, A. Shen, A. Bhiwandiwalla, A. Tao, Ahmad, W. Du, W. Ping, W. Zhou, W. Jennings, W. Zhang, A. Guan, A. Mandarwal, A. Mehta, A. Aithal, A. PooW. Prazuch, X. Ren, Y. Karnati, Y. Choi, Y. Meyer, Y.-F. jary, A. Ahamed, A. K. Thekkumpate, A. Dattagupta, Wu, Y. Zhang, Y. Lin, Y. Geifman, Y. Fu, Y. Subara, B. Zhu, B. Sadeghi, B. Simkin, B. Lanir, B. SchifY. Suhara, Y. Gao, Z. Moshe, Z. Dong, Z. Liu, Z. Chen, ferer, B. Nushi, B. Kartal, B. D. Rouhani, B. Ginsand Z. Yan, “Nemotron 3 nano: Open, efficient mixtureburg, B. Norick, B. Soubasis, B. Kisacanin, B. Yu, of-experts hybrid mamba-transformer model for agentic B. Catanzaro, C. del Mundo, C. Hwang, C. Wang, C.reasoning,” 2025. P. Hsieh, C. Zhang, C. Yu, C. Mungekar, C. Patel, [25] Q. Team, “Qwen3.5: Accelerating productivity with native C. Alexiuk, C. Parisien, C. Neale, D. Mosk-Aoyama, multimodal agents,” February 2026. D. Su, D. Corneil, D. Afrimi, D. Rohrer, D. Serebrenik, [26] C. Farabet, “Gemma 4: Byte for byte, the most capable D. Gitman, D. Levy, D. Stosic, D. Mosallanezhad, open models,” 04 2026. D. Narayanan, D. Nathawani, D. Rekesh, D. Yared, D. Kakwani, D. Ahn, D. Riach, D. Stosic, E. Minasyan, E. Lin, E. Long, E. P. Long, E. Lantz, E. Evans, E. Ning, A PPENDIX E. Chung, E. Harper, E. Tramel, E. Galinkin, E. Pounds, E. Briones, E. Bakhturina, F. Ladhak, F. Wang, F. Jia, F. Soares, F. Chen, F. Galko, F. Siino, G. H. Agam, G. Aj- A. Prompt Template for Evaluation janagadde, G. Bhatt, G. Prasad, G. Armstrong, G. Shen, G. Batmaz, G. Nalbandyan, H. Qian, H. Sharma, H. Ross, We use a single continuation-style prompt template across the H. Ngo, H. Sahota, H. Wang, H. Soni, H. Upadhyay, main sub-benchmarks: the model is provided the question, the H. Mao, H. C. Nguyen, H. Q. Nguyen, I. Cunningham, answer options, and a partial (half) reasoning trace, and is I. Shahaf, I. Gitman, I. Loshchilov, I. Moshkov, I. Put- instructed to continue the reasoning and output exactly one terman, J. Kautz, J. P. Scowcroft, J. Casper, J. Mitra, final answer line in option-index format (Final Answer: J. Glick, J. Chen, J. Oliver, J. Zhang, J. Zeng, J. Lou, <n>). For the Auxiliary (TeleMath) experiment, we keep the J. Zhang, J. Huang, J. Conway, J. Guman, J. Kamalu, same continuation structure (question + partial reasoning + J. Greco, J. Cohen, J. Jennings, J. Daw, J. V. Vialard, strict one-line final output), but change the required output J. Yi, J. Parmar, K. Xu, K. Zhu, K. Briski, K. Cheung, format from an option index to a numeric value, i.e., Final K. Luna, K. Santhanam, K. Shih, K. Kong, K. Bhardwaj, Answer: <number> in decimal or scientific notation. K. C. Puvvada, K. Pawelec, K. Anik, L. McAfee, L. Sleiman, L. Derczynski, L. Ding, L. Liebenwein, Shared Generation Template L. Vega, M. Grover, M. V. Segbroeck, M. R. de Melo, You are a telecom engineering assistant. You started M. N. Sreedhar, M. Kilaru, M. Ashkenazi, M. Romeijn, reasoning about a multiple-choice question but stopped M. Cai, M. Kliegl, M. Moosaei, M. Novikov, M. Samadi, mid-way. Continue from the partial reasoning in your M. Corpuz, M. Wang, M. Price, M. Boone, M. Evans, thinking stream, finish your analysis, and give the final M. Martinez, M. Chrzanowski, M. Shoeybi, M. Patwary, answer. N. Mulepati, N. Hereth, N. Assaf, N. Habibi, N. Zmora, Question: {QUESTION} N. Haber, N. Sessions, N. Bhatia, N. Jukar, N. Pope, Options: {OPTIONS} N. Ludwig, N. Tajbakhsh, N. Juluru, O. Hrinchuk, Partial reasoning (incomplete — continue from here): O. Kuchaiev, O. Delalleau, O. Olabiyi, O. U. Argov, {HALF_TRACE} O. Xie, P. Chadha, P. Shamis, P. Molchanov, P. Morkisz, Use the model’s thinking stream to extend this partial P. Dykas, P. Jin, P. Xu, P. Januszewski, P. P. Thombre, trace. When you are done reasoning, your final visible P. Varshney, P. Gundecha, Q. Miao, R. K. Mahabadi, reply MUST be exactly one line and nothing else: R. El-Yaniv, R. Zilberstein, R. Shafipour, R. Harang, Final Answer: <n> where <n> is the option number R. Izzo, R. Shahbazyan, R. Garg, R. Borkar, R. Gala, from 1 to {NUM_OPTIONS} (integer only). Do not R. Islam, R. Waleffe, R. Watve, R. Koren, R. Zhang, add any other text before or after that line in the final R. J. Hewett, R. Prenger, R. Timbrook, S. Mahdavi, reply. S. Modi, S. Kriman, S. Kariyappa, S. Satheesh, S. Kaji, S. Pasumarthi, S. Narentharen, S. Narenthiran, S. Bak,
10
B. Sample Outputs: Successful Flips (Resilience Demonstrated)
SixG_Bench Model: qwen3.5:9b Sample ID: ot-lite:SixG_Bench:27 Question (truncated): The UAV is currently operating under URLCC slice with 9ms latency, 1.2ms jitter, 0.03% loss, 150Mbps throughput, and 0.35 edge load after recovering from mMTC degradation. The mission requires inference with 12ms SLA latency and 0.1% max loss. Onboard inference consumes 15W and adds 2ms processing delay. Edge infere... Correct Label Text: Use onboard inference: adds 2ms delay but avoids network risk; total latency 2ms, within 12ms SLA; battery drain 15W over 3 turns. Half-trace cue (truncated): Thinking Process: 1. **Analyze the Request:** * Role: Telecom engineering assistant. * Task: Solve a multiple-choice question about UAV network slicing (URLCC slice). * Input Data: * Current State: URLCC slice, 9ms latency, 1.2ms jitter, 0.03% loss, 150Mbps throughput, 0.35 edge load. * Mission Requirement: Inference with 12ms SLA latency, 0.1% max loss. * Onboard Inference: 15W power, 2ms processing delay. * Edge Inference (Current): 9ms latency, 0.03% loss. * Edge Inference (Next Turn Risk): 15ms latency, 0.1% loss (due to edge load rising to 0.45). * UAV Speed: 18m/s. * Time to Next Waypoint: 3 turns. * Goal: Select the best option (1, 2, 3, or 4) based on SLA compliance, risk manageme... Model output (truncated): The user wants me to complete a reasoning trace based on a provided partial thought process. The partial trace analyzes a scenario involving latency, SLA (Service Level Agreement), battery consumption, and network reliability for a drone mission. There are four options presented: 1. **Onboard inference:** Adds 2ms processing delay. Total latency = 9ms (network) + 2ms (onboard) = 11ms. Loss = 0.03%. Battery = 15W. 2. **Use edge inference now:** Latency = 9ms. Risk of next turn hitting 15ms lat... Flip outcome: Corrected to gold (pred=1: Use onboard inference: adds 2ms delay but avoids network risk; total latency 2ms, within 12ms SLA; battery drain 15W over 3 turns., gold=Use onboard inference: adds 2ms delay but avoids network risk; total latency 2ms, within 12ms SLA; battery drain 15W over 3 turns.).
TeleQnA Model: qwen3.5:4b Sample ID: ot-lite:TeleQnA:82 Question (truncated): What is the formula to compute the number of pad bits, NPAD, in the sequence? [IEEE 802.15.4] Correct Label Text: NPAD = ND * LENGTH / 8 Half-trace cue (truncated): Thinking Process: 1. **Analyze the Request:** * Role: Telecom engineering assistant. * Task: Solve a multiple-choice question about IEEE 802.15.4. * Question: What is the formula to compute the number of pad bits, NPAD, in the sequence? * Options: 5 specific formulas involving ND (Number of Data Bits?), LENGTH, and 8. * Constraint: The final visible reply must be exactly one line: "Final Answer: <n>" where <n> is the option number (1-5). No other text before or after. 2. **Analyze the Subject Matter (IEEE 802.15.4):** * IEEE 802.15.4 is a standard for low-power wireless personal area networks (WPANs). * It defines the MAC layer and physical layer. * The question asks about "pad bits" (NPA... Model output (truncated): The user wants me to continue the reasoning process for a multiple-choice question about IEEE 802.15.4 and provide the final answer in a specific format. **1. Analyze the Question:** * **Topic:** IEEE 802.15.4 (Wireless Personal Area Networks). * **Subject:** Formula for Number of Pad Bits ($NPAD$). * **Options:** 1. $NPAD = ND \{}times 8 \{}times LENGTH$ 2. $NPAD = ND / 8 \{}times LENGTH$ 3. $NPAD = ND \{}times LENGTH / 8$ 4. $NPAD = ND / 8 + LENGTH$ 5. $NPAD = ND + 8 \{}times LENGTH$ **2. Recall IEEE... Flip outcome: Corrected to gold (pred=3: NPAD = ND * LENGTH / 8, gold=NPAD = ND * LENGTH / 8).
TeleLogs
3GPP_TSG Model: nemotron-3-nano:4b Sample ID: ot-lite:3GPP_TSG:67 Question (truncated): As a distinguished expert in telecommunication domain you are skilled in understanding and classifying 3GPP techincal documents. Please help user to classify text into 3GPP working group. Give answer in this format: {"WORKING GROUP": "working group name"}. Do not include any other information. Classify the following... Correct Label Text: SA4 Half-trace cue (truncated): The user wants me to classify a specific text into a 3GPP working group based on a provided list of options. 1. **Analyze the text:** * The text mentions "MPEGDASH". * It mentions "3GPP SA4". * It discusses "MPD linking mechanism", "DAM3", "AMD2", "Rel-14 work". * It mentions "ISO/IEC 23009-1 and ISO/IEC 23009-5". * It mentions "MPEGDASH". * It mentions "MPEG" (Moving Picture Experts Group). * It mentions "SA4" explicitly in the text ("3GPP SA4 would like to confirm..."). * It mentions "Work Item on MBMS Transport Protocol and APIs (TRAPI)" attached in S4-1602XX. * It mentions "Improved Streaming QoE Reporting in 3GPP Services and Networks (IQoE)" attached in S4-1602XX. * It mentions "S... Model output (truncated): The text explicitly mentions “3GPP SA4”. Among the options, SA4 corresponds to option 14. Therefore the correct choice is 14. Final Answer: 14 Flip outcome: Corrected to gold (pred=14: SA4, gold=SA4).
Model: gemma4:31b Sample ID: ot-lite:TeleLogs:68 Question (truncated): Analyze the 5G wireless network drive-test user plane data and engineering parameters. Identify the reason for the throughput dropping below 600Mbps in certain road sections. From the following 8 potential root causes, select the most likely one and enclose its number in \{}boxed{{}} in the final answer. C1: The servi... Correct Label Text: C3 Half-trace cue (truncated): The user wants to identify the correct option (C1 to C8) based on the provided engineering parameters data. The input data consists of two columns of data: 1. A header row with column names. 2. A data section with rows containing timestamps, coordinates, and other parameters. 3. A second data section with more rows. 4. A third data section with more rows. Wait, looking at the structure: The first block of text looks like a pipe-delimited CSV or similar format. ‘000000|128.145272|32.616028|18|697|72.66|5.04|1181.78|591|240|208|-|-|-78.6|-87.73|-92.05|-|-|202.72‘ This looks like a specific format, possibly from a specific dataset or a specific model’s output. However, the prompt says "Enge... Model output (truncated): To determine the correct option, we analyze the provided data. The input consists of three distinct sections: a set of configuration parameters (Engineering parameters), a set of measurement data (likely signal strength or quality), and a set of target/reference values. 1. **Engineering Parameters**: We have a set of cells with specific configurations (Azimuth, Tilt, Height, etc.). 2. **Measurement Data**: We have a set of measurements (RSRP, RSRQ, SINR) associated with different cells. 3. **... Flip outcome: Corrected to gold (pred=3: C3, gold=C3).
11
TeleTables
ORANBench
Model: gemma4:31b Sample ID: ot-lite:TeleTables:21 Question (truncated): What is the maximum SNR difference between FRC G-FR1-A3-33A and G-FR1-A4-29A in HST Scenario 1-NR500? Correct Label Text: 12.3 dB Half-trace cue (truncated): Thinking Process: 1. **Analyze the Request:** * Role: Telecom engineering assistant. * Task: Solve a multiple-choice question about a specific scenario (HST Scenario 1-NR500) involving specific FR (Frequency Range) identifiers (GFR1-A3-33A and G-FR1-A4-29A). * Question: What is the maximum SNR difference between FRC G-FR1-A3-33A and G-FR1-A4-29A in HST Scenario 1-NR500? * Options: 1. 12.3 dB, 2. 5.5 dB, 3. 17.8 dB, 4. 9.1 dB, 5. 8.7 dB. * Constraint: The final visible reply must be exactly one line: "Final Answer: <n>" where <n> is the option number (1-5). No other text before or after. 2. **Evaluate the Problem:** * This looks like a specific technical question related to a specific dat... Model output (truncated): * Question: Maximum SNR difference between FRC G-FR1-A3-33A and G-FR1-A4-29A in HST Scenario 1-NR500. * Options: 1. 12.3 dB, 2. 5.5 dB, 3. 17.8 dB, 4. 9.1 dB, 5. 8.7 dB. * Context: HST Scenario 1-NR500, FRCs G-FR1-A3-33A and G-FR1-A4-29A. * "HST" likely refers to High-Speed Train. * "Scenario 1-NR500" likely refers to a specific test scenario for 5G NR (New Radio) at 500 km/h or a similar high-speed condition. * "FRC" stands for Frequency Range Configuration or similar. * "G-FR1-A3-33A" and "... Flip outcome: Corrected to gold (pred=1: 12.3 dB, gold=12.3 dB).
Model: qwen3.5:27b Sample ID: ot-lite:ORANBench:19 Question (truncated): Which of the following protocols can be used for external user to group mapping in NETCONF endpoints according to the provided requirements? Correct Label Text: 4. All of the above Half-trace cue (truncated): Thinking Process: 1. **Analyze the Request:** * Role: Telecom engineering assistant. * Task: Solve a multiple-choice question. * Question: Which of the following protocols can be used for external user to group mapping in NETCONF endpoints according to the provided requirements? * Options: 1. 1. LDAP with StartTLS 2. 2. OAuth 2.0 3. 3. RADIUS with EAP 4. 4. All of the above * Constraint: The final visible reply must be exactly one line: "Final Answer: <n>" where <n> is the option number (1-4). No other text before or after. 2. **Analyze the Question Content:** * The question asks about "external user to group mapping in NETCONF endpoints". * It references "provided requirements", but no s... Model output (truncated): The user wants me to identify the correct option for a multiple-choice question about NETCONF endpoints and external user to group mapping. 1. **Analyze the Input:** * The user provides a snippet of reasoning ("You: ’NETCONF endpoints’..."). * The reasoning discusses NETCONF, RADIUS, OAuth 2.0, and LDAP. * It mentions "According to the provided requirements" but notes no requirements are actually provided in the prompt. * It leans towards "All of the above" because all three protocols (LDAP, ... Flip outcome: Corrected to gold (pred=4: 4. All of the above, gold=4. All of the above).
C. Sample Outputs: Failed Flips (Error Propagation) SixG_Bench
srsRANBench Model: gemma4:31b Sample ID: ot-lite:srsRANBench:119 Question (truncated): What is the purpose of the csi_resource_config struct? Correct Label Text: 1. It defines a group of NZP-CSI-RSResourceSet, CSI-IM-ResourceSet and/or CSI-SSB-ResourceSet. Half-trace cue (truncated): The user wants me to identify the purpose of the ‘csi_resource_config‘ struct in the context of telecommunications engineering, specifically related to 5G NR (New Radio) or similar standards (likely 3GPP). 1. **Analyze the Question:** The question asks for the purpose of the ‘csi_resource_config‘ struct. 2. **Context:** In 5G NR, CSI (Channel State Information) reporting involves configuring resources for CSI-RS (Channel State Information Reference Signal), CSI-IM (Interference Measurement Resource), and CSI-SSB (Synchronization Signal Block). 3. **Search/Recall Knowledge:** * In the 3GPP specification (specifically NR R15/R16), there is a structure called ‘csi_resource_config‘. * This st... Model output (truncated): Based on the provided options, all four choices are identical. The purpose of the ‘csi-ResourceConfig‘ (or ‘csi-ResourceConfig‘ in some contexts) in 5G NR/LTE is to configure the resources used for Channel State Information (CSI) reporting. Specifically, it defines the sets of resources that the UE should monitor and report on, which includes the configuration of the Resource Sets and the associated Resource Indicators. In the context of the provided options, it is used to define the groups o... Flip outcome: Corrected to gold (pred=1: 1. It defines a group of NZP-CSI-RS-ResourceSet, CSI-IM-ResourceSet and/or CSI-SSBResourceSet., gold=1. It defines a group of NZP-CSI-RS-ResourceSet, CSI-IM-ResourceSet and/or CSI-SSB-ResourceSet.).
Model: qwen3.5:27b Sample ID: ot-lite:SixG_Bench:32 Question (truncated): At turn 8 the UAV is in icing conditions with wind at 8.0 mps while descending from an altitude of 500.0 within policy bounds of 100.0 to 3500.0. The battery is at 100.0 percent and speed is 28.0 mps, but recent coordination and avoidance required sustained networking. Across turns 7 to 9 the URLLC slice shows laten... Correct Label Text: Downgrade to reduced autonomy with sensor fusion limited to radar and GNSS, keep URLLC active, and cap coordination messages given throughput between 100 and 130 Mbps and edge load near 0.3. Half-trace cue (truncated): The user wants me to solve a multiplechoice question based on a scenario involving a UAV in icing conditions. **Scenario Analysis:** * **Current State (Turn 8):** * Conditions: Icing, Wind 8.0 mps, Descending from 500.0m (Policy bounds 100.0 to 3500.0). * Battery: 100.0%. * Speed: 28.0 mps. * Networking: Sustained networking required. * URLLC Metrics (Turn 7-9): Latency 6-8ms, Jitter 0.9-1.3ms, Loss 0.04-0.08%, Throughput 100-130 Mbps, Edge Load 0.25-0.32. * Turn 6 mMTC Metrics: Latency 30ms, Jitter 10ms, Loss 2.1%, Edge Load 0.8. * Turn 9 Collision Avoidance: Required. * **Constraints & Risks:** * Icing implies control margins shrink in 1-3 turns if recovery doesn’t occur. * Worst case:... Model output (truncated): The user wants me to complete a reasoning trace based on a provided partial trace and then output a specific final answer format. **1. Analyze the Input:** * **Context:** The input looks like a snippet from a multi-turn conversation or a simulation log involving network metrics (URLLC, Edge Load, Control Margins). * **Partial Trace:** The text provided is a repetitive loop of reasoning about "Turn 10 Edge Load". It calculates the increment from Turn 8 to Turn 9 and applies it to Turn 9 to get... Observed error propagation: pred=4: Command an immediate full autonomy landing sequence within the geofence to minimize exposure to icing despite current URLLC latency of 6 to 8 ms., gold=Downgrade to reduced autonomy with sensor fusion limited to radar and GNSS, keep URLLC active, and cap coordination messages given throughput between 100 and 130 Mbps and edge load near 0.3..
12
srsRANBench
3GPP_TSG
Model: gemma4:31b Sample ID: ot-lite:srsRANBench:108 Question (truncated): What is the purpose of the ‘stress_traffic_sink‘ class? Correct Label Text: 1. To generate traffic for stress testing. Half-trace cue (truncated): Thinking Process: 1. **Analyze the Request:** * Role: Telecom engineering assistant. * Task: Solve a multiple-choice question about a specific class (‘stress_traffic_sink‘). * Question: What is the purpose of the ‘stress_traffic_sink‘ class? * Options: 1. To generate traffic for stress testing. 2. To receive traffic from the stress test source. 3. To log traffic information. 4. To manage the PDCP upper data notifier. * Constraint: The final visible reply must be exactly one line: ‘Final Answer: <n>‘ where ‘<n>‘ is the option number (1-4). No other text before or after. 2. **Identify the Context:** * The question refers to a specific class ‘stress_traffic_sink‘ in a specific software conte... Model output (truncated): Based on the terminology used in network simulation and testing, a "sink" is a component designed to receive and consume data, whereas a "source" is a component that generates data. In the context of a stress test or a performance test, a ‘stress_traffic_source‘ would generate the load, and a ‘stress_traffic_sink‘ would be the destination that receives that load to verify delivery or measure throughput. Therefore, the function of a ‘stress_traffic_sink‘ (or ‘stress_traffic_sink‘ as implied by... Observed error propagation: pred=2: 2. To receive traffic from the stress test source., gold=1. To generate traffic for stress testing..
Model: qwen3.5:4b Sample ID: ot-lite:3GPP_TSG:16 Question (truncated): As a distinguished expert in telecommunication domain you are skilled in understanding and classifying 3GPP techincal documents. Please help user to classify text into 3GPP working group. Give answer in this format: {"WORKING GROUP": "working group name"}. Do not include any other information. Classify the following... Correct Label Text: SA5 Half-trace cue (truncated): Thinking Process: 1. **Analyze the Request:** * Role: Telecom engineering assistant. * Task: Classify a text snippet from a 3GPP technical document into a specific 3GPP working group (WG). * Input Text: A paragraph discussing RAN overload, temporary stops, QoE reports, and SA4’s request to RAN2. * Output Format: ‘{"WORKING GROUP": "working group name"}‘. * Constraint 1: Do not include any other information in the final visible reply (except the JSON-like format specified in the prompt’s instruction for the final answer line). * Constraint 2: The final visible reply MUST be exactly one line: ‘Final Answer: <n>‘ where ‘<n>‘ is the option number (1-16). * Wait, there’s a conflict in the prom... Model output (truncated): The user wants me to identify the correct option number based on the provided text and instructions. 1. **Analyze the Request:** * **Task:** Classify the text into a 3GPP working group. * **Constraint:** The final visible reply MUST be exactly one line: ‘Final Answer: <n>‘. * **Options:** Numbered 1 to 16. * **Text Content:** The text mentions "SA4", "RAN2", "SA4 is able to address this...", "SA4 kindly asks RAN2". * **Reasoning:** The text explicitly discusses SA4 (Service Access Architectur... Observed error propagation: pred=4: CT6, gold=SA5.
TeleQnA Model: qwen3.5:27b Sample ID: ot-lite:TeleQnA:53 Question (truncated): According to the document, how do higher frequencies of RF fields affect eye pathology? [IEEE C95.1] Correct Label Text: Higher frequencies have greater potential for effects on structures near the outer surface of the eye, including the lens. Half-trace cue (truncated): Thinking Process: 1. **Analyze the Request:** * Role: Telecom engineering assistant. * Task: Solve a multiple-choice question based on a document (specifically IEEE C95.1). * Question: How do higher frequencies of RF fields affect eye pathology? * Options: 5 specific statements regarding higher frequencies and eye pathology. * Constraint: The final visible reply must be exactly one line with the option number (1-5) only. No other text before or after. * Reasoning: Use the model’s thinking stream for detailed reasoning. 2. **Analyze the Document (IEEE C95.1):** * I need to recall or infer the content of IEEE C95.1 (IEEE Standard for Safety of Electrical Equipment - Eye Safety). * Wait, IEE... Model output (truncated): The user wants me to identify the correct option based on the provided text and reasoning. The reasoning provided in the prompt’s "thought" section (which I am simulating/continuing) points to Option 2. The reasoning cites IEEE C95.1-2000, Section 10.1.2, stating: "Higher frequencies of RF fields have a greater potential for effects on structures deep within the eye, including the lens." This matches Option 2: "Higher frequencies of RF fields have a greater potential for effects on structures... Observed error propagation: pred=2: Higher frequencies have greater potential for effects on structures deep within the eye, including the lens., gold=Higher frequencies have greater potential for effects on structures near the outer surface of the eye, including the lens..
TeleLogs Model: qwen3.5:27b Sample ID: ot-lite:TeleLogs:66 Question (truncated): Analyze the 5G wireless network drive-test user plane data and engineering parameters. Identify the reason for the throughput dropping below 600Mbps in certain road sections. From the following 8 potential root causes, select the most likely one and enclose its number in \{}boxed{{}} in the final answer. C1: The servi... Correct Label Text: C3 Half-trace cue (truncated): The user wants to identify the correct option (C1 to C8) based on a provided dataset of engineering parameters and a specific model’s thinking process. However, the prompt asks me to "Use the model’s thinking stream for detailed reasoning." This implies I need to act as the model generating the reasoning, but the final output must be strictly "Final Answer: <n>". Wait, looking at the prompt structure, it seems like a classification task where I need to determine which option corresponds to the data provided. The data provided contains timestamps, signal strength (RSSI-like values), and various engineering parameters (Cell ID, Longitude, Latitude, etc.). The "Engeneering parameters data" s... Model output (truncated): The user wants me to identify a specific option (C1-C8) based on the provided data. However, the prompt provided in the "second block" description is actually just a table of engineering parameters data. Wait, looking closely at the prompt structure provided in the "User" message: The user says "You second block is the ’Engeneering parameters data’ table." Then they provide a list of rows (Row 1 to Row 4) which look like time-series data (timestamps, coordinates, azimuths, etc.). Then they de... Observed error propagation: pred=4: C4, gold=C3.
13
ORANBench Model: qwen3.5:9b Sample ID: ot-lite:ORANBench:133 Question (truncated): Which of the following standards is specifically mentioned in the provided context as being relevant to Open Radio Access Networks (O-RAN)? Correct Label Text: 3. 3GPP TS 28.313: Management and orchestration; Self-Organizing Networks (SON) for 5G networks Half-trace cue (truncated): Thinking Process: 1. **Analyze the Request:** * Role: Telecom engineering assistant. * Task: Solve a multiple-choice question based on a "provided context". * Constraint: The user *has not* provided a context. The prompt says "based on the provided context" but there is no text provided in the input. * Constraint: The final visible reply must be exactly one line: "Final Answer: <n>" where <n> is the option number (1-4). * Constraint: Do not add any other text before or after that line. 2. **Evaluate the Input:** * Input: "You are a telecom engineering assistant. Solve this multiple-choice question.\{}n\{}nQuestion:\{}nWhich of the following standards is specifically mentioned in the provided co... Model output (truncated): The user is asking me to complete a reasoning trace for a multiple-choice question about O-RAN standards. The user explicitly states: "Use the model’s thinking stream for detailed reasoning. When you are done reasoning, your final visible reply MUST be exactly one line and nothing else: Final Answer: <n>". However, the user also says: "You I violate the ’Final Answer: <n>’ constraint." in the prompt text provided in the example. Wait, looking at the prompt structure: The prompt starts with "Y... Observed error propagation: pred=4: 4. RFC 8348: A YANG Data Model for Hardware Management, gold=3. 3GPP TS 28.313: Management and orchestration; Self-Organizing Networks (SON) for 5G networks.
TeleTables Model: gemma4:31b Sample ID: ot-lite:TeleTables:3 Question (truncated): Which consecutive taps exhibit the highest increase in power? Correct Label Text: Tap 1 to Tap 2 Half-trace cue (truncated): Thinking Process: 1. **Analyze the Request:** * Role: Telecom engineering assistant. * Task: Solve a multiple-choice question. * Question: "Which consecutive taps exhibit the highest increase in power?" * Options: 1. Tap 10 to Tap 11 2. Tap 1 to Tap 2 3. Tap 13 to Tap 15 4. Tap 21 to Tap 22 5. Tap 9 to Tap 10 * Constraint: The final visible reply must be exactly one line: "Final Answer: <n>" where <n> is the option number (1-5). No other text before or after. * Missing Information: The question *as stated* does not provide the data (e.g., a table of tap powers, a signal diagram, or a specific scenario description). This looks like a question from a specific dataset or context (likely a GS... Model output (truncated): * Role: Telecom engineering assistant. * Question: "Which consecutive taps exhibit the highest increase in power?" * Options: 1. Tap 10 to Tap 11 2. Tap 1 to Tap 2 3. Tap 13 to Tap 15 4. Tap 21 to Tap 22 5. Tap 9 to Tap 10 * Constraint: Final answer must be "Final Answer: <n>". * The prompt provides a "Partial reasoning" section. * The partial reasoning acknowledges that the data (the power levels for each tap) is missing. * It hypothesizes that this is a question from a known benchmark (like... Observed error propagation: pred=1: Tap 10 to Tap 11, gold=Tap 1 to Tap 2.