arXiv CSopen access

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation SHIDE ZHOU, Huazhong University of Science and Technology, China KAILONG WANG∗ , Huazhong University of Science and Technology, China and National University of

arXiv:2607.04784v1 [cs.SE] 6 Jul 2026

Singapore, Singapore LING SHI, Nanyang Technological University, Singapore HAOYU WANG, Huazhong University of Science and Technology, China Defining the reasoning boundaries and ensuring the reliability of Large Reasoning Models (LRMs) remains a critical challenge. Current benchmarks primarily rely on static datasets susceptible to data contamination or synthetic tasks lacking fine-grained difficulty control. Furthermore, standard outcome-based evaluations often conceal reasoning flaws by neglecting the reasoning process. To address these limitations, we introduce TRACE, a testing framework that models temporal reasoning as constraint satisfaction problems via Allen’s Interval Algebra. This approach enables precise regulation of logical complexity and incorporates a Trace-Based Verification Oracle to validate reasoning faithfulness. Using this framework, we construct TRACEBench, an extensive benchmark comprising 1,200 synthesized test instances across graded difficulty levels. We employ TRACE to evaluate eight widely used LRMs on TRACEBench. The results confirm a strong negative correlation between model performance and our difficulty metric (Pearson’s 𝑟 ≈ −0.96), validating the effectiveness of our difficulty control mechanism. Moreover, our trace-based analysis exposes significant discrepancies between reasoning validity and final answers, revealing a high spurious guessing rate of approximately 28% in mid-sized models. In addition, we diagnose scale-dependent failure modes, ranging from Degenerative Loops in small models to Reasoning Explosion in advanced architectures. TRACE thus provides a robust, automated platform for benchmarking the true temporal reasoning capabilities of LRMs. CCS Concepts: • Computing methodologies → Temporal reasoning; Natural language generation; • Software and its engineering → Software verification and validation. Additional Key Words and Phrases: Large Reasoning Models, Temporal Reasoning, Automated Testing, Test Generation

1

Introduction

The evolution of Large Language Models (LLMs) has culminated in the emergence of Large Reasoning Models (LRMs), such as the DeepSeek-R1 family [7, 8] and OpenAI’s o-series [19], designed specifically for complex problem-solving. In this domain, temporal reasoning is a fundamental capability, demanding strict logical consistency rather than approximate retrieval. While Chain-ofThought(CoT) strategies [15, 28, 30] have yielded significant performance gains, a fundamental question persists: Do these improvements reflect genuine deduction or merely sophisticated pattern matching? This uncertainty complicates reliability assessments [9], underscoring the urgent need for a specialized benchmarking framework. ∗ Corresponding author.

Authors’ Contact Information: Shide Zhou, Huazhong University of Science and Technology, Wuhan, China, shidez@ hust.edu.cn; Kailong Wang, Huazhong University of Science and Technology, Wuhan, China and National University of Singapore, Singapore, Singapore, [email protected]; Ling Shi, Nanyang Technological University, Singapore, Singapore, [email protected]; Haoyu Wang, Huazhong University of Science and Technology, Wuhan, China, [email protected]. cn. 2026. ACM XXXX-XXXX/2026/7-ART https://doi.org/10.1145/nnnnnnn.nnnnnnn , Vol. 1, No. 1, Article . Publication date: July 2026.

2

Shide Zhou, Kailong Wang, Ling Shi, and Haoyu Wang

Table 1. Comparison of Our Work with Existing Benchmarks. TRAM

TimeBench

Test of Time t-BEN

Our Work

Data Type Difficulty Control Verification Method Data Type: Static Corpus; Dynamic Synthetic. Difficulty Control: Taxonomy-based; Coarse-grained; Fine-grained. Verification Method: Outcome-based; Dual Verification (Trace + Answer).

Current benchmarks in temporal reasoning exhibit significant structural limitations, as summarized in Table 1. First, the reliance on static dataset aggregation in suites like TRAM [29] and TimeBench [5] introduces severe data contamination risks, allowing models to exploit memorization rather than engaging in genuine deduction [20]. Second, while synthetic frameworks such as Test of Time [10] and t-BEN [27] mitigate these risks, they typically employ coarse-grained difficulty proxies. Lacking precise regulation of logical complexity, these methods struggle to pinpoint specific breakdown points in a model’s reasoning capabilities [33]. Third, prevailing evaluation methodologies remain strictly outcome-centric. By prioritizing final answer accuracy over process validity, they fail to detect spurious guessing, leaving the faithfulness of the reasoning trace largely unexamined [18]. Our Work. To overcome these structural deficits, a robust framework must fulfill three core design objectives: ensuring dynamic data generation to prevent contamination, implementing finegrained difficulty control [23] for precise boundary detection, and incorporating dual verification to validate both reasoning traces and final outcomes. To this end, we introduce TRACE (Temporal Reasoning Automated Controllable Evaluator). This framework builds upon Allen’s Interval Algebra [1, 2], which encompasses all 13 fundamental temporal relations, to model reasoning tasks as Constraint Satisfaction Problems (CSPs) [14]. TRACE operates through three primary modules corresponding to the design objectives. The Difficulty-Aware Constraint Generator constructs constraint graphs. It allows users to strictly control logical complexity by setting a target difficulty and then adjusting the number of events and the types of temporal relations in the graph. The Task Constructor translates these graphs into natural language contexts. It utilizes the explicit edges of the graph as known premises and selects the implicit, inferred edges as questions to ensure the task requires reasoning. Finally, the Trace-Based Verifier assesses the model’s logic. Unlike traditional methods that compare outputs against a fixed reference, this module verifies each step of the generated reasoning trace against the algebraic closure implied by the ground-truth constraint network. We utilize TRACE to construct TRACEBench, a graded benchmark comprising 1,200 synthesized test instances spanning six distinct difficulty levels. To ensure a focused assessment of LRMs’ intrinsic logical reasoning rather than their tool-calling abilities, we purposely restrict the use of external solvers, thereby measuring pure deductive capabilities. Using TRACEBench, we conduct a comprehensive evaluation of eight widely used LRMs, ranging from distilled variants to advanced proprietary models. The results confirm the precision of our difficulty modeling. We observe a strong negative correlation between the difficulty score and model accuracy, with Pearson’s r approximately −0.96. This result validates that the framework effectively generates tasks with a controllable gradient of complexity. Notably, our trace-based analysis exposes significant discrepancies between reasoning validity and final answer correctness. We find that mid-sized models exhibit a high spurious guessing rate of approximately 28%, where they frequently arrive at the correct final label despite relying on invalid reasoning steps. Conversely, smaller architectures suffer from Answer Misalignment, where valid logic leads to incorrect final labels. Furthermore, we diagnose , Vol. 1, No. 1, Article . Publication date: July 2026.

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

3

scale-dependent structural failure modes under extreme complexity. Small and mid-sized models often fall into Degenerative Loops of repetitive generation, whereas advanced models mainly face Reasoning Explosion, with valid reasoning chains becoming too long and eventually exceeding the context window. Contributions. In summary, this work makes the following contributions: • A difficulty-controllable testing framework. We introduce TRACE, a novel framework that leverages graph-based generation and Allen’s Interval Algebra to synthesize logically consistent temporal reasoning tasks with precisely tunable complexity, incorporating a trace-based oracle for faithfulness verification. • An extensive graded benchmark. We construct TRACEBench, containing 1,200 instances across six difficulty levels, designed to systematically probe the reasoning capabilities of LRMs. • Empirical insights and failure taxonomy. Through extensive evaluation, we uncover a high spurious guessing rate in mid-sized models and identify scale-dependent failure modes, including Degenerative Loops in smaller models and Reasoning Explosion in advanced models. 2 2.1

Background and Related Work Large Reasoning Model

The recent shift from scaling training parameters to scaling test-time computation has enabled the emergence of LRMs [3, 24]. Unlike standard models that generate immediate responses, LRMs utilize an extended inference phase to decompose complex problems into intermediate logical steps. While initial capabilities are elicited through prompt engineering methods like Chain-ofThought [11, 15, 28, 30, 31], state-of-the-art models such as the DeepSeek-R1 family [7, 8] internalize this deliberative process through specialized post-training optimization, including Reinforcement Learning and rejection sampling [18, 32]. As a result, these models produce an output structured into two clear parts: an internal thinking process that traces intermediate reasoning and a final conclusion representing the definitive outcome. 2.2

Evolution of Reasoning Benchmarks

The evaluation of reasoning in language models has evolved from general competency tests to specialized, logic-intensive benchmarks. Initial efforts focused on static datasets designed to probe mathematical and symbolic reasoning. Canonical benchmarks such as GSM8K [6] and MATH [13] established the standard for evaluating multi-step mathematical derivation. Similarly, BIG-Bench Hard [25] and MMLU [12] extended this scope to encompass broad cross-domain reasoning tasks, while specialized datasets like SCAN [16] and CLUTRR [23] focused on compositional and inductive logic, respectively. To mitigate the inherent risks of contamination and coarse granularity in such static datasets, recent research has shifted towards synthetic and difficulty-controllable evaluation frameworks. Approaches such as PrOntoQA [22] and DyVal [33] utilize generated logic puzzles to prevent memorization. More recently, GRADE [17] advanced this direction in retrieval-augmented systems by modeling task difficulty via a 2D matrix of reasoning depth and semantic distance, validating that error rates strictly follow structural constraints. In the specific domain of temporal reasoning, benchmarks have progressed from static aggregation to dynamic synthesis. Works such as TRAM [29] construct comprehensive evaluation suites by unifying multiple datasets covering aspects like frequency, duration, and event ordering. Similarly, TimeBench [5] introduces a hierarchical taxonomy, organizing tasks into symbolic, commonsense, , Vol. 1, No. 1, Article . Publication date: July 2026.

4

Shide Zhou, Kailong Wang, Ling Shi, and Haoyu Wang

and event-based reasoning categories to probe different dimensions of temporal logic. To address static limitations, recent efforts such as Test of Time [10] and t-BEN [27] have advanced this field by leveraging synthetic generation mechanisms, creating dynamic datasets to mitigate the memorization risks inherent in static corpora. Despite these advancements, current methodologies largely remain outcome-centric or rely on coarse-grained difficulty proxies, leaving the critical need for fine-grained complexity control and trace-level faithfulness verification unaddressed. 2.3

Allen Interval Algebra Table 2. The 13 Basic Relations in Allen Interval Algebra. Relation / Inverse Relation

Symbol (Rel/Inv)

before / after meets / met-by overlaps / overlapped-by starts / started-by during / contains finishes / finished-by equals

B / BI M / MI O / OI S / SI D / DI F / FI E

Visualization X

Y

X

Y

X

Y

X Y X Y Y X Y

X

Endpoint Conditions 𝑋𝑒 < 𝑌𝑠 𝑋𝑒 = 𝑌𝑠 𝑋𝑠 < 𝑌𝑠 < 𝑋𝑒 < 𝑌𝑒 𝑋𝑠 = 𝑌𝑠 , 𝑋𝑒 < 𝑌𝑒 𝑌𝑠 < 𝑋𝑠 , 𝑋𝑒 < 𝑌𝑒 𝑋𝑒 = 𝑌𝑒 , 𝑋𝑠 > 𝑌𝑠 𝑋𝑠 = 𝑌𝑠 , 𝑋𝑒 = 𝑌𝑒

TRACE is built upon Allen’s Interval Algebra [1], which treats time intervals as primitive objects and provides a robust reasoning framework for temporal relations. An interval 𝑋 is defined by its start and end points, denoted as [𝑋𝑠 , 𝑋𝑒 ] where 𝑋𝑠 < 𝑋𝑒 . The algebra defines a set of 13 mutually exclusive basic relations that capture all possible qualitative positions between two intervals. As shown in Table 2, these include six asymmetric relations (e.g., before) with their corresponding inverses (e.g., after), and one symmetric relation (equals). A temporal constraint is formed by assigning one of these relations to a pair of intervals (e.g., 𝑋 overlaps 𝑌 ). Consequently, a reasoning task can be modeled as a network where each node corresponds to an event represented as a time interval, and each edge encodes a temporal constraint. A fundamental property of this algebra lies in its ability to support deductive reasoning through transitivity. Intuitively, if event 𝑋 happens before 𝑌 , and 𝑌 happens before 𝑍 , logic dictates that 𝑋 must be before 𝑍 . Formally, this is determined by the composition of relations: Given two 𝑟1 𝑟2 constraints 𝑋 −→ 𝑌 and 𝑌 −→ 𝑍 , where 𝑟 1, 𝑟 2 ∈ R𝐴𝑙𝑙𝑒𝑛 denote the specific relations, the valid relationship between 𝑋 and 𝑍 is constrained to the composition set 𝑟 1 ◦ 𝑟 2 ⊆ R𝐴𝑙𝑙𝑒𝑛 . To serve as a valid reasoning task, a constraint graph must strictly satisfy Path Consistency. We enforce path consistency as a generation-time constraint: for every triple of nodes (𝑖, 𝑗, 𝑘), the direct constraint between 𝑖 and 𝑘 does not conflict with the transitive constraints inferred through 𝑗. We apply this property to rule out impossible scenarios (e.g., a cycle where 𝐴 is before 𝐵 and 𝐵 is before 𝐴), ensuring the resulting instances are logically contradiction-free and admit a consistent temporal interpretation. 3

Methodology

In this section, we present the design and implementation of TRACE, a difficulty-controllable testing framework specifically tailored for evaluating LRMs in complex temporal reasoning scenarios. TRACE enables the dynamic generation of reasoning tasks with tunable complexity, coupled with a rigorous verification mechanism to assess the faithfulness of the reasoning process. We first provide a high-level overview of the system architecture. Subsequently, we elaborate on the three core modules: Difficulty-Aware Constraint Graph Generation, Temporal Reasoning Task Construction, and the Trace-Based Verification Oracle. , Vol. 1, No. 1, Article . Publication date: July 2026.

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

3.1

5

Overview of TRACE

The overall architecture of TRACE is illustrated in Figure 1. The workflow begins with a difficulty configuration, where the user sets a target difficulty level. Based on this configuration, the DifficultyAware Constraint Generator (Section 3.2) constructs a series of mathematically consistent constraint graphs, which serve as the logical backbone of the test cases. These abstract graphs are then processed by the Task Constructor (Section 3.3), which translates the constraints into a natural language context. Specifically, the system formulates questions based on implicit relations, which are logical consequences not explicitly stated in the context. This design ensures that the tasks strictly require deductive reasoning. In the final stage, the generated tasks are executed by the target LRM. During this phase, the model operates independently without access to external tools or solvers, ensuring the evaluation isolates pure internal reasoning. Through structured prompting, the model is required to output both the formal reasoning process and the final answer. TRACE then employs a Trace-Based Verifier (Section 3.4) to extract the reasoning traces from the model’s responses and uses a constraint solver to validate the correctness of each reasoning step. This mechanism enables TRACE to distinguish genuine reasoning from spurious guesses, comprehensively assessing reasoning faithfulness. 1. Difficulty-Aware Constraint Graph Generation

2. Temporal Reasoning Task Construction

3. Trace-Based Verification Oracle

Question Templatization

Candidate Constraint Config Generation Difficulty Model

Target Difficulty

Candidate Config

(𝒏, 𝒎, 𝓛)

Event Scale Estimation

Edge Sampling

Greedy Calibration

Relation Sampling

Constraint Graph Generation

Prompt Template

Temporal Benchmark

LRM

Thinking Process

JSON Object

Extraction of Facts and Questions

Premise Facts

Ground Truth

Candidate Queries

Reasoning Trace Verification

Answer Verification

True Reasoning

Spurious Guessing

Explicit Constraints Implicit Constraints

1. Skeleton Construction

2. Relation Augmentation

3. Isomorphism Elimination

be

A

re fo

B

be

fo re

C

before

Temporal Constraint Graph

Answer Misalignment

Complete Failure

Fig. 1. The Workflow of TRACE.

3.2

Difficulty-Aware Constraint Graph Generation

3.2.1 Difficulty Modeling. To systematically quantify the difficulty of a temporal reasoning task, we propose a model based on two primary dimensions: event-set scale and constraint complexity. Problem Formalization. We model a temporal reasoning task as a constraint graph G = (𝑉 , 𝐸), where 𝑉 = {𝑣 1, 𝑣 2, . . . , 𝑣𝑛 } denotes the set of 𝑛 unique events and 𝐸 denotes the set of temporal constraints given as premises. Each constraint (𝑣𝑖 , 𝑣 𝑗 ) ∈ 𝐸 is labeled with a relation type 𝑟 ∈ R𝐴𝑙𝑙𝑒𝑛 (e.g., B for before, O for overlaps). Note that 𝐸 contains only the explicit constraints provided as context, excluding any implicit relations to be inferred. Difficulty Definition. We posit that task difficulty is jointly determined by event-set scale and constraint complexity. Specifically, we define the difficulty score D for a graph G as: , Vol. 1, No. 1, Article . Publication date: July 2026.

6

Shide Zhou, Kailong Wang, Ling Shi, and Haoyu Wang

© 1 ∑︁ ª D (G) = |𝑉 |𝛼 · ­ 𝑤 (𝑟𝑖 𝑗 ) ® |𝐸| (𝑣𝑖 ,𝑣 𝑗 ) ∈𝐸 « ¬ | {z }

(1)

𝑤

where |𝑉 | is the number of events (event-set scale), 𝛼 controls how scale amplifies difficulty, and 𝑤 (𝑟 ) maps each relation type to a scalar complexity score. Design Rationale. This formulation captures two orthogonal components of difficulty: ❶ Eventset scale (|𝑉 |𝛼 ): As the number of events increases, the number of possible event permutations grows super-linearly. The exponent 𝛼 models this expansion, reflecting how the increase in context length and reasoning depth impacts the difficulty. ❷ Constraint complexity (𝑤): Not all relations are equally hard to process. The term 𝑤 represents the average inference cost per edge. For instance, determining a precise ordering under a complex constraint (e.g., overlaps) requires handling more uncertainty than a simple constraint (e.g., equals). This term distinguishes graphs with the same number of events but different average constraint complexity. Complexity Analysis and Model Calibration. To instantiate the difficulty model, we assign a weight 𝑤 (𝑟 ) to each Allen relation and calibrate the scaling exponent 𝛼. Our weights reflect the cognitive load required to maintain path consistency, determined by the degrees of freedom and boundary precision. Relations that reduce the number of independent variables are treated as easier, while those requiring the simultaneous satisfaction of multiple coupled inequalities without explicit anchors are considered harder. 1) Relation Complexity Weights. We group the 13 Allen relations into four tiers based on their impact on the reasoning state space; for each inverse pair, we use the same weight, i.e., 𝑤 (𝑟 ) = 𝑤 (𝑟 −1 ): • Coincidence Constraints (𝑤 = 0.8): The equals (𝐸) relation enforces that two events share identical start and end points (𝑆𝑡𝑎𝑟𝑡𝐴 = 𝑆𝑡𝑎𝑟𝑡𝐵 ∧ 𝐸𝑛𝑑𝐴 = 𝐸𝑛𝑑𝐵 ). Although this involves two conditions, logically, it collapses two distinct events into a single temporal entity. This effective dimensionality reduction simplifies the constraint graph, lowering the memory and inference burden required to track the timeline. • Precedence Constraints (𝑤 = 1.0): Relations before (𝐵) and after (𝐵𝐼 ) express a basic precedence condition (e.g., 𝐸𝑛𝑑𝐴 < 𝑆𝑡𝑎𝑟𝑡𝐵 ). Given its high degrees of freedom, we treat this ubiquitous “A before B” pattern as the baseline unit of difficulty. • No-gap Adjacency Constraints (𝑤 = 1.1): Relations meets (𝑀) and met-by (𝑀𝐼 ) strengthen precedence by requiring exact boundary alignment (e.g., 𝐸𝑛𝑑𝐴 = 𝑆𝑡𝑎𝑟𝑡𝐵 ). We assign a slightly higher weight to reflect the additional precision needed to satisfy and verify this zero-gap condition. • Endpoint-interleaving Constraints (𝑤 ∈ {1.5, 2.0}): Relations such as starts (𝑆) and finishes (𝐹 ) include an endpoint equality (e.g., 𝑆𝑡𝑎𝑟𝑡𝐴 = 𝑆𝑡𝑎𝑟𝑡𝐵 or 𝐸𝑛𝑑𝐴 = 𝐸𝑛𝑑𝐵 ) together with an inequality, which partially anchors the configuration; we assign them 𝑤 = 1.5. In contrast, relations such as overlaps (𝑂) and during (𝐷) require satisfying multiple coupled inequalities simultaneously (e.g., overlaps: 𝑆𝑡𝑎𝑟𝑡𝐴 < 𝑆𝑡𝑎𝑟𝑡𝐵 < 𝐸𝑛𝑑𝐴 < 𝐸𝑛𝑑𝐵 ), which is typically harder to deduce; we assign them 𝑤 = 2.0. Although extreme cases (e.g., all equals) may slightly distort the difficulty score, our strategy of diverse constraint sampling helps the score better reflect the actual reasoning complexity in practice. 2) Calibrating the Scale Exponent. To set 𝛼 in an interpretable way, we anchor the metric to a standard reference task: the smallest transitive inference problem in which three events are given , Vol. 1, No. 1, Article . Publication date: July 2026.

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

7

with two explicit relations, and the model must infer the third relation. We define the difficulty of this atomic task as D𝑟𝑒 𝑓 = 10. Assuming a uniform distribution over relation types, the mean relation weight is 𝑤 ≈ 1.46. Substituting into our formula yields 10 = 3𝛼 · 1.46, and solving gives 𝛼 ≈ 1.75. This calibration grounds the metric in a solvable baseline case and ensures difficulty grows smoothly with problem scale. Algorithm 1 Candidate Constraint Configuration Generation Require: Target difficulty Dtar , relation set R𝐴𝑙𝑙𝑒𝑛 with weights 𝑤 (·), scale exponent 𝛼, slack Δ, density 𝜌, softmax width 𝜎, tolerance 𝜏, max adjust steps 𝑇 . Ensure: Candidate configuration (𝑛, 𝑚, L). 1: Phase 1: Event Scale Estimation & Edge Sampling

Í 𝑛 ← max 3, round (Dtar /𝑤 prior ) 1/𝛼 2: 𝑤 prior ← | R 1 | 𝑟 ∈ R𝐴𝑙𝑙𝑒𝑛 𝑤 (𝑟 ); 𝐴𝑙𝑙𝑒𝑛
𝑛 3: 𝑁 pair ← 2 ; 𝑚 exp ← round(𝜌 · 𝑁 pair ) 4: Compute bounds [𝑚 lo, 𝑚 hi ] around 𝑚 exp 5: Sample 𝑚 ∼ Unif ({𝑚 lo, . . . , 𝑚 hi }) 6: Phase 2: Relation Sampling Compute 𝑝 (𝑟 ) ∝ exp(−(𝑤 (𝑟 ) − 𝑤 tar ) 2 /2𝜎 2 ) 7: 𝑤 tar ← Dtar /𝑛𝛼 ; 8: Sample multiset L of size 𝑚 from distributions 𝑝 (·) 9: Phase 3: Greedy Calibration 1 Í tar | 𝜀 ← |𝑤 ach𝑤−𝑤 10: 𝑤 ach ← 𝑚 ; 𝑡 ←0 𝑟 ∈ L 𝑤 (𝑟 ); tar 11: while 𝜀 > 𝜏 and 𝑡 < 𝑇 do 12: Swap one extreme-weight relation in L to move 𝑤 ach towards 𝑤 tar 13: Update 𝑤 ach and 𝜀; 𝑡 ← 𝑡 + 1 14: end while 15: return (𝑛, 𝑚, L) 3.2.2 Candidate Constraint Configuration Generation. Based on the difficulty model proposed in Section 3.2.1, TRACE first converts the target difficulty Dtar into a candidate constraint configuration that is expected to yield an achieved difficulty Dach close to Dtar . Algorithm 1 summarizes the complete procedure. Concretely, a configuration is represented as a tuple (𝑛, 𝑚, L), where 𝑛 is the number of events, 𝑚 is the number of explicit constraints to be presented as premises, and L is a multiset of 𝑚 Allen relations. The generation starts by inferring a reasonable event-set size. Let 𝑤 prior denote the average weight over the allowed relation set, ∑︁ 1 𝑤 prior = 𝑤 (𝑟 ). (2) |R𝐴𝑙𝑙𝑒𝑛 | 𝑟 ∈ R 𝐴𝑙𝑙𝑒𝑛

Following Eq. (1), we invert the relationship between scale and difficulty to estimate the event count:   1 !! Dtar 𝛼 𝑛 = max 3, round (3) 𝑤 prior The lower bound of three guarantees the smallest setting in which transitive inference is meaningful. Given 𝑛, TRACE determines the number of explicit constraints 𝑚 by controlling the edge density.
The number of possible unordered event pairs is 𝑁 pair = 𝑛2 . When 𝑚 is too large (close to 𝑁 pair ), the context becomes nearly complete and leaves little room for deductive inference; when 𝑚 is too small, the graph tends to be under-constrained or disconnected, which makes deterministic , Vol. 1, No. 1, Article . Publication date: July 2026.

8

Shide Zhou, Kailong Wang, Ling Shi, and Haoyu Wang

reasoning unreliable. We therefore use a density parameter 𝜌 ∈ (0, 1) to set an expected edge count 𝑚 exp = round(𝜌 · 𝑁 pair ) and sample 𝑚 from a bounded interval around it: 𝑚 lo = max(𝑛 − 1, 𝑚 exp − Δ),

(4)

𝑚 hi = min(𝑁 pair − 1, 𝑚 exp + Δ),
𝑚 ∼ Unif {𝑚 lo, . . . , 𝑚 hi } ,

(5) (6)

where Δ is a small slack constant. The lower bound 𝑛 − 1 encourages connectivity, while the upper bound 𝑁 pair − 1 avoids producing a fully saturated graph. After fixing (𝑛, 𝑚), TRACE computes the target mean relation weight implied by Eq. (1): 𝑤 tar =

Dtar . 𝑛𝛼

(7)

Intuitively, 𝑤 tar specifies the average constraint complexity required at scale 𝑛 to match the target difficulty. To bias relation sampling toward this target, TRACE assigns each relation 𝑟 a probability based on the distance between 𝑤 (𝑟 ) and 𝑤 tar . Specifically, we use a Gaussian-shaped softmax:   (𝑤 (𝑟 ) − 𝑤 tar ) 2 𝑝 (𝑟 ) ∝ exp − (8) 2𝜎 2 so that relations with weights closer to 𝑤 tar are more likely to be selected. We then draw 𝑚 relations from this distribution to obtain an initial multiset L. Because finite sampling can deviate from the intended mean, TRACE measures the relative error between the achieved mean weight 𝑤 ach of L and 𝑤 tar : 𝜀 =

|𝑤 ach − 𝑤 tar | . 𝑤 tar

(9)

If 𝜀 exceeds a preset tolerance 𝜏, we apply a greedy calibration to adjust the multiset composition while keeping 𝑚 fixed. When 𝑤 ach > 𝑤 tar , we repeatedly replace one occurrence of a currently highest-weight relation with the minimum-weight relation; when 𝑤 ach < 𝑤 tar , we symmetrically replace one minimum-weight relation with a maximum-weight relation. This bounded adjustment monotonically moves 𝑤 ach toward 𝑤 tar and yields the final relation multiset L. Finally, TRACE computes the achieved difficulty Dach = 𝑛𝛼 · 𝑤 ach and outputs the candidate configuration (𝑛, 𝑚, L), which is instantiated into a path-consistent constraint graph in the next stage. 3.2.3 Constraint Graph Generation. Given the candidate constraint configuration C = (𝑛, 𝑚, L) derived in Section 3.2.2, our objective is to instantiate these parameters into a concrete, logically consistent temporal constraint graph G = (𝑉 , 𝐸). This process is non-trivial; a naive stochastic assignment of relations to arbitrary event pairs frequently induces logical contradictions (e.g., before before a cycle 𝐴 −−−−→ 𝐵 −−−−→ 𝐴), rendering the graph mathematically invalid. To address this, we propose a constructive generation procedure, formalized in Algorithm 2, which builds the network incrementally to guarantee satisfiability. The procedure initiates with Skeleton Construction. Our primary goal is to establish a connected backbone for the event set 𝑉 = {𝑣 1, . . . , 𝑣𝑛 } without introducing structural conflicts. To ensure that every event is reachable and integrated into the reasoning chain, we generate a random spanning tree 𝐸𝑡𝑟𝑒𝑒 consisting of 𝑛 − 1 edges based on a Prufer sequence.1 1 The Prufer sequence provides a bijection between the set of labeled trees on 𝑛 vertices and sequences of length 𝑛 − 2,

thereby ensuring an unbiased sampling of tree topologies. , Vol. 1, No. 1, Article . Publication date: July 2026.

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

9

Algorithm 2 Path-Consistent Constraint Graph Construction Require: Candidate configuration (𝑛, 𝑚, L), History signatures Σ. Ensure: Path-consistent constraint graph G, Canonical signature S. 1: Phase 1: Skeleton Construction 2: Initialize G ← (𝑉 , 𝐸 tree ) via random Prufer sequence spanning tree 3: Partition L into Lskel (size 𝑛 − 1) and Lrem 4: Map Lskel to 𝐸 tree ; Compute degrees deg(𝑣) for all 𝑣 ∈ 𝑉 5: Phase 2: Relation Augmentation 6: 𝑃 free ← Unconnected pairs sorted by ascending cost 𝑘 (𝑢, 𝑣) = deg(𝑢) + deg(𝑣) 7: for 𝑟 ∈ Lrem do 8: Find first (𝑢, 𝑣) ∈ 𝑃 free satisfying IsConsistent(G ∪ {(𝑢, 𝑣, 𝑟 )}) 9: if no valid pair found then return Failure 10: end if 11: Add (𝑢, 𝑣, 𝑟 ) to G; Update degrees; Remove (𝑢, 𝑣) from 𝑃free 12: end for 13: Phase 3: Isomorphism Elimination 14: if not IsConsistent(G) then return Failure 15: end if 16: S ← CanonicalHash(G) 17: if S ∈ Σ then return Failure 18: end if 19: return (G, S) Subsequently, we align the relation set with this topology. We partition the candidate relation multiset L into a backbone set L𝑠𝑘𝑒𝑙 and a remainder set L𝑟𝑒𝑚 : L = L𝑠𝑘𝑒𝑙 ⊎ L𝑟𝑒𝑚 ,

where |L𝑠𝑘𝑒𝑙 | = 𝑛 − 1.

(10)

Since a tree topology is inherently acyclic, any one-to-one assignment of relations from L𝑠𝑘𝑒𝑙 to the edges in 𝐸𝑡𝑟𝑒𝑒 is guaranteed to be path-consistent. Consequently, we assign the relations in L𝑠𝑘𝑒𝑙 directly to the skeleton edges, yielding a sparse but connected initial graph G0 = (𝑉 , 𝐸𝑡𝑟𝑒𝑒 ). Following the skeleton initialization, we proceed to Relation Augmentation. At this stage, the graph contains only 𝑛 − 1 edges; the target complexity requires embedding the remaining relations L𝑟𝑒𝑚 into the network. However, arbitrarily placing these constraints can lead to structural biases, where specific nodes accumulate disproportionate constraints, resulting in an uneven distribution of reasoning difficulty. To mitigate this, we dynamically prioritize unconnected pairs that link sparsely populated regions of the graph. Formally, for any candidate pair (𝑢, 𝑣) in the set of unconnected pairs 𝑃 𝑓 𝑟𝑒𝑒 , we define a selection cost 𝑘 (𝑢, 𝑣) based on the current nodal degrees: 𝑘 (𝑢, 𝑣) = deg(𝑢) + deg(𝑣).

(11)

We first sort 𝑃 𝑓 𝑟𝑒𝑒 in ascending order of 𝑘. Then, for each pending relation 𝑟 ∈ L𝑟𝑒𝑚 , we traverse the sorted candidate pairs to find the first (𝑢, 𝑣) that satisfies path consistency: IsConsistent(G ∪ {(𝑢, 𝑣, 𝑟 )}) = true.

(12)

If the check passes, the edge is committed, and (𝑢, 𝑣) is removed from the pool. This strategy promotes a uniform distribution of complexity by prioritizing structurally simpler connections, thereby maximizing the likelihood that all required relation types are successfully embedded. , Vol. 1, No. 1, Article . Publication date: July 2026.

10

Shide Zhou, Kailong Wang, Ling Shi, and Haoyu Wang

The final stage is Isomorphism Elimination. To ensure the benchmark evaluates diverse reasoning patterns rather than memorization of repetitive structures, we must filter out topologically identical graphs. First, we execute a global path consistency check to confirm that the accumulated local constraints result in a globally valid network. Subsequently, we compute a canonical signature S that identifies the graph’s topological structure. To guarantee dataset diversity, we compare S against a history set Σ and retain the graph only if its signature is novel (i.e., S ∉ Σ). If Algorithm 2 returns Failure during any phase, the framework discards only the current randomized attempt rather than the underlying candidate configuration. An outer loop continues to initiate independent trials until the target number of valid graphs is constructed. Naturally, higher difficulty levels impose stricter constraints, necessitating more attempts. For example, successfully generating 40 valid graphs at Difficulty 80 required approximately 800 attempts. Because constraint graph generation is a one-time, parallelizable offline preprocessing step, these discarded attempts incur minimal computational overhead and do not impact the practical usability of the benchmark. 3.3

Temporal Reasoning Task Construction

3.3.1 Extraction of Facts and Questions. Based on the temporal constraint graph G = (𝑉 , 𝐸) generated in Section 3.2.3, TRACE systematically constructs a dataset of reasoning tasks. For each graph, we formalize the output as a structure T = (F , Q, A), consisting of a shared set of premise facts F , a set of candidate natural language queries Q, and their corresponding ground truth answers A. The construction process begins with Fact Extraction, which establishes the explicit context F . The edges 𝐸 in the constraint graph represent the known temporal premises. For every explicit constraint (𝑢, 𝑣, 𝑟 ) ∈ 𝐸, where 𝑟 ∈ R𝐴𝑙𝑙𝑒𝑛 is the symbolic relation (e.g., O), we map it to its corresponding natural language descriptor 𝑟 𝑛𝑙 (e.g., "overlaps") using a predefined lexicon. The collection of these textual statements constitutes the fact set F , providing the necessary and sufficient conditions to solve the graph. To formulate meaningful questions that require deductive reasoning rather than simple retrieval, we employ Constraint Propagation. We execute the path consistency algorithm on G to compute the transitive closure of the network. Let R𝑖𝑛𝑓 𝑒𝑟𝑟𝑒𝑑 (𝑢, 𝑣) denote the set of permissible relations between any two events 𝑢 and 𝑣 after propagation. We iterate through all event pairs and filter out those already explicitly defined in 𝐸. The remaining pairs represent implicit relationships that can only be determined by inferring through the chain of facts in F . From these implicit pairs, we select candidates for Question Formulation to populate Q and A. To ensure precise evaluation, we focus strictly on deterministic inferences, which are cases where the interaction of constraints narrows the relationship down to a single possibility. A pair (𝑢, 𝑣) is selected if and only if |R𝑖𝑛𝑓 𝑒𝑟𝑟𝑒𝑑 (𝑢, 𝑣)| = 1. Let 𝑟𝑡𝑟𝑢𝑒 be the unique inferred relation. For each standard Allen relation 𝑟𝑘 ∈ R𝐴𝑙𝑙𝑒𝑛 , we construct a binary verification question 𝑞𝑘 : “Is 𝑢 𝑟𝑘𝑛𝑙 𝑣?”. The corresponding ground truth label 𝑎𝑘 is determined as: ( YES if 𝑟𝑘 = 𝑟𝑡𝑟𝑢𝑒 𝑎𝑘 = . (13) NO otherwise The collection of all generated 𝑞𝑘 constitutes the query set Q, and the corresponding 𝑎𝑘 form the answer set A. This mechanism generates a diverse pool of candidate queries for each inferred fact. 3.3.2 Question Templatization. To transform the logical task T into an executable input for Large Language Models, we employ a standardized prompt template. This design not only provides the model with the necessary context, including formal definitions of the 13 Allen relations, but also strictly enforces a structured output format. By requiring the model to generate a JSON object , Vol. 1, No. 1, Article . Publication date: July 2026.

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

11

containing a step-by-step "reasoning" trace alongside the "final_answer", we enable the parser to extract and verify the underlying logic in the subsequent stage. Specifically, the prompt is structured into six components. The # Constraint section defines the reasoner’s role and provides context along with restrictions, while # Allen relations provides formal interval definitions to ensure semantic precision. The # Facts and # Question sections present the explicit premises and the target inference query, respectively. Subsequently, # Instructions enforces the strict JSON schema for the reasoning trace, and # Example provides a concrete demonstration of the expected output format to ensure parsing compatibility. An abbreviated example of the prompt structure is illustrated below, where [FACTS] is instantiated with statements such as “A after C, B during C,” and [QUESTION] is instantiated with questions such as “Is A after B?” The complete prompt templates and comprehensive task examples are publicly available on our project website [26]. Prompt Template # Constraint You are a temporal reasoner. Use ONLY event IDs (A, B, C, ...) and these 13 Allen relations: {before, after, meets, met-by, ...}. Do not invent events/relations beyond what is stated or logically entailed. # Allen relations Let an event X be an interval [Xs, Xe] with Xs < Xe. ... - equals: A equals B < − > As = Bs and Ae = Be. # Facts: [FACTS] # Question: [QUESTION] # Instructions: 1) Produce a step-by-step reasoning as a JSON array named "reasoning". Each step is an object { "lhs": "X", "rel": "allen_relation", "rhs": "Y" } ... # Example { "reasoning": [ ... ], "final_answer": "YES" }

3.4 Trace-Based Verification Oracle To rigorously assess whether the model derives the correct answer through valid temporal logic rather than statistical shortcuts, we propose a trace-based verification mechanism. This module executes the reasoning tasks, parses the structured traces, and validates the logical soundness of each derivation step against the ground truth constraints. The evaluation process begins by feeding the generated prompts into the target LRM. Under the constraints imposed by the prompt template, the LRM produces an internal thinking process (e.g., within <think>...</think>) and then outputs a structured JSON object that contains a step-by-step reasoning chain and the final answer. To handle potential formatting irregularities common in LLM outputs, we employ a robust regex-based parser to extract the reasoning trace 𝑇 = [𝑠 1, 𝑠 2, . . . , 𝑠𝑘 ] from the JSON object. Each step 𝑠𝑖 is formalized as a triplet (𝑢𝑖 , 𝑟𝑖 , 𝑣𝑖 ), representing a claim that "Event 𝑢𝑖 has relation 𝑟𝑖 to Event 𝑣𝑖 ." Simultaneously, the predicted final answer is extracted and normalized into a standard label 𝑦ˆ ∈ {YES, NO, Unsure}. Unlike traditional methods that typically compare model outputs against a predefined reference or rule-based solution, TRACE verifies each reasoning step against the algebraic closure implied by the ground-truth constraint network. For every extracted triplet 𝑠𝑖 = (𝑢𝑖 , 𝑟𝑖 , 𝑣𝑖 ) from the reasoning chain, we invoke the underlying path-consistency solver on the ground truth graph G to compute the propagated relation set R 𝑣𝑎𝑙𝑖𝑑 (𝑢𝑖 , 𝑣𝑖 ), which contains all Allen relations that remain feasible between 𝑢𝑖 and 𝑣𝑖 after constraint propagation from the premise facts. To align verification with b𝑖 : the semantics of Allen’s algebra, we interpret the model’s predicted relation 𝑟𝑖 as a relation set R b𝑖 = {𝑟𝑖 }; if the model outputs a disjunctive form such as if 𝑟𝑖 is a single basic Allen relation, then R , Vol. 1, No. 1, Article . Publication date: July 2026.

12

Shide Zhou, Kailong Wang, Ling Shi, and Haoyu Wang

B|M, we parse it as the corresponding union of basic relations. A step 𝑠𝑖 is considered valid if and only if the model’s claimed relation set matches the solver-implied closure exactly: b𝑖 = R 𝑣𝑎𝑙𝑖𝑑 (𝑢𝑖 , 𝑣𝑖 ). R

(14)

This strict criterion filters out ambiguous or incorrect intermediate claims, while avoiding any dependence on a single canonical proof. In other words, any derivation route is accepted as long as every step agrees with the relation set implied by the constraint-solver closure. Based on these verification results, we classify the model’s performance into four distinct categories to provide a comprehensive faithfulness assessment. We define the correctness of the reasoning chain, 𝐶𝑟𝑒𝑎𝑠 , as a strict conjunction where 𝐶𝑟𝑒𝑎𝑠 = 1 if and only if all steps in 𝑇 are valid. Combining this with the binary correctness of the final answer (𝐶𝑎𝑛𝑠 ), we define the following metrics: • True Reasoning (𝐶𝑎𝑛𝑠 = 1, 𝐶𝑟𝑒𝑎𝑠 = 1): The model correctly answers the question based on a logically valid derivation chain. • Spurious Guessing (𝐶𝑎𝑛𝑠 = 1, 𝐶𝑟𝑒𝑎𝑠 = 0): The model predicts the correct label, but the reasoning process contains at least one logical error. • Answer Misalignment (𝐶𝑎𝑛𝑠 = 0, 𝐶𝑟𝑒𝑎𝑠 = 1): The model performs sound logical derivation throughout the reasoning chain but produces an incorrect final answer label. • Complete Failure (𝐶𝑎𝑛𝑠 = 0, 𝐶𝑟𝑒𝑎𝑠 = 0): The model fails in both the reasoning process and the final prediction. 4

Evaluation

In this section, we evaluate the performance of TRACE. We first outline the experimental setup, including the evaluated LRMs and the dataset generation configuration. Specifically, our evaluation addresses the following research questions: • RQ1: How effective is TRACE in generating temporal reasoning tasks with controllable difficulty? • RQ2: How do different LLM architectures and sizes perform on the temporal reasoning benchmark TRACEBench? • RQ3: To what extent do traditional outcome-based metrics overestimate the reasoning faithfulness of LLMs? • RQ4: What are the characteristic failure modes of LLMs in complex temporal reasoning tasks? 4.1

Evaluation Setup

4.1.1 Evaluated Models. We select a representative set of Large Reasoning Models for evaluation, covering both open-weights distilled models and advanced models. Specifically, we include: • Open-Weights Distilled Models: We evaluate the DeepSeek-R1-Distill family to analyze the impact of model scale and base architecture. This includes the Qwen-based variants (7B, 14B, 32B) and the Llama-based variant (Llama-8B). All distilled models are deployed locally. • Advanced Models: We evaluate Gemini-2.5-Flash, DeepSeek-R1, GPT-5-mini, and Claude-Sonnet-4.6 via their respective APIs. These models serve as the topline performance baseline. All models are configured with a maximum generation length of 8,192 tokens. To minimize answer stochasticity, the temperature is set to 0 for all models that support this parameter (for some API-based models that do not support temperature settings, we use their default configuration). , Vol. 1, No. 1, Article . Publication date: July 2026.

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

13

4.1.2 Dataset Configuration. We utilize TRACE to construct TRACEBench, a temporal reasoning benchmark spanning multiple levels of complexity. We generate reasoning tasks across six distinct difficulty levels: Dtar ∈ {10, 45, 80, 115, 150, 185}. This progression ranges from elementary transitive inference (Difficulty 10) to highly complex temporal networks (Difficulty 185). For each difficulty tier, we generate 40 distinct constraint graphs and sample 200 reasoning questions in total (across these graphs), resulting in 1,200 test samples. To ensure structural consistency across difficulties, we fix the constraint density 𝜌 = 0.5 and enforce a strict difficulty alignment with a tolerance threshold 𝜏 = 0.1. 4.2

RQ1: Analysis of Difficulty Controllability

Table 3. Average Nodes, Edges, and Achieved Difficulty of Constraint Graphs for Target Difficulty from 10 to 185

Target Difficulty

Metric

True Reasoning Accuracy (%)

Nodes Edges Achieved Difficulty

D-10

D-45

D-80

D-115

D-150

D-185

3.00 2.00 9.36

7.00 10.18 44.97

10.00 22.03 81.14

12.00 32.98 114.33

14.00 45.98 148.32

16.00 59.98 185.40

DeepSeek-R1-Distill-Llama-8B (r=-0.98) DeepSeek-R1-Distill-Qwen-7B (r=-0.86) DeepSeek-R1-Distill-Qwen-14B (r=-0.99) DeepSeek-R1-Distill-Qwen-32B (r=-0.95) Gemini-2.5-Flash (r=-0.86)

100

DeepSeek-R1 (r=-0.85) GPT-5-mini (r=-0.94) Claude-Sonnet-4.6 (r=-0.92) Average (r=-0.96)

80 60 40 20 0

10

45

80

Difficulty

115

150

185

Fig. 2. True Reasoning Accuracy of Evaluated LRMs Across Six Target Difficulty Levels. The bars represent individual model performance, while the dashed black line indicates the average accuracy trajectory. Pearson correlation coefficients (𝑟 ) are reported in the legend to quantify the negative correlation between difficulty and True Reasoning Accuracy.

To verify whether TRACE can effectively generate temporal reasoning tasks with controllable difficulty, we conduct a two-step validation. First, we calculate the structural statistics of the generated constraint graphs to verify their alignment with our difficulty specifications. Second, we evaluate the performance of eight different LRMs on these datasets across varying difficulty levels to see if the theoretical difficulty score effectively reflects the relative reasoning load on the models, as indicated by actual performance drops. Structural Analysis of Constraint Graphs. Table 3 presents the statistics of the generated graphs. As the target difficulty increases from 10 to 185, the graph size increases in line with the , Vol. 1, No. 1, Article . Publication date: July 2026.

14

Shide Zhou, Kailong Wang, Ling Shi, and Haoyu Wang

configuration. The average number of nodes grows from 3.00 to 16.00, and the number of edges increases from 2.00 to 59.98. The actual achieved difficulty scores are almost identical to the target scores. For example, at the target difficulty of 45, the achieved difficulty is 44.97, and at target difficulty 185, it is 185.40. This confirms that the generation algorithm precisely follows the target difficulty settings. Verification of Difficulty-Performance Correlation. Figure 2 illustrates the True Reasoning Accuracy of different models across these difficulty levels. We observe a consistent downward trend for all models as the difficulty increases. For instance, the accuracy of DeepSeek-R1-DistillQwen-32B drops from 67.50% at difficulty 10 to 29.50% at difficulty 185. Similarly, the smaller model DeepSeek-R1-Distill-Llama-8B starts at 46.00% and drops to 15.50% at the hardest level. Even the high-performing GPT-5-mini shows a clear decline, dropping from 93.50% to 72.50%. The Pearson correlation coefficients further quantify this trend, ranging from -0.85 to -0.99 across all models, with an average of -0.96. This strong negative correlation demonstrates that our metric effectively serves as a relative indicator of the actual reasoning load imposed on the models. Answer to RQ1: TRACE generates temporal reasoning tasks with precise difficulty control. The structural statistics of the generated graphs closely correspond to the target difficulty settings. Furthermore, the performance of all evaluated LRMs shows a strong negative correlation (𝑟 𝑎𝑣𝑔 = −0.96) with the relative difficulty metric, demonstrating that the generated tasks effectively provide a controllable gradient of difficulty. 4.3

RQ2: Benchmarking LLM Performance

To address RQ2, we evaluate the performance of eight LRMs in TRACEBench. The goal is to benchmark their temporal reasoning capabilities and understand the impact of different model architectures and parameter scales on temporal reasoning performance. Figure 3 presents a finegrained breakdown of model outputs, with the green bar (True Reasoning) serving as the primary indicator of effective performance. Impact of Parameter Scaling. First, by examining the DeepSeek-R1-Distill-Qwen series, we observe a clear positive correlation between temporal reasoning capability and model scale. At the lowest difficulty (Difficulty 10), the performance gap among the three models is relatively small. The 7B, 14B, and 32B models achieve True Reasoning accuracies of 54.50%, 67.00%, and 67.50%, respectively. However, as the difficulty increases, the 7B model’s performance degrades sharply. In contrast, the 14B and 32B models demonstrate much stronger resilience. Although their accuracy also declines, the gap between the two larger models remains narrow, with both significantly outperforming the 7B variant. Overall, the 32B model demonstrates the best robustness, achieving an average accuracy of 45.42% across all six difficulty levels, followed by the 14B model at 41.08%, and the 7B model at 17.42%. Impact of Model Architecture. Second, we examine the influence of base architecture by comparing DeepSeek-R1-Distill-Llama-8B with the DeepSeek-R1-Distill-Qwen series. At the lowest difficulty (Difficulty 10), the Qwen-7B model holds a performance advantage, achieving 54.50% True Reasoning Accuracy compared to 46.00% for Llama-8B. However, as the task complexity escalates, Llama-8B demonstrates significantly stronger robustness. Unlike Qwen-7B, which suffers a steep decline, Llama-8B maintains a more stable degradation curve. As difficulty increases, the performance of Llama-8B progressively converges with that of the much larger DeepSeek-R1Distill-Qwen-14B. The accuracy gap between them narrows consistently from 21.00% at Difficulty 10 to 16.50%, 14.50%, 9.00%, and 6.00% across intermediate levels, ultimately shrinking to just 5.00% at Difficulty 185. , Vol. 1, No. 1, Article . Publication date: July 2026.

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

Breakdown Percentage (%)

True Reasoning Spurious Guessing

Answer Misalignment Complete Failure

Difficulty 10

100

Breakdown Percentage (%)

Parse Fail

Difficulty 45

80 60 40 20 0

i a-8B wen-7B en-14B en-32Bmini-2.5Seek-R1 T-5-min ude-4.6 Q Qw Qw Ge eep GP Cla D

Llam

i a-8B wen-7B en-14B en-32Bmini-2.5Seek-R1 T-5-min ude-4.6 Q Qw Qw Ge eep GP Cla D

Llam

Difficulty 80

100

Difficulty 115

80 60 40 20 0

i a-8B wen-7B en-14B en-32Bmini-2.5Seek-R1 T-5-min ude-4.6 Q Qw Qw Ge eep GP Cla D

Llam

Breakdown Percentage (%)

15

i a-8B wen-7B en-14B en-32Bmini-2.5Seek-R1 T-5-min ude-4.6 Q Qw Qw Ge eep GP Cla D

Llam

Difficulty 150

100

Difficulty 185

80 60 40 20 0

i a-8B wen-7B en-14B en-32Bmini-2.5Seek-R1 T-5-min ude-4.6 Q Qw Qw Ge eep GP Cla D

Llam

i a-8B wen-7B en-14B en-32Bmini-2.5Seek-R1 T-5-min ude-4.6 Q Qw Qw Ge eep GP Cla D

Llam

Fig. 3. Fine-Grained Performance Breakdown of Evaluated LRMs Across Six Difficulty Levels. The 100% stacked bar charts visualize the composition of model outputs, categorized into True Reasoning, Spurious Guessing, Answer Misalignment, Complete Failure, and Parse Fail.

Answer to RQ2-1: Reasoning performance generally improves with model scale. However, the base architecture is a critical factor: a smaller model (e.g., Llama-8B) can achieve performance that closely approaches that of larger models (e.g., Qwen-14B) as task difficulty increases. Evaluation of Model Capability and Boundaries. Finally, we integrate the analysis of response categories with performance boundaries to evaluate the capability thresholds of different models. First, regarding small-scale models, specifically DeepSeek-R1-Distill-Llama-8B and DeepSeekR1-Distill-Qwen-7B, we observe a distinct "Answer Misalignment" phenomenon. Across all six difficulty levels, these two models exhibit the highest rate of misalignment, averaging 17.83 and 16.17 samples per level, respectively. This indicates that while they may occasionally derive correct reasoning, they struggle to map this logic to the final label consistently. In terms of performance boundaries, these models hit their limits early. DeepSeek-R1-Distill-Qwen-7B drops rapidly, with its True Reasoning Accuracy dropping to just 8.00% at Difficulty 80. While DeepSeek-R1-DistillLlama-8B is relatively more robust, it also struggles at the extreme, achieving only 15.50% True Reasoning Accuracy at Difficulty 185. , Vol. 1, No. 1, Article . Publication date: July 2026.

16

Shide Zhou, Kailong Wang, Ling Shi, and Haoyu Wang

Second, for mid-sized models like DeepSeek-R1-Distill-Qwen-14B and DeepSeek-R1-DistillQwen-32B, the primary issue shifts to "Spurious Guessing." These models exhibit a pronounced tendency to predict correct labels without valid reasoning, averaging 56 and 52 spurious samples per level, respectively (the highest among all models). This behavior leads to a significant inflation of performance metrics when relying solely on answer correctness, a discrepancy we quantify in Section 4.4. Regarding performance boundaries, these models demonstrate extended capability but eventually succumb to complexity. At the maximum difficulty of 185, the True Reasoning Accuracy for the 14B and 32B models falls to 20.50% and 29.50%, respectively, indicating a clear ceiling in their ability to handle complex temporal dependencies. Finally, the advanced models establish the performance ceiling for this benchmark, though they exhibit varying operational boundaries. Claude-Sonnet-4.6, DeepSeek-R1, and GPT-5-mini demonstrate exceptional capabilities with average True Reasoning Accuracies of 83.00%, 80.50%, and 79.83%, respectively. Under extreme conditions, their stability diverges. Both Claude-Sonnet4.6 and GPT-5-mini exhibit strong architectural stability. At Difficulty 185, they maintain high accuracies of 75.50% and 72.50%, and neither model produces any Parse Failures throughout the benchmark, indicating robust instruction compliance. In contrast, while DeepSeek-R1 achieves a highly competitive overall average, it encounters a sharper performance drop at the maximum difficulty, falling to 60.50%. This degradation is primarily driven by context window bottlenecks, where excessive reasoning chain lengths lead to Parse Failures, a specific failure mechanism that we dissect in Section 4.5. Gemini-2.5-Flash performs slightly below this top tier, with an average accuracy of 58.75%. Moreover, its performance degrades rapidly as task difficulty increases, placing it between mid-sized open-weights models and the highest-performing advanced models. Answer to RQ2-2: We identify distinct failure modes across model sizes. Small models exhibit higher levels of Answer Misalignment, while mid-sized models are prone to Spurious Guessing. Advanced models remain robust but encounter difficulties at extreme complexity, largely due to the challenge of processing extended reasoning contexts. 4.4

RQ3: Assessment of Reasoning Faithfulness

To address RQ3, we investigate the prevalence of Spurious Guessing, which refers to instances where models predict the correct final label despite failing to provide a valid reasoning trace. This analysis aims to quantify the discrepancy between outcome-based accuracy and true reasoning performance. If a model relies on statistical shortcuts rather than logical deduction, outcome-based metrics serve as an inflated proxy for actual performance. Table 4 illustrates the Spurious Guessing Rate across different difficulty levels. We observe three distinct behavioral patterns: High Overestimation in Mid-Sized Models. The most significant inflation of performance metrics appears in mid-sized distilled models. DeepSeek-R1-Distill-Qwen-14B and 32B exhibit consistently high spurious guessing rates, averaging 28.00% and 26.00% respectively across all difficulties. Notably, the 14B model’s spurious rate climbs with complexity, peaking at 35.00% at Difficulty 150. This indicates that these models possess sufficient parameter capacity to exploit statistical shortcuts for label prediction, yet they frequently fail to construct sound and complete reasoning traces. Consequently, relying solely on answer correctness for this model class yields the most misleading assessment. Instability in Small Models. The smaller DeepSeek-R1-Distill-Qwen-7B displays a volatile "early-peak" pattern. Its spurious guessing surges to 28.50% at Difficulty 45 but declines sharply as complexity increases. This decline does not reflect improved faithfulness; rather, it aligns with , Vol. 1, No. 1, Article . Publication date: July 2026.

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

17

Table 4. Spurious Guessing Rate Across Six Difficulty Levels. Llama-8B and Qwen-7B/14B/32B refer to DeepSeek-R1-Distill models. Target Difficulty

Model D-10

D-45

D-80

D-115

Llama-8B 18.50% Qwen-7B 7.50% Qwen-14B 16.00% Qwen-32B 25.00% Gemini-2.5-Flash 7.00% DeepSeek-R1 2.00% GPT-5-mini 4.50% Claude-Sonnet-4.6 3.50%

27.00% 28.50% 24.00% 23.00% 18.50% 6.50% 10.50% 14.50%

22.50% 24.50% 31.50% 25.00% 20.50% 8.50% 13.50% 14.00%

21.50% 28.50% 24.50% 19.50% 15.50% 11.00% 30.50% 35.00% 31.00% 33.00% 26.50% 23.50% 24.50% 24.00% 25.00% 15.50% 12.50% 2.00% 13.00% 16.50% 15.00% 12.50% 17.00% 22.00%

D-150

D-185

the performance collapse observed in RQ2. As the model becomes unable to predict correct labels reliably under high complexity, the occurrence of spurious successes naturally diminishes. High Faithfulness in Advanced Models. In contrast, DeepSeek-R1 demonstrates superior reasoning faithfulness. Its spurious guessing rate remains low, with an average of 7.83%, showing only minor fluctuations in the intermediate difficulty range. This confirms that advanced models are highly consistent: when they output a correct answer, it is almost invariably grounded in a valid logical derivation. Similarly, GPT-5-mini and Claude-Sonnet-4.6 maintain relatively low and stable spurious rates (averaging 12.17% and 13.92%, respectively), further supporting that stronger reasoners depend less on guesswork. Gemini-2.5-Flash also shows a moderate spurious rate of 19.92%, remaining more faithful than the mid-sized distilled models. Answer to RQ3: Traditional outcome-based metrics significantly overestimate model performance, particularly for mid-sized models which show the highest average spurious guessing rates (up to 28.00%). In contrast, advanced models demonstrate high consistency between answer correctness and logical validity, highlighting the necessity of TRACE’s trace-based verification for precise evaluation. 4.5

RQ4: Diagnosis of Failure Modes

To address RQ4, we investigate the characteristic failure modes of LLMs in complex temporal reasoning tasks. We conduct a manual inspection of a representative subset of failure cases sampled from the experimental results. Based on this analysis, we summarize the observed errors into two primary categories: Logical Failures, where models generate valid formats but yield incorrect reasoning or answers, and Structural Failures, where models fail to produce parsable outputs due to format violations or context limitations. 4.5.1 Logical Failures. Through a detailed inspection of samples where models produce validly formatted outputs but yield incorrect reasoning or answers, we categorize the observed logical failures into three primary classes: Direct Inference Error, Reasoning Stagnation, and Hallucination under Complexity. The first two types represent deficiencies in reasoning capability and efficiency. The most common error across all models is Direct Inference Error. In this case, the model draws an incorrect conclusion from the given premises by misapplying transitivity or temporal constraints, which leads to an incorrect final answer. The second type is Reasoning Stagnation. In these cases, the model generates a lengthy reasoning trace that appears structurally valid but merely rephrases , Vol. 1, No. 1, Article . Publication date: July 2026.

18

Shide Zhou, Kailong Wang, Ling Shi, and Haoyu Wang

or permutes the provided facts. This failure to progress logically prevents the model from deriving the target conclusion, effectively forcing it to guess the final answer despite a long chain of thought. The third type is Hallucination under Complexity, which manifests as a breakdown in the model’s grounding to the provided context. We observe that models occasionally introduce undefined elements into the JSON reasoning trace, including non-existent events (e.g., "As", "?") or non-standard relations that are not defined in the prompt (e.g., "<", "=", "less_than", "cannot overlap"). This phenomenon appears across all model scales, suggesting that under high complexity, models may hallucinate external knowledge or symbols to bridge logical gaps. Table 5. Distribution of Logical Failure Modes Across Evaluated LRMs.

Model Llama-8B Qwen-7B Qwen-14B Qwen-32B Gemini-2.5-Flash DeepSeek-R1 GPT-5-mini Claude-Sonnet-4.6

Direct Inference Error Reasoning Stagnation 68.92% 50.00% 81.57% 94.77% 96.83% 79.65% 89.26% 96.08%

15.97% 17.65% 9.41% 4.04% 2.68% 3.54% 9.09% 3.92%

Hallucination 15.10% 32.35% 9.02% 1.19% 0.49% 16.81% 1.65% 0.00%

To further substantiate these observations, we developed automated scripts based on our initial manual inspection to classify all Logical Failures. As detailed in Table 5, the quantitative distribution confirms our qualitative analysis. Direct Inference Error is the overwhelmingly dominant failure mode, accounting for at least 50% of logical errors across every evaluated model. Meanwhile, Reasoning Stagnation remains a persistent baseline issue across the distilled models. Finally, while Hallucination under Complexity affects all scales, smaller architectures like Qwen-7B struggle with it most significantly (32.35%). A representative example of this hallucinatory behavior is illustrated below: Case Study: Hallucination in Reasoning Trace (GPT-5-mini, Difficulty = 115) ... {"lhs": "J", "rel": "overlapped-by", "rhs": "K"}, {"lhs": "Cs", "rel": "less_than", "rhs": "Gs"}, {"lhs": "Cs", "rel": "less_than", "rhs": "Hs"} ...

Answer to RQ4-1: Logical failures manifest in three distinct modes: (1) Direct Inference Errors, which are the most prevalent; (2) Reasoning Stagnation, where models parrot facts without logical progress; and (3) Hallucinations, where models invent undefined events or relations. 4.5.2 Structural Failures. Through an inspection of samples where models failed to produce parsable outputs, we categorize the observed structural failures into two primary classes: Format Non-Compliance and Context Window Exhaustion. These failures represent a breakdown in the model’s instruction-following capabilities and resource management when facing extreme reasoning complexity. , Vol. 1, No. 1, Article . Publication date: July 2026.

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

19

The first type is Format Non-Compliance, which is predominantly observed in small and mid-sized models. In these instances, the model fails to strictly adhere to the required JSON schema. The second type is Context Window Exhaustion, where the generation is truncated due to exceeding the token limit. We identify two distinct mechanisms driving this exhaustion. The first mechanism is Reasoning Explosion, primarily driven by the exponential growth of the necessary inference chain as difficulty rises. Notably, DeepSeek-R1’s context failures are primarily attributed to this phenomenon. The incidence spikes dramatically at Difficulty 185, affecting 74 out of 200 samples, whereas other difficulty levels record a maximum of only 18 cases. The second mechanism is Degenerative Loops, typically found in small and mid-sized distilled models, as well as Gemini-2.5-Flash. As task difficulty escalates, these models often fall into a "cognitive collapse," infinitely repeating a single reasoning step or phrase until the context window is exhausted. This phenomenon is similar to observations reported in prior studies [4, 21]. Table 6. Distribution of Structural Failure Modes Across Evaluated LRMs (Based on sampled parse-failure cases).

Model Llama-8B Qwen-7B Qwen-14B Qwen-32B Gemini-2.5-Flash DeepSeek-R1

Format Non-Compliance

Reasoning Explosion

Degenerative Loops

20.00% 10.00% 0.00% 0.00% 0.00% 0.00%

5.00% 15.00% 5.00% 10.00% 0.00% 100.00%

75.00% 75.00% 95.00% 90.00% 100.00% 0.00%

To quantify these structural breakdowns, we manually inspected 20 randomly sampled parsefailure cases per model (excluding GPT-5-mini and Claude-Sonnet-4.6, which recorded zero parse failures). As summarized in Table 6, format issues are primarily confined to smaller models. Degenerative Loops constitute the most severe problem for open-weights distilled models and Gemini-2.5-Flash, whereas DeepSeek-R1’s structural failures are entirely (100.00%) driven by Reasoning Explosion. A representative example of such a loop is illustrated below: Case Study: Degenerative Loop (DeepSeek-R1-Distill-Qwen-7B, Difficulty = 185) ... But J is equal to K, which is from I's start to I's end. So I's start is before A's end, and I's end is J's end, which is K's end. So I is from (before A's end) to K's end. But K is equal to J, which is equal to I's finish. So I's finish is K's end. So I is from (before A's end) to K's end. But K is equal to J, which is equal to I's finish. ... [Repeated indefinitely until context limit]

Answer to RQ4-2: Structural failures reveal scale-dependent limitations. Format NonCompliance is mainly concentrated in small models, whereas Degenerative Loops are prevalent in small and mid-sized models as well as Gemini-2.5-Flash. In contrast, advanced models like DeepSeek-R1 primarily suffer from Reasoning Explosion, driven by valid but excessive deductive chains. , Vol. 1, No. 1, Article . Publication date: July 2026.

20

5

Shide Zhou, Kailong Wang, Ling Shi, and Haoyu Wang

Discussion

The Shift to Process-Centric Verification. This study highlights a fundamental limitation in current benchmarks: the inability to distinguish between genuine deductive reasoning and the "Clever Hans" effect, where models rely on statistical correlations rather than logic. Our findings suggest that as LRMs integrate into high-stakes domains, outcome-based accuracy is an insufficient metric for safety and reliability. Effective evaluation requires treating reasoning as a verifiable trajectory rather than a singular output. TRACE demonstrates that rigorous, trace-level auditing serves as a pivotal mechanism to ensure models are not merely mimicking logic, but actively executing it. Guidance for LRM Development. Our diagnosed failure modes may inform future LRM training and alignment. First, to address Spurious Guessing, developers can introduce Process Reward Models alongside widely-used Outcome Reward Models to explicitly supervise intermediate deductive steps, rather than relying solely on final answers. Second, to mitigate Answer Misalignment, strict consistency penalties should be applied for mismatches between the generated reasoning trace and the final label. Finally, for Reasoning Explosion, a task-complexity estimation mechanism can be introduced to dynamically allocate and constrain the length of the CoT, thereby effectively reducing redundancy and context exhaustion. Evaluating Pure Reasoning. TRACE focuses exclusively on the intrinsic logical capabilities of LRMs. While practical applications often delegate complex constraint satisfaction to external solvers, allowing such tool augmentation in this context would shift the evaluation target from intrinsic deduction to tool-use proficiency. By enforcing a strict, self-contained reasoning environment across varying complexities, TRACE functions as a targeted diagnostic instrument. It is precisely this isolation that allows us to expose and analyze fundamental cognitive bottlenecks, such as degenerative loops and reasoning explosions, which would otherwise remain masked if the reasoning load were offloaded. Trade-offs in Controlled Synthesis. To ensure precise difficulty gradients, we prioritize logical control over naturalistic diversity. By synthesizing tasks from strictly defined algebraic rules, we isolate pure deductive reasoning capacity from the confounding variables of natural language. We acknowledge that this design lacks the semantic ambiguity, linguistic nuance, and "noisy" context inherent in real-world communication. Consequently, TRACE serves as a diagnostic instrument for intrinsic reasoning robustness, rather than a complete substitute for benchmarks grounded in unstructured, open-domain scenarios. 6

Conclusion

In this work, we introduce TRACE, a framework leveraging Allen’s Interval Algebra to generate difficulty-controllable tasks for benchmarking LRMs. Based on this framework, we construct TRACEBench, a comprehensive graded benchmark designed to systematically probe reasoning boundaries. Our evaluation confirms a precise alignment between task complexity and model performance, while exposing the inadequacy of outcome-based metrics due to the prevalence of Spurious Guessing in mid-sized models. Furthermore, we diagnose distinct, scale-dependent failure modes under extreme complexity: Degenerative Loops in smaller architectures and Reasoning Explosion in advanced models. These findings demonstrate the effectiveness of TRACE and establish a rigorous foundation for assessing the true reasoning boundaries of AI systems. 7

Data Availability

The code, data, and additional information relevant to this study are available at https://anonymous. 4open.science/r/TRACE-2061 and our project website [26]. , Vol. 1, No. 1, Article . Publication date: July 2026.

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

21

References [1] James F. Allen. 1983. Maintaining knowledge about temporal intervals. Commun. ACM 26, 11 (Nov. 1983), 832–843. doi:10.1145/182.358434 [2] Christian Bessière. 1996. A Simple Way to Improve Path Consistency Processing in Interval Algebra Networks. In Proceedings of the Thirteenth National Conference on Artificial Intelligence and Eighth Innovative Applications of Artificial Intelligence Conference, AAAI 96, IAAI 96, Portland, Oregon, USA, August 4-8, 1996, Volume 1, William J. Clancey and Daniel S. Weld (Eds.). AAAI Press / The MIT Press, 375–380. http://www.aaai.org/Library/AAAI/1996/aaai96-057.php [3] Bradley C. A. Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. 2024. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. CoRR abs/2407.21787 (2024). arXiv:2407.21787 doi:10.48550/ARXIV.2407.21787 [4] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2024. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs. CoRR abs/2412.21187 (2024). arXiv:2412.21187 doi:10.48550/ARXIV.2412.21187 [5] Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2024. TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 1204–1228. doi:10.18653/V1/2024.ACL-LONG.66 [6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. CoRR abs/2110.14168 (2021). arXiv:2110.14168 https://arxiv.org/abs/2110.14168 [7] DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. CoRR abs/2412.19437 (2024). arXiv:2412.19437 doi:10.48550/ARXIV. 2412.19437 [8] DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. CoRR abs/2501.12948 (2025). arXiv:2501.12948 doi:10.48550/ARXIV.2501.12948 [9] Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating Large Language Models in Class-Level Code Generation. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 81:1–81:13. doi:10.1145/3597503.3639219 [10] Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. 2025. Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https://openreview.net/forum?id=44CoQe6VCq [11] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-Based Prompting for Multi-step Reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview.net/forum?id=yf1icZHC-l9 [12] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. https://openreview.net/forum?id=d7KBjmI3GmQ [13] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, Joaquin Vanschoren and Sai-Kit Yeung (Eds.). https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html [14] Amar Isli and Hachemi Bennaceur. 1996. Networks of Qualitative Interval Relations: Combining Circuit Consistency and Path Consistency in the Search for a Solution. In Proceedings of the Third International Workshop on Temporal Representation and Reasoning, TIME-96, Key West, Florida, USA, May 19-20, 1996, Luca Chittaro, Scott D. Goodwin, Howard J. Hamilton, and Angelo Montanari (Eds.). IEEE Computer Society, 57–62. doi:10.1109/TIME.1996.555682 [15] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). http://papers.nips.cc/paper_files/paper/ 2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html [16] Brenden M. Lake and Marco Baroni. 2018. Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 (Proceedings of Machine Learning Research, Vol. 80), Jennifer G. Dy and Andreas Krause (Eds.). PMLR, 2879–2888. http://proceedings.mlr.press/v80/lake18a.html , Vol. 1, No. 1, Article . Publication date: July 2026.

22

Shide Zhou, Kailong Wang, Ling Shi, and Haoyu Wang

[17] Jeongsoo Lee, Daeyong Kwon, and Kyohoon Jin. 2025. GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluation. CoRR abs/2508.16994 (2025). arXiv:2508.16994 doi:10.48550/ARXIV.2508.16994 [18] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s Verify Step by Step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id= v8L0pN6EOi [19] OpenAI. 2025. Introducing OpenAI o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/. Accessed: 2026-01-26. [20] Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. 2024. Proving Test Set Contamination in Black-Box Language Models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=KS8mIvetg2 [21] Keqin Peng, Liang Ding, Yuanxin Ouyang, Meng Fang, and Dacheng Tao. 2025. Revisiting Overthinking in Long Chain-of-Thought from the Perspective of Self-Doubt. CoRR abs/2505.23480 (2025). arXiv:2505.23480 doi:10.48550/ ARXIV.2505.23480 [22] Abulhair Saparov and He He. 2023. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview.net/forum?id=qFVVBzXxR2V [23] Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 4505–4514. doi:10.18653/V1/D19-1458 [24] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. CoRR abs/2408.03314 (2024). arXiv:2408.03314 doi:10.48550/ARXIV.2408.03314 [25] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 13003–13051. doi:10.18653/V1/2023.FINDINGS-ACL.824 [26] TRACE. 2026. A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation. https://sites.google.com/view/tracebench Accessed: 2026-01-28. [27] Dingmin Wang, Bocheng Zou, Zhen Han, and zhiqiang xu. 2025. t-BEN: A Temporal Logic Guided Approach for Temporal Reasoning Benchmark Generation. https://openreview.net/forum?id=XkzGgKJAA2 [28] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview.net/forum?id=1PL1NIMMrw [29] Yuqing Wang and Yun Zhao. 2024. TRAM: Benchmarking Temporal Reasoning for Large Language Models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 6389–6415. doi:10.18653/V1/2024.FINDINGS-ACL.382 [30] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-AbstractConference.html [31] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.cc/paper_files/paper/2023/hash/271db9922b8d1f4dd7aaef84ed5ac703-Abstract-Conference.html [32] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. STaR: Bootstrapping Reasoning With Reasoning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). http://papers.nips.cc/paper_files/paper/2022/hash/ 639a9a172c044fbb64175b5fad42e9a5-Abstract-Conference.html , Vol. 1, No. 1, Article . Publication date: July 2026.

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

23

[33] Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2024. DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=gjfOL9z5Xr

, Vol. 1, No. 1, Article . Publication date: July 2026.