Reason, Reward, Refine: Step-Level Errors Corrections with Structured Feedback for Physics Reasoning in Small Language Models Raj Jaiswal1∗ , Dhruv Jain2∗ , Rishabh Dhawan1∗ , Sree Krishna Uppalapati1∗ , Shin’ichi Satoh4 , Tanuja Ganu5 , Rajiv Ratn Shah3 1 IIIT Delhi 2 IIT BHU 3 IIT Kanpur 4 NII Tokyo 5 Microsoft Research India {jaiswalp, rishabh23002, sree23533}@iiitd.ac.in [email protected], [email protected], [email protected], [email protected] ∗
Equal contribution
arXiv:2607.05199v1 [cs.AI] 6 Jul 2026
Abstract Physics reasoning fails structurally in small language models: an error at any step propagates forward, corrupting every inference that follows. Limited domain knowledge, hallucination under multi-step derivation, and distributional sensitivity compound this failure. We propose a step-level reward framework that identifies the first reasoning error, generates targeted structured feedback, and trains the model to revise its solution via policy gradient with KL regularization, without exposing it to ground truth solutions as generation targets. Unlike annotation-dependent step-level methods, no preference data construction is required and the external verifier operates exclusively at training time. Across five physics benchmarks, our framework delivers accuracy gains of 17–20% over CoT prompting and 10–16% over the strongest baseline, reduces calculation errors from 56.9% to 23.5%, and reduces miscomprehension errors from 22.3% to 12.0% in the best observed cases. Conceptual errors reduce from 89.7% to 68.7%, yet persist as the hardest failure mode across all conditions. Code, prompts, and experimental details are provided in the Appendix section (§9).
1
Introduction
Physics reasoning is inherently sequential: each solution step depends on the correctness of all preceding steps, and an error at any point invalidates every inference that follows (Jaiswal et al., 2024; Anand et al., 2024; Ding et al., 2023). Language models have made substantial progress on reasoning tasks (Wei et al., 2022; Fu et al., 2023; Zhang et al., 2022), yet physics reasoning remains an open challenge for small language models (SLMs) (Srivastava et al., 2025; Boye and Myrberg, 2025). Larger models benefit from knowledge compression across broad training distributions; smaller models do not, and the reasoning failures that
Table 1: Our Framework Achieves the Highest Average Accuracy Across All Conditions. Per-model average (%) across five benchmarks; Cell shading indicates per-row rank across conditions. Discussed in §5. Model
CoT
RAG
SFT
DPO
Ours
Qwen 2.5 1.5B LLaMA 3.2 1B LLaMA 3.2 3B Phi 3.5 Mini 3.8B
46.1 32.5 44.8 50.5
53.3 41.1 51.2 54.3
41.0 37.5 44.0 51.0
40.7 37.5 44.1 52.1
58.5 53.9 64.4 61.5
emerge are structurally distinct from those observed at scale (Zhang et al., 2024; Srivastava et al., 2025). Prior work has progressively improved physics reasoning in language models, with each approach addressing a distinct limitation (Ding et al., 2023; Pang et al., 2024; Anand et al., 2024). Chain-ofThought prompting (Wei et al., 2022) improves reasoning transparency by externalizing intermediate steps, though step-level error propagation across dependent steps remains unaddressed. RetrievalAugmented Generation (Lewis et al., 2020) provides access to relevant domain knowledge at inference time, though correct retrieval does not guarantee correct application of retrieved content (Anand et al., 2024). Supervised fine-tuning on expert reasoning trajectories (Ho et al., 2022; Luo et al., 2023) improves structural response quality, though models trained under token prediction objectives have been shown to reproduce surface reasoning patterns without internalizing the underlying logic (Lobo et al., 2024; McCloskey and Cohen, 1989). Direct Preference Optimization (Rafailov et al., 2023) introduces comparative signal between correct and incorrect solution pairs, though preference is computed over complete responses and does not identify which reasoning step produced the incorrect outcome (Lai et al., 2024a; Chen et al., 2024). Step-level preference optimization (Xu et al., 2025; Lai et al., 2024a) assigns rewards at indi-
Table 2: Comparison with Prior Work. Prior work covers at most three of the five dimensions; this work is the first to address all five jointly Physics Domain
Step-Level Localization
Structured Feedback
Step-DPO (Lai et al., 2024a)
✗
✓
✗
✗
✗
Full-Step-DPO (Xu et al., 2025)
✗
✓
✗
✓
✓
SVPO (Chen et al., 2024)
✗
✓
✗
✓
✓
Step-KTO (Lin et al., 2025)
✗
✓
✗
✓
✗
Self-Refine (Madaan et al., 2024)
✗
✓
✓
✓
✗
This Work
✓
✓
✓
✓
✓
Work
vidual reasoning steps, but depends on externally annotated step-level data and a reliable verifier. In scientific domains, both are prohibitively costly, and small language models have been shown to be unreliable self-verifiers (Li et al., 2024; Zhang et al., 2024). Our contributions are as follows: • Step-Level Reward Mechanism: We propose r = efirst /(n + 1), penalizing earlier reasoning failures more heavily and enabling correction at the precise point of failure rather than over the complete response, without requiring annotated step-level preference data. (§3.2) • Error-Type Reduction via Structured Feedback: Error-type-conditioned feedback at training time reduces Calculation errors from 56.9% to 23.5%, Problem Miscomprehension from 22.3% to 12.0%, and Conceptual Misapplication from 89.7% to 68.7% in the best observed case per error type; Conceptual Misapplication persists across all conditions. (§3.3, Table 7) • Empirical Gains Across Models and Benchmarks: Our framework delivers accuracy gains of 17–20% over CoT prompting and 10– 16% over all evaluated baselines across four open-source language models and five physics benchmarks, with a peak gain of 27.1% on JEEBench for LLaMA 3.2 3B. (Table 3, Table 4)
2
Related Work
Reinforcement learning from human feedback (Ouyang et al., 2022; Christiano et al., 2017) introduced reward-based policy optimization as an alternative to imitation, training models against
No Preference Data
Position Reward
a scalar reward signal rather than fixed output targets. Direct Preference Optimization (Rafailov et al., 2023) reformulates this as a closed-form preference objective over chosen and rejected response pairs, removing the need for an explicit reward model. Both frameworks treat the complete response as a single unit; the reward or preference signal is computed over the output as a whole and does not identify which reasoning step produced an incorrect outcome (Lai et al., 2024b). Step-DPO (Lai et al., 2024b) and Full-StepDPO (Xu et al., 2025) extend preference optimization to individual reasoning steps, assigning stepwise rewards across the reasoning chain. Steplevel supervision has been shown to improve over response-level signal on mathematical reasoning benchmarks (Xu et al., 2025), though both methods require separately annotated step-level preference data and an external verifier to evaluate intermediate steps (Xu et al., 2025; Cobbe et al., 2021). Self-Refine (Madaan et al., 2024) and Reflexion (Shinn et al., 2024) implement iterative refinement through self-generated feedback, while SCoRe (Kumar et al., 2024) trains models to selfcorrect via reinforcement learning. Small language models have been shown to be unreliable verifiers of their own reasoning in sub-4B parameter regimes (Zhang et al., 2024; Tyen et al., 2024; Li et al., 2024; Huang et al., 2023), making selfcorrection an insufficient mechanism at this scale.
3
Methodology
We propose a step-level reward training framework for physics reasoning in small language models, illustrated in Figure 1. The framework operates in three stages. Stage 1 (§3.1) initializes the policy πθ through supervised fine-tuning on numbered step-by-step chain-of-thought solutions. Stage 2 (§3.2) enters an iterative training loop: the model
Figure 1: Step-Level Reward Penalizes Earlier Failures More Heavily; Feedback Targets the Precise Point of Failure. πθ generates y1 , receives error-type-conditioned feedback p1 from GPT-4o, and revises to y2 ; reward r = efirst /(n + 1) drives policy gradient update. Discussed §3.
generates an initial solution y1 , an external verifier (GPT-4o) identifies the first reasoning error against ground truth y ∗ and computes a step-level reward r = efirst /(n + 1). Stage 3 (§3.3) generates errorconditioned feedback p1 ; the model produces a revised solution y2 from which the policy is updated via policy gradient with KL regularization (§3.4). The model is never exposed to correct solutions as generation targets — it receives only structured feedback identifying the first error step and its type. 3.1
Stage 1: Supervised Fine-Tuning Warm-up
The fine-tuning corpus comprises 2,494 physics problems sourced from JEE preparation materials (Pandey, 2020; Tipler, 1999; Pinsky, 1989; Halliday et al., 2014), distinct from all evaluation benchmarks. Each problem is paired with a numbered chain-of-thought solution across five domains: Mechanics, Electromagnetism, Thermodynamics, Waves & Optics, and Modern Physics. The policy πθ is initialized via LoRA adapters (Hu et al., 2022) on this corpus. This stage internalizes the step-indexed format required for the verifier to reliably identify individual reasoning steps; without it, reward computation lacks the structural consistency it depends on. Upon completion, the adapter weights are frozen as the reference policy πref , serving as the KL regularization anchor throughout Stage 2, and πθ is reinitialized from these same weights as the starting point for reward-guided training. Hyperparameter details are
in Appendix A. 3.2
Stage 2: Step-Level Reward Mechanism
Physics reasoning failures exhibit three recurring patterns (Jaiswal et al., 2024): Problem Miscomprehension (MC), misidentified objective or misread quantities; Conceptual Misapplication (CM), wrong governing law or principle applied outside validity conditions; and Calculation Error (CE), correct setup but arithmetic or algebraic error in execution. Each error type is interdependent; classification follows MC, CM, CE priority order to route feedback accordingly (§3.3). Verifier. Let x denote a physics problem and y ∗ its ground truth solution. An external verifier receives a model-generated solution y1 and identifies the first step at which reasoning deviates from y ∗ , returning a structured error tuple (e1 , c1 , expl1 ): the index of the first error step e1 , the error category c1 ∈ {MC, CM, CE}, and a natural language explanation expl1 . The verifier is GPT-4o, queried via the OpenAI API. The verifier operates exclusively at training time; no external model is queried at inference. Reward.
The step-level reward is defined as: r=
efirst n+1
(1)
where efirst is the step index of the first reasoning error and n is the total number of reasoning steps. This assigns higher reward to solutions that sustain
Figure 2: Targeted Feedback at the First Error Step Enables the Model to Reason Toward a Correct Solution. Each column shows one Baseline responses on the same problem; red annotations mark the first error step and type, green checkmarks in <final_answer> indicate correct quantities. Structured feedback on the identified error step progressively corrects reasoning across revision passes. Discussed in §5.3 & §6.1
correct reasoning further into the chain, penalizing earlier failures more heavily regardless of error type. If the verifier returns no error (e1 = 0), reward is set to 1.0 and the sample is skipped in the training loop.
tion errors, Python code is generated to correctly perform the computation at the identified error step; the execution output is incorporated into feedback clarifying the correct calculation and the original arithmetic error.
3.3
3.4
Stage 3: Feedback Generation
The feedback generator F receives the error tuple (e1 , c1 , expl1 ) and produces structured feedback p1 conditioned on error type c1 via one of three channels, all implemented using GPT-4o; feedback is generated only when r1 ≤ τ , skipping solutions that already meet the reward threshold. For Problem Miscomprehension errors, a structured prompt instructs the model to re-read the problem statement and correct its understanding of the given conditions before regenerating. For Conceptual Misapplication errors, a retrieval query is issued against a vector store built from a physics formula corpus (National Testing Agency, 2023) using sentence-transformer embeddings (Reimers and Gurevych, 2019) to supply grounded governing principles and formulas rather than relying on GPT-4o generation alone; the retrieved content is combined with the error explanation into feedback identifying the correct physical law. For Calcula-
Policy Update
The policy πθ generates a revised solution y2 ∼ πθ (· | x, y1 , p1 ), conditioned on the original problem x, the first attempt y1 , and structured feedback p1 . The revised solution is evaluated by the verifier to produce a second error tuple (e2 , c2 , expl2 ) and reward r2 = e2 /(n + 1), where e2 is the first error step index in y2 evaluated against the ground truth solution y ∗ ; if the verifier returns no error, r2 = 1.0. The policy is updated via a policy gradient objective with KL regularization against the frozen reference policy πref — the Stage 1 SFT checkpoint: h L(θ) = − log πθ (y2 | x, y1 , p1 ) − log πθ (y1 | x) · r2 i − β DKL πθ (· | x) ∥ πref (· | x) (2)
where β = 0.1 controls KL regularization strength
90
Algorithm 1 : Step-Level Reward Training for Physics Reasoning in Small LMs
and τ = 0.9 determines early stopping; full sensitivity analysis is in Appendix A. Gradients are computed with respect to y2 only; the log-probability of y1 serves as a fixed baseline and is detached from the computation graph. The objective incentivizes improvement from y1 to y2 by weighting the logprobability gain by the step-level reward r2 . The KL term prevents the policy from deviating excessively from the SFT initialization. The KL term is computed on πθ (· | x) rather than on the full conditioning context (x, y1 , p1 ), anchoring the policy to the base reasoning distribution established during warm-up independent of feedback quality. Full hyperparameter and code details are provided in Appendix A.
4
Experiments
4.1
Benchmark Datasets
We evaluate on five physics benchmarks spanning foundational to advanced reasoning difficulty: SciEval Static (Sun et al., 2024) (164 MCQ, introductory to intermediate), MMLU High School and College Physics (Hendrycks et al., 2020) (170 and 118 MCQ, foundational to undergraduate), JEEBench (Arora et al., 2023) (123 questions, multi-concept synthesis and mathematical derivation), and PhysicsQA (Jaiswal et al., 2024) (370 problems with verified chain-of-thought solutions enabling step-level evaluation, introduced in prior work). All benchmarks are used without filtering or subset selection; representative samples are provided in Appendix E.
70 Proportion of Dataset (%)
Require: Training dataset D, policy πθ (warm-started from Stage 1 SFT), reference policy πref (Stage 1 checkpoint, frozen), external verifier V (GPT-4o), feedback generator F, threshold τ = 0.9 1: for each training step do 2: Sample (x, y ∗ ) ∼ D 3: Generate y1 ∼ πθ (· | x) 4: (e1 , c1 , expl1 ) ← V(y1 , y ∗ ) ▷ Verify against ground truth 5: r1 ← e1 / (n + 1) ▷ Step-level reward 6: if r1 > τ then 7: continue ▷ Initial solution correct; skip 8: end if 9: p1 ← F(x, y1 , e1 , c1 , expl1 ) ▷ Structured feedback 10: Generate y2 ∼ πθ (· | x, y1 , p1 ) ▷ Revised solution 11: (e2 , c2 , expl2 ) ← V(y2 , y ∗ ) 12: r2 ← e2 / (n + 1) 13: Update θ via Eq. 2 ▷ Policy gradient + KL 14: end for
SFT Training SciEval MMLU-High MMLU-College PhysicsQA
80
60 50 40 30 20 10 0
Mechanics
Electromagnetism
Thermodynamics
Waves & Optics
Modern Physics
Figure 3: Common Topic Distributions Confirm Data similarity Across Training and Benchmarks. Topic distribution (%) across the SFT training corpus and evaluation benchmarks; the same five domains appear in both but at substantially different proportions.
4.2
Models
We evaluate four open-source language models: Qwen 2.5 1.5B Instruct (Qwen Team, 2024), LLaMA 3.2 (1B and 3.2 3B) Instruct (Grattafiori et al., 2024), and Phi 3.5 Mini 3.8B Instruct (Microsoft Research, 2024). Each model is evaluated across five conditions: CoT, RAG, SFT, DPO, and our framework. All fine-tuned conditions (SFT, DPO, and our framework) are trained from identical pre-trained base weights, ensuring performance differences are attributable to the training objective rather than initialization. CoT and RAG baselines are evaluated via the Nebius API in default configuration; RAG augments the same inference with retrieved context from the physics formula corpus described in §3.3. 4.3
Baseline Setup
Chain-of-Thought (CoT). CoT prompting elicits step-by-step reasoning without any parameter update. Each model is prompted zero-shot with: “You are an expert physics assistant. Given a question, generate the final solution. Let’s think step by step.” This reflects each model’s intrinsic reasoning capability under standard inference conditions. Retrieval-Augmented Generation (RAG). RAG augments inference with externally retrieved physics knowledge. The retrieval corpus contains formulas and physical constants across five domains, with no worked examples or narrative explanations. The corpus is divided into 500-character segments with a 50-character overlap and embedded offline using text- embedding- ada- 002. The resulting embeddings are stored in a local FAISS vector store. At inference time, the three most similar
Table 3: Our Framework Leads on Easy Benchmarks; Knowledge Grounding via RAG Outperforms FineTuning Approaches. Final answer accuracy (%) confirms RAG improves over CoT across all models while SFT and DPO fail to consistently improve over CoT. Cell shading indicates per-row rank across conditions. Discussed in §5.2. Model
SciEval-Static
MMLU-High
MMLU-College
CoT
RAG
FT
DPO
Ours
CoT
RAG
FT
DPO
Ours
CoT
RAG
FT
DPO
Ours
Qwen 2.5 1.5B
62.73
68.12
57.93
59.32
79.29
47.05
59.41
41.18
46.47
55.03
52.62
62.27
43.22
43.22
53.64
LLaMA 3.2 1B
44.51
57.37
55.49
56.93
68.09
30.58
38.02
42.35
39.41
55.29
33.63
45.82
38.14
36.44
57.89
LLaMA 3.2 3B
62.26
66.87
61.59
57.93
81.52
50.00
55.29
48.29
47.65
67.20
53.63
61.82
45.76
50.85
69.09
Phi 3.5 Mini 3.8B
62.43
61.01
59.17
56.87
65.19
55.17
60.53
55.97
56.12
65.83
65.63
67.27
62.77
64.34
73.12
Table 4: Our Framework Consistently Outperforms on Hard Benchmarks Where All Baselines Remain Below 42%. Final answer accuracy (%) shows CoT falling below 40%, SFT and DPO failing to surpass CoT, and RAG gains narrowing relative to Table 3. Cell shading indicates per-row rank across conditions. Discussed in §5.2. Model
JEEBench
PhysicsQA
CoT
RAG
FT
DPO
Ours
CoT
RAG
FT
DPO
Ours
Qwen 2.5 1.5B
37.53
40.00
39.02
30.08
54.86
30.64
36.91
23.51
24.32
49.62
LLaMA 3.2 1B
32.52
35.50
28.46
35.77
49.17
21.35
28.64
23.24
18.92
39.02
LLaMA 3.2 3B
30.21
40.83
36.59
38.21
57.34
27.67
31.21
27.84
25.95
46.73
Phi 3.5 Mini 3.8B
35.90
41.25
38.04
41.65
50.23
33.35
41.59
39.19
41.49
53.22
100 90
SciEval-Static MMLU-High School MMLU-College
JEEBench PhysicsQA
Easy
80 CoT Accuracy (%)
chunks are retrieved using cosine similarity and inserted into the prompt. No external API is queried during inference. Full configuration details are provided in Appendix E.
70
Hard
60 50 40 30
Supervised Fine-Tuning (SFT). SFT trains each model on the 2,494-problem corpus using LoRA adapters (Hu et al., 2022) via the Hugging Face PEFT framework, pairing each problem with a structured chain-of-thought solution enforcing explicit reasoning steps. Training uses AdamW with learning rate 5e-5, batch size 4, gradient accumulation 8, for 3 epochs on a single H100 GPU. Full configuration details are in Appendix E. Direct Preference Optimization (DPO). DPO is applied on top of the SFT checkpoint using preference pairs from the same 2,494-problem corpus. Chosen solutions are verified for physical correctness; rejected solutions have errors injected at reasoning execution steps, directing the preference signal toward execution rather than problem interpretation. IPO loss is used in place of standard sigmoid DPO as it does not assume a consistent preference gap across pairs (Azar et al., 2023). Training uses AdamW with learning rate 2e-6, batch size 2, gradient accumulation 16, for 1 epoch on a single H100 GPU. Full configuration details are in Appendix E.
20
2.5 en .5B Qw 1
3.2B MA 1
LLa
3.2B MA 3
LLa
3.5B Phii 3.8 Min
3.1B MA 8
LLa
Phi
ed 3 M14B
2 ma27B
Gem
2.5 en 72B
Qw
o T-4
GP
Figure 4: Accuracy Improves With Model Size; Gains Are Larger on Easy Than Hard Benchmarks. CoT accuracy (%) across models (1B–72B and GPT-4o); across SciEval, MMLU-High, MMLU-College (easy) and JEEBench, PhysicsQA (hard). Discussed in §5.1.
4.4
Evaluation
Final answer accuracy is the primary metric across all five benchmarks. We employ a three-step shadow protocol: Step 1 verifies answer correctness via string matching; Step 2 checks structural compliance across seven required XML tags; Step 3, applied only to structurally compliant solutions, attributes errors across the three error types using four human evaluators with qualifying scores in physics, chemistry, and mathematics at the national engineering entrance level. Cohen’s Kappa ranges from 0.75 to 0.89 across all models and benchmarks. Full protocol details and agreement scores are in Table 6 and Appendix E.
Table 5: Calculation Error Reduces Most Consistently; Conceptual Misapplication Dominates Incorrect Predictions Across All Conditions. Reasoning error distribution (%) among incorrect predictions on Benchmarks; cell shading indicates per-row error severity. LLaMA 3.2 1B shows simultaneous reduction across all three error types under our framework. Discussed in §5.3. Model
MC (%) — Problem Miscomprehension CoT
RAG
SFT
DPO
CM (%) — Conceptual Misapplication
CE (%) — Calculation Error
Ours
CoT
RAG
SFT
DPO
Ours
CoT
RAG
SFT
DPO
Ours
Qwen 2.5 1.5B
9.3
15.0
4.6
16.1
4.1
38.6
61.4
49.5
83.9
66.1
52.1
24.5
35.0
38.9
31.4
LLaMA 3.2 1B
22.3
34.1
38.7
25.3
12.0
89.7
79.5
93.0
97.3
68.7
55.0
45.5
49.6
30.0
27.5
LLaMA 3.2 3B
7.9
5.9
6.4
9.1
12.0
73.8
47.2
55.8
80.7
42.7
37.8
44.9
40.4
38.7
38.5
Phi 3.5 Mini 3.8B
8.1
6.0
6.7
8.3
6.4
50.8
57.9
44.6
55.6
56.7
56.9
26.4
62.5
27.8
23.5
Table 6: Human Evaluation Confirms Reliability of Step-Level Assessment Across All Benchmarks. Cohen’s Kappa between four human evaluators for steplevel scoring across all evaluation samples. Discussed in §4.4. Model
SciEval
MMLU
PhysicsQA
JEEBench
Qwen 2.5 1.5B
0.84
0.82
0.81
0.77
LLaMA 3.2 1B
0.81
0.79
0.78
0.75
LLaMA 3.2 3B
0.87
0.85
0.84
0.82
Phi 3.5 Mini 3.8B
0.89
0.88
0.86
0.84
5
Results
5.1
Accuracy Increases With Model Scale.
We evaluate CoT accuracy across nine models spanning 1B to 72B parameters across easy benchmarks (SciEval, MMLU-High, MMLU-College) and hard benchmarks (JEEBench, PhysicsQA), shown in Figure 4. Accuracy improves consistently as model scale increases on both benchmark types. On easy benchmarks, gains are steep — Qwen 2.5 72B reaches 98.3% on SciEval. On hard benchmarks, the same scaling trend holds but at lower absolute values — no model exceeds 80% on PhysicsQA regardless of scale, leaving step-level physics reasoning as an open challenge across all model sizes evaluated. 5.2
Accuracy Across Conditions.
We report final answer accuracy across five conditions on all five benchmarks across four models (Tables 3, 4). Our framework achieves the highest accuracy across all models and benchmarks; on easy benchmarks, LLaMA 3.2 3B reaches 81.52% on SciEval under our framework against 62.26% under CoT. RAG ranks second consistently on both easy and hard benchmarks, with gains over CoT higher on easy benchmarks and narrowing on hard benchmarks. SFT produces accuracy at or below
CoT across the majority of model-benchmark combinations; DPO shows the same pattern, with the most pronounced degradation observed for Qwen 2.5 1.5B on MMLU-College and PhysicsQA. Our framework does not rank first on MMLU-High and MMLU-College for Qwen 2.5 1.5B, where RAG ranks above it — the only two such conditions across both tables. 5.3
Reasoning Error Distribution Across Conditions.
We analyze the distribution of Problem Miscomprehension (MC), Conceptual Misapplication (CM), and Calculation Error (CE) among incorrect predictions on PhysicsQA across all five conditions (Table 5). DPO produces the largest CM increase of any condition: for Qwen 2.5 1.5B, CM rises from 38.6% under CoT to 83.9% under DPO, a 45.3 percentage point increase; for LLaMA 3.2 1B, CM rises from 89.7% to 97.3%, the highest proportion observed across any model or condition in this table. Under our framework, CE reduces consistently across three of four models — from 52.1% to 31.4% for Qwen 2.5 1.5B, from 55.0% to 27.5% for LLaMA 3.2 1B, and from 56.9% to 23.5% for Phi 3.5 Mini 3.8B — while LLaMA 3.2 3B shows no meaningful change in CE (37.8% to 38.5%). MC reduces for three of four models under our framework, with the exception of LLaMA 3.2 3B where MC increases from 7.9% to 12.0%. LLaMA 3.2 1B is the only model where all three error types reduce simultaneously, with MC dropping from 22.3% to 12.0%, CM from 89.7% to 68.7%, and CE from 55.0% to 27.5%. CM remains the largest error category across all models and conditions; under our framework, CM stays above 42% across all four models and increases relative to CoT for Qwen 2.5 1.5B and Phi 3.5 Mini 3.8B.
Table 7: LLaMA 3.2 1B Benefits Across All Feedback Channels; Calculation Feedback Shows Largest Gains. Ablation on PhysicsQA across feedback channels and baselines; green shading indicates error reduction under our framework, red shading indicates error increase. Discussed in §6.1. Ours − Baseline (%) Feedback Type
Qwen 2.5 1.5B CoT
RAG
Problem Statement (MC)
+5.2
Concept (CM)
−27.5
Calculation (CE)
+20.7
LLaMA 3.2 1B RAG
DPO
CoT
RAG
SFT
DPO
CoT
RAG
SFT
DPO
+10.9
+0.5
+12.0 +10.3 +22.1 +26.7 +13.3
−4.1
−6.1
−5.6
−4.7
−2.9
+1.7
−0.4
+0.3
+1.9
−16.6 +17.8 +21.0 +10.8 +24.3 +28.6 +31.1 +4.5 +13.1 +38.0
−5.9
+1.2 −12.1 −1.1
−6.9
+3.6
+27.5 +18.0 +22.1
6
Discussion
6.1
Structured Error Feedback Improvement.
The reduction patterns in Table 7 are consistent across all four baseline comparisons, not only relative to CoT. Calculation feedback is the most consistent of the three channels, reducing Calculation errors across three of four models regardless of which baseline is compared. Concept feedback is most effective where DPO caused the most damage — models with the highest Conceptual Misapplication proportions under DPO (Table 5) show the largest reductions under our framework. Statement feedback is the least consistent, reducing Problem Miscomprehension errors for LLaMA 3.2 1B across all baselines but increasing them for LLaMA 3.2 3B across all baselines. Concept feedback increases Conceptual Misapplication errors for Qwen 2.5 1.5B relative to most baselines — the only model where this channel consistently shows negative values. These negative values indicate that feedback does not always correct the identified error; when feedback does not target the actual source of failure, the revised solution y2 may introduce new errors. LLaMA 3.2 1B, which shows the highest baseline error rates, benefits from all three channels across all baseline comparisons. 6.2
SFT
Phi 3.5 Mini 3.8B
DPO
+7.5
CoT
LLaMA 3.2 3B
SFT
Where Step-Level Reward and Feedback Fall Short.
Despite consistent accuracy gains, three failure patterns emerge from the results. Conceptual Misapplication errors persist above 42% across all models under our framework and increase relative to CoT for Qwen 2.5 1.5B and Phi 3.5 Mini 3.8B (Table 5); retrieval feedback supplies the correct governing principle but does not guarantee its correct application. Problem Miscomprehension errors increase for LLaMA 3.2 3B across all baseline comparisons (Table 7), where Conceptual Misapplication dominates at 73.8% under CoT and the training loop’s prioritization of conceptual correction may alter
+2.5
−0.7
+6.4
+1.9
+0.2
+33.4 +2.9 +39.0 +4.3
problem interpretation as a secondary effect. The step-level reward signal depends on the verifier correctly classifying the error type; when classification is incorrect, feedback is routed to the wrong channel and y2 is conditioned on feedback that does not address the actual failure.
7
Conclusion
Step-level reward training for physics reasoning in small language models identifies the first reasoning error in a solution, generates targeted structured feedback by error type, and trains the model via policy gradient with KL regularization, without access to correct solutions as generation targets and without preference data construction. Across four models and five physics benchmarks, the framework delivers accuracy gains of 17–20% over CoT prompting and 10–16% over all evaluated baselines, reducing Calculation errors by up to 33.4% and Problem Miscomprehension errors by up to 10.3%. Conceptual Misapplication errors persist above 42% across all conditions, remaining the most consistent unresolved failure mode across all evaluated settings.
8
Limitations
Our evaluation is restricted to physics benchmarks in English; whether the Problem Miscomprehension, Conceptual Misapplication, and Calculation Error taxonomy and the step-level reward signal generalize to other scientific domains or nonEnglish corpora remains untested. Results are reported for single training runs of 2,494 samples over 60 epochs without multi-seed validation. The framework depends on GPT-4o for both error verification and structured feedback generation. This coupling means systematic GPT-4o classification errors would corrupt both the reward signal and feedback conditioning y2 , without being detectable from accuracy metrics alone. Structured JSON output is required for reward computation; parse failures default to minimal reward and their fre-
quency is not reported. The step-level reward depends on regex-based step counting and LoRA adapters trained on seven-tag structured solutions; non-standard formatting would produce incorrect reward signals without detection. The CM and CE channels invoke GPT-4o via external API while baselines operate without equivalent resources, introducing a parity gap. Individual channels were not ablated; contribution is inferred from errortype reduction patterns. Per-iteration accuracy between y1 and y2 was not logged. The proportion of training samples skipped due to r1 > τ was not logged during training; reporting this rate would clarify how often the early-stopping threshold is triggered and its effect on effective training corpus size. While topic distribution analysis confirms common domain proportions between the SFT corpus and evaluation benchmarks (Figure 3), no exact string matches were found between the SFT training corpus and JEEBench; however, formal semantic deduplication has not been performed and topical overlap cannot be fully ruled out.
9
Ethical Considerations
All datasets used for training and evaluation are publicly available and contain no personal data, sensitive information, or harmful content. The physics formula corpus is publicly available educational material. The 2,494 training samples are sourced from existing publicly available JEE physics problem sets; no new human annotation was conducted in this work. GPT-4o (OpenAI API) is used as the external verifier at training time; all four models are accessed via Nebius API for CoT inference. Experiments were conducted on H100 GPUs. Evaluation is restricted to English-language physics benchmarks; generalization to other languages or scientific domains isvnot claimed. Code, prompts, and training configurations are provided in the Appendix (§9).
References Avinash Anand, Kritarth Prasad, Chhavi Kirtani, Ashwin R Nair, Mohit Gupta, Saloni Garg, Anurag Gautam, Snehal Buldeo, and Rajiv Ratn Shah. 2024. Enhancing llms for physics problem-solving using reinforcement learning with human-ai feedback. Preprint, arXiv:2412.06827. Daman Arora, Himanshu Gaurav Singh, and 1 others. 2023. Have LLMs advanced enough? A challenging problem solving benchmark for large language models. arXiv preprint arXiv:2305.15074.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. 2023. A general theoretical paradigm to understand learning from human preferences. Preprint, arXiv:2310.12036. Johan Boye and Birger Myrberg. 2025. Large language models for physics reasoning. arXiv preprint arXiv:2502.11537. Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024. Step-level value preference optimization for mathematical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Jingzhe Ding, Yan Cen, and Xinyuan Wei. 2023. Using large language model to solve and explain physics word problems approaching human level. arXiv preprint arXiv:2309.08182. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, and 1 others. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. David Halliday, Robert Resnick, and Jearl Walker. 2014. Fundamentals of Physics, 10th edition. Wiley. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01848.
Raj Jaiswal, Dhruv Jain, Harsh Parimal Popat, Avinash Anand, Abhishek Dharmadhikari, Atharva Marathe, and Rajiv Ratn Shah. 2024. Improving physics reasoning in large language models using mixture of refinement agents. Preprint, arXiv:2412.00821. Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, and 1 others. 2024. Training language models to selfcorrect via reinforcement learning. arXiv preprint arXiv:2409.12917.
sequential learning problem. In Psychology of Learning and Motivation, volume 24, pages 109–165. Elsevier. Microsoft Research. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. National Testing Agency. 2023. NEET physics formula sheet. Physics formula reference used for retrieval corpus construction.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024a. Step-DPO: Stepwise preference optimization for long-chain reasoning of LLMs. arXiv preprint arXiv:2406.18629.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024b. Step-DPO: Stepwise preference optimization for long-chain reasoning of LLMs. arXiv preprint arXiv:2406.18629.
D. C. Pandey. 2020. IIT JEE Physics: 35 Years Chapterwise Solved Papers. Arihant Publications.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33:9459–9474. Xiaoze Li, Feifei Sun, Yixin Zhang, Xinlong Xu, Jielong Jiang, Bhaskar Mitra, and Lucian Popa. 2024. Evaluating the factuality of large language models using large-scale knowledge graphs. arXiv preprint arXiv:2404.00216. Yen-Ting Lin, Di Jin, Tengyu Xu, Tianhao Wu, Sainbayar Sukhbaatar, Chen Zhu, Yun He, Yun-Nung Chen, Jason Weston, Yuandong Tian, Arash Rahnama, Sinong Wang, Hao Ma, and Han Fang. 2025. Step-kto: Optimizing mathematical reasoning through stepwise binary feedback. Preprint, arXiv:2501.10799. Elita Lobo, Chirag Agarwal, and Himabindu Lakkaraju. 2024. On the impact of fine-tuning on chain-ofthought reasoning. arXiv preprint arXiv:2411.15382. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. WizardMath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583.
Xinyu Pang, Ruixin Hong, Zhanke Zhou, and 1 others. 2024. Physics reasoner: Knowledge-augmented reasoning for solving physics problems with large language models. arXiv preprint arXiv:2412.13791. A. A. Pinsky. 1989. Problems in Physics. Mir Publishers. Qwen Team. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728– 53741. Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3982–3992. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36. Aarohi Srivastava and 1 others. 2025. Reasoning ability of small language models. arXiv preprint arXiv:2502.11462.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36.
Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. 2024. SciEval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19053–19061.
Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The
Paul A. Tipler. 1999. Physics for Scientists and Engineers, 4th edition. W. H. Freeman and Company.
Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen, and Tony Mak. 2024. LLMs cannot find reasoning errors, but can correct them given the error location. In Findings of the Association for Computational Linguistics: ACL 2024. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824– 24837. Huimin Xu, Xin Mao, Feng-Lin Li, Xiaobao Wu, Wang Chen, Wei Zhang, and Anh Tuan Luu. 2025. Fullstep-DPO: Self-supervised preference optimization with step-wise rewards for mathematical reasoning. arXiv preprint arXiv:2502.14356. Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. 2024. Small language models need strong verifiers to self-correct reasoning. arXiv preprint arXiv:2404.17140. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493.
Appendix A
Training Dataset Samples
The fine-tuning corpus comprises 2,494 high school-level physics problems sourced from standard Indian JEE preparation materials. Each problem is paired with a structured chain-of-thought solution formatted using seven XML tags: <problem_analysis>, <principle>, <governing_equation>, <value_identification>, <substitution>, <calculation>, and <final_answer>. This tag structure enforces explicit separation between physical reasoning and algebraic execution, and is consistent with the inference prompt used at evaluation time. Solutions were generated using an automated structuring prompt (see Appendix C) and verified for tag-level correctness before inclusion. Four representative samples are shown below, spanning Thermodynamics, Electromagnetism, Nuclear Physics, and Classical Mechanics.
Sample 1 (ID: 2162) Thermodynamics — Adiabatic compression followed by isochoric heating, multi-part symbolic derivation Two moles of an ideal monatomic gas, initially at pressure p1 and volume V1 , undergo adiabatic compression until its volume is V2 . The gas is then given heat Q at constant volume V2 . (a) Sketch the complete process on a P -V diagram. (b) Find the total work done by the gas, the total change in its internal energy, and the final temperature of the gas. [Give answers in terms of p1 , V1 , V2 , Q and R.] Chain-of-Thought Solution — Sample 1 <problem_analysis> Two sequential processes: (1) adiabatic compression V1->V2, (2) isochoric heating at V2 with heat Q. Monatomic ideal gas, n=2 mol => gamma=5/3, Cv=3R/2. Find: total W, total DU, final temperature T3. </problem_analysis> <principle> First Law of Thermodynamics: DQ = DU + DW. Adiabatic: DQ=0 => DU=-W. Isochoric: DW=0 => DU=DQ. </principle> <governing_equation> Adiabatic: p1*V1ˆgamma = p2*V2ˆgamma W_adiabatic = (p1*V1 - p2*V2) / (gamma - 1) Isochoric: W2=0, DU2=Q </governing_equation> <value_identification> n=2 mol, gamma=5/3, Cv=(3/2)R Initial: pressure p1, volume V1 Heat added at constant volume: Q </value_identification>
After compression: volume V2
<substitution> Substitute p2=p1*(V1/V2)ˆgamma into W1: W1 = [p1*V1/(gamma-1)] * [1-(V1/V2)ˆ(gamma-1)] = (3*p1*V1/2) * [1-(V1/V2)ˆ(2/3)] [gamma=5/3] W2=0 => W_total=W1 DU1=-W1=(3*p1*V1/2)*[(V1/V2)ˆ(2/3)-1], DU2=Q </substitution> <calculation> W = (3*p1*V1/2)*[1-(V1/V2)ˆ(2/3)] DU = (3*p1*V1/2)*[(V1/V2)ˆ(2/3)-1] + Q From DU1=n*Cv*(T2-T1) and Q=n*Cv*(T3-T2), with n=2: T3 = Q/(3R) + (p1*V1)/(2R)*(V1/V2)ˆ(2/3) </calculation> <final_answer> W = (3p1V1/2)[1-(V1/V2)ˆ(2/3)] DU = (3p1V1/2)[(V1/V2)ˆ(2/3)-1] + Q T3 = Q/(3R) + (p1V1/2R)*(V1/V2)ˆ(2/3) </final_answer>
Sample 2 (ID: 166)
Electromagnetism — Electron in crossed E and B fields, coupled ODEs reducing to cyclotron SHM
An electron is released from the origin where a uniform electric field E exists along the negative y-axis and a uniform magnetic field B exists along the negative z-axis. Find the displacement of the electron along the y-axis when its velocity becomes perpendicular to the electric field for the first time. Chain-of-Thought Solution — Sample 2 <problem_analysis> Electron (charge -e, mass m) starts from rest at origin. E=-E*j, B=-B*k. Velocity in xy-plane: v=vx*i+vy*j. Condition: vy=0 (v perpendicular to E). Find y at first vy=0 after t=0. </problem_analysis> <principle> Lorentz force: F=q(E+v x B). Coupled ODEs in vx,vy reduce to SHM in vy (cyclotron motion). </principle> <governing_equation> F = -e(E + v x B) omega = eB/m (cyclotron frequency) vy(t) = (E/B)*sin(omega*t) y(t) = (E/B*omega)*(1-cos(omega*t)) </governing_equation> <value_identification> E=electric field magnitude, B=magnetic field magnitude e=electron charge, m=electron mass, omega=eB/m Initial: vx(0)=vy(0)=0 </value_identification> <substitution> F = eE*j + eB*(vy*i - vx*j) dvx/dt=(eB/m)*vy ...(1) dvy/dt=(e/m)*(E-vx*B) ...(2) Differentiate (2), substitute (1): d2vy/dt2 = -omegaˆ2*vy => SHM With vy(0)=0, dvy/dt(0)=eE/m: vy(t)=(E/B)*sin(omega*t), y(t)=(E/B*omega)*(1-cos(omega*t)) </substitution> <calculation> vy=0 first at: omega*t=pi => t*=pi*m/(eB) y at t=t*: y=(E/B*omega)*(1-cos(pi))=2E/(B*omega)=2Em/(eBˆ2) </calculation> <final_answer> y = 2Em/(eBˆ2) </final_answer>
Sample 3 (ID: 889)
Nuclear Physics — Radioactive decay chain, rate equations, population maximization (JEE 2001)
1 −1 s . Initially only X nuclei are A radioactive nucleus X decays to Y with λX = 0.1 s−1 . Y decays to stable Z with λY = 30 N0 λX 20 present, N0 = 10 . (a) Set up rate equations for NX , NY , NZ . (b) Given NY (t) = λX −λY [e−λY t − e−λX t ], find t at which NY is maximum and determine NX , NZ at that instant.
Chain-of-Thought Solution — Sample 3 <problem_analysis> Decay chain X->Y->Z (Z stable). N_X(0)=1e20, N_Y(0)=N_Z(0)=0. Part (a): write coupled ODEs. Part (b): maximize N_Y(t).
</problem_analysis> <principle> Radioactive decay rate proportional to current population. N_Y sourced by X decay, depleted by Y decay. </principle> <governing_equation> dN_X/dt = -lambda_X * N_X dN_Y/dt = lambda_X*N_X - lambda_Y*N_Y dN_Z/dt = lambda_Y * N_Y </governing_equation> <value_identification> lambda_X=0.1 sˆ-1, lambda_Y=1/30 sˆ-1, N_X(t)=N0*exp(-lambda_X*t) </value_identification>
...(i) ...(ii) ...(iii)
N0=1e20
<substitution> dN_Y/dt=0 => lambda_X*N_X = lambda_Y*N_Y Substitute N_X, N_Y into (iv) and simplify: lambda_X/lambda_Y = exp((lambda_X-lambda_Y)*t) t* = ln(lambda_X/lambda_Y) / (lambda_X-lambda_Y) </substitution>
...(iv)
<calculation> t* = ln(3)/(2/30) = 15*ln(3) = 16.48 s N_X = 1e20*exp(-0.1*16.48) = 1.92e19 N_Y = N_X*lambda_X/lambda_Y = 1.92e19*3 = 5.76e19 N_Z = N0 - N_X - N_Y = 2.32e19 </calculation> <final_answer> t* = 15ln(3) = 16.48 s N_X = 1.92 x 10ˆ19, N_Z = 2.32 x 10ˆ19 </final_answer>
Sample 4 (ID: 2248)
Classical Mechanics — Compound Atwood machine, constraint analysis, Newton’s laws
Three blocks of masses m1 , m2 and m3 are connected via a compound pulley system (pulley B is itself suspended from a string over a fixed pulley). All surfaces are frictionless; strings and pulleys are massless. Find the acceleration of m1 . Chain-of-Thought Solution — Sample 4 <problem_analysis> m1 on frictionless table, string over fixed pulley to movable pulley B. m2 and m3 hang from B on either side. a0=accel. of m1 (right)=downward accel. of B. a=accel. of m2/m3 relative to B. Ground-frame: m2->(a0-a) down, m3->(a0+a) down. </problem_analysis> <principle> Newton’s Second Law for each body. Fixed string lengths impose kinematic constraints. Massless pulley B: net force=0. </principle> <governing_equation> Pulley B: 2T’=T => T’=T/2 m1: T=m1*a0 m2: m2*g-T/2=m2*(a0-a) m3: m3*g-T/2=m3*(a0+a) </governing_equation> <value_identification>
...(ii) ...(iii) ...(iv)
m1,m2,m3=block masses T=upper string tension, T’=T/2=lower string tension a0=accel. of m1 and B, a=relative accel. of m2/m3 w.r.t. B </value_identification> <substitution> Substitute T=m1*a0 into (iii),(iv): a0-a = g - m1*a0/(2*m2) a0+a = g - m1*a0/(2*m3) Add (iii’) and (iv’) to eliminate a: 2*a0 = 2g - (m1*a0/2)*(1/m2+1/m3) </substitution>
...(iii’) ...(iv’)
<calculation> a0*[1+(m1/4)*(1/m2+1/m3)] = g a0 = g / [1+(m1/4)*(1/m2+1/m3)] </calculation> <final_answer> a0 = g / [1 + (m1/4)*(1/m2 + 1/m3)] </final_answer>
All 2,494 training samples follow this structure. The dataset spans six physics domains: Mechanics, Electromagnetism, Thermodynamics, Waves, Optics, and Modern Physics. Difficulty is restricted to high school and early undergraduate level, consistent with the JEE source material.
A
Verifier Prompt
The external verifier (GPT-4o) receives the model-generated solution y1 and the ground truth solution y ∗ and returns a structured JSON object containing the first error step efirst , error type c1 ∈ {MC, CM, CE}, and a natural language explanation expl1 .
Listing 1: Verifier prompt used by GPT-4o to identify and classify the first reasoning error. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
eval_prompt = " " " You are a physics reasoning evaluator . Identify and explain the first reasoning error ( if any ) in the model 's solution . Classify the error as one of : - " Problem Miscomprehension " : misreading given values or problem conditions before any physics is applied . - " Conceptual Error " : wrong physics law , formula , or principle applied . - " Computational Error " : arithmetic , calculus , or algebraic mistake . Return JSON : { " error_step " : < first error step number > , " error_type " : " < one of the three categories > " , " error_explanation " : " < plain text explanation > " } If no error : { " error_step " : 0 , " error_type " : " no_error " , " error_explanation " : " No error in your solution . " } Question : { question } Ground Truth CoT : { cot_solution } LLM Generated Solution : { error_solution } """
B
Step-Level Reward Computation
The reward r = efirst /(n + 1) is computed from the verifier output. A solution with no error receives r = 1.0. Solutions where the first error occurs later in the chain receive higher reward.
Listing 2: Step-level reward computation from verifier output. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
def generate_reward ( sample , eval_model ) : decoded = eval_model . invoke ( eval_prompt . format ( question = sample [ ' question '] , cot_solution = sample [ ' cot_solution '] , error_solution = sample [ ' incorrect_solution '] ) ). content decoded = ast . literal_eval ( decoded ) error_step = decoded [ ' error_step '] if error_step == 0: reward = 1.0 # No error - full reward else : step_pattern = r " (?:#+\ s *) ? Step \ s +\ d +: " matches = re . findall ( step_pattern , sample [ ' incorrect_solution '] ) total_steps = len ( matches ) if matches else 1 # Earlier failure means lower reward reward = error_step / ( total_steps + 1) return reward , decoded
C
Structured Feedback Generation
Feedback is routed to one of three channels based on the classified error type. MC errors receive statement-level feedback directly from GPT-4o. CM errors invoke FeedbackRAGAgent using LLaMA 3.2 3B via Nebius API with ChromaDB retrieval over the NEET physics corpus. CE errors invoke CodeAgent using LLaMA 3.2 3B via Nebius API to execute corrected Python code.
Listing 3: Feedback routing by error type. MC errors receive statement feedback; CM errors invoke retrievalaugmented feedback; CE errors invoke code-execution feedback. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
def generate_feedback ( decoded , question ) : if decoded [ ' error_step '] == 0: return " There is no error in your solution . " if decoded [ ' error_type '] == ' Problem Miscomprehension ': # Statement feedback - direct prompt feedback = feedback_prompt . format ( error_step = decoded [ ' error_step '] , error_type = decoded [ ' error_type '] , error_explanation = decoded [ ' error_explanation '] ) elif decoded [ ' error_type '] == ' Conceptual Error ': # Retrieval - augmented feedback via LLaMA 3.2 3 B agent = FeedbackRAGAgent ( pdf_path = " physics - formulas - for - neet -2023. pdf " , chroma_path = " ./ chroma_db_feedback " , embedding_model_name = " sentence - transformers / all - MiniLM - L6 - v2 " , llm_model_name = " meta - llama / Llama -3.2 -3 B - Instruct " , llm_api_base = " https :// api . studio . nebius . com / v1 / " ) feedback = agent . run_feedback_cycle ( question = question , error_step = decoded [ " error_step " ] , error_explanation = decoded [ " error_explanation " ] )
40 41 42
elif decoded [ ' error_type '] == ' Computational Error ': # Code - execution feedback via LLaMA 3.2 3 B agent = CodeAgent ( question , decoded [ ' error_step '] , decoded [ ' error_explanation '] , llm_model_name = " meta - llama / Llama -3.2 -3 B - Instruct " , llm_api_base = " https :// api . studio . nebius . com / v1 / " ) feedback = agent . run ()
43 44
return feedback
32 33 34 35 36 37 38 39
D
Policy Gradient Training Objective
The policy is updated using REINFORCE with KL regularization. The loss penalizes the difference between attempt 1 and attempt 2 log-probabilities weighted by the step-level reward, minus a KL divergence term from the reference policy.
Listing 4: Policy gradient loss with KL regularization. r2 is the step-level reward on the revised solution y2 ; beta2 controls the KL penalty weight. 1 2 3 4 5 6 7 8 9
loss = -( ( ( att2_log_probs * attempt2_answer_mask [: , 1:]) . sum ( -1) / attempt2_answer_mask [: , 1:]. sum ( -1) ( att1_log_probs [: , 1:] * attempt1_answer_mask [: , 1:]) . sum ( -1) / attempt1_answer_mask [: , 1:]. sum ( -1) ) * reward2_tensor - beta2 * kl_div ). mean ()
E
Training Hyperparameters Hyperparameter
Value
Batch size
5
Stage 1 epochs (SFT)
10
Stage 2 epochs (RL)
60
Learning rate
5 × 10−6
KL penalty β
0.1
Early stopping τ
0.9
Max prompt length
512 tokens
Max attempt length
1,240 tokens
LoRA rank
16
LoRA α
32
Temperature
1.0
Table 8: Training hyperparameters used across all four models. Discussed in §3.
B
SFT Training Configuration
Supervised fine-tuning is performed on all four ultra-small models using LoRA (Hu et al., 2022) adapters via the Hugging Face PEFT framework. The warm-up stage trains each model on the 2,494-problem fine-tuning corpus for a fixed number of epochs, after which the LoRA adapter is frozen and used as the initialisation point for DPO training. All experiments use a fixed seed of 42 throughout for full reproducibility. Effective batch size equals per_device_batch_size × gradient_accumulation_steps. All models use cosine learning rate scheduling, weight_decay=0.01, bf16=True, and are trained on a single H100 GPU.
Shared Configuration — All Models task_type = CAUSAL_LM target_modules = q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj lora_dropout = 0.05 bias = none packing = True lr_scheduler_type = cosine warmup_ratio = 0.05 weight_decay = 0.01 precision = bf16 hardware = single H100 GPU seed = 42 alpha rule = alpha = 2 × r (consistent LoRA update magnitude)
Per-Model Hyperparameters Model
r
alpha
LR
Epochs
Batch
Grad Accum
Eff. Batch
Seq Len
Qwen 2.5 1.5B LLaMA 3.2 1B LLaMA 3.2 3B Phi 3.5 Mini 3.8B
32 32 32 64
64 64 64 128
1e-4 1e-4 1e-4 5e-5
3 3 3 3
4 4 4 2
8 8 8 16
32 32 32 32
4096 4096 4096 4096
All models share max_grad_norm=1.0. alpha = 2 × r is maintained throughout for consistent LoRA update magnitude scaling and cross-model comparability.
Training Prompt Format — Inference-Consistent Instruction Template You are an expert physics assistant. You are given a physics problem. Your task is to generate the complete solution using exactly the following seven XML tags in order: <problem_analysis> Restate the givens, unknowns, domain, and assumptions. <principle> Name the physical law in plain English. <governing_equation> Give the symbolic formula only and define all symbols. <value_identification> List every given value explicitly with its units. <substitution> Substitute the values line by line and show each step. <calculation> Show the arithmetic step by step, carrying the units throughout. <final_answer> Give the answer with units and use \boxed{}.
Critical rules: – Never skip a tag and never add extra tags. – The <governing_equation> tag must use symbolic form only, with no numerical values. – The <substitution> tag must reference the values listed in <value_identification> exactly. – If the solution has multiple parts, repeat the complete tag structure for each part. Question: {question}
SFT Solution Structuring Prompt — Automated XML Conversion You are converting physics solutions into a structured educational format. Your output must use exactly these seven XML tags in exactly this order. Never skip a tag. Never add extra tags. <problem_analysis> Restate the givens, unknowns, domain, and assumptions. <principle> Name the physical law in plain English. <governing_equation> Give the symbolic formula only and define all symbols. <value_identification> List every given value explicitly with its units. <substitution> Substitute the values line by line and show each step. <calculation> Show the arithmetic step by step, carrying the units throughout. <final_answer> Give the answer with units and use \boxed{}. Critical rules: – The <value_identification> tag must explicitly list EVERY numerical value from the problem statement. – The <governing_equation> tag must contain symbolic expressions only, with no numerical values. – The <substitution> tag must reference the values listed in <value_identification> exactly. – Preserve all LaTeX mathematical notation. – If the solution has multiple parts, repeat the complete tag structure for each part. Problem: {question} Original Solution: {solution}
Training time per model ranged from 45 to 60 minutes on a single H100 GPU. The resulting LoRA adapter is saved as a checkpoint and used directly as the starting point for DPO training, with no intermediate merging into the base model weights.
C
DPO Dataset Samples and Configuration
Direct Preference Optimization is applied on top of the SFT LoRA checkpoint. Each DPO training pair consists of a chosen solution (fully correct, all seven tags structurally and physically consistent) and a rejected solution (structurally identical but with one critical error injected at a single tag). The error propagates as a domino through all subsequent tags, producing a fluent, complete, but physically incorrect solution. The <problem_analysis> and <principle> tags are held identical across chosen and rejected in every pair, isolating the preference signal to the reasoning execution rather than problem interpretation.
DPO Rejected Sample Generation Prompt You are a physics expert tasked with generating a REJECTED (incorrect) response for a physics problem. This is meant to be used as a negative example in Direct Preference Optimization (DPO). Your task is to generate a rejected solution that: – closely follows the structure and reasoning style of the correct solution – introduces a subtle but critical mistake, such as a physics misconception, mathematical error, or incorrect substitution, midway through the steps – clearly results in an INCORRECT final answer – maintains the same level of detail and formatting as the correct solution – makes sure the LENGTH is similar to the accepted solution, with a similar token count – uses exactly the same XML tags as the correct solution: <problem_analysis> <principle> <governing_equation> <value_identification> <substitution> <calculation> <final_answer> DO NOT include any meta-commentary. Output ONLY the flawed solution. Problem: {question} Correct Solution (for reference): {solution} Now generate ONLY the flawed (rejected) solution:
Pair 1 (ID: 2162)
Thermodynamics — Adiabatic compression + isochoric heating
Error injected at <governing_equation>: γ set to 7/5 (diatomic) instead of 5/3 (monatomic). Domino propagates through substitution, calculation, and final answer. Chosen — Correct <problem_analysis> Two sequential processes: (1) adiabatic compression V1->V2, (2) isochoric heating at V2 with heat Q. Monatomic ideal gas, n=2 mol => gamma=5/3, Cv=3R/2. </problem_analysis> <principle> First Law of Thermodynamics: DQ=DU+DW. Adiabatic: DQ=0 => DU=-W. Isochoric: DW=0 => DU=DQ. </principle> <governing_equation> Adiabatic: p1*V1ˆgamma = p2*V2ˆgamma [gamma=5/3, monatomic] W_adiabatic = (p1*V1 - p2*V2)/(gamma-1)
</governing_equation> <value_identification> n=2 mol, gamma=5/3, Cv=(3/2)R Initial: p1, V1. After compression: V2. Heat added: Q. </value_identification> <substitution> W1 = (3*p1*V1/2)*[1-(V1/V2)ˆ(2/3)] [from gamma=5/3] DU1=-W1=(3*p1*V1/2)*[(V1/V2)ˆ(2/3)-1], DU2=Q </substitution> <calculation> W = (3*p1*V1/2)*[1-(V1/V2)ˆ(2/3)] DU = (3*p1*V1/2)*[(V1/V2)ˆ(2/3)-1] + Q T3 = Q/(3R) + (p1*V1)/(2R)*(V1/V2)ˆ(2/3) </calculation> <final_answer> W=(3p1V1/2)[1-(V1/V2)ˆ(2/3)], DU=(3p1V1/2)[(V1/V2)ˆ(2/3)-1]+Q T3=Q/(3R)+(p1V1/2R)*(V1/V2)ˆ(2/3) </final_answer>
Rejected — Error at <governing_equation>: γ = 7/5 used instead of 5/3 <problem_analysis> [Identical to chosen] </problem_analysis> <principle> [Identical to chosen] </principle> <governing_equation> Adiabatic: p1*V1ˆgamma = p2*V2ˆgamma [ERROR: gamma=7/5, diatomic value] W_adiabatic = (p1*V1 - p2*V2)/(gamma-1) </governing_equation> <value_identification> n=2 mol, gamma=7/5, Cv=(5/2)R [ERROR: wrong Cv for monatomic gas] Initial: p1, V1. After compression: V2. Heat added: Q. </value_identification> <substitution> W1 = (5*p1*V1/2)*[1-(V1/V2)ˆ(2/5)] [wrong exponent from gamma=7/5] DU1=-W1=(5*p1*V1/2)*[(V1/V2)ˆ(2/5)-1], DU2=Q </substitution> <calculation> W = (5*p1*V1/2)*[1-(V1/V2)ˆ(2/5)] [WRONG] DU = (5*p1*V1/2)*[(V1/V2)ˆ(2/5)-1] + Q [WRONG] T3 = Q/(5R) + (p1*V1)/(2R)*(V1/V2)ˆ(2/5) [WRONG] </calculation> <final_answer> W=(5p1V1/2)[1-(V1/V2)ˆ(2/5)], DU=(5p1V1/2)[(V1/V2)ˆ(2/5)-1]+Q T3=Q/(5R)+(p1V1/2R)*(V1/V2)ˆ(2/5) [INCORRECT] </final_answer>
Pair 2 (ID: 166)
Electromagnetism — Electron in crossed E and B fields
Error injected at <substitution>: force sign flipped — electron treated as positive charge. Domino propagates through velocity, displacement, and final answer. Chosen — Correct <problem_analysis> Electron (charge -e, mass m) from rest at origin. E=-E*j, B=-B*k. Find y when vy=0 first time after t=0. </problem_analysis>
<principle> Lorentz force: F=q(E+v x B). Coupled ODEs reduce to SHM in vy. </principle> <governing_equation> F=-e(E+v x B), omega=eB/m vy(t)=(E/B)*sin(omega*t), y(t)=(E/B*omega)*(1-cos(omega*t)) </governing_equation> <value_identification> E,B=field magnitudes, e=electron charge, m=electron mass omega=eB/m, Initial: vx(0)=vy(0)=0 </value_identification> <substitution> F=-e(-E*j+v x (-B*k))=eE*j+eB*(vy*i-vx*j) dvx/dt=(eB/m)*vy ...(1), dvy/dt=(e/m)*(E-vx*B) ...(2) d2vy/dt2=-omegaˆ2*vy => vy=(E/B)*sin(omega*t) </substitution> <calculation> vy=0 at omega*t=pi => t*=pi*m/(eB) y=(E/B*omega)*(1-cos(pi))=2Em/(eBˆ2) </calculation> <final_answer> y = 2Em/(eBˆ2) </final_answer>
Rejected — Error at <substitution>: charge sign flipped, q = +e
<problem_analysis> [Identical to chosen] </problem_analysis> <principle> [Identical to chosen] </principle> <governing_equation> [Identical to chosen] </governing_equation> <value_identification> [Identical to chosen] </value_identification> <substitution> F=+e(-E*j+v x (-B*k)) [ERROR: sign flipped, q=+e used] =-eE*j+eB*(...) dvx/dt=-(eB/m)*vy ...(1), dvy/dt=(e/m)*(vx*B-E) ...(2) d2vy/dt2=-omegaˆ2*vy => vy=-(E/B)*sin(omega*t) [wrong sign] </substitution> <calculation> vy=0 at omega*t=pi => t*=pi*m/(eB) [same time, wrong trajectory] y=-(E/B*omega)*(1-cos(pi))=-2Em/(eBˆ2) [WRONG: negative displacement] </calculation> <final_answer> y = -2Em/(eBˆ2) </final_answer>
[INCORRECT: electron displaced in wrong direction]
All DPO pairs follow this structure. The <problem_analysis> and <principle> tags are identical across chosen and rejected in every pair. The error is introduced at exactly one tag and propagates consistently through all subsequent tags, producing a fluent, structurally complete, but physically incorrect rejected solution. This controlled design ensures the preference signal targets reasoning quality rather than structural formatting.
DPO Training Configuration — Shared (All Models) loss_type = ipo label_smoothing = 0.05 max_prompt_length = 256 lr_scheduler_type = cosine warmup_ratio = 0.1 gradient_checkpointing = True reference_free = False optimizer = AdamW epochs = 1 seed = 42
Per-Model Hyperparameters Model
beta
LR
Batch
Grad Accum
Eff. Batch
max_grad_norm
max_length
Qwen 2.5 1.5B LLaMA 3.2 1B LLaMA 3.2 3B Phi 3.5 Mini 3.8B
0.10 0.10 0.10 0.05
2e-6 2e-6 2e-6 1e-6
2 2 2 1
16 16 16 16
32 32 32 16
1.0 1.0 1.0 0.5
4096 4096 4096 4096
beta is set lower for Phi 3.5 Mini to preserve pretrained physics knowledge acquired during SFT. Learning rate decreases with model size as larger models are more sensitive to weight updates post-LoRA. Each DPO training run takes approximately 1.5–2 hours on a single H100 GPU. Early stopping is applied based on validation loss to prevent over-optimisation.
E
Retrieval-Augmented Generation (RAG) Setup
RAG is used as a strong baseline that provides models with external physics knowledge at inference time, without any parameter updates. The design intentionally constrains what is retrievable to isolate the model’s reasoning ability from its recall ability — the system can retrieve the correct formula but must still apply it correctly.
Knowledge Base The retrieval corpus is a concise physics formula sheet covering six domains: Mechanics, Waves, Optics, Heat and Thermodynamics, Electricity and Magnetism, and Modern Physics, together with a table of standard physical constants. The corpus contains only symbolic formulas, definitions, and physical constants — it explicitly excludes narrative explanations, worked examples, and problem-solving strategies. This design ensures the model cannot locate and reproduce a pre-existing solution. The model must still perform all critical reasoning steps: identifying the relevant formula from context, performing correct substitution, and executing the calculation. The full formula sheet used as the RAG knowledge base is shown in Figure 5.
Chunking and Embedding Configuration The formula sheet is chunked into segments of 500 characters with a 50-character overlap to preserve context across formula boundaries. Chunks are embedded using OpenAI’s text-embedding-ada-002 model and stored in a vector store built with LangChain. Chunking strategy = 500-character segments, 50-character overlap Embedding model = text-embedding-ada-002 (OpenAI) Vector store = LangChain FAISS Retrieval = top-k = 3 most similar chunks Similarity metric = cosine similarity
Inference Prompt — RAG Instruction Template You are an expert physics assistant. Use the provided context to solve the physics problem below. Think step by step. Context (retrieved physics formulas): {retrieved_chunks} Question: {question} Using the context above, provide a complete step-by-step solution. Your final answer should be clearly stated at the end.
The RAG baseline receives identical prompting structure to all other conditions. The only difference is the injected context block. No parameter updates are performed during RAG inference — the knowledge base is static across all models and benchmarks.
Motion in a straight line with constant a: v = u + at,
Gravitation constant Boltzmann constant Molar gas constant Avogadro’s number Charge of electron Permeability of vacuum Permitivity of vacuum 0 1 Coulomb constant 4π0 Faraday constant F Mass of electron me Mass of proton mp Mass of neutron mn Atomic mass unit u Atomic mass unit u Stefan-Boltzmann σ constant Rydberg constant R∞ Bohr magneton µB Bohr radius a0 Standard atmosphere atm Wien displacement b constant
3 × 108 m/s 6.63 × 10−34 J s 1242 eV-nm 6.67×10−11 m3 kg−1 s−2 1.38 × 10−23 J/K 8.314 J/(mol K) 6.023 × 1023 mol−1 1.602 × 10−19 C 4π × 10−7 N/A2
u u sin θ
c h hc G k R NA e µ0
Projectile Motion:
x
H
R
y = ut sin θ − 12 gt2 g y = x tan θ − 2 x2 2u cos2 θ 2u sin θ u2 sin 2θ u2 sin2 θ T = , R= , H= g g 2g x = ut cos θ,
8.85 × 10−12 F/m 9 × 109 N m2 /C2 96485 C/mol 9.1 × 10−31 kg 1.6726 × 10−27 kg 1.6749 × 10−27 kg 1.66 × 10−27 kg 931.49 MeV/c2 5.67×10−8 W/(m2 K4 )
1.3: Newton’s Laws and Friction Linear momentum: p~ = m~v Newton’s first law: inertial frame. p Newton’s second law: F~ = d~ dt ,
1.097 × 107 m−1 9.27 × 10−24 J/T 0.529 × 10−10 m 1.01325 × 105 Pa 2.9 × 10−3 m K
F~ = m~a
Newton’s third law: F~AB = −F~BA Frictional force: fstatic, max = µs N, Banking angle:
2
µ+tan θ v v rg = tan θ, rg = 1−µ tan θ 2
ac = vr
Pseudo force: F~pseudo = −m~a0 ,
MECHANICS
fkinetic = µk N
2
2
Fcentrifugal = − mv r
2
Minimum speed to complete vertical circle: p p vmin, bottom = 5gl, vmin, top = gl
1.1: Vectors Notation: ~a = ax ı̂ + ay ̂ + az k̂ q Magnitude: a = |~a| = a2x + a2y + a2z
Conical pendulum: T = 2π
Dot product: ~a · ~b = ax bx + ay by + az bz = ab cos θ Cross product:
y
θ u cos θ
O
Centripetal force: Fc = mv r ,
1
v 2 − u2 = 2as
Relative Velocity: ~vA/B = ~vA − ~vB
0.1: Physical Constants Speed of light Planck constant
s = ut + 12 at2 ,
~ a × ~b
l l cos θ g
~ a
k̂
mg ̂
~a ×~b = (ay bz − az by )ı̂ + (az bx − ax bz )̂ + (ax by − ay bx )k̂ |~a × ~b| = ab sin θ
θ
θ T
ı̂
~b θ
q
1.4: Work, Power and Energy ~ = F S cos θ, Work: W = F~ · S
W = 2
p Kinetic energy: K = 12 mv 2 = 2m
R
~ F~ · dS
Potential energy: F = −∂U/∂x for conservative forces.
1.2: Kinematics Average and Instantaneous Vel. and Accel.: ~vav = ∆~r/∆t,
~vinst = d~r/dt
~aav = ∆~v /∆t
~ainst = d~v /dt
Ugravitational = mgh,
Uspring = 12 kx2
Work done by conservative forces is path independent and depends only on initial and final points: H F~conservative · d~r = 0. Work-energy theorem: W = ∆K
Figure 5: The physics formula sheet used as the RAG knowledge base. The corpus is strictly limited to formulas, definitions, and physical constants. It explicitly lacks narrative explanations, problem-solving strategies, or worked examples, ensuring the RAG system acts as a formula reference rather than a solution retriever.
Mechanical energy: E = U + K. Conserved if forces are conservative in nature. Power Pav = ∆W ∆t ,
Pinst = F~ · ~v
R
P Pxi mi , mi
ω = ω0 + αt,
θ = ωt + 12 αt2 ,
Moment of Inertia: I =
1.5: Centre of Mass and Collision Centre of mass: xcm =
Rotation about an axis with constant α:
xdm
xcm = R dm
P
i mi ri
2
,
ω 2 − ω0 2 = 2αθ I=
R
r2 dm
2 1 2 2 2 mr m(a +b ) 12
mr 2
2 1 2 mr
2 2 3 mr
2 2 5 mr
2 1 12 ml
mr 2
ring
disk
shell
sphere
rod
hollow
b
a
CM of few useful configurations: m1
m2
r
1. m1 , m2 separated by r:
C m2 r m1 +m2
m1 r m1 +m2
2. Triangle (CM ≡ Centroid) yc = h3
C
3. Semicircular ring: yc = 2r π 4.
solid rectangle
Theorem of Perp. Axes: Iz = Ix + Iy 2r π
Radius of Gyration: k =
5. Hemispherical shell: yc = 2r
r
6. Solid Hemisphere: yc = 3r 8
r
cm z
h 3
C
r
Ic d
h
r
4r Semicircular disc: yc = 3π
Ik
Theorem of Parallel Axes: Ik = Icm + md2
C
p
x
I/m
4r 3π
~ = ~r × p~, Angular Momentum: L
C
r 2
Torque: ~τ = ~r × F~ ,
C
3r 8
7. Cone: the height of CM from the base is h/4 for the solid cone and h/3 for the hollow cone.
y
~
~τ = ddtL ,
~ = I~ L ω y
P θ ~ F ~ r x
τ = Iα O
~ ~τext = 0 =⇒ L ~ = const. Conservation of L: P~ P Equilibrium condition: F = ~0, ~τ = ~0 Kinetic Energy: Krot = 12 Iω 2
Motion of the CM: M = ~vcm = Impulse: J~ = Collision:
P R
mi~vi , M
P
Dynamics:
mi
p~cm = M~vcm ,
~acm =
~τcm = Icm α ~,
F~ext M
F~ext = m~acm , p~cm = m~vcm ~ L = Icm ω ~ + ~rcm × m~vcm
K = 12 mvcm 2 + 12 Icm ω 2 ,
F~ dt = ∆~ p 1.7: Gravitation Before collision After collision m1
m2
v1
v2
m1 v10
m2 v20
Momentum conservation: m1 v1 +m2 v2 = m1 v10 +m2 v20 2 2 Elastic Collision: 12 m1 v1 2+ 12 m2 v2 2 = 12 m1 v10 + 12 m2 v20 Coefficient of restitution: −(v10 − v20 ) 1, completely elastic e= = 0, completely in-elastic v1 − v2 If v2 = 0 and m1 m2 then v10 = −v1 . If v2 = 0 and m1 m2 then v20 = 2v1 . Elastic collision with m1 = m2 : v10 = v2 and v20 = v1 .
2 Gravitational force: F = G mr1 m 2
m1
F F r
Potential energy:
U = − GMr m
Gravitational acceleration: g = GM R2 h Variation of g with depth: ginside ≈ g 1 − R
Variation of g with height: goutside ≈ g 1 − 2h R Effect of non-spherical earth shape on g: gat pole > gat equator (∵ Re − Rp ≈ 21 km) Effect of earth rotation on apparent weight:
1.6: Rigid Body Dynamics Angular velocity: ωav = ∆θ ∆t ,
ω = dθ dt ,
~v = ω ~ × ~r
Angular Accel.: αav = ∆ω ∆t ,
α = dω dt ,
~a = α ~ × ~r
Figure 5 continued (Page 2).
m2
Waves
2 4. 1st overtone/2nd harmonics: ν1 = 2L
2.1: Waves Motion
q
2
2
∂ y 1 ∂ y General equation of wave: ∂x 2 = v 2 ∂t2 .
T µ
q
3 5. 2nd overtone/3rd harmonics: ν2 = 2L
6. All harmonics are present.
Notation: Amplitude A, Frequency ν, Wavelength λ, Period T , Angular Frequency ω, Wave Number k, 1 2π T = = , ν ω
2π k= λ
v = νλ,
y = f (t − x/v),
+x;
N
A
1. Boundary conditions: y = 0 at x = 0
−x
y A
x
λ
λ 2
y = A sin(kx − ωt) = A sin(2π (x/λ − t/T )) 2.2: Waves on a String
2. Allowed Freq.: L = (2n + 1) λ4 , ν = 2n+1 4L 0, 1, 2, . . .. q 1 T 3. Fundamental/1st harmonics: ν0 = 4L µ q 3 T 4. 1st overtone/3rd harmonics: ν1 = 4L µ q 5 nd th 5. 2 overtone/5 harmonics: ν2 = 4L Tµ
q
T µ,
n =
6. Only odd harmonics are present.
Speed of waves on a string with mass per unit length µ p and tension T : v = T /µ Transmitted power: Pav = 2π 2 µvA2 ν 2
Sonometer: ν ∝ L1 , ν ∝
Interference:
2.3: Sound Waves
y1 = A1 sin(kx − ωt),
A
N
λ/2
y = f (t + x/v),
Progressive sine wave:
T µ
L
String fixed at one end:
Progressive wave travelling with speed v:
y2 = A2 sin(kx − ωt + δ)
y = y1 + y2 = A sin(kx − ωt + ) q A = A1 2 + A2 2 + 2A1 A2 cos δ
Standing Waves:
A
N
√
n T , ν ∝ √1µ . ν = 2L
q
T µ
Displacement wave: s = s0 sin ω(t − x/v) Pressure wave: p = p0 cos ω(t − x/v), p0 = (Bω/v)s0 Speed of sound waves: s s B Y vliquid = , vsolid = , ρ ρ
A2 sin δ tan = A1 + A2 cos δ 2nπ, constructive; δ= (2n + 1)π, destructive. 2A cos kx
2
2
2
vgas =
s
γP ρ
2
p0 0 v Intensity: I = 2πv B s0 2 ν 2 = p2B = 2ρv
A
N
A
x
Standing longitudinal waves:
λ/4
y1 = A1 sin(kx − ωt),
p1 = p0 sin ω(t − x/v),
y2 = A2 sin(kx + ωt)
p2 = p0 sin ω(t + x/v)
p = p1 + p2 = 2p0 cos kx sin ωt
y = y1 + y2 = (2A cos kx) sin ωt n + 12 λ2 , nodes; n = 0, 1, 2, . . . x= n λ2 , antinodes. n = 0, 1, 2, . . . Closed organ pipe:
L
String fixed at both ends:
N
A
N
A
L
N
1. Boundary condition: y = 0 at x = 0
λ/2
1. Boundary conditions: y = 0 at x = 0 and at x = L q n T 2. Allowed Freq.: L = n λ2 , ν = 2L µ , n = 1, 2, 3, . . .. q 1 T 3. Fundamental/1st harmonics: ν0 = 2L µ
v 2. Allowed freq.: L = (2n + 1) λ4 , ν = (2n + 1) 4L , n= 0, 1, 2, . . . v 3. Fundamental/1st harmonics: ν0 = 4L 3v 4. 1st overtone/3rd harmonics: ν1 = 3ν0 = 4L
Figure 5 continued (Page 4).
3
Optics
h
Lens maker’s formula: f1 = (µ − 1)
3.1: Reflection of Light
1 1 R1 − R2
i f
normal
Laws of reflection:
i r
incident
(i)
reflected
Lens formula: v1 − u1 = f1 ,
m = uv u
Incident ray, reflected ray, and normal lie in the same plane (ii) ∠i = ∠r
Power of the lens:
v
P = f1 , P in diopter if f in metre.
Two thin lenses separated by distance d:
Plane mirror: d
d
(i) the image and the object are equidistant from mirror (ii) virtual image of real object
1 1 1 d = + − F f1 f2 f1 f2
d f1
f2
I
Spherical Mirror:
O f
3.3: Optical Instruments
v
u
Simple microscope: m = D/f in normal adjustment.
1. Focal length f = R/2
Eyepiece
Objective
2. Mirror equation: v1 + u1 = f1 3. Magnification: m = − uv
Compound microscope:
∞
O
u
v
3.2: Refraction of Light
fe D
speed of light in vacuum c Refractive index: µ = speed of light in medium = v incident µ1 i
µ2 sin i Snell’s Law: sin r = µ1
µ2
1. Magnification in normal adjustment: m = uv fDe
reflected
fo
r
fe
refracted d0 d I O
real depth d Apparent depth: µ = apparent depth = d0
Critical angle: θc = sin−1 µ1
θ 1 2. Resolving power: R = ∆d = 2µ sin λ
µ
Astronomical telescope:
1. In normal adjustment: m = − ffoe , L = fo + fe
θc
1 1 2. Resolving power: R = ∆θ = 1.22λ
A
3.4: Dispersion
δ
Deviation by a prism:
i
r
r
i0
0
µ 0
δ = i + i − A,
m sin A+δ 2 µ= , sin A2
1. Mean deviation: δy = (µy − 1)A
general result 0
i = i for minimum deviation for small A
δm
µ2
P
Q
O u
m=
Dispersion without deviation:
i
µ1
µ2 µ1 µ2 − µ1 − = , v u R
2. Angular dispersion: θ = (µv − µr )A θ r Dispersive power: ω = µµvy−µ −1 ≈ δy (if A and i small)
i0
Refraction at spherical surface:
A>0
Dispersion by prism with small A and i:
δ
δm = (µ − 1)A,
Cauchy’s equation: µ = µ0 + λA2 ,
v
(µy − 1)A + (µ0y − 1)A0 = 0 Deviation without dispersion: (µv − µr )A = (µ0v − µ0r )A0
µ1 v µ2 u
Figure 5 continued (Page 6).
A
µ0
µ
A0
4
Heat and Thermodynamics
4.4: Theromodynamic Processes First law of thermodynamics: ∆Q = ∆U + ∆W
4.1: Heat and Temperature Temp. scales: F = 32 + 95 C,
Work done by the gas:
K = C + 273.16
Ideal gas equation: pV = nRT ,
n : number of moles a van der Waals equation: p + V 2 (V − b) = nRT
∆W = p∆V,
Wisobaric = p(V2 − V1 ) p1 V1 − p2 V2 Wadiabatic = γ−1 Wisochoric = 0
F ∆l A =Y l
Thermal stress of a material:
Z V2
pdV V 1 V2 Wisothermal = nRT ln V1
Thermal expansion: L = L0 (1 + α∆T ), A = A0 (1 + β∆T ), V = V0 (1 + γ∆T ), γ = 2β = 3α
W =
4.2: Kinetic Theory of Gases General: M = mNA , k = R/NA
T1 Q1
Efficiency of the heat engine:
n
W Q2 T2
Maxwell distribution of speed: vp v̄ vrms
RMS speed: vrms = Average speed: v̄ =
q
q
3kT m
=
8kT πm
=
Most probable speed: vp = 2 Pressure: p = 13 ρvrms
q
q
q
v
3RT M
work done by the engine Q1 − Q2 = heat supplied to it Q1 Q2 T2 ηcarnot = 1 − =1− Q1 T1 η=
8RT πM
T1 Q1
Coeff. of performance of refrigerator:
2kT m
W Q2 T2
1 2 kT
Equipartition of energy: K = for each degree of freedom. Thus, K = f2 kT for molecule having f degrees of freedoms. Internal energy of n moles of an ideal gas is U = f2 nRT .
Q2 2 COP = Q W = Q1 −Q2
Entropy: ∆S = ∆Q T , Sf − Si = Const. T : ∆S = Q T,
R f ∆Q i
T
T
Varying T : ∆S = ms ln Tfi
Adiabatic process: ∆Q = 0, pV γ = constant 4.3: Specific Heat
4.5: Heat Transfer
Q Specific heat: s = m∆T
∆T Conduction: ∆Q ∆t = −KA x
Latent heat: L = Q/m
x Thermal resistance: R = KA
∆Q Specific heat at constant volume: Cv = n∆T
Specific heat at constant pressure: Cp =
V
∆Q n∆T
Rseries = R1 + R2 = A1 p
Relation between Cp and Cv : Cp − Cv = R Ratio of specific heats:
x1 x2 K1 + K2
K1
K2
x1
x2
1 1 1 1 Rparallel = R1 + R2 = x (K1 A1 + K2 A2 )
K2
A2
K1
A1
x
γ = Cp /Cv
Relation between U and Cv : ∆U = nCv ∆T
E
emissive power body Kirchhoff ’s Law: absorptive power = abody = Eblackbody
Specific heat of gas mixture: n1 Cv1 + n2 Cv2 Cv = , n1 + n2
A
Eλ
n1 Cp1 + n2 Cp2 γ= n1 Cv1 + n2 Cv2
Molar internal energy of an ideal gas: U = f2 RT , f = 3 for monatomic and f = 5 for diatomic gas.
Wien’s displacement law: λm T = b λm 4 Stefan-Boltzmann law: ∆Q ∆t = σeAT
Newton’s law of cooling: dT dt = −bA(T − T0 )
Figure 5 continued (Page 7).
λ
5
Electricity and Magnetism
5.3: Capacitors Capacitance: C = q/V
5.1: Electrostatics 1 q1 q2 Coulomb’s law: F~ = 4π 2 r̂ 0 r
q1
~ r) = 1 q2 r̂ Electric field: E(~ 4π0 r
r
q
q2
Electrostatic potential: V ~ · ~r, dV = −E
~ r
Spherical capacitor:
Z ~r ∞
+q d
A
r2
1 q = 4π 0 r
V (~r) = −
A
~ E
1 q1 q2 U = − 4π r 0
Electrostatic energy:
−q
Parallel plate capacitor: C = 0 A/d
~ · d~r E
0 r1 r2 C = 4π r2 −r1
−q +q
2π0 l Cylindrical capacitor: C = ln(r 2 /r1 )
r1
r2
l
r1
p ~
Electric dipole moment: p~ = q d~
−q
+q
d
Capacitors in parallel: Ceq = C1 + C2
A C1
C2
B 1 p cos θ Potential of a dipole: V = 4π r2 0
V (r)
θ r
Capacitors in series: C1eq = C11 + C12
p ~ Er
Field of a dipole:
θ r
Eθ
p ~ 1 2p cos θ Er = 4π , r3 0
C2 B
Force between plates of a parallel plate capacitor: Q2 F = 2A 0 2
Q Energy stored in capacitor: U = 12 CV 2 = 2C = 12 QV
1 p sin θ Eθ = 4π r3 0
Energy density in electric field E: U/V = 12 0 E 2
~ ~τ = p~ × E ~ Torque on a dipole placed in E: ~ U = −~ ~ Pot. energy of a dipole placed in E: p·E
Capacitor with dielectric: C = 0 KA d
5.2: Gauss’s Law and its Applications H ~ · dS ~ Electric flux: φ = E H ~ · dS ~ = qin /0 Gauss’s law: E
5.4: Current electricity Current density: j = i/A = σE i Drift speed: vd = 12 eE m τ = neA
Resistance of a wire: R = ρl/A, where ρ = 1/σ
Field of a uniformly charged ring on its axis: qx 1 EP = 4π 2 2 3/2 0 (a +x )
C1 A
q
a x
P
~ E
Temp. dependence of resistance: R = R0 (1 + α∆T ) Ohm’s law: V = iR
E and V (of a uniformly charged sphere: 1 Qr 4π0 R3 , for r < R E E= 1 Q for r ≥ R 4π0 r 2 , O ( Q r2 3 − , for r < R V R2 0R V = 8π 1 Q for r ≥ R 4π0 r , O
R
R
E and V of a uniformly charged spherical shell: 0, for r < R E E= 1 Q 4π0 r 2 , for r ≥ R O R ( 1 Q , for r < R 4π0 R V V = 1 Q 4π0 r , for r ≥ R O
R
Field of a line charge: E = 2πλ0 r
r
r
Kirchhoff ’s Laws: (i) The Junction Law: The algebraic sum of all the currents directed towards a node is zero i.e., Σnode Ii = 0. (ii)The Loop Law: The algebraic sum of all the potential differences along a closed loop in a circuit is zero i.e., Σloop ∆ Vi = 0. Resistors in parallel: R1eq = R11 + R12
r
Resistors in series: Req = R1 + R2
A R1
A
R1
R2
R1
r
Wheatstone bridge:
Balanced if R1 /R2 = R3 /R4 .
Field in the vicinity of conducting surface: E = σ0
Electric Power: P = V 2 /R = I 2 R = IV
Figure 5 continued (Page 8).
B R2
↑ G R3
R4 V
Field of an infinite sheet: E = 2σ0
R2
B
6
Modern Physics
N N0
Population at time t: N = N0 e−λt
6.1: Photo-electric effect
N0 2
O
Photon’s energy: E = hν = hc/λ
t1/2
Photon’s momentum: p = h/λ = E/c
Half life: t1/2 = 0.693/λ
Max. KE of ejected photo-electron: Kmax = hν − φ
Average life: tav = 1/λ
Threshold freq. in photo-electric effect: ν0 = φ/h
Population after n half lives: N = N0 /2n .
Stopping potential: Vo = hc e
1 λ
Mass defect: ∆m = [Zmp + (A − Z)mn ] − M
V0 hc e
− φe −φ e
φ hc
t
1 λ
Binding energy: B = [Zmp + (A − Z)mn − M ] c2 Q-value: Q = Ui − Uf
de Broglie wavelength: λ = h/p
= ∆mc2
Energy released in nuclear reaction: ∆E where ∆m = mreactants − mproducts .
6.2: The Atom 6.4: Vacuum tubes and Semiconductors
Energy in nth Bohr’s orbit: 2 4
En = −
2
mZ e , 80 2 h2 n2
13.6Z eV n2
Half Wave Rectifier:
a0 = 0.529 Å
Full Wave Rectifier:
En = −
D
rn =
0 h2 n2 , πmZe2
rn =
n2 a0 , Z
R Output
˜
Radius of the nth Bohr’s orbit:
˜
Output
Quantization of the angular momentum: l = nh 2π
Grid
Triode Valve:
Photon energy in state transition: E2 − E1 = hν E2
Plate
E2
hν E1
Cathode Filament
hν E1 Absorption
Emission
∆V
Plate resistance of a triode: rp = ∆ipp
Wavelength of emitted radiation: for from nth to mth state: 1 1 1 = RZ 2 2 − 2 λ n m
a
∆Vp =0
∆ip =0
Relation between rp , µ, and gm : µ = rp × gm Kα Kβ Ie λmin
λα
λ
Ic
Current in a transistor: Ie = Ib + Ic Ib
ν = a(Z − b)
α and β parameters of a transistor: α = Ic α Ib , β = 1−α
X-ray diffraction: 2d sin θ = nλ Heisenberg uncertainity principle: ∆p∆x ≥ h/(2π), ∆E∆t ≥ h/(2π)
Nuclear radius: R = R0 A1/3 ,
Ic Ie ,
∆Ic Transconductance: gm = ∆V be
Logic Gates:
6.3: The Nucleus
Decay rate:
∆Vg =0
∆ip ∆Vg
∆V
I
Moseley’s law:
Transconductance of a triode: gm = Amplification by a triode: µ = − ∆Vpg
hc X-ray spectrum: λmin = eV
√
transition
R0 ≈ 1.1 × 10−15 m
A 0 0 1 1
B 0 1 0 1
AND AB 0 0 0 1
dN dt = −λN
Figure 5 continued (Page 11).
OR A+B 0 1 1 1
NAND AB 1 1 1 0
NOR A+B 1 0 0 0
XOR AB̄ + ĀB 0 1 1 0
β =
F
Evaluation Pipeline
We evaluate model outputs using a three-step shadow evaluation protocol that combines automated verification with LLM-based error analysis. Each sample produces a shadow list [S1 , S2 , S3 ] where each entry is No (correct / no error) or Yes (incorrect / error present).
Shadow List Classification Logic S1 (Answer)
S2 (Tags)
S3 (Errors)
Classification
No
No
No
True Positive — fully correct
Yes
Yes
Yes
True Negative — all errors present
Yes
No
Yes
True Negative — needs human verification
Yes
No
No
Ambiguous — wrong answer, no detectable error
S1 = answer correctness, S2 = tag compliance, S3 = step-level error presence.
F.1
Three-Step Protocol
Step 1 — Answer Verification (automated). The \boxed{} content is extracted from the <final_answer> tag and normalised (whitespace, LATEX formatting, case). Returns No (correct) or Yes (wrong / no boxed answer found). No LLM is involved. Step 2 — Tag Compliance (automated). All seven required XML tags are checked for presence and closure: <problem_analysis>, <principle>, <governing_equation>, <value_identification>, <substitution>, <calculation>, <final_answer>. Returns No (all present) or Yes (any tag missing). Step 3 is skipped if Step 2 returns Yes. Step 3 — Error Analysis (LLM-based). An LLM judge evaluates the model’s solution across three independent dimensions: Problem Miscomprehension (MC), Conceptual Misapplication (CM), and Calculation Error (CE).
F.2
Step 3 Error Analysis Prompt
Question: {question} Generated Solution: {solution} Original Answer: {answer} Problem ID: {problem_id} 1. Problem Comprehension: 1. Does the solution attempt to address the correct objective asked in the question? 2. Are the correct values, variables, and notations from the question being used? 2. Concept Application: 1. Check the solution against the relevant physics concepts and formulas required. 2. Verify whether the correct physics concepts and formulas are applied. Important: Do not verify mathematical reasoning or calculations. Focus only on whether the correct physics concepts are applied. 3. Calculation Accuracy: Check each step and verify all mathematical calculations, including arithmetic, algebraic manipulation, substitutions, integration, differentiation, fractions, exponents, and numerical approximations. Output format (strictly follow; no extra text): Problem Miscomprehension Flag: [Yes/No] Concept Error Flag: [Yes/No]
Calculation Error Flag: [Yes/No] Judgement Key: [Correct/Incorrect]
F.3
Erroneous Solution Examples
Three samples from PhysicsQA, one per error type. The erroneous tag is highlighted in each rejected solution to show exactly what the Step 3 judge detects.
Example 1
Problem Miscomprehension (MC) — model swaps given quantity
A gas can be taken from A to B via two processes ACB and ADB. Path ACB: Q = 60 J, W = 30 J. Path ADB: W = 10 J. Find heat flow in path ADB. Ground Truth: (c) 40 J Correct Solution <value_identification> QACB = 60 J, WACB = 30 J; WADB = 10 J, QADB =? </value_identification> <substitution> ∆U = 60 − 30 = 30 J; 30 = QADB − 10 </substitution> <calculation> QADB = 40 J </calculation> <final_answer> (c) 40 J </final_answer>
Erroneous — MC: W = 10 J misread as Q = 10 J
<value_identification> Path ADB: Q=10 J [ERROR: W=10 J misread as Q=10 J] </value_identification> <substitution> ∆U = 30 J; 30 = 10 − WADB </substitution> <final_answer> (d) 20 J [INCORRECT] </final_answer>
Problem Miscomprehension Flag: Judgement Key: Incorrect
Example 2
Yes
Concept Error Flag:
No
Calculation Error Flag:
No
Conceptual Misapplication (CM) — monatomic Cp applied to diatomic gas
Diatomic rigid gas does W = 10 J at constant pressure. Find heat absorbed.
Ground Truth: (c) 35 J
Correct Solution <governing_equation> rigid diatomic: Cv = 5R/2, Cp = 7R/2; </governing_equation> <substitution> ∆Q = 10 × (7/2) = 35 J </substitution>
∆Q/∆W = Cp /R = 7/2
<final_answer> (c) 35 J </final_answer>
Erroneous — CM: monatomic Cp = 5R/2 used instead of diatomic 7R/2
<governing_equation> Cv = 3R/2, Cp = 5R/2 [ERROR: monatomic values for diatomic gas] </governing_equation> <substitution> ∆Q = 10 × (5/2) = 25 J [WRONG] </substitution>
<final_answer> (d) 20 J [INCORRECT] </final_answer>
Problem Miscomprehension Flag: Judgement Key: Incorrect
Example 3
No
Concept Error Flag:
Yes
Calculation Error Flag:
No
Calculation Error (CE) — correct concept, arithmetic error in final multiplication
Helium in fixed 67.2 L cylinder at STP. Heat for ∆T = 20 K? [R = 8.31 J mol−1 K−1 ]
Ground Truth: (c) 748 J
Correct Solution <calculation> ∆Q = 3 × 1.5 × 8.31 × 20 = 747.9 ≈ 748 J </calculation> <final_answer> (c) 748 J </final_answer>
Erroneous — CE: dropped factor of 2 in multiplication <problem_analysis> [Identical] </problem_analysis> <principle> [Identical] </principle> <governing_equation> [Identical] </governing_equation> <value_identification> [Identical] </value_identification> <substitution> [Identical] </substitution> <calculation> = 4.5 × 8.31 × 20 = 37.395 × 20 = 374 J [WRONG — dropped factor of 2] </calculation> <final_answer> (d) 374 J [INCORRECT] </final_answer>
Problem Miscomprehension Flag: No Concept Error Flag: No Calculation Error Flag: Yes Judgement Key: Incorrect
All three examples demonstrate the independence of the three error flags. The Step 3 judge evaluates each dimension separately — a solution can fail on MC without CM or CE, and vice versa. This independence enables the error distribution analysis to attribute performance gaps to specific failure modes rather than conflating them into a single accuracy metric.
D
Benchmark Dataset Samples
We evaluate on five benchmarks spanning foundational to advanced physics reasoning. Two to three representative samples per benchmark are shown below, reproduced in full. All samples are drawn directly from the evaluation sets used in our experiments.
D.1
PhysicsQA
N = 370 | JEE-sourced MCQ with CoT solutions | Intermediate difficulty
PhysicsQA comprises 370 intermediate-level high school physics problems sourced from Indian JEE preparation materials (2000–2010). Each problem is a four-option MCQ accompanied by a verified chain-of-thought solution, enabling step-level evaluation beyond final answer accuracy.
Sample 1
Heat Transfer — Path-independent internal energy, First Law
A gas can be taken from A to B via two different processes ACB and ADB. When path ACB is used, 60 J of heat flows into the system and 30 J of work is done by the system. If path ADB is used, work done by the system is 10 J. The heat flow into the system in path ADB is: Options: (a) 100 J
(b) 80 J
(c) 40 J
(d) 20 J
Solution: ∆Q = ∆U + ∆W ⇒ ∆U = ∆Q − ∆W Internal energy is a state function: (∆U )ACB = (∆U )ADB 60 − 30 = ∆QADB − 10 ⇒ ∆QADB = 40 J Answer: (c) 40 J
Sample 2
Thermodynamics — Polytropic process, molar heat capacity
In a process, temperature and volume of one mole of an ideal monatomic gas vary according to V T = K (constant). The temperature increases by ∆T . The heat absorbed by the gas is (R = gas constant): Options: (a) (2K/3)∆T
(b) 21 R∆T
(c) 32 R∆T
(d) 12 KR∆T
Solution: V T = K and P V = RT ⇒ P V 2 = const. (polytropic, n = 2) R R Molar heat capacity: C = 1−n + Cv = 1−2 + 3R = R 2 2 1 ∆Q = nC∆T = 2 R∆T Answer: (b) 12 R∆T
Sample 3
Thermodynamics — Isochoric heating of helium, molar heat capacity
A cylinder with fixed capacity 67.2 L contains helium gas at STP. The heat needed to raise the temperature by 20◦ C is [R = 8.31 J mol−1 K−1 ]: Options: (a) 350 J
(b) 700 J
(c) 748 J
(d) 374 J
Solution: n = 67.2/22.4 = 3 mol ∆Q = nCv ∆T = 3 × 32 R × 20 = 3 × 1.5 × 8.31 × 20 = 747.9 ≈ 748 J Answer: (c) 748 J
D.2
SciEval-Static Physics
N = 164 | Conceptual MCQ | Introductory–Intermediate
SciEval-Static is the static physics subset of the SciEval benchmark, comprising 164 multiple-choice questions across Fluid Mechanics, Forces, and Waves. Each item has a single correct answer label; no reference solution is provided. Ability tags classify questions as Base Knowledge or Scientific Calculation.
Sample 1
Fluid Mechanics — Buoyancy (Base Knowledge)
How do buoyant forces occur? A. Buoyant forces occur when an object is exposed to air, causing air pressure to create a lifting force. B. Buoyant forces occur when an object is partially or completely submerged in a liquid, equal to the weight of liquid displaced. C. Buoyant forces occur when an object is heated, causing it to expand and displace liquid. D. Buoyant forces occur when an object is compressed, reducing its volume and displacing liquid. Answer: B
Sample 2
Topic: Fluid Mechanics
Ability: Base Knowledge
Fluid Mechanics — Bernoulli’s Principle (Scientific Calculation)
How does changing the speed of a fluid affect its pressure? A. Fluid pressure is constant regardless of velocity. B. Direct proportionality between fluid pressure and velocity. C. No relationship between fluid pressure and velocity. D. Inverse proportionality between fluid pressure and velocity. Answer: D
Sample 3
Topic: Fluid Mechanics
Ability: Scientific Calculation
Forces — Vector resultant acceleration (Scientific Calculation)
An object with mass 2 kg is acted on by two forces. F1 = ⟨−9 N, 8 N⟩ and F2 = ⟨−7 N, −4 N⟩. What is the object’s rate and direction of acceleration? A. 10.3 m/s2 , 124◦ clockwise from x-axis B. 7.8 m/s2 , 95◦ clockwise from x-axis C. 8.25 m/s2 , 104◦ clockwise from x-axis D. 6.5 m/s2 , 86◦ clockwise from x-axis Answer: C
D.3
Topic: Forces and Newton’s Laws
MMLU High School Physics
Ability: Scientific Calculation
N = 170 | Conceptual MCQ | High School
The MMLU High School Physics subset comprises 170 four-option multiple-choice questions testing foundational physics knowledge. No reference solutions are provided; evaluation is based on final answer accuracy only.
Sample 1
Electromagnetism — Capacitor charge calculation
The plates of a capacitor are charged to a potential difference of 5 V. If the capacitance is 2 mF, what is the charge on the positive plate? A. 0.005 C
B. 0.01 C
C. 0.02 C
D. 0.5 C
Answer: B
Sample 2
Topic: Electromagnetism
Electromagnetism — Field decay with distance
Which of these quantities decreases as the inverse square of distance for distances far from the objects producing the fields? A. The electric field produced by a finite-length charged rod B. The electric field produced by an infinitely long charged cylinder C. The electric field produced by an infinite plane of charge D. The magnetic field produced by an infinitely long, straight current wire Answer: A
Sample 3
Topic: Electromagnetism
Electromagnetism — Field near a non-uniformly charged conductor
A solid metal object is isolated from other charges and has non-uniform charge distribution on its surface. It may be correctly concluded that the: A. electric field outside the object is zero B. electric field outside equals the field inside the object C. external field is directly proportional to distance from the centre of mass D. external field very close to the surface equals the surface charge density at any location divided by the permittivity of free space Answer: D
D.4
Topic: Electromagnetism
MMLU College Physics
N = 118 | Conceptual MCQ | Undergraduate
The MMLU College Physics subset comprises 118 questions at undergraduate difficulty, covering Modern Physics, Optics, Thermodynamics, and Mechanics. Problems frequently require synthesis of multiple concepts and dimensional reasoning.
Sample 1
Modern Physics — Quantum efficiency and Poisson statistics
The quantum efficiency of a photon detector is 0.1. If 100 photons are sent into the detector one after the other, the detector will detect photon: A. an average of 10 times, with an rms deviation of about 4 B. an average of 10 times, with an rms deviation of about 3 C. an average of 10 times, with an rms deviation of about 1 D. an average of 10 times, with an rms deviation of about 0.1 Answer: B
Sample 2
Topic: Modern Physics
Note: Binomial, n = 100, p = 0.1; σ =
p √ np(1 − p) = 9 = 3
Optics — Thin film interference, reflected wavelength
White light is normally incident on a puddle of water (refractive index 1.33). A thin 500 nm layer of oil (refractive index 1.5) floats on the surface. Of the following, the most strongly reflected wavelength is: A. 500 nm
B. 550 nm
C. 600 nm
D. 650 nm
Answer: C
Sample 3
Topic: Optics
Thermodynamics — Reversible process and entropy
Which of the following is true about any system that undergoes a reversible thermodynamic process? A. There are no changes in the internal energy of the system. B. The temperature of the system remains constant during the process. C. The entropy of the system and its environment remains unchanged. D. The entropy of the system and its environment must increase. Answer: C
D.5
Topic: Thermodynamics
JEEBench
N = 123 | JEE Advanced MCQ | Highest difficulty
JEEBench comprises 123 physics problems drawn from JEE Advanced examination papers (2016–2023). Problems require multi-concept synthesis, mathematical fluency, and derivation under examination conditions. Ground truth is the correct answer option only; no reference solutions are provided.
Sample 1 (JEE Adv 2016, Paper 1)
Modern Physics — Planck’s constant from photoelectric stopping potential data
In a historical experiment to determine Planck’s constant, a metal surface was irradiated with light of different wavelengths. The stopping potentials measured are: λ (µm)
V0 (V)
0.3 0.4 0.5
2.0 1.0 0.4
Given c = 3 × 108 m s−1 , e = 1.6 × 10−19 C. Planck’s constant found from this experiment is: (A) 6.0 × 10−34 J s
(B) 6.4 × 10−34 J s
(C) 6.6 × 10−34 J s
(D) 6.8 × 10−34 J s
Gold Answer: B
Sample 2 (JEE Adv 2016, Paper 1)
Mechanics — Stick on inclined wall, torque and friction equilibrium
A uniform wooden stick of mass 1.6 kg and length l rests inclined on a smooth vertical wall of height h < l such that a small portion extends beyond. The wall reaction is perpendicular to the stick; the stick makes 30◦ with the wall; the bottom rests on a rough floor. The wall reaction equals the floor reaction in magnitude. (g = 10 m s−2 ) √ √ (A) h/l = 3/16, f = 16 √ 3/3 N (B) h/l = 3/16, f = 16 3/3 √ √ N 3/3 N (C) h/l = 3 √3/16, f = 8 √ (D) h/l = 3 3/16, f = 16 3/3 N Gold Answer: D
Sample 3 (JEE Adv 2016, Paper 2)
Nuclear Physics — Electrostatic energy and nuclear radius from binding energy difference
2
The electrostatic energy of Z protons uniformly distributed in a spherical nucleus of radius R is E = 35 Z(Z−1)e . 4πε0 R 15 15 N Measured masses: neutron = 1.008665 u, 11 H = 1.007825 u, 15 7 N = 15.000109 u, 8 O = 15.003065 u. Radii of and 15 O are equal. [1 u = 931.5 MeV/c2 , e2 /(4πε0 ) = 1.44 MeV fm, 1 fm = 10−15 m] Assuming the binding energy difference between 15 N and 15 O is purely electrostatic, the radius of either nucleus is: (A) 2.85 fm
(B) 3.03 fm
(C) 3.42 fm
(D) 3.80 fm
Gold Answer: C
All benchmark samples are used as-is from their respective datasets with no modification. Models are prompted to produce a structured seven-tag solution before selecting a final answer, enabling step-level error analysis alongside final answer accuracy evaluation.