Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA
arXiv:2605.29277v1 [cs.SE] 28 May 2026
Jun Zhang
JianYing Qu
Hanwen Du Qiao Zhao Baidu Inc.
Zhongkai Sun
Yehua Yang
May 27, 2026
Abstract We present Code-QA-Bench, a fully automated framework for synthesizing repositorylevel code understanding benchmarks that separates genuine code comprehension from documentation recall and pretraining memorization. The framework makes two methodological contributions: (1) an answer-first generation pipeline where a tool-equipped agent explores source code to produce verified gold answers before deriving questions, ensuring every task is grounded in real code structure; and (2) a three-condition experimental design evaluating agents under closed-book (no repository), code-only (documentation removed), and documented (full repository) conditions, with deltas directly quantifying documentation utility and memorization. We generate 528 code-derivable and 100 doc-dependent tasks across 10 Python repositories from SWE-Bench, scored by an LLM judge on accuracy, completeness, and specificity. Experiments on four frontier models reveal that code access is the dominant factor (+0.23 mean gain over closed-book), documentation provides modest additional benefit (+0.071 on doc-dependent tasks, p < 0.003), and code-only ≈ documented on code-derivable tasks (∆ < 0.01), validating the design. The framework is open-source and applicable to any well-documented Python repository.
Keywords: code comprehension, repository-level QA, benchmark, documentation utility, LLM evaluation, code reasoning
1
Introduction
The dominant use of AI coding assistants in practice is not writing code from scratch, but understanding existing codebases [27]. Code understanding is broad: it includes reasoning about the input/output behavior of a function, locating the code responsible for a given feature, tracing control flow across modules, and building enough context to confidently propose a bug fix. Yet the most prominent benchmarks for AI coding agents (HumanEval [4], MBPP [2], and SWE-Bench [12]) focus on code generation or issue resolution, leaving code comprehension under-evaluated. This gap matters. An agent that can generate correct patches may still fail to explain why a function behaves a certain way, or to locate the right file for a given concept in a 500-file repository. Code understanding is a prerequisite for safe and reliable code modification, and measuring it independently provides signal that generation benchmarks miss. CRUXEval [7] highlighted this by showing that code generation ability does not imply code reasoning ability, but operates only at the function level. RepoReason [17] extended reasoning evaluation to the repository level, revealing that aggregating information across files is the primary cognitive 1
bottleneck for frontier models. Recent work on repository-level code QA [24, 3] has made progress on question taxonomies and difficulty calibration, but evaluates agents on repositories with full documentation intact, making it difficult to separate genuine code reading from documentation recall or pretraining memorization. We introduce Code-QA-Bench, a benchmark designed around four principles: (1) a threecondition experimental design evaluating each task under closed-book (no repository), codeonly (documentation removed), and documented (full repository) conditions, with deltas directly quantifying documentation utility and memorization; (2) documentation removal as an environment-level control, removing all docstrings, comments, and documentation files to force code-structural reasoning while retaining semantic signal from identifiers, imports, and type annotations (Section 8); (3) answer-first task generation where a tool-equipped agent explores source code to produce verified gold answers before deriving questions, ensuring every task is grounded in real code structure; and (4) unified continuous scoring via an LLM judge on three axes (accuracy, completeness, specificity), each 0–5, normalized to a single 0–1 score. Code-QA-Bench uses 10 Python repositories from SWE-Bench [12] (Table 1), pinned to specific commits for reproducibility. Tasks follow a four-type taxonomy (What/Why/Where/How) with 12 subtypes from SWE-QA [24], balanced per repository. The entire pipeline is fully automated and repo-agnostic: it can be applied to any well-documented Python repository without modification.
2
Related Work
Function-level and repository-level code benchmarks. Early code understanding benchmarks operate at the function level: CRUXEval [7], CodeQA [21], CS1QA [13], LiveCodeBench [10], and EvalPlus [19]. None capture the cross-file reasoning demands of real-world development. At the repository level, SWE-Bench [12] and its extensions [1, 29, 5] evaluate code generation via patches, while others address cross-file completion [6, 22], lifecycle tasks [14, 15], and agent architectures [30, 28]. In all cases, code understanding is a means to generation, not the measured outcome. Repository-level code QA. SWE-QA [24] is the first large-scale repository-level code QA benchmark (576 questions, 12 repositories, 4-type taxonomy). SWE-QA-Pro [3] improves on it by selecting 26 long-tail repositories and applying difficulty calibration to filter questions answerable without tools, demonstrating that many existing items do not require repository interaction. Other work evaluates code reasoning through assertion verification [17], execution prediction [20], multi-dimensional frameworks [32], and automated benchmark synthesis [11]. Our work addresses the memorization problem from a different angle: rather than filtering easy questions or selecting unfamiliar repositories, we directly measure memorization via three evaluation conditions and adopt SWE-QA’s taxonomy with balanced distribution and an answerfirst generation pipeline. Evaluation methodology. We use rubric-guided LLM-as-judge evaluation [33, 18] with a 3-axis rubric emphasizing concreteness. Our documentation-removal approach to contamination control relates to CodeSearchNet’s [8] separation of code from natural-language descriptions, and complements temporal methods [10] and difficulty calibration [3] by removing the naturallanguage layer most susceptible to memorization.
2
Table 1: Benchmark repositories, drawn from SWE-Bench [12]. Task counts are allocated by repository size tier. Repository
Domain
.py Files
Commit SHA
Tasks
django pylint sympy
Web framework Static code analysis Symbolic mathematics
2,894 2,366 1,589
856c9153... b715af6c... 693a559a...
72 72 72
scikit-learn astropy matplotlib sphinx pytest xarray
Machine learning Astronomy library Plotting / visualization Documentation generator Testing framework N-D labeled arrays
993 987 913 774 262 237
f3182980... 30686862... cc6cead6... cc7c6f43... 8ecf49ec... 92601de1...
48 48 48 48 48 48
seaborn
Statistical visualization
151
32088bbc...
24
Total
3
Benchmark Design
3.1
Repository Selection and Pinning
528
We select 10 of the 12 SWE-Bench Python repositories [12] (Table 1), excluding Flask and Requests due to insufficient documentation volume. Unlike SWE-QA-Pro [3], which avoids popular repositories, we retain well-known projects and address memorization via the threecondition design (Section 5.1). Each repository is pinned to a specific git commit SHA for reproducibility. Task counts are allocated by size tier: large (>1,500 .py files) receive 72 tasks, medium (200–1,000) receive 48, and small (<200) receive 24, yielding 528 total. The pipeline makes no assumptions specific to these repositories and can be applied to any well-documented Python project.
3.2
Documentation Removal
To isolate code comprehension from documentation recall, we create a code-only version of each repository that removes three categories of natural-language content: 1. Docstrings. We use Python’s Abstract Syntax Tree (AST) module to identify docstrings (string literals that are the first statement in a module, class, or function body). These are removed from the source. When removing a docstring would leave an empty body (e.g., a function whose only statement is its docstring), a pass statement is inserted to maintain syntactic validity. 2. Comments. We use Python’s tokenize module to identify comment tokens. Full-line comments (where only whitespace precedes the #) are removed entirely. Inline comments are truncated at the # position. 3. Documentation files. The directories docs/, doc/, and .git/ are deleted. Files matching README*, *.md, *.rst, CHANGELOG*, and CONTRIBUTING* are removed. The removal is deterministic and preserves all executable code, imports, type annotations, and string literals. Unlike SWE-QA-Pro’s question-level difficulty calibration [3], our environmentlevel approach ensures all questions require genuine code reading.
3
Repository extract Doc Chunks (documented)
score
Selected Chunks (≥9.0) + category
Gold Answer + Evidence
Task Gen Agent (read, list, search)
fact-check + code-only audit
Verification
Final Task Q + Rubric + Answer
Leak-Free Answer
Question Agent
Figure 1: The answer-first task generation pipeline. Documentation chunks are extracted and scored (Table 2); chunks ≥ 9.0 are selected. The Task Generation Agent explores source code to produce a gold answer, which is verified and code-only audited (§4.4); a question and rubric are then derived from the verified answer.
3.3
Task Categories
We adopt the four-type taxonomy from SWE-QA [24], with 12 Level-2 subtypes derived from their analysis of 77,100 real GitHub issue questions: • What (factual inquiry): architecture exploration, concept definition, dependency tracing. Example: “What components make up the caching subsystem?” • Why (causal explanation): design rationale, purpose exploration, performance. Example: “Why does the ORM use lazy evaluation for querysets?” • Where (location identification): data/control flow, feature location, identifier location. Example: “Where is the request routing logic implemented?” • How (procedural explanation): system design, algorithm implementation, API/framework support. Example: “How does the template engine resolve variable lookups?” Each repository’s tasks are split evenly across the four Level-1 categories (25% each) via round-robin assignment during chunk selection. Within each category, subtypes are distributed evenly via a secondary round-robin. This balanced design ensures that no category dominates and enables per-category analysis of model strengths and weaknesses. We hypothesize that Why questions, which often require understanding design rationale typically found in comments and documentation, will show the largest gap between code-only and documented conditions. Categories serve as metadata for breakdown analysis but do not affect the overall score.
4
Task Generation Pipeline
The task generation pipeline follows an answer-first design: we generate code-grounded gold answers before deriving questions. This inverts the typical approach used in SWE-QA [24] (question templates from taxonomy) and SWE-QA-Pro [3] (question synthesis from issue clusters), and ensures every question has a verifiable, specific reference answer rooted in actual source code. The pipeline consists of five stages (Figure 1).
4
4.1
Stage 1: Documentation Chunk Extraction
We extract documentation from the documented (full) repository from three sources: • README chunks. README files are split by ## markdown headings. Each heading–body pair becomes a chunk. • Documentation file chunks. Files under docs/ and doc/ (Markdown and reStructuredText) are split by headings. For RST files, we detect underline-style headings (====, ––, ˜˜˜, etc.) in addition to Markdown-style headings, using whichever method finds more sections. This is critical for RST-heavy projects like Django (724 doc files, mostly RST). • Docstring chunks. All Python files are AST-parsed to extract docstrings. Rather than aggregating all docstrings from a single module into one mega-chunk, we emit one chunk per top-level class (with its methods’ docstrings grouped) and one chunk per top-level function. Module-level docstrings become their own chunks. This finer granularity produces more focused, class-specific chunks that ground better task generation. RST directive code references (:func:, :class:, :meth:, :mod:) are recognized alongside backtick identifiers when scoring chunks (Section 4.2), ensuring that RST-formatted documentation is not disadvantaged relative to Markdown.
4.2
Stage 2: Chunk Scoring and Selection
Each chunk receives a deterministic heuristic score based on multiple signals organized into three groups: content quality, path-based, and content bonuses/penalties. Table 2: Chunk scoring heuristic. Signals are grouped into baseline content signals, pathbased penalties that demote non-core code, and content-quality bonuses/penalties that reward structured documentation and penalize low-information chunks. Signal
Pts
Rationale
Baseline content signals Length 200–8000 chars Code references (backticks, dotted paths) Behavioral keywords (returns, raises, calls, when, if ) Source is docstring
+2 +3 +2 +1
Generic heading (install, license, contributing)
−2
Enough detail, not too noisy Grounds in specific code Describes runtime behavior More code-proximal than README Not code-understanding
Path-based penalties Test file (/test or test_ in path) Vendored code (extern, vendor) Example/gallery in path
−2 −2 −1
Test behavior, not design Third-party, not repo’s own Tutorial-like
Content-quality bonuses and penalties Structured sections (Params + Returns) Doctest examples (>>>) Deep module path (≥3 levels) Stacked one-liners (>8, avg <80 chars) RST directive-heavy (>30% of lines)
+1 +1 +1 −2 −1
Strong QA grounding Concrete usage Implementation-focused Low information density Mostly markup
Chunks are ranked by score and the top N per repository are selected (where N follows the size tier in Table 1), with at most 2 chunks per source file. Categories are assigned via round-robin to ensure 25% per category within each repository.
5
4.3
Stage 3: Agentic Gold Answer Generation
For each selected chunk, the Task Generation Agent produces a gold answer grounded in actual source code. The agent operates in a multi-turn tool-use loop with three read-only tools: read_file, list_directory, and search_code (regex over all .py files). A 200K-character context budget prevents overflow. The agent receives the documentation chunk as a topic guide but must go beyond it: the system prompt requires tracing at least one level deeper (callees, parent classes, or related modules) and including facts not in the documentation. Per-category guidance steers exploration strategy (e.g., Where traces call chains; How follows implementations line by line). The agent outputs structured JSON with a detailed answer, key file paths, and ≥3 code evidence items citing specific files and functions. Key files are validated against the filesystem; missing paths trigger a re-prompt with correction hints.
4.4
Stage 4: Verification and Code-Only Verification
Two automated quality gates filter the generated answers. First, a fact-check LLM call verifies the answer against the documentation chunk; tasks with significant contradictions are dropped (Pass/Warn/Fail). Second, a code-only verification pass addresses doc-leakage: since agents are evaluated on repositories with documentation removed, the gold answer must not contain claims only recoverable from documentation. An LLM auditor receives the gold answer alongside the code-only key files and classifies each claim as Keep (verifiable from code), Remove (documentation-dependent), or Rewrite (partially verifiable). The revised answer replaces the original, and the doc-leakage fraction is recorded per task (full prompt in Appendix E).
4.5
Dual Task Set Design
Code-only-verified tasks should yield similar scores under code-only and documented conditions; we call these code-derivable tasks (528 tasks). To measure documentation utility, we additionally generate 100 doc-dependent tasks (10 per repository) using a different branch: gold answers are produced in a single LLM turn from documentation only (no code tools), are not code-only-verified, and intentionally require documentation to fully address. Code-derivable tasks validate the design (code-only ≈ documented expected); doc-dependent tasks quantify documentation utility (documented > code-only expected).
4.6
Stage 5: Question Generation
In its second phase, the Task Generation Agent derives a natural question from the verified gold answer. The agent uses the same tool set (allowing it to re-check details against the code) but operates under a different system prompt and output schema: it produces a short (∼25 words), conversational question and a rubric of key points. The agent is instructed to write the kind of question a developer would actually ask, avoiding academic-sounding phrasing like “trace through” or “provide a comprehensive analysis.” The two-phase design (answer first, question second) ensures that answer quality is established and verified before a question is derived from it, reducing the risk of ill-posed or unanswerable questions. The rubric produced in this phase is used by the LLM judge during evaluation.
5
Evaluation
5.1
Three-Condition Experimental Design
All 528 tasks are evaluated under three conditions for every model: The key analysis deltas are: 6
Table 3: Three experimental conditions. All tasks are evaluated under all conditions. Condition
Agent Input
Measures
Closed-book Code-only (primary) Documented
Question only, no repo access Question + code-only repo Question + full repo with docs
Memorization / prior knowledge Code-structural reasoning Code reasoning + documentation
• Documented − Code-only = documentation utility for AI code comprehension. • Code-only − Closed-book = genuine contribution of code reading beyond memorization. • Per-category breakdown: we hypothesize that Why questions benefit most from documentation (rationale in comments), while Where questions may not (requires code tracing regardless). Tasks with high closed-book scores are flagged as potentially contaminated. The code-only condition is the primary benchmark metric; the other two conditions provide diagnostic context.
5.2
Scoring
Every agent answer is evaluated by an LLM judge on three axes: • Accuracy (0–5): Are the factual claims in the answer correct? Points are deducted for incorrect statements about the code. • Completeness (0–5): Does the answer cover all key points in the rubric? 5 = all points covered, 0 = none. • Specificity (0–5): Does the answer reference specific files, functions, classes, or code patterns? 5 = very specific with file/function names, 0 = entirely vague. The per-question score is the mean of the three axes, normalized to [0, 1]: accuracyi + completenessi + specificityi 15 The overall benchmark score is the simple mean across all N questions: si =
(1)
N
S=
1 X si N
(2)
i=1
Our 3-axis rubric differs from the 5-dimension evaluation used in SWE-QA [24] and SWEQA-Pro [3] (correctness, completeness, relevance, clarity, reasoning). We deliberately omit “clarity” and “reasoning” as evaluation axes because in a code-only setting, we care more about whether the agent found the right code and reported accurate facts than about the prose quality of its explanation. Note on specificity saturation. When agents have repository access (code-only or documented conditions), specificity saturates near 5.0 (≥4.78 for all models), because agents naturally reference concrete file paths and function names. In practice, the composite score in agentic conditions is primarily driven by accuracy and completeness; specificity serves as a discriminator only in the closed-book condition where models cannot ground their answers in specific code locations. We add “specificity” as a dedicated axis because referencing concrete file paths and function names is the primary evidence that an agent genuinely navigated the codebase rather than providing a plausible-sounding but vague response. 7
The simple mean treats every question equally. Category breakdowns (What/Why/Where/How) and per-condition breakdowns (closed-book/code-only/documented) remain available for diagnostic analysis.
5.3
Judge Model
We use GPT-5.4 as the LLM judge, deliberately choosing a model from a different provider and model family than the task generation model (Claude Opus 4.6) to mitigate self-evaluation bias. The judge receives: the original question, the gold reference answer, the rubric, and the agent’s answer. It returns a structured JSON response with the three scores and a brief explanation. To mitigate known biases in LLM judges [33], including position bias and verbosity bias, the judge prompt uses a structured rubric with explicit scoring criteria. SWE-QA [24] additionally anonymizes candidates and randomizes answer order; we adopt similar practices. For the relative comparisons that form our primary analysis (condition deltas, model rankings), judge consistency matters more than absolute calibration.
6
Experimental Setup
6.1
Agent Architecture
Code-QA-Bench ships with a built-in evaluation agent that reuses the same infrastructure as the Task Generation Agent (Section 4.3): the same tool definitions (read_file, list_directory, search_code), the same tool execution engine, and the same multi-turn loop (up to 60 turns). The evaluation agent uses condition-specific system prompts: • Code-only condition: The agent is told that “the repository has been stripped of all docstrings, comments, and documentation files” and must “derive understanding from code structure, naming conventions, control flow, and type signatures alone.” • Documented condition: The agent is told that “the repository includes full docstrings, comments, and documentation files” and is instructed to “start by searching for relevant documentation (README, docs/, CHANGELOG, .rst/.md files)” before diving into code. Since gold answers were produced by an agent with exactly these tools, any performance gap reflects code understanding ability rather than tooling differences. The benchmark also exposes a pluggable agent interface: any async callable matching (repo_path, question) -> answer can substitute the built-in agent.
6.2
Models Evaluated
Table 4: Models evaluated. All use the same built-in agent with identical tool access (60 max turns, 4096 max tokens per response). Model
API Identifier
Provider
Context
Claude Opus 4.6 DeepSeek-V4-Pro Kimi-K2.6 Gemini-3.1-Pro
claude-opus-4-6 deepseek-v4-0324 kimi-k2.6-0528 gemini-3.1-pro-preview
Anthropic DeepSeek Moonshot AI Google
200K 128K 128K 1M
Claude Opus 4.6 also serves as the task generation model, creating a potential circularity. We mitigate this via three design choices: (1) a separate GPT-5.4 judge (different provider, no self-evaluation); (2) generation uses documented code while evaluation uses code-only code; 8
(3) rubric-based scoring constrains evaluation to factual key points. Empirically, the bias is negligible on the primary metric: DeepSeek (0.892) matches Claude (0.891) on code-derivable tasks in the code-only condition despite having no role in task generation. All models are evaluated through a unified inference gateway with identical tool definitions and system prompts.
6.3
Evaluation Protocol
Each model is evaluated under all three conditions (Section 5.1). Generation parameters: max_tokens = 4096, max_turns = 60, context_budget = 200,000 characters. All agent and judge calls include retry logic with exponential backoff.
7
Results
We present results on both the code-derivable (528 tasks) and doc-dependent (100 tasks) task sets across all four models and three conditions. All reported p-values use paired bootstrap (10,000 resamples) with Bonferroni correction (αadj = 0.003 for 16 comparisons); full statistical details and 95% confidence intervals are in Appendix H. Score distributions in the code-only condition are left-skewed with 5–12% of tasks at ceiling (≥0.95) and 1–15% below 0.70; closed-book distributions have 87–97% of tasks below 0.70, confirming that most tasks genuinely require code access.
7.1
Doc-Dependent Task Results
Table 5 shows results on the 100 doc-dependent tasks. This task set is designed to measure documentation utility: gold answers contain information only available in documentation, so agents with documentation access (documented condition) should outperform those without. Table 5: Results on doc-dependent tasks (100 tasks). Score is the normalized mean of accuracy, completeness, and specificity (each 0–5, normalized to 0–1). ∆doc = documented − code-only measures documentation utility. All ∆doc values are statistically significant (p < 0.003, paired bootstrap test, 10K resamples; full CIs in Appendix H). Closed-book
Code-only
Documented
∆doc
Claude Opus 4.6 Kimi-K2.6 DeepSeek-V4-Pro Gemini-3.1-Pro
0.682 0.639 0.557 0.636
0.873 0.859 0.867 0.840
0.953 0.950 0.928 0.893
+0.080 +0.091 +0.061 +0.053
Mean
0.629
0.860
0.931
+0.071
Model
Three clear patterns emerge: Documentation provides consistent, measurable benefit. Across all four models, the documented condition outperforms code-only by +0.053 to +0.091 (mean +0.071). This gap is consistent across model families and confirms that agents can effectively use documentation when questions genuinely require it. The gap is moderate rather than dramatic, suggesting that much of the answer can be partially inferred from code structure, but documentation provides the final details (design rationale, deprecation warnings, edge case caveats) that push scores from ∼0.86 to ∼0.93.
9
Code access dominates over memorization. The code-only−closed-book gap (mean +0.231) is three times larger than the documented−code-only gap (mean +0.071), demonstrating that reading code contributes far more to understanding than documentation alone. Even for questions explicitly designed to require documentation, code structure provides substantial signal. Significant parametric knowledge in closed-book. Models score 0.56–0.68 without any repository access, indicating that frontier models have memorized substantial information about these well-known Python libraries during pretraining. Claude Opus achieves the highest closedbook score (0.682), consistent with its larger pretraining corpus.
7.2
Code-Derivable Task Results
Table 6 shows results on the 528 code-derivable tasks. These tasks are code-only-verified: gold answers are recoverable from code structure alone. Table 6: Results on code-derivable tasks (528 tasks). Tasks are code-only-verified, so code-only ≈ documented is expected. ∆doc is near zero for three models (p > 0.03); Gemini’s negative ∆ is marginally significant (p = 0.004, just above Bonferroni-adjusted α = 0.003); Claude’s positive ∆ reflects generator advantage (p < 0.001). Full bootstrap CIs in Appendix H. Closed-book
Code-only
Documented
∆doc
Claude Opus 4.6 DeepSeek-V4-Pro Kimi-K2.6 Gemini-3.1-Pro
0.560 0.442 0.514 0.482
0.891 0.892 0.873 0.772
0.918 0.899 0.882 0.755
+0.026 +0.007 +0.008 −0.018
Mean
0.500
0.857
0.864
+0.006
Model
The key validation: code-only ≈ documented on code-derivable tasks (mean ∆doc = +0.007). For DeepSeek and Kimi, the documented−code-only delta is not statistically significant (p > 0.03, paired bootstrap), confirming that code-only verification produces tasks whose answers are recoverable from code alone. The near-zero documentation gap validates our experimental design: any gap observed on doc-dependent tasks (mean ∆doc = +0.071, p < 0.003 for all models) is genuinely attributable to documentation utility, not an artifact of the evaluation setup. All code-only−closed-book deltas are highly significant (p < 0.001), confirming that code access provides substantial information beyond parametric knowledge. Several model-specific patterns emerge: Claude Opus 4.6 leads across conditions. As the task generation model, Claude Opus achieves the highest scores in both code-only (0.891) and documented (0.918). Its closed-book score (0.560) is also highest, reflecting both its large pretraining corpus and the advantage of having generated the gold answers from the same codebase knowledge. DeepSeek-V4-Pro: strong code reader, weak memorizer. DeepSeek achieves near-top code-only performance (0.892) with the lowest closed-book score (0.442), the largest gap of any model (+0.450). This suggests DeepSeek relies heavily on active code exploration rather than parametric recall. Gemini shows a negative documentation effect. Gemini-3.1-Pro is the only model where documented (0.755) underperforms code-only (0.772), yielding ∆doc = −0.018 (p = 0.004,
10
marginally significant at Bonferroni-adjusted α = 0.003). Documentation presence likely diverts the agent’s exploration strategy toward README/docs files that provide no signal for code-derivable questions, consuming turns from a limited budget. Closed-book scores are lower than on doc-dependent tasks. Mean closed-book on codederivable (0.500) is below doc-dependent (0.629), suggesting that implementation-level details (function signatures, control flow, class hierarchies) are harder to recall from pretraining than documentation-level knowledge (design rationale, usage patterns).
7.3
Axis-Level Analysis
Table 7 breaks down the doc-dependent scores by evaluation axis. Table 7: Per-axis scores on doc-dependent tasks (0–5 scale). Specificity saturates near ceiling with code access; completeness shows the most variation across conditions. Model
Condition
Accuracy
Completeness
Specificity
Claude Opus 4.6
Closed-book Code-only Documented
3.05 4.15 4.62
3.13 3.98 4.68
4.05 4.97 5.00
DeepSeek-V4-Pro
Closed-book Code-only Documented
2.43 4.22 4.53
2.38 3.95 4.55
3.54 4.97 4.99
Kimi-K2.6
Closed-book Code-only Documented
3.02 4.33 4.80
2.75 3.71 4.46
3.82 4.85 4.99
Gemini-3.1-Pro
Closed-book Code-only Documented
3.28 4.13 4.58
2.71 3.65 4.18
3.55 4.82 4.78
Three axis-level findings: • Specificity saturates. With code access (code-only or documented), all models achieve specificity ≥4.78, indicating that agents reliably reference concrete file paths and function names once they can explore the repository. Specificity is primarily a closed-book discriminator and contributes negligible signal for distinguishing model performance in agentic conditions. Future rubric iterations could replace or supplement specificity with a more discriminative axis such as integration depth (how many cross-file connections the answer traces). • Completeness benefits most from documentation. The largest documented−code-only gains appear on the completeness axis (+0.53 to +0.75 across models), because documentation provides the missing details (caveats, edge cases, design choices) needed to fully address the rubric. • Accuracy improves steadily. Accuracy rises monotonically from closed-book through codeonly to documented, reflecting that more information sources reduce factual errors.
7.4
Per-Category Breakdown
Table 8 reports scores by question category on code-derivable tasks, testing the hypothesis (Section 3.3) that Why questions show the largest documentation gap. 11
Table 8: Per-category scores on code-derivable tasks (pooled across 4 models; 132 tasks per category × 4 models = 528 observations per cell). ∆ = documented − code-only; d = Cohen’s d. Bonferroni-adjusted α = 0.0125 (4 tests). Category
Closed-book
Code-only
Documented
∆
d
p
0.484 0.536 0.479 0.500
0.860 0.837 0.863 0.869
0.863 0.840 0.874 0.876
+0.003 +0.003 +0.011 +0.008
0.03 0.03 0.13 0.06
0.278 0.268 0.003 0.069
What Why Where How
Contrary to our hypothesis, Why questions do not show the largest documentation gap on code-derivable tasks (∆=+0.003, p=0.268). Instead, Where questions exhibit the only statistically significant delta (∆=+0.011, p=0.003, d=0.13), suggesting that documentation helps most when agents need to locate features across a repository (feature location, dependency tracing), where README files and module docstrings serve as a navigation index. The negligible gap for Why questions is consistent with our design: code-derivable tasks have gold answers grounded entirely in source code, so even design-rationale questions are answerable from code patterns (defensive checks, fallback logic) without requiring documentation. This validates the task generation pipeline: code-derivable tasks genuinely do not require documentation regardless of question type. Additionally, Why questions have the highest closed-book scores (0.536 vs. 0.479–0.500), confirming that design rationale is most susceptible to memorization. Claude excels on How (0.912 code-only) while DeepSeek excels on What (0.903), suggesting complementary strengths.
7.5
Ceiling Effects and Documentation Utility
While agents benefit from documentation (∆doc = +0.071 mean on doc-dependent tasks), the gain is modest: agents achieve 0.84–0.87 on code-only, suggesting they infer much documented information from code structure alone. Documentation provides incremental benefit primarily for completeness—filling in design rationale that cannot be inferred from code patterns. Three of four models achieve code-only scores above 0.87 on code-derivable tasks, approaching a practical ceiling. Contributing factors: specificity saturates near 5.0 with repository access, rubric-based scoring rewards key-point coverage that agents with 60 turns reliably achieve, and frontier models are genuinely strong code-structural readers. The fully automated pipeline supports periodic regeneration from newer repositories, tighter exploration budgets, or deeper architectural questions to maintain discrimination as models improve.
8
Discussion
What “code-only” actually measures. The code-only condition preserves identifier names, string literals, imports, and type annotations, all of which carry semantic information (e.g., validate_email_format communicates purpose via naming alone). Thus, we measure codestructural reasoning rather than pure syntactic reasoning: the ability to derive understanding from executable source code minus natural-language annotations. In the program comprehension literature, this maps to a blend of bottom-up comprehension [23] (tracing control flow and data flow from code) and top-down cues from identifier semantics [25], corresponding to the practical task of reading unfamiliar or undocumented code where developers switch fluidly between strategies [26]. The high code-only scores (0.77–0.89) likely reflect models leveraging descriptive naming conventions alongside control-flow analysis. This is a deliberate design choice: reading undocumented but well-named code is the practical task we aim to measure. An 12
obfuscation ablation (replacing identifiers with opaque tokens) would quantify the contribution of naming conventions vs. pure structural reasoning; we leave this for future work. Importantly, the code-access gain (+0.23 mean over closed-book) is almost entirely attributable to accuracy and completeness improvements: a 2-axis composite excluding specificity yields deltas within 3% of the 3-axis composite (Appendix I), confirming that the headline number is not inflated by specificity saturation. Validation logic and circularity. The dual task set creates a falsifiable prediction: codederivable tasks should show code-only ≈ documented, while doc-dependent tasks should show documented > code-only. Both predictions are confirmed (Tables 6–5). We acknowledge that code-derivable tasks are designed to be code-answerable via the code-only verification pass, so the near-zero delta on these tasks is partially self-confirming. However, the differential prediction is not circular: the same design predicts a positive delta on doc-dependent tasks, which could have failed if documentation were genuinely unhelpful. The consistent, significant doc-dependent gap across all four models (mean +0.071, p < 0.003) provides non-trivial evidence that the benchmark distinguishes the two task types as intended. Category balance. We enforce 25% per question type via round-robin allocation (Section 3.3). While real developer questions are not uniformly distributed—SWE-QA’s analysis of 77,100 GitHub questions shows substantial skew—balanced allocation enables equal-powered per-category comparisons and avoids confounding category effects with difficulty. Results under balanced allocation remain valid for ranking models; users who prefer ecological weighting can re-weight using per-category scores (Table 8). Limitations. The benchmark has several known limitations. Judge reliability: We use a single LLM judge (GPT-5.4) without inter-judge agreement or human correlation analysis; a reliability study stratified by condition is needed. Generator-evaluatee overlap: Claude generates gold answers and scores highest on some conditions, though the bias is negligible on the primary codeonly metric (DeepSeek matches at 0.892 vs. 0.891). No human validation: The pipeline relies on automated quality gates; a stratified human evaluation is planned. The fact-check verification produces 38.4% pass and 61.6% warn verdicts; “warn” indicates minor discrepancies (e.g., slight imprecision) that do not rise to factual contradiction, and only “fail” verdicts are dropped. While warned tasks are retained, the high warn rate suggests future iterations should tighten the verification threshold or add human review for borderline cases. Scope: The benchmark is Python-only (AST-based documentation removal is language-specific), uses single-commit snapshots, and removal may break tests that inspect __doc__ attributes. Ceiling effects: Three of four models score 0.87–0.92 on code-derivable tasks in the code-only condition, with 5–12% of tasks at ceiling (≥0.95); periodic regeneration or tighter exploration budgets are needed to maintain discrimination as models improve.
9
Conclusion
We presented Code-QA-Bench, a fully automated benchmark that separates code comprehension from documentation memorization via a three-condition experimental design and dual task set (528 code-derivable + 100 doc-dependent tasks across 10 repositories). Experiments on four frontier models show that code access is the dominant factor (+0.23 over closed-book), documentation provides consistent but moderate benefit (+0.071 on doc-dependent tasks, p < 0.003), and code-only ≈ documented on code-derivable tasks validates the methodology. The pipeline is repo-agnostic and can be applied to any well-documented Python repository, serving both as a reusable evaluation framework and as a source of verified training data [3]. The 13
three-condition design complements SWE-QA’s [24] taxonomic coverage and SWE-QA-Pro’s [3] difficulty calibration by providing environment-level control with quantitative contamination analysis. Code and data are open-source.
References [1] Aleithan, R., Kang, M.J., and Kamalloo, E. SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv preprint arXiv:2410.06992, 2024. [2] Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732, 2021. [3] Cai, S., Lyu, Z., Ni, Y., Chen, X., Zhou, B., Zhu, S., Lu, Y., Wang, H., Ruan, C., Schneider, B., Zhang, W., Li, X., Zheng, A., Zhang, Y., Nie, P., and Chen, W. SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding. arXiv preprint arXiv:2603.16124, 2025. [4] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374, 2021. [5] Chou, J., Liu, A., Deng, Y., Zeng, Z., Zhang, T., Zhu, H., Cai, J., Mao, Y., Zhang, C., Tan, L., Xu, Z., Zhai, B., Liu, H., Zhu, S., Zhou, W., and Lian, F. AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators. arXiv preprint arXiv:2508.09101, 2025. [6] Ding, Y., Wang, Z., Ahmad, W.U., Ramanathan, M.K., Nallapati, R., Bhatia, P., Roth, D., and Xiang, B. CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion. In NeurIPS, 2024. [7] Gu, A., Rozière, B., Leather, H., Solar-Lezama, A., Synnaeve, G., and Wang, S.I. CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution. arXiv preprint arXiv:2401.03065, 2024. [8] Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., and Brockschmidt, M. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. arXiv preprint arXiv:1909.09436, 2019. [9] Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., and Steinhardt, J. Measuring Coding Challenge Competence with APPS. In NeurIPS, 2021. [10] Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S.I., Solar-Lezama, A., Sen, K., and Stoica, I. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv preprint arXiv:2403.07974, 2024. [11] Jain, N., Shetty, M., Zhang, T., Han, K., Sen, K., and Stoica, I. R2E: Turning any Github Repository into a Programming Agent Environment. In ICML, 2024. [12] Jimenez, C.E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In ICLR, 2024. [13] Lee, J., Seo, J., Ahn, J., and Seo, M. CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course. arXiv preprint arXiv:2210.14921, 2022. 14
[14] Li, J., Qi, G., Li, Y., Dong, Y., and Guo, D. DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories. In ACL, 2024. [15] Li, B., Fang, T., Cui, Y., Jiang, Y., Wu, J., Gong, Y., Ding, L., Sun, J., and Tao, D. DevBench: A Comprehensive Benchmark for Software Development. arXiv preprint arXiv:2403.08604, 2024. [16] Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. Competition-Level Code Generation with AlphaCode. Science, 378(6624):1092–1097, 2022. [17] Li, J., Su, Y., and Lyu, M.R. From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level. arXiv preprint arXiv:2601.03731, 2025. [18] Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Zhu, B., Gonzalez, J.E., and Stoica, I. From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv preprint arXiv:2406.11939, 2024. [19] Liu, J., Xia, C.S., Wang, Y., and Zhang, L. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In NeurIPS, 2024. [20] Liu, T., Fang, J., Wen, Y., and Xie, T. CodeMind: A Framework to Challenge Large Language Models for Code Reasoning. arXiv preprint arXiv:2402.09664, 2024. [21] Liu, J., Wan, C., Tao, C., Zhao, K., and Sun, C. CodeQA: A Question Answering Dataset for Source Code Comprehension. In Findings of EMNLP, 2021. [22] Liu, T., Xu, C., and McAuley, J. RepoBench: Benchmarking Repository-Level Code AutoCompletion Systems. In ICLR, 2024. [23] Pennington, N. Stimulus Structures and Mental Representations in Expert Comprehension of Computer Programs. Cognitive Psychology, 19(3):295–341, 1987. [24] Peng, W., Shi, Y., Wang, Y., Zhang, X., Shen, B., and Gu, X. SWE-QA: Can Language Models Answer Repository-level Code Questions? arXiv preprint arXiv:2509.14635, 2025. [25] Schankin, A., Berger, A., Holt, D.V., Hofmeister, J.C., Riedel, T., and Beigl, M. Descriptive Compound Identifier Names Improve Source Code Comprehension. In ICPC, 2018. [26] von Mayrhauser, A. and Vans, A.M. Program Comprehension During Software Maintenance and Evolution. IEEE Computer, 28(8):44–55, 1995. [27] Xia, C.S., Deng, Y., and Zhang, L. A Survey on Large Language Models for Code Generation. arXiv preprint arXiv:2406.00515, 2024. [28] Xia, C.S., Deng, Y., Dunn, S., and Zhang, L. Agentless: Demystifying LLM-based Software Engineering Agents. arXiv preprint arXiv:2407.01489, 2024. [29] Xie, Y., Liu, Z., Chen, Y., Li, J., et al. FeatureBench: Benchmarking Agentic Coding for Complex Feature Development. arXiv preprint, 2025. [30] Yang, J., Jimenez, C.E., Wettig, A., Liber, K., Yao, S., Narasimhan, K., and Press, O. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv preprint arXiv:2405.15793, 2024.
15
[31] Li, J., Zhang, G., Li, Y., Dong, Y., Luo, L., Zhu, M., Guo, Y., and He, Q. EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories. arXiv preprint arXiv:2404.00599, 2024. [32] Yan, W., Liu, H., Wang, Y., Li, Z., Zhao, Q., Wei, F., Liu, T., and Sui, Z. CodeScope: An Execution-Based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation. In ACL, 2024. [33] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.P., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In NeurIPS, 2023.
A
Example Tasks
We present four example tasks, one per category, drawn from different repositories. Where / data_control_flow (sympy) Question: Where does ipartfrac get called, and how does its data flow through migcdex, igcdex, and gcdext? Key files: sympy/functions/elementary/_trigonometric_special.py, trigonometric.py, sympy/core/intfunc.py Rubric (first 3 points): 1. ipartfrac is called in cos._eval_rewrite_as_sqrt (not directly in sin’s version) 2. ipartfrac uses functools.reduce with a mul function to compute the product of all denominators 3. It builds complement values (denom // x for each x) and passes them to migcdex Gold answer (abridged): The ipartfrac function flows data through the following call chain: it is called in cos._eval_rewrite_as_sqrt() in trigonometric.py. Internally it uses functools.reduce to compute total product of denominators, builds complement values, and passes them to migcdex which calls igcdex pairwise... How / system_design (pytest) Question: How does pytest handle finalizer failures during fixture teardown, and how are multiple exceptions from different teardown stages combined? Key files: src/_pytest/fixtures.py, src/_pytest/runner.py, src/_pytest/skipping.py Rubric (first 3 points): 1. FixtureDef.finish() runs ALL finalizers even if some fail, collecting exceptions in a list 2. Finalizers are popped in LIFO order (using .pop()) 3. Multiple exceptions are wrapped in a BaseExceptionGroup with exceptions reversed
16
Gold answer (abridged): FixtureDef.finish() runs all finalizers even if some fail. Finalizers are popped in LIFO order. If multiple exceptions occur, they are wrapped in a BaseExceptionGroup with exceptions[::-1]. After finalization, cached_result is set to None and _finalizers.clear() is called even if finalization fails... What / architecture_exploration (xarray) Question: What is the role and architecture of _normalize_path in xarray’s backend system, and where is it used across the codebase? Key files: xarray/backends/common.py, xarray/core/utils.py, xarray/backends/api.py Rubric (first 3 points): 1. Defined in common.py with three @overload signatures and a single implementation accepting os.PathLike | str | T
2. Performs two transformations: os.fspath() for PathLike, and os.path.abspath(os.path.expandus for local strings 3. Remote URIs detected via is_remote_uri() (regex-based in utils.py) are left unmodified Gold answer (abridged): _normalize_path normalizes file paths throughout xarray’s backend I/O system. It has three @overload signatures: PathLike→str, str→str, and generic T→T. It converts PathLike objects via os.fspath(), expands local strings, and passes remote URIs through unchanged... Why / purpose_exploration (django) Question: Why is ogrinspect split into a public function and a private _ogrinspect generator, and how does the management command exploit that design? Key files: django/contrib/gis/utils/ogrinspect.py, management/commands/ogrinspect.py, gdal/geomtype.py Rubric (first 3 points): 1. _ogrinspect is a generator (uses yield) producing model definition lines one at a time; ogrinspect joins them with newlines 2. The management command calls _ogrinspect directly to collect lines into a list and append the mapping dictionary before joining 3. The command uses get_func_args(_ogrinspect) to dynamically filter CLI options to accepted parameters Gold answer (abridged): The separation serves two purposes: (1) streaming vs. string output — _ogrinspect yields lines one at a time, allowing the management command to append additional output before joining; (2) dynamic argument filtering — the command uses get_func_args to introspect accepted parameters...
B
Documentation Removal Examples
The stripping procedure removes all docstrings (via AST), all comments (via tokenize), and all documentation files (README, .md, .rst, docs/ directories). Below are two representative transformations. 17
Example 1: Docstring and comment removal. Before: def calculate_distance(point_a, point_b): """Calculate Euclidean distance between two points. Args: point_a: A tuple (x, y). point_b: A tuple (x, y). Returns: The Euclidean distance. """ # Compute squared differences dx = point_a[0] - point_b[0] dy = point_a[1] - point_b[1] return (dx**2 + dy**2) ** 0.5 After: def calculate_distance(point_a, point_b): dx = point_a[0] - point_b[0] dy = point_a[1] - point_b[1] return (dx**2 + dy**2) ** 0.5 Example 2: Docstring-only body replaced with pass. Before: class Validator: """Base class for all validators.""" def validate(self, value): """Validate the given value. Subclasses must override.""" raise NotImplementedError def get_help_text(self): """Return a description of what this validator checks."""
After: class Validator: def validate(self, value): raise NotImplementedError
def get_help_text(self): pass Note that get_help_text had only a docstring as its body, so the stripper inserts pass to maintain syntactic validity. The validate method retains its raise statement because it was a real code statement, not just documentation. Identifier names, type annotations, string literals, and import statements are preserved unchanged.
18
C
Judge Prompt
The full prompt given to the LLM judge: You are an expert judge evaluating an AI agent’s answer about a code repository. ## Question {question} ## Reference Answer (Gold) {gold_answer} ## Key Points the Answer Should Cover (Rubric) {rubric} ## Agent’s Answer {agent_answer} ## Instructions Score the agent’s answer on three axes, each from 0 to 5: 1. Accuracy (0-5): Are the factual claims correct? 2. Completeness (0-5): Does the answer cover all rubric points? 3. Specificity (0-5): Does the answer reference specific files/functions? Return JSON: {"accuracy": N, "completeness": N, "specificity": N, "explanation": "..."}
D
Comparison with SWE-QA and SWE-QA-Pro Table 9: Feature comparison of repository-level code QA benchmarks.
Feature
SWE-QA
SWE-QA-Pro
Code-QA-Bench
Repositories Questions Taxonomy Question source Doc in eval
12 (popular) 576 4-type, 12 sub GitHub issues Full
26 (long-tail) 260 4-type, 12 sub Issue clusters Full
Anti-memorization Answer generation Human validation Scoring axes Eval conditions Training data
None RAG + human Yes 5 1 No
Difficulty calibration Claude Code + human Yes 5 1 Yes (SFT+RL)
10 (popular, 3 cond.) 528 + 100 doc-dep 4-type, 12 sub (balanced) Doc chunks Code-only / Documented / None 3-condition design Answer-first agent Automated (LLM) 3 3 Pipeline supports
E
Code-Only Verification Prompt
The code-only verification pass (Section 4.4) uses a single LLM call with the following prompt structure. The system prompt establishes the auditor role: 19
You are an auditor for a code-understanding benchmark. Your job is to identify "doc-leakage" -- claims in a gold answer that come from documentation but are NOT recoverable by reading the code-only source code alone. Stripped code has ALL docstrings, comments, and documentation files removed. Only bare Python source remains: function/class definitions, logic, imports. The user prompt presents the gold answer and the code-only versions of the key files: Gold answer: --- answer --{answer text} --- end answer --Key files referenced (stripped versions shown below): --- path/to/file.py --{stripped source code} --- end path/to/file.py --For each factual claim in the gold answer, decide: - KEEP: verifiable from code-only code (function names, class hierarchies, imports, control flow, algorithm logic) - REMOVE: relies on documentation (design rationale from docstrings, purpose descriptions from README, parameter semantics) - REWRITE: partially verifiable -- keep the code-grounded part Respond with JSON: { "claims": [{"claim": "...", "verdict": "keep|remove|rewrite", "reason": "..."}], "revised_answer": "answer with REMOVE claims deleted", "doc_leakage_fraction": 0.0 to 1.0, "summary": "one-line summary" } The key files are stripped using the same AST-based procedure as the evaluation repository (Section 3.2), ensuring the auditor sees exactly the code that agents will encounter during evaluation.
F
Answer Generation Prompt
The answer generation agent receives two prompts. The system prompt establishes the agent’s role and exploration strategy: You are generating a gold-standard answer for a code-understanding benchmark. You have tools to explore the repository: read_file, list_directory, search_code. Your answer will be used as the ground truth to judge AI agents. It must contain specific, verifiable facts that can ONLY be known by 20
reading the actual source code. Exploration strategy: 1. Start by locating the code referenced in the documentation chunk 2. Read the primary file(s) and identify the key functions/classes 3. Go ONE LEVEL DEEPER: trace at least one callee, one parent class, or one related module to understand how the code connects to the broader system 4. Your answer must include facts you discovered from code that are NOT stated in the documentation chunk -- this is what makes the benchmark challenging The user prompt provides the documentation chunk as a topic guide and specifies the target category: The following documentation chunk from "{repo_name}" identifies the TOPIC for your answer. Use it to know WHAT to investigate -- but your answer must go beyond it by reading actual code. --- chunk --[{source}: {file_path}] {heading} {content} --- end chunk --Target question type: "{category}" (sub-type: "{sub_type}") Exploration guidance by category: - "what": Read the module structure, class hierarchies, and imports to map architecture and dependencies - "why": Look for code patterns that reveal design decisions -defensive checks, performance optimizations, fallback logic - "where": Trace the call chain: who calls this, what does it call, where does data flow through - "how": Read the implementation line by line -- understand the algorithm, the state transitions, the edge case handling Your job: 1. Use tools to find and read the source code 2. Go deeper: trace at least one callee, parent class, or import 3. Write an answer that includes specific code details NOT found in the documentation chunk above When done exploring, respond with JSON (no tool calls): { "answer": "...", "key_files": ["relative/path.py", ...], "code_evidence": [ "specific fact verified by reading code (file + function)", ... ] } Your answer MUST include at least 3 items in code_evidence. 21
G
Question Generation Prompt
The question generation agent derives a natural question from the gold answer. The system prompt: You are writing questions for a code-understanding benchmark. The questions should sound like a curious developer asking a colleague -short, natural, and conversational. Rules: - One sentence, max ~25 words - Do NOT mention file paths, directory names, or line numbers - Do NOT use phrases like "provide a comprehensive analysis", "describe in detail", "trace through", "for each step identify" - Use natural language: "What happens when...", "How does X work?" - The agent being tested will see code with ALL docstrings, comments, and docs removed - You have tools to read files, list directories, and search code The user prompt provides the gold answer and category guidance: Gold answer (what the correct response should cover): {answer} Target question type: "{category}" (sub-type: "{sub_type}") Write a short, natural "{category}" question that this answer would correctly respond to. Category guidance -- your question MUST match the target type: - "what": Ask about structure, definition, or relationships. Examples: "What components make up the caching subsystem?" - "why": Ask about rationale, purpose, or design decisions. Examples: "Why does the ORM use lazy evaluation?" - "where": Ask about location, data flow, or identifiers. Examples: "Where is the request routing logic implemented?" - "how": Ask about implementation, algorithms, or operation. Examples: "How does the template engine resolve lookups?" Bad examples (DO NOT write like these): - "Trace through function X in file Y and describe in detail..." - "Provide a comprehensive architectural analysis of..." The question must be answerable ONLY by reading source code (no docs). Respond with JSON: { "question": "...", "rubric": ["point1", ...] }
22
H
Bootstrap Confidence Intervals
All means and deltas reported in the main text are accompanied by 95% bootstrap confidence intervals computed via 10,000 resamples with replacement (seed = 42). For paired comparisons (e.g., documented−code-only), we use paired bootstrap: resampling task indices jointly and computing the delta on each resample. p-values are computed as the proportion of bootstrap samples where the delta crosses zero. Table 10: 95% bootstrap CIs for code-derivable task scores (528 tasks). CI width of ±0.006– 0.014 reflects tight estimation. Model
Closed-book
Code-only
Documented
Claude Opus 4.6 DeepSeek-V4-Pro Kimi-K2.6 Gemini-3.1-Pro
0.560 [.548, .571] 0.442 [.430, .455] 0.514 [.501, .527] 0.482 [.468, .497]
0.891 [.885, .897] 0.892 [.884, .899] 0.873 [.867, .880] 0.772 [.761, .784]
0.918 [.912, .922] 0.899 [.893, .905] 0.882 [.874, .889] 0.755 [.742, .766]
Table 11: 95% bootstrap CIs for doc-dependent task scores (100 tasks). Wider CIs reflect smaller sample size. Model
Closed-book
Code-only
Documented
Claude Opus 4.6 DeepSeek-V4-Pro Kimi-K2.6 Gemini-3.1-Pro
0.682 [.643, .719] 0.557 [.514, .597] 0.639 [.597, .681] 0.636 [.592, .679]
0.873 [.847, .898] 0.867 [.833, .898] 0.859 [.825, .890] 0.840 [.808, .871]
0.953 [.941, .965] 0.928 [.903, .947] 0.950 [.937, .962] 0.893 [.864, .917]
Table 12: Paired bootstrap test for ∆doc (documented − code-only). Code-derivable deltas are near zero for most models; doc-dependent deltas are consistently positive and significant. ∆
95% CI
p
Code-deriv.
Claude Opus 4.6 DeepSeek-V4-Pro Kimi-K2.6 Gemini-3.1-Pro
+0.026 +0.007 +0.008 −0.018
[+0.021, +0.033] [−0.001, +0.016] [−0.001, +0.017] [−0.031, −0.005]
<0.001 0.039 0.033 0.004
Doc-dep.
Claude Opus 4.6 Kimi-K2.6 DeepSeek-V4-Pro Gemini-3.1-Pro
+0.080 +0.091 +0.061 +0.053
[+0.059, +0.103] [+0.062, +0.123] [+0.024, +0.099] [+0.015, +0.091]
<0.001 <0.001 <0.001 0.002
Task Set
Model
The code-derivable results validate the code-only verification design: for DeepSeek and Kimi, the documented−code-only CI includes zero, confirming that gold answers are code-recoverable. Claude’s positive delta (+0.026, p < 0.001) likely reflects generator advantage rather than documentation dependency (Section 6). Gemini’s negative delta (−0.018, p = 0.004, marginally significant) suggests that documentation occasionally distracts from code-grounded reasoning for this model. For doc-dependent tasks, all models show significant positive deltas (minimum p = 0.002), confirming that documentation utility is real and measurable. The wider CIs (reflecting N = 100) mean that model-to-model differences in ∆doc are not individually significant, but the consistent directionality across all four models provides strong evidence for the aggregate effect. 23
I
Per-Repository Breakdown
Table 13 shows code-only scores by repository for all four models. Scores are consistent across repositories: within each model, the range spans only 0.04–0.07 points, confirming that aggregate results are not driven by a subset of easy or hard repositories. Table 13: Per-repository scores in the code-only condition on code-derivable tasks. Range = max − min within each model. Repository
Claude
DeepSeek
Kimi
Gemini
django pylint sympy scikit-learn astropy matplotlib sphinx pytest xarray seaborn
0.889 0.889 0.886 0.885 0.893 0.893 0.897 0.889 0.907 0.886
0.881 0.894 0.892 0.886 0.906 0.904 0.890 0.894 0.904 0.850
0.876 0.876 0.868 0.879 0.879 0.881 0.844 0.868 0.886 0.883
0.786 0.783 0.787 0.750 0.736 0.793 0.751 0.739 0.792 0.803
Range
0.022
0.056
0.042
0.067
24