arXiv:2606.29955v1 [cs.SE] 29 Jun 2026
S PREADSHEET B ENCH 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows
Jian Zhu1∗ Yuzheng Zhang1∗ Zeyao Ma1 Bohan Zhang1 Armin Schoepf2 Daniel Woloch2 Peter Yiliu Wang3 Guangyu Robert Yang3 Samuel Jacob4 Siddharth Nagisetty4 Abhiram Chundru4 Jean Lin4 Spencer Mateega4 Jing Zhang1† 1 School of Information, Renmin University of China 2 Aptura AI 3 Shortcut AI 4 AfterQuery
Abstract Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making. However, most existing spreadsheet benchmarks evaluate isolated operations such as single-formula generation or local cell edits, and therefore fail to capture end-to-end workflows in realistic business settings. We introduce S PREADSHEET B ENCH 2, a workflow-level benchmark for spreadsheet agents that covers three task categories: generation, debugging, and visualization. The benchmark is constructed from authentic business data, including financial reports and corporate filings, and is annotated and validated by domain experts. The benchmark contains 321 tasks; each instance averages 11.8 worksheets and requires 593.5 cell modifications, reflecting large multi-sheet workbooks with cross-sheet dependencies. We evaluate eight frontier large language models under a unified multi-turn agent scaffold, and additionally include several LLM-based spreadsheet products as complementary baselines. Results show that current systems remain far from reliable on real-world workflows: the best model achieves 34.89% overall task accuracy, and debugging accuracy is as low as 12.00%. Trajectory analysis and a failure taxonomy further indicate that insufficient spreadsheet inspection and incorrect target-cell selection are the dominant bottlenecks. Together, these findings position S PREADSHEET B ENCH 2 as a challenging testbed for advancing reliable spreadsheet automation. Project page: https://spreadsheetbench.github.io/
1
Introduction
Spreadsheets serve as a core infrastructure for structured data processing in modern organizations, supporting tasks such as financial modeling, reporting, and decision-making [1, 2, 3, 4, 5, 6, 7]. Recent advances in large language models (LLMs) have enabled the development of spreadsheet agents capable of generating formulas, manipulating data, and automating a wide range of spreadsheet operations [8, 9, 10, 11, 12, 13]. However, despite strong performance on existing benchmarks, these agents remain far from reliable in real-world spreadsheet scenarios. A key limitation lies in current evaluation benchmarks, which primarily assess spreadsheet agents at the level of isolated operations rather than complete workflows [8, 14, 15]. Existing work often focuses on small or simplified workbooks with limited cross-sheet dependencies [16, 17, 18, 19], emphasizing local cell-level manipulations instead of end-to-end task completion. In addition, many datasets are derived from synthetic sources or community forums, resulting in tasks that are less representative of real-world, professional business spreadsheet workflows [15, 20, 21, 22, 23]. ∗ Equal contribution. † Corresponding author. [email protected]
Preprint.
Debugging
Financial Modeling / Template
Data Visualization
Instruction (excerpt)
Instruction (excerpt)
Instruction (excerpt)
• Audit and repair the spreadsheet. • Identify and fix formula errors (#REF!, #VALUE!). • Correct cascading reference issues.
• Complete the financial model based on the provided assumptions. • Fill in the missing quarterly figures. • Calculate D/E Ratio for Q1–Q2.
• Create a scatter chart from my blood sugar data.. • Three categories: Low (<7), Normal (7-12), and High (>12). • Add data labels showing the value.
Summary
Cost Drivers
Equity Returns
Ratios Analysis
Summary
Dashboard
Margin Analysis
Cash Flow
Equity Returns
Corrupted Spreadsheet
Incomplete Financial Model
Coorrected Spreadsheet
Completed Financial Model
Visualization Chart
Figure 1: S PREADSHEET B ENCH 2 consists of three representative task categories: Debugging, Generation (Financial Modeling and Template), and Data Visualization. Debugging tasks focus on identifying and repairing errors; Generation tasks (Financial Modeling and Template) involve completing or constructing spreadsheets; Visualization tasks require producing analysis-ready charts. Detailed examples of each category are provided in Appendix E. In practice, spreadsheet usage follows a structured workflow. Tasks commonly involve constructing or completing spreadsheet artifacts, identifying and correcting errors, and presenting results through visualization and reporting. These stages introduce challenges such as cross-sheet reasoning, longrange dependencies, and multi-step coordination, which are largely absent from existing benchmarks. To better reflect these characteristics, we introduce S PREADSHEET B ENCH 2, a challenging benchmark for evaluating spreadsheet agents on workflow-level tasks grounded in real-world, professional business scenarios. As illustrated in Figure 1, the benchmark organizes tasks into three representative categories: generation (financial modeling and template completion), debugging, and visualization, each corresponding to a key stage in spreadsheet workflows. S PREADSHEET B ENCH 2 is constructed from authentic business data sources, including financial reports and corporate filings, and paired with expert-annotated instructions, requiring over 1,500 hours of expert effort (Appendix B.2). The benchmark consists of 321 tasks, each defined by a natural language instruction and an input spreadsheet representing an in-progress business artifact. These tasks involve complex multi-sheet structures, rich cross-sheet dependencies, and substantial modification requirements, making them significantly more challenging than prior datasets. We evaluate eight state-of-the-art large language models under a unified multi-turn agent scaffold, where models interact with spreadsheets through a command-line interface. In addition, we include several LLM-based spreadsheet products (e.g., Claude for Excel, ChatGPT for Excel) as complementary baselines. Despite strong performance on existing benchmarks, current LLMs and spreadsheet products achieve limited success on S PREADSHEET B ENCH 2, with overall accuracy remaining below 35% and substantially lower performance on debugging tasks. These results highlight a significant gap between current model capabilities and the requirements of real-world spreadsheet workflows. Further analysis reveals that failures are primarily driven by insufficient inspection and incorrect selection of target cells for modification. Moreover, to isolate the impact of scaffold design, we fix GLM-5 and find that our SWE-agent-based scaffold outperforms three coding agent scaffolds on 50 samples. While modification scores are comparable across scaffolds, task-level accuracy is substantially lower for general-purpose coding scaffolds, indicating weaker end-to-end correctness on spreadsheet workflows. Our contributions are summarized as follows: (1) We introduce S PREADSHEET B ENCH 2, a workflowlevel benchmark grounded in real-world business settings, featuring expert-annotated tasks over 2
complex multi-sheet spreadsheets with cross-sheet dependencies, thereby shifting evaluation from isolated operations to end-to-end task completion. (2) We provide a systematic evaluation of state-ofthe-art large language models under a unified multi-turn agent scaffold, together with complementary evaluation of LLM-based spreadsheet products, and we further benchmark different agent scaffolds to quantify how scaffold design affects end-to-end performance, revealing substantial performance gaps and key failure modes in real-world spreadsheet automation.
2
S PREADSHEET B ENCH 2
S PREADSHEET B ENCH 2 is designed to evaluate spreadsheet agents on realistic, workflow-level tasks in business settings. To achieve this goal, the benchmark is constructed around three key aspects: (1) task formulation based on real-world spreadsheet workflows, (2) data construction grounded in authentic business data, and (3) evaluation protocols that capture both cell-level correctness and task-level outcomes. In this section, we describe task categories, dataset construction, and evaluation metrics. 2.1
Task Categories
Real-world spreadsheet usage follows a structured workflow rather than isolated operations. In business settings, tasks typically involve three stages: constructing spreadsheet artifacts, validating and debugging logical dependencies, and communicating insights through visualization and reporting. These stages define the lifecycle of spreadsheet usage in practice. Motivated by this workflow perspective, S PREADSHEET B ENCH 2 organizes tasks into three categories: Generation, Debugging, and Visualization. Generation (Financial Modeling and Template). Generation tasks evaluate the ability to construct or complete spreadsheet artifacts from partially specified inputs or high-level objectives. These tasks involve cross-sheet reasoning, formula propagation, and the construction of structured spreadsheet. They correspond to the construction phase of spreadsheet workflows, where users design and populate structured workbooks from scratch or incomplete specifications. We include two levels of difficulty: Template tasks involve simpler, typically single-structure completion (avg. 1.2 sheets), while financial modeling tasks represent more complex scenarios with multiple interdependent components across sheets (avg. 15.3 sheets), requiring coordinated reasoning over large spreadsheet structures. Debugging. Debugging tasks evaluate the ability to identify and repair errors in existing spreadsheets. In real-world settings, the causes and locations of errors are often unknown. Therefore, instructions do not explicitly specify error types or target cells, requiring models to autonomously locate and fix issues. These tasks cover ten error types, including double counting (values are unintentionally aggregated multiple times, leading to systematic overestimation), incorrect references (misconfigured cell links or lookup ranges that break dependencies), formula errors (syntactic or logical mistakes that produce invalid or unintended outputs), and others (Appendix B.4). They correspond to the validation and correction phase of spreadsheet workflows and are particularly challenging, as they require holistic reasoning over complex, interdependent spreadsheet structures without explicit information about error locations or types. Visualization. Visualization tasks evaluate the ability to transform spreadsheet data into analysisready visual artifacts, such as charts and pivot-based summaries. In business contexts, spreadsheets function not only as computational tools but also as communication instruments for analysis and decision-making. Unlike standard visualization benchmarks, the input data in these tasks is often irregularly structured, distributed across multiple sheets, or embedded within semi-structured tables, requiring models to correctly interpret and extract relevant information. The target visualizations can involve complex structures, such as multiple data series, hierarchical axes, and customized formatting. These tasks correspond to the presentation stage of spreadsheet workflows and require both accurate data selection and precise visual construction within spreadsheet-native formats. 2.2
Benchmark Construction
As shown in Figure 2, S PREADSHEET B ENCH 2 is constructed through a three-stage pipeline: data collection, task construction, and data validation. 3
Data Collection Collect Real-World Business Data
Task Construction Expert Annotation
Data Validation Independent Review by Two Experts Expert A
Expert B
Financial Modeling / Template Annual Reports, Financial Statements, Investment, etc.
Transform into GoldStandard Spreadsheets
Build valuation models, forecasts, financial analyses using spreadsheet formulas.
Generate Solution
Generate Solution
Compare with Ground Truth
Compare with Ground Truth
Compare Outputs Between Experts
Agreement & Consistency Check
Debugging
Please audit and fix this file thoroughly, identify and correct errors in formulas, references, etc.
Visualization
Financial Experts Annotate and Build Gold-Standard Spreadsheet Solutions
Create charts and visual summaries within spreadsheets to clearly present key patterns and trends in the data.
SPREADSHEETBENCH 2
Figure 2: The benchmark construction pipeline of S PREADSHEET B ENCH 2.
Data Collection. We construct S PREADSHEET B ENCH 2 using data from authentic business sources, including publicly available financial reports and corporate filings from the NYU Stern (Damodaran) dataset, Screener.in, Bloomberg, Bseindia (More details in Appendix B.1). Financial experts curate these materials and build complete, internally consistent spreadsheets as gold-standard solutions, with correct formulas, cross-sheet references, and domain-appropriate logic. To capture the diversity of real-world spreadsheet workflows, the dataset covers a wide range of financial and accounting scenarios, including accounting tasks (e.g., consolidation, deferred tax, equity investments) and finance-related analyses such as discounted cash flow (DCF), leveraged buyout (LBO), mergers and acquisitions (M&A), and comparable company analysis. Task Construction. Based on the gold-standard spreadsheets, each task is created by transforming a complete workbook into a partially specified input spreadsheet paired with a natural-language instruction. Depending on the task category, this transformation involves selectively removing target regions, injecting controlled errors, or specifying visualization objectives, such that the resulting task requires the model to recover or produce the missing content. All tasks are manually designed and annotated by financial experts to reflect realistic business spreadsheet workflows. The construction process enforces several quality constraints: each task must admit a unique, deterministic solution given only the input spreadsheet and instruction; the instruction must be self-contained without requiring external knowledge beyond standard financial domain conventions; and the task must involve multi-step reasoning over complex spreadsheet structures rather than isolated cell-level edits. As a result, solving a single task typically requires coordinated operations across multiple sheets, including formula propagation, reference resolution, structural inference, and consistency verification, mirroring the cognitive demands of real-world spreadsheet work in professional settings. Data Validation. To ensure the overall quality of the benchmark, each task is independently reviewed by two experts who were not involved in its construction. Reviewers are provided only with the input spreadsheet and the natural-language instruction, and are asked to solve the task independently. Their solutions are then compared against the original ground-truth spreadsheet. If discrepancies arise, the task is iteratively revised through expert discussion and correction until consistent solutions are obtained. This cross-validation process ensures that task instructions are unambiguous, all required operations are well-defined, and the final solution is uniquely determined by the given spreadsheet and instruction. 2.3
Benchmark Statistics
Data Statistics. S PREADSHEET B ENCH 2 comprises 321 tasks across three categories: generation (financial modeling and template), debugging, and visualization. Table 1 summarizes the key statistics. The benchmark comprises an average of 11.8 worksheets and 593.5 modified cells per task, with Financial Modeling and Debugging emerging as the most challenging categories, averaging 1,164.5 and 656.6 modified cells per task, respectively. Solving a single task thus requires reasoning over 4
Table 1: Overview statistics of S PREADSHEET B ENCH 2. Avg. Sheets and Max. Sheets correspond to the mean and maximum number of worksheets per task file, respectively. Avg. Words indicates the length of the input instructions, while Modif. Cells denotes the average number of cells requiring modification. Category
#Tasks
Avg. Sheets
Max. Sheets
Avg. Words
Modif. Cells
Financial Modeling Debugging Template Visualization
100 100 97 24
15.3 21.1 1.2 1.7
43 99 3 8
705.9 387.0 139.9 617.7
1164.5 656.6 65.7 88.3
Overall
321
11.8
99
429.0
593.5
Table 2: Comparison of S PREADSHEET B ENCH 2 with existing benchmarks, emphasizing real-world tasks, multi-sheet and cross-sheet reasoning, and high domain expertise. Aspect Realism Total Tasks Avg. Instr. Words Multi-Sheet Tasks Domain Expertise Avg. Sheets/File Error Taxonomy Chart Tasks Real Workflow
SheetCopilot [8] InstructExcel [14] SpreadsheetBench [15] Synthetic 221 27.9 Rare Low ∼1 ✗ ✗ ✗
Synthetic 4,850 9.8 Limited Low ∼2 ✗ ✗ ✗
Semi-real 912 85.7 Limited Low ∼1.4 ✗ ✗ ✗
Ours Real-world + Expert-curated 321 429.0 Extensive High 11.8 ✓ ✓ ✓
cross-worksheet dependencies and coordinating hundreds of cell-level modifications, a level of complexity characteristic of real-world professional spreadsheet use. Comparison with existing benchmarks. Table 2 compares S PREADSHEET B ENCH 2 with representative spreadsheet benchmarks across multiple dimensions. First, S PREADSHEET B ENCH 2 is grounded in real-world business data curated by domain experts, whereas SheetCopilot [8] and InstructExcel [14] rely on synthetic instructions and SpreadsheetBench [15] draws from semi-real forum posts. Second, our tasks exhibit substantially higher structural complexity: an average of 11.8 sheets per file compared to roughly 1–2 in prior work, with instructions averaging 429.0 words that specify multi-step business objectives rather than single atomic operations. Third, S PREADSHEET B ENCH 2 introduces task dimensions absent from existing benchmarks, including a systematic error taxonomy for debugging, native spreadsheet chart generation for visualization, and end-to-end workflow-level evaluation that requires sustained cross-sheet reasoning and domain-specific knowledge. These properties make S PREADSHEET B ENCH 2 a more realistic and challenging testbed for evaluating agent capabilities in professional spreadsheet environments. 2.4
Evaluation Metrics
We adopt two evaluation protocols tailored to different output types. For Financial Modeling, Template, and Debugging tasks, we compare the agent’s output against the golden spreadsheet. For Visualization tasks, whose outputs are charts, we employ a VLM-based evaluation. Spreadsheet Execution Metrics. By comparing the input spreadsheet with the golden file, we identify the set of target cells—those whose values or formulas must be changed to complete the task. We report two complementary metrics that operate at different granularities. Modification is a cell-level metric: for each task, we compute the fraction of target cells whose computed values match the golden file; we then report the average across all tasks in a category. It captures how precisely the agent performs the required edits, independent of side effects elsewhere. Accuracy is a task-level metric: a task is scored 1 if every cell in the output spreadsheet matches the golden file (covering both target and non-target cells), and 0 otherwise; we report the fraction of tasks that achieve this full match. VLM-as-a-Judge. Rule-based verification is impractical for visualization tasks due to the diversity of chart types, layouts, and formatting. Instead, we design fine-grained rubrics for each task along 5
Table 3: Performance of open-source and closed-source models across four spreadsheet task categories. Modif. reports the average fraction of target cells whose computed values match the golden file. Acc. reports the fraction of tasks where every cell in the output matches the golden file. For Visualization tasks, Acc. instead reports the average rubric pass rate (fraction of expert-designed rubric satisfied per task). The Overall column aggregates across all applicable tasks per metric. Model
Template Modif.
Acc.
Financial Modeling Modif.
Acc.
Debugging Modif.
Acc.
Visualization Acc.
Overall Modif.
Acc.
Closed-source Models Claude Opus 4.6
91.97
52.58
89.69
34.00
50.38
12.00
62.50
77.20
34.89
GPT-5.2
83.84
35.05
87.07
33.00
39.07
8.00
45.83
69.85
26.79
Gemini 3.1 Pro
86.06
28.87
85.26
31.00
41.86
7.00
41.67
70.91
23.68
Open-source Models GLM-5
66.60
17.53
78.44
22.00
29.06
7.00
37.50
57.95
17.14
Deepseek-V3.2
75.19
25.77
59.78
7.00
28.71
10.00
33.33
54.35
15.58
Kimi K2.5
67.08
18.56
64.86
15.00
19.48
4.00
41.67
50.31
14.64
Qwen3.5-397B-A17B
65.62
17.53
69.59
10.00
24.06
3.00
25.00
52.96
11.22
MiniMax M2.5
50.14
7.22
62.05
8.00
16.76
4.00
16.67
42.91
7.17
two dimensions: Data Correctness (e.g., whether plotted values match the source data) and Format Compliance (e.g., chart type, axis labels, color schemes). All rubrics are constructed and validated by domain experts (Appendix B.3). We use GLM-4.6V [24] as the judge to produce a binary PASS/FAIL signal per criterion, and report the rubric pass rate as the task score.
3
Experiments
3.1
Experimental Settings
LLMs. We evaluate eight large language models via API-based inference, including five opensource models (GLM-5.0 [25], MiniMax M2.5 3 , Kimi K2.5 [26], Qwen3.5-397B-A17B [27], and DeepSeek-V3.2 [28]) and three closed-source models (GPT-5.2 4 , Gemini 3.1 Pro 5 , and Claude Opus 4.6 6 ). All models are evaluated in thinking mode, with GPT-5.2 and Gemini 3.1 Pro configured to use high reasoning effort. In addition, we include several LLM-based spreadsheet tools (Kimi Sheet, GLM in Excel, Claude for Excel, and ChatGPT for Excel) for complementary comparison; as these products do not expose a unified API interface compatible with our evaluation pipeline, we evaluate them through manual execution on a subset of 30 representative examples. Agent Scaffold. Our agent scaffold is built on SWE-agent [29] and retains its iterative observe– reason–act workflow [30]. In the main setting, the agent interacts with spreadsheets through three tools: bash, view_xlsx, and submit. The bash tool executes local commands for programmatic workbook operations, view_xlsx provides read-only inspection of workbook structure and sheet contents (including formulas and evaluated values), and submit finalizes the episode for scoring. We set the maximum number of interaction turns to 50 and keep the scaffold fixed across models by using the same prompt template and tool specifications (additional implementation details are provided in Appendix C). We further compare GLM-5 across different agent scaffolds, including our scaffold, Claude Code, Kilo Code, and Cline, to study the impact of scaffold design.
3 https://www.minimax.io/news/minimax-m25 4 https://openai.com/index/introducing-gpt-5-2/ 5 https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/ 6 https://www.anthropic.com/news/claude-opus-4-6
6
66.1
63.0
60
55.1
Overall Modif. Overall Acc.
54.4
54.2
Score (%)
45.4 40
19.8
20
us
e ud
a Cl
Op
4.6 i in
G
em
40.0
39.8
39.5
30.9
28.8
0
45.2
3.1
15.4
o Pr
x
l ce
e ud
fo
rE
14.3
T-
GP
a Cl
30.1
5.2
13.2
2.5
iK
m Ki
11.1
9.8
l ce
-5
M
GL
9.8
T GP
t ha
r fo
Ex
M
GL
in
l ce Ex
4.3
i
m
Ki
t ee Sh
Qw
C
4.3
-A 7B 39 .53 n
e
3.2
B 17
ax
iM in
M
2.1
2.5
M
k
De
e se
.2 V3
ep
Figure 3: Performance on a 30-example representative subset covering Financial Modeling, Debugging, and Visualization tasks. We compare different models and four LLM-based spreadsheet products. The results show that LLM-based spreadsheet products do not outperform models under our agent scaffold, with Claude for Excel achieving the best performance among the spreadsheet products.
3.2
Main Results
S PREADSHEET B ENCH 2 remains challenging even for frontier models. As shown in Table 3, the best-performing model, Claude Opus 4.6, achieves only 34.89% overall accuracy, and six of the eight evaluated models fall below 25%. Closed-source models hold a clear advantage over their opensource counterparts: the top three closed-source models reach 23.68%–34.89% accuracy, substantially outperforming all open-source models (7.17%–17.14%). The gap is equally pronounced within individual categories: Claude Opus 4.6 achieves 89.69% Modification but only 34.00% Accuracy on Financial Modeling, and 50.38% Modification but only 12.00% Accuracy on Debugging. This suggests that while current LLMs can perform individual cell-level edits with moderate success, they struggle to maintain cross-cell consistency and end-to-end correctness across multi-step workflows— indicating that the bottleneck lies not in isolated operations but in coordinating coherent modifications over complex, interdependent spreadsheet structures. Performance varies significantly across task categories. Table 3 presents the accuracy scores of various models across different task categories. The results reveal a pronounced performance disparity among task types: visualization tasks exhibit the lowest difficulty, with Claude Opus 4.6 achieving the highest accuracy of 62.5%; generation tasks, specifically template and financial modeling, follow with moderate performance; whereas debugging tasks pose the greatest challenge, with even the best-performing model, Claude Opus 4.6, attaining only 12% accuracy. These findings indicate that while current large language models have developed strong capabilities in spreadsheet visualization and foundational structure generation, they still face significant performance bottlenecks in real-world business scenarios that demand complex logical reasoning, error diagnosis, and automated correction. LLM-based spreadsheet products do not show clear performance advantages. As shown in Figure 3, we compare four LLM-based spreadsheet products—Kimi Sheet, GLM in Excel, Claude for Excel, and ChatGPT for Excel—against foundation models on a challenging subset of 30 examples spanning Financial Modeling, Debugging, and Visualization (10 each). For a fair comparison, each product is tested by human operators given only the task instruction and input file. The results show that none of the spreadsheet products surpass the foundation models, with Claude for Excel achieving the highest accuracy among them at only 15.4%, suggesting that the workflow-level complexity of S PREADSHEET B ENCH 2 poses significant challenges for current LLM-based spreadsheet products. 3.3
Analysis and Discussions
Ten Error Types of Debugging Tasks. In Figure 4, we compare model performance across ten debugging error subcategories (see Table 6). Difficulty varies: Incorrect Index Match and Incorrect Sign are relatively tractable, while Errors is the most challenging, with even the best model barely exceeding 20%. This category involves diagnosing and fixing formulas that produce explicit spreadsheet error values (e.g., #REF!, #NUM!), which often stem from complex cross-sheet dependencies or invalid references, making them particularly difficult to localize and resolve. Models exhibit complementary strengths, with Claude Opus 4.6 performing best on Incorrect Sign and GPT-5.2 leading on Double 7
Claude Opus 4.6
40.3
39.5
18.4
48.5
27.3
43.7
57.6
86.6
56.5
45.0
GPT-5.2
56.6
31.3
22.3
61.9
37.7
44.1
48.3
20.3
32.4
31.9
Gemini 3.1 Pro
37.6
24.0
11.0
39.4
17.9
30.6
44.9
39.9
22.7
20.9
Deepseek V3.2
28.8
31.7
15.9
30.0
21.1
8.2
53.4
42.1
30.9
22.1
GLM-5
16.3
34.4
16.1
20.0
21.5
25.8
49.9
36.0
29.3
32.5
Qwen3.5-397B-A17B
32.6
12.7
8.7
8.3
33.1
32.9
33.3
35.8
12.6
30.6
Kimi K2.5
20.2
16.5
12.4
10.0
11.5
22.8
38.5
17.8
29.6
11.6
15.7
12.2
5.7
10.4
10.9
39.4
14.7
30.8
10.3
MiniMax M2.5
Double Embedded Counting Hardcodes
15.9 Errors
Inconsistent Incorrect Cross-Sheet Incorrect Incorrect Rel vs Abs Unit Color Refs Index Match Sign Refs Mismatch Average
Figure 4: Modification scores across 10 error subcategories (Appendix B.4) in Debugging tasks.
Counting and Inconsistent Color, and no single model dominates across all subcategories. Notably, performance drops markedly on Errors, highlighting the persistent difficulty of diagnosing and fixing formula-level failures in spreadsheets. Interaction Turns and Effective Step Ratio. We compare Claude Opus 4.6 and MiniMax M2.5 in Figure 6(a). Claude Opus 4.6 consistently achieves higher modification scores and accuracy across all task categories than MiniMax M2.5. It also requires fewer average interaction steps across four task types, suggesting that increased interaction-step count does not necessarily improve performance in spreadsheet-based tasks. To explain this efficiency gap, we define an effective step as one executed without errors (e.g., tool-call failures, code execution errors, or bash failures). Figure 6(b) reveals that Claude Opus 4.6 maintains a substantially higher effective-step ratio, confirming that its advantage stems from more reliable per-step execution rather than a larger interaction budget: each step more consistently translates into meaningful task progress. Failure Taxonomy. To elucidate the underlying causes of agent failures, we conducted a systematic analysis of failure trajecto- Figure 5: Failure Taxonomy disries, categorizing errors into six distinct types, as summarized tribution for Claude Opus 4.6 train Table 7, where the detailed definitions of each type are pro- jectories of unresolved tasks. vided. Leveraging Claude Code for automated trajectory analysis Turn Limit Other Exceeded 3.4% of Claude Opus 4.6, we derived the error distribution illustrated Format/Output Error in Figure 5. Our results indicate that the predominant error cat4.7% 5.4% egories are Insufficient Inspection and Wrong Target Selection. Task This suggests that agents struggle to adequately comprehend the Misunderstanding 42.3% 11.4% intricate interdependencies within spreadsheets—such as formuInsufficient Inspection laic, numerical, and domain-specific logical relationships—and 32.9% fail to accurately pinpoint the target cells requiring modification. Wrong Target Selection Notably, these challenges reflect common pain points experienced by humans in real-world spreadsheet workflows. Different Agent Scaffold. Using GLM-5 as the fixed backbone, we compare our SWE-agent-based scaffold against three coding Table 4: Overall performance agent scaffolds on a subset of 50 samples (Table 4). Our scaffold comparison across different agent achieves the highest modification score (46.32%) and accuracy scaffolds. (15.45%), with Claude Code as the closest competitor (43.47% / 14.20%). While modification scores remain within a narrow 4- Agent Scaffold Modif. Acc. 46.32 15.45 point band, accuracy diverges sharply: Cline and Kilo Code attain Our Scaffold 43.47 14.20 only 8.66%, suggesting that general-purpose coding scaffolds can Claude Code Cline 41.98 8.66 produce plausible edits but struggle with end-to-end correctness Kilo 43.12 8.66 on spreadsheet tasks. Tool Usage Patterns. To understand the behavioral patterns of agents in spreadsheet tasks, we categorize tool usage trajectories into three functional phases (Figure 6(c)): Inspect (reading contents and examining formulas), Implement (writing cells and executing code), and Verify (re-reading modified cells to confirm correctness). Claude Opus 4.6 exhibits a higher inspection proportion than 8