SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows

arXiv:2606.29955v1 [cs.SE] 29 Jun 2026

S PREADSHEET B ENCH 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows

Jian Zhu1∗ Yuzheng Zhang1∗ Zeyao Ma1 Bohan Zhang1 Armin Schoepf2 Daniel Woloch2 Peter Yiliu Wang3 Guangyu Robert Yang3 Samuel Jacob4 Siddharth Nagisetty4 Abhiram Chundru4 Jean Lin4 Spencer Mateega4 Jing Zhang1† 1 School of Information, Renmin University of China 2 Aptura AI 3 Shortcut AI 4 AfterQuery

Abstract Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making. However, most existing spreadsheet benchmarks evaluate isolated operations such as single-formula generation or local cell edits, and therefore fail to capture end-to-end workflows in realistic business settings. We introduce S PREADSHEET B ENCH 2, a workflow-level benchmark for spreadsheet agents that covers three task categories: generation, debugging, and visualization. The benchmark is constructed from authentic business data, including financial reports and corporate filings, and is annotated and validated by domain experts. The benchmark contains 321 tasks; each instance averages 11.8 worksheets and requires 593.5 cell modifications, reflecting large multi-sheet workbooks with cross-sheet dependencies. We evaluate eight frontier large language models under a unified multi-turn agent scaffold, and additionally include several LLM-based spreadsheet products as complementary baselines. Results show that current systems remain far from reliable on real-world workflows: the best model achieves 34.89% overall task accuracy, and debugging accuracy is as low as 12.00%. Trajectory analysis and a failure taxonomy further indicate that insufficient spreadsheet inspection and incorrect target-cell selection are the dominant bottlenecks. Together, these findings position S PREADSHEET B ENCH 2 as a challenging testbed for advancing reliable spreadsheet automation. Project page: https://spreadsheetbench.github.io/

1

Introduction

Spreadsheets serve as a core infrastructure for structured data processing in modern organizations, supporting tasks such as financial modeling, reporting, and decision-making [1, 2, 3, 4, 5, 6, 7]. Recent advances in large language models (LLMs) have enabled the development of spreadsheet agents capable of generating formulas, manipulating data, and automating a wide range of spreadsheet operations [8, 9, 10, 11, 12, 13]. However, despite strong performance on existing benchmarks, these agents remain far from reliable in real-world spreadsheet scenarios. A key limitation lies in current evaluation benchmarks, which primarily assess spreadsheet agents at the level of isolated operations rather than complete workflows [8, 14, 15]. Existing work often focuses on small or simplified workbooks with limited cross-sheet dependencies [16, 17, 18, 19], emphasizing local cell-level manipulations instead of end-to-end task completion. In addition, many datasets are derived from synthetic sources or community forums, resulting in tasks that are less representative of real-world, professional business spreadsheet workflows [15, 20, 21, 22, 23]. ∗ Equal contribution. † Corresponding author. zhang-jing@ruc.edu.cn

Preprint.

Debugging

Financial Modeling / Template

Data Visualization

Instruction (excerpt)

Instruction (excerpt)

Instruction (excerpt)

• Audit and repair the spreadsheet. • Identify and fix formula errors (#REF!, #VALUE!). • Correct cascading reference issues.

• Complete the financial model based on the provided assumptions. • Fill in the missing quarterly figures. • Calculate D/E Ratio for Q1–Q2.

• Create a scatter chart from my blood sugar data.. • Three categories: Low (<7), Normal (7-12), and High (>12). • Add data labels showing the value.

Summary

Cost Drivers

Equity Returns

Ratios Analysis

Summary

Dashboard

Margin Analysis

Cash Flow

Equity Returns

Corrupted Spreadsheet

Incomplete Financial Model

Coorrected Spreadsheet

Completed Financial Model

Visualization Chart

Figure 1: S PREADSHEET B ENCH 2 consists of three representative task categories: Debugging, Generation (Financial Modeling and Template), and Data Visualization. Debugging tasks focus on identifying and repairing errors; Generation tasks (Financial Modeling and Template) involve completing or constructing spreadsheets; Visualization tasks require producing analysis-ready charts. Detailed examples of each category are provided in Appendix E. In practice, spreadsheet usage follows a structured workflow. Tasks commonly involve constructing or completing spreadsheet artifacts, identifying and correcting errors, and presenting results through visualization and reporting. These stages introduce challenges such as cross-sheet reasoning, longrange dependencies, and multi-step coordination, which are largely absent from existing benchmarks. To better reflect these characteristics, we introduce S PREADSHEET B ENCH 2, a challenging benchmark for evaluating spreadsheet agents on workflow-level tasks grounded in real-world, professional business scenarios. As illustrated in Figure 1, the benchmark organizes tasks into three representative categories: generation (financial modeling and template completion), debugging, and visualization, each corresponding to a key stage in spreadsheet workflows. S PREADSHEET B ENCH 2 is constructed from authentic business data sources, including financial reports and corporate filings, and paired with expert-annotated instructions, requiring over 1,500 hours of expert effort (Appendix B.2). The benchmark consists of 321 tasks, each defined by a natural language instruction and an input spreadsheet representing an in-progress business artifact. These tasks involve complex multi-sheet structures, rich cross-sheet dependencies, and substantial modification requirements, making them significantly more challenging than prior datasets. We evaluate eight state-of-the-art large language models under a unified multi-turn agent scaffold, where models interact with spreadsheets through a command-line interface. In addition, we include several LLM-based spreadsheet products (e.g., Claude for Excel, ChatGPT for Excel) as complementary baselines. Despite strong performance on existing benchmarks, current LLMs and spreadsheet products achieve limited success on S PREADSHEET B ENCH 2, with overall accuracy remaining below 35% and substantially lower performance on debugging tasks. These results highlight a significant gap between current model capabilities and the requirements of real-world spreadsheet workflows. Further analysis reveals that failures are primarily driven by insufficient inspection and incorrect selection of target cells for modification. Moreover, to isolate the impact of scaffold design, we fix GLM-5 and find that our SWE-agent-based scaffold outperforms three coding agent scaffolds on 50 samples. While modification scores are comparable across scaffolds, task-level accuracy is substantially lower for general-purpose coding scaffolds, indicating weaker end-to-end correctness on spreadsheet workflows. Our contributions are summarized as follows: (1) We introduce S PREADSHEET B ENCH 2, a workflowlevel benchmark grounded in real-world business settings, featuring expert-annotated tasks over 2

complex multi-sheet spreadsheets with cross-sheet dependencies, thereby shifting evaluation from isolated operations to end-to-end task completion. (2) We provide a systematic evaluation of state-ofthe-art large language models under a unified multi-turn agent scaffold, together with complementary evaluation of LLM-based spreadsheet products, and we further benchmark different agent scaffolds to quantify how scaffold design affects end-to-end performance, revealing substantial performance gaps and key failure modes in real-world spreadsheet automation.

2

S PREADSHEET B ENCH 2

S PREADSHEET B ENCH 2 is designed to evaluate spreadsheet agents on realistic, workflow-level tasks in business settings. To achieve this goal, the benchmark is constructed around three key aspects: (1) task formulation based on real-world spreadsheet workflows, (2) data construction grounded in authentic business data, and (3) evaluation protocols that capture both cell-level correctness and task-level outcomes. In this section, we describe task categories, dataset construction, and evaluation metrics. 2.1

Task Categories

Real-world spreadsheet usage follows a structured workflow rather than isolated operations. In business settings, tasks typically involve three stages: constructing spreadsheet artifacts, validating and debugging logical dependencies, and communicating insights through visualization and reporting. These stages define the lifecycle of spreadsheet usage in practice. Motivated by this workflow perspective, S PREADSHEET B ENCH 2 organizes tasks into three categories: Generation, Debugging, and Visualization. Generation (Financial Modeling and Template). Generation tasks evaluate the ability to construct or complete spreadsheet artifacts from partially specified inputs or high-level objectives. These tasks involve cross-sheet reasoning, formula propagation, and the construction of structured spreadsheet. They correspond to the construction phase of spreadsheet workflows, where users design and populate structured workbooks from scratch or incomplete specifications. We include two levels of difficulty: Template tasks involve simpler, typically single-structure completion (avg. 1.2 sheets), while financial modeling tasks represent more complex scenarios with multiple interdependent components across sheets (avg. 15.3 sheets), requiring coordinated reasoning over large spreadsheet structures. Debugging. Debugging tasks evaluate the ability to identify and repair errors in existing spreadsheets. In real-world settings, the causes and locations of errors are often unknown. Therefore, instructions do not explicitly specify error types or target cells, requiring models to autonomously locate and fix issues. These tasks cover ten error types, including double counting (values are unintentionally aggregated multiple times, leading to systematic overestimation), incorrect references (misconfigured cell links or lookup ranges that break dependencies), formula errors (syntactic or logical mistakes that produce invalid or unintended outputs), and others (Appendix B.4). They correspond to the validation and correction phase of spreadsheet workflows and are particularly challenging, as they require holistic reasoning over complex, interdependent spreadsheet structures without explicit information about error locations or types. Visualization. Visualization tasks evaluate the ability to transform spreadsheet data into analysisready visual artifacts, such as charts and pivot-based summaries. In business contexts, spreadsheets function not only as computational tools but also as communication instruments for analysis and decision-making. Unlike standard visualization benchmarks, the input data in these tasks is often irregularly structured, distributed across multiple sheets, or embedded within semi-structured tables, requiring models to correctly interpret and extract relevant information. The target visualizations can involve complex structures, such as multiple data series, hierarchical axes, and customized formatting. These tasks correspond to the presentation stage of spreadsheet workflows and require both accurate data selection and precise visual construction within spreadsheet-native formats. 2.2

Benchmark Construction

As shown in Figure 2, S PREADSHEET B ENCH 2 is constructed through a three-stage pipeline: data collection, task construction, and data validation. 3

Data Collection Collect Real-World Business Data

Task Construction Expert Annotation

Data Validation Independent Review by Two Experts Expert A

Expert B

Financial Modeling / Template Annual Reports, Financial Statements, Investment, etc.

Transform into GoldStandard Spreadsheets

Build valuation models, forecasts, financial analyses using spreadsheet formulas.

Generate Solution

Generate Solution

Compare with Ground Truth

Compare with Ground Truth

Compare Outputs Between Experts

Agreement & Consistency Check

Debugging

Please audit and fix this file thoroughly, identify and correct errors in formulas, references, etc.

Visualization

Financial Experts Annotate and Build Gold-Standard Spreadsheet Solutions

Create charts and visual summaries within spreadsheets to clearly present key patterns and trends in the data.

SPREADSHEETBENCH 2

Figure 2: The benchmark construction pipeline of S PREADSHEET B ENCH 2.

Data Collection. We construct S PREADSHEET B ENCH 2 using data from authentic business sources, including publicly available financial reports and corporate filings from the NYU Stern (Damodaran) dataset, Screener.in, Bloomberg, Bseindia (More details in Appendix B.1). Financial experts curate these materials and build complete, internally consistent spreadsheets as gold-standard solutions, with correct formulas, cross-sheet references, and domain-appropriate logic. To capture the diversity of real-world spreadsheet workflows, the dataset covers a wide range of financial and accounting scenarios, including accounting tasks (e.g., consolidation, deferred tax, equity investments) and finance-related analyses such as discounted cash flow (DCF), leveraged buyout (LBO), mergers and acquisitions (M&A), and comparable company analysis. Task Construction. Based on the gold-standard spreadsheets, each task is created by transforming a complete workbook into a partially specified input spreadsheet paired with a natural-language instruction. Depending on the task category, this transformation involves selectively removing target regions, injecting controlled errors, or specifying visualization objectives, such that the resulting task requires the model to recover or produce the missing content. All tasks are manually designed and annotated by financial experts to reflect realistic business spreadsheet workflows. The construction process enforces several quality constraints: each task must admit a unique, deterministic solution given only the input spreadsheet and instruction; the instruction must be self-contained without requiring external knowledge beyond standard financial domain conventions; and the task must involve multi-step reasoning over complex spreadsheet structures rather than isolated cell-level edits. As a result, solving a single task typically requires coordinated operations across multiple sheets, including formula propagation, reference resolution, structural inference, and consistency verification, mirroring the cognitive demands of real-world spreadsheet work in professional settings. Data Validation. To ensure the overall quality of the benchmark, each task is independently reviewed by two experts who were not involved in its construction. Reviewers are provided only with the input spreadsheet and the natural-language instruction, and are asked to solve the task independently. Their solutions are then compared against the original ground-truth spreadsheet. If discrepancies arise, the task is iteratively revised through expert discussion and correction until consistent solutions are obtained. This cross-validation process ensures that task instructions are unambiguous, all required operations are well-defined, and the final solution is uniquely determined by the given spreadsheet and instruction. 2.3

Benchmark Statistics

Data Statistics. S PREADSHEET B ENCH 2 comprises 321 tasks across three categories: generation (financial modeling and template), debugging, and visualization. Table 1 summarizes the key statistics. The benchmark comprises an average of 11.8 worksheets and 593.5 modified cells per task, with Financial Modeling and Debugging emerging as the most challenging categories, averaging 1,164.5 and 656.6 modified cells per task, respectively. Solving a single task thus requires reasoning over 4

Table 1: Overview statistics of S PREADSHEET B ENCH 2. Avg. Sheets and Max. Sheets correspond to the mean and maximum number of worksheets per task file, respectively. Avg. Words indicates the length of the input instructions, while Modif. Cells denotes the average number of cells requiring modification. Category

#Tasks

Avg. Sheets

Max. Sheets

Avg. Words

Modif. Cells

Financial Modeling Debugging Template Visualization

100 100 97 24

15.3 21.1 1.2 1.7

43 99 3 8

705.9 387.0 139.9 617.7

1164.5 656.6 65.7 88.3

Overall

321

11.8

99

429.0

593.5

Table 2: Comparison of S PREADSHEET B ENCH 2 with existing benchmarks, emphasizing real-world tasks, multi-sheet and cross-sheet reasoning, and high domain expertise. Aspect Realism Total Tasks Avg. Instr. Words Multi-Sheet Tasks Domain Expertise Avg. Sheets/File Error Taxonomy Chart Tasks Real Workflow

SheetCopilot [8] InstructExcel [14] SpreadsheetBench [15] Synthetic 221 27.9 Rare Low ∼1 ✗ ✗ ✗

Synthetic 4,850 9.8 Limited Low ∼2 ✗ ✗ ✗

Semi-real 912 85.7 Limited Low ∼1.4 ✗ ✗ ✗

Ours Real-world + Expert-curated 321 429.0 Extensive High 11.8 ✓ ✓ ✓

cross-worksheet dependencies and coordinating hundreds of cell-level modifications, a level of complexity characteristic of real-world professional spreadsheet use. Comparison with existing benchmarks. Table 2 compares S PREADSHEET B ENCH 2 with representative spreadsheet benchmarks across multiple dimensions. First, S PREADSHEET B ENCH 2 is grounded in real-world business data curated by domain experts, whereas SheetCopilot [8] and InstructExcel [14] rely on synthetic instructions and SpreadsheetBench [15] draws from semi-real forum posts. Second, our tasks exhibit substantially higher structural complexity: an average of 11.8 sheets per file compared to roughly 1–2 in prior work, with instructions averaging 429.0 words that specify multi-step business objectives rather than single atomic operations. Third, S PREADSHEET B ENCH 2 introduces task dimensions absent from existing benchmarks, including a systematic error taxonomy for debugging, native spreadsheet chart generation for visualization, and end-to-end workflow-level evaluation that requires sustained cross-sheet reasoning and domain-specific knowledge. These properties make S PREADSHEET B ENCH 2 a more realistic and challenging testbed for evaluating agent capabilities in professional spreadsheet environments. 2.4

Evaluation Metrics

We adopt two evaluation protocols tailored to different output types. For Financial Modeling, Template, and Debugging tasks, we compare the agent’s output against the golden spreadsheet. For Visualization tasks, whose outputs are charts, we employ a VLM-based evaluation. Spreadsheet Execution Metrics. By comparing the input spreadsheet with the golden file, we identify the set of target cells—those whose values or formulas must be changed to complete the task. We report two complementary metrics that operate at different granularities. Modification is a cell-level metric: for each task, we compute the fraction of target cells whose computed values match the golden file; we then report the average across all tasks in a category. It captures how precisely the agent performs the required edits, independent of side effects elsewhere. Accuracy is a task-level metric: a task is scored 1 if every cell in the output spreadsheet matches the golden file (covering both target and non-target cells), and 0 otherwise; we report the fraction of tasks that achieve this full match. VLM-as-a-Judge. Rule-based verification is impractical for visualization tasks due to the diversity of chart types, layouts, and formatting. Instead, we design fine-grained rubrics for each task along 5

Table 3: Performance of open-source and closed-source models across four spreadsheet task categories. Modif. reports the average fraction of target cells whose computed values match the golden file. Acc. reports the fraction of tasks where every cell in the output matches the golden file. For Visualization tasks, Acc. instead reports the average rubric pass rate (fraction of expert-designed rubric satisfied per task). The Overall column aggregates across all applicable tasks per metric. Model

Template Modif.

Acc.

Financial Modeling Modif.

Acc.

Debugging Modif.

Acc.

Visualization Acc.

Overall Modif.

Acc.

Closed-source Models Claude Opus 4.6

91.97

52.58

89.69

34.00

50.38

12.00

62.50

77.20

34.89

GPT-5.2

83.84

35.05

87.07

33.00

39.07

8.00

45.83

69.85

26.79

Gemini 3.1 Pro

86.06

28.87

85.26

31.00

41.86

7.00

41.67

70.91

23.68

Open-source Models GLM-5

66.60

17.53

78.44

22.00

29.06

7.00

37.50

57.95

17.14

Deepseek-V3.2

75.19

25.77

59.78

7.00

28.71

10.00

33.33

54.35

15.58

Kimi K2.5

67.08

18.56

64.86

15.00

19.48

4.00

41.67

50.31

14.64

Qwen3.5-397B-A17B

65.62

17.53

69.59

10.00

24.06

3.00

25.00

52.96

11.22

MiniMax M2.5

50.14

7.22

62.05

8.00

16.76

4.00

16.67

42.91

7.17

two dimensions: Data Correctness (e.g., whether plotted values match the source data) and Format Compliance (e.g., chart type, axis labels, color schemes). All rubrics are constructed and validated by domain experts (Appendix B.3). We use GLM-4.6V [24] as the judge to produce a binary PASS/FAIL signal per criterion, and report the rubric pass rate as the task score.

3

Experiments

3.1

Experimental Settings

LLMs. We evaluate eight large language models via API-based inference, including five opensource models (GLM-5.0 [25], MiniMax M2.5 3 , Kimi K2.5 [26], Qwen3.5-397B-A17B [27], and DeepSeek-V3.2 [28]) and three closed-source models (GPT-5.2 4 , Gemini 3.1 Pro 5 , and Claude Opus 4.6 6 ). All models are evaluated in thinking mode, with GPT-5.2 and Gemini 3.1 Pro configured to use high reasoning effort. In addition, we include several LLM-based spreadsheet tools (Kimi Sheet, GLM in Excel, Claude for Excel, and ChatGPT for Excel) for complementary comparison; as these products do not expose a unified API interface compatible with our evaluation pipeline, we evaluate them through manual execution on a subset of 30 representative examples. Agent Scaffold. Our agent scaffold is built on SWE-agent [29] and retains its iterative observe– reason–act workflow [30]. In the main setting, the agent interacts with spreadsheets through three tools: bash, view_xlsx, and submit. The bash tool executes local commands for programmatic workbook operations, view_xlsx provides read-only inspection of workbook structure and sheet contents (including formulas and evaluated values), and submit finalizes the episode for scoring. We set the maximum number of interaction turns to 50 and keep the scaffold fixed across models by using the same prompt template and tool specifications (additional implementation details are provided in Appendix C). We further compare GLM-5 across different agent scaffolds, including our scaffold, Claude Code, Kilo Code, and Cline, to study the impact of scaffold design.

3 https://www.minimax.io/news/minimax-m25 4 https://openai.com/index/introducing-gpt-5-2/ 5 https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/ 6 https://www.anthropic.com/news/claude-opus-4-6

6

66.1

63.0

60

55.1

Overall Modif. Overall Acc.

54.4

54.2

Score (%)

45.4 40

19.8

20

us

e ud

a Cl

Op

4.6 i in

G

em

40.0

39.8

39.5

30.9

28.8

0

45.2

3.1

15.4

o Pr

x

l ce

e ud

fo

rE

14.3

T-

GP

a Cl

30.1

5.2

13.2

2.5

iK

m Ki

11.1

9.8

l ce

-5

M

GL

9.8

T GP

t ha

r fo

Ex

M

GL

in

l ce Ex

4.3

i

m

Ki

t ee Sh

Qw

C

4.3

-A 7B 39 .53 n

e

3.2

B 17

ax

iM in

M

2.1

2.5

M

k

De

e se

.2 V3

ep

Figure 3: Performance on a 30-example representative subset covering Financial Modeling, Debugging, and Visualization tasks. We compare different models and four LLM-based spreadsheet products. The results show that LLM-based spreadsheet products do not outperform models under our agent scaffold, with Claude for Excel achieving the best performance among the spreadsheet products.

3.2

Main Results

S PREADSHEET B ENCH 2 remains challenging even for frontier models. As shown in Table 3, the best-performing model, Claude Opus 4.6, achieves only 34.89% overall accuracy, and six of the eight evaluated models fall below 25%. Closed-source models hold a clear advantage over their opensource counterparts: the top three closed-source models reach 23.68%–34.89% accuracy, substantially outperforming all open-source models (7.17%–17.14%). The gap is equally pronounced within individual categories: Claude Opus 4.6 achieves 89.69% Modification but only 34.00% Accuracy on Financial Modeling, and 50.38% Modification but only 12.00% Accuracy on Debugging. This suggests that while current LLMs can perform individual cell-level edits with moderate success, they struggle to maintain cross-cell consistency and end-to-end correctness across multi-step workflows— indicating that the bottleneck lies not in isolated operations but in coordinating coherent modifications over complex, interdependent spreadsheet structures. Performance varies significantly across task categories. Table 3 presents the accuracy scores of various models across different task categories. The results reveal a pronounced performance disparity among task types: visualization tasks exhibit the lowest difficulty, with Claude Opus 4.6 achieving the highest accuracy of 62.5%; generation tasks, specifically template and financial modeling, follow with moderate performance; whereas debugging tasks pose the greatest challenge, with even the best-performing model, Claude Opus 4.6, attaining only 12% accuracy. These findings indicate that while current large language models have developed strong capabilities in spreadsheet visualization and foundational structure generation, they still face significant performance bottlenecks in real-world business scenarios that demand complex logical reasoning, error diagnosis, and automated correction. LLM-based spreadsheet products do not show clear performance advantages. As shown in Figure 3, we compare four LLM-based spreadsheet products—Kimi Sheet, GLM in Excel, Claude for Excel, and ChatGPT for Excel—against foundation models on a challenging subset of 30 examples spanning Financial Modeling, Debugging, and Visualization (10 each). For a fair comparison, each product is tested by human operators given only the task instruction and input file. The results show that none of the spreadsheet products surpass the foundation models, with Claude for Excel achieving the highest accuracy among them at only 15.4%, suggesting that the workflow-level complexity of S PREADSHEET B ENCH 2 poses significant challenges for current LLM-based spreadsheet products. 3.3

Analysis and Discussions

Ten Error Types of Debugging Tasks. In Figure 4, we compare model performance across ten debugging error subcategories (see Table 6). Difficulty varies: Incorrect Index Match and Incorrect Sign are relatively tractable, while Errors is the most challenging, with even the best model barely exceeding 20%. This category involves diagnosing and fixing formulas that produce explicit spreadsheet error values (e.g., #REF!, #NUM!), which often stem from complex cross-sheet dependencies or invalid references, making them particularly difficult to localize and resolve. Models exhibit complementary strengths, with Claude Opus 4.6 performing best on Incorrect Sign and GPT-5.2 leading on Double 7

Claude Opus 4.6

40.3

39.5

18.4

48.5

27.3

43.7

57.6

86.6

56.5

45.0

GPT-5.2

56.6

31.3

22.3

61.9

37.7

44.1

48.3

20.3

32.4

31.9

Gemini 3.1 Pro

37.6

24.0

11.0

39.4

17.9

30.6

44.9

39.9

22.7

20.9

Deepseek V3.2

28.8

31.7

15.9

30.0

21.1

8.2

53.4

42.1

30.9

22.1

GLM-5

16.3

34.4

16.1

20.0

21.5

25.8

49.9

36.0

29.3

32.5

Qwen3.5-397B-A17B

32.6

12.7

8.7

8.3

33.1

32.9

33.3

35.8

12.6

30.6

Kimi K2.5

20.2

16.5

12.4

10.0

11.5

22.8

38.5

17.8

29.6

11.6

15.7

12.2

5.7

10.4

10.9

39.4

14.7

30.8

10.3

MiniMax M2.5

Double Embedded Counting Hardcodes

15.9 Errors

Inconsistent Incorrect Cross-Sheet Incorrect Incorrect Rel vs Abs Unit Color Refs Index Match Sign Refs Mismatch Average

Figure 4: Modification scores across 10 error subcategories (Appendix B.4) in Debugging tasks.

Counting and Inconsistent Color, and no single model dominates across all subcategories. Notably, performance drops markedly on Errors, highlighting the persistent difficulty of diagnosing and fixing formula-level failures in spreadsheets. Interaction Turns and Effective Step Ratio. We compare Claude Opus 4.6 and MiniMax M2.5 in Figure 6(a). Claude Opus 4.6 consistently achieves higher modification scores and accuracy across all task categories than MiniMax M2.5. It also requires fewer average interaction steps across four task types, suggesting that increased interaction-step count does not necessarily improve performance in spreadsheet-based tasks. To explain this efficiency gap, we define an effective step as one executed without errors (e.g., tool-call failures, code execution errors, or bash failures). Figure 6(b) reveals that Claude Opus 4.6 maintains a substantially higher effective-step ratio, confirming that its advantage stems from more reliable per-step execution rather than a larger interaction budget: each step more consistently translates into meaningful task progress. Failure Taxonomy. To elucidate the underlying causes of agent failures, we conducted a systematic analysis of failure trajecto- Figure 5: Failure Taxonomy disries, categorizing errors into six distinct types, as summarized tribution for Claude Opus 4.6 train Table 7, where the detailed definitions of each type are pro- jectories of unresolved tasks. vided. Leveraging Claude Code for automated trajectory analysis Turn Limit
 Other
 Exceeded 3.4% of Claude Opus 4.6, we derived the error distribution illustrated Format/Output 
 Error in Figure 5. Our results indicate that the predominant error cat4.7% 5.4% egories are Insufficient Inspection and Wrong Target Selection. Task
 This suggests that agents struggle to adequately comprehend the Misunderstanding 42.3% 11.4% intricate interdependencies within spreadsheets—such as formuInsufficient
 Inspection laic, numerical, and domain-specific logical relationships—and 32.9% fail to accurately pinpoint the target cells requiring modification. Wrong Target
 Selection Notably, these challenges reflect common pain points experienced by humans in real-world spreadsheet workflows. Different Agent Scaffold. Using GLM-5 as the fixed backbone, we compare our SWE-agent-based scaffold against three coding Table 4: Overall performance agent scaffolds on a subset of 50 samples (Table 4). Our scaffold comparison across different agent achieves the highest modification score (46.32%) and accuracy scaffolds. (15.45%), with Claude Code as the closest competitor (43.47% / 14.20%). While modification scores remain within a narrow 4- Agent Scaffold Modif. Acc. 46.32 15.45 point band, accuracy diverges sharply: Cline and Kilo Code attain Our Scaffold 43.47 14.20 only 8.66%, suggesting that general-purpose coding scaffolds can Claude Code Cline 41.98 8.66 produce plausible edits but struggle with end-to-end correctness Kilo 43.12 8.66 on spreadsheet tasks. Tool Usage Patterns. To understand the behavioral patterns of agents in spreadsheet tasks, we categorize tool usage trajectories into three functional phases (Figure 6(c)): Inspect (reading contents and examining formulas), Implement (writing cells and executing code), and Verify (re-reading modified cells to confirm correctness). Claude Opus 4.6 exhibits a higher inspection proportion than 8

$YHUDJHVWHSV

'EJ )LQ



 

 

9L] 

 

7HPS

 0LQL0D[0

&ODXGH2SXV

(a)

  



 

7HPS 9L]

 

&ODXGH2SXV 0LQL0D[0 ,QVSHFW ,PSOHPHQW 9HULI\

&ODXGH2SXV

)LQ

'EJ

 



 0LQL0D[0

&ODXGH2SXV

(b)

3HUFHQW



 

 0LQL0D[0

&ODXGH2SXV

(IIHFWLYHVWHSUDWLR

0LQL0D[0

   

7HPS

)LQ

'EJ

9L]

(c)

Figure 6: Overview of trajectory-level behavior across four task domains. (a) Average number of steps per task for each model. (b) Effective-step ratio, measuring the fraction of steps that directly contribute to task progress. (c) Process distribution, showing the proportion of inspect, implement, and verify phases within each domain.

MiniMax M2.5 in Template, Visualization, and Financial Modeling tasks, but this trend reverses in Debugging tasks, where both models allocate over 60% of their trajectory to inspection. We attribute this to the absence of explicit step-by-step instructions in Debugging tasks, which necessitates extensive sheet exploration and imposes significant demands on the agent’s ability to localize information within complex spreadsheet environments. Case Study. We highlight a representative Financial Modeling failure (details in Appendix D.4, Figure 15). The task requires filling six groups of empty cells across a multi-sheet banking model. Claude Opus 4.6 derived formulas from general financial knowledge (e.g., “Book Value per Share = Total Equity / Shares Outstanding”) rather than reverse-engineering the conventions already established in the workbook; the formulas are textbook-correct but the cell references did not match the actual row layout, yielding only 4.1% modification accuracy. This reveals that current agents can retrieve relevant domain knowledge but lack the ability to ground it in a specific workbook’s dependency structure, a capability gap that future spreadsheet agents must address. Five additional cases spanning Debugging and Visualization are provided in Appendix D.4.

4

Related Work

Spreadsheet and Table Benchmarks. A substantial body of work has developed benchmarks for table understanding, spanning question answering over structured tables, fact verification against tabular evidence, and numerical reasoning over hybrid tabular-textual data [31, 32, 33, 34, 35, 36, 37, 20]. On the spreadsheet side, prior efforts address two complementary directions: table encoding and representation learning for spreadsheet structures [38, 39], and benchmarking spreadsheet manipulation from natural language instructions, including formula generation, cell editing, and tasklevel completion [8, 14, 40, 15]. However, these benchmarks predominantly target isolated atomic operations, such as generating a single formula or editing specific cells, rather than the multi-step, cross-sheet workflows that characterize real business spreadsheet usage. S PREADSHEET B ENCH 2 addresses this gap with end-to-end workflow-level tasks spanning generation, debugging, and visualization, averaging 11.8 sheets and 593.5 modified cells per instance. LLM-based Spreadsheet Agents. Recent advances in LLM-based agents have demonstrated the potential of autonomous tool use for complex tasks, with agent frameworks and benchmarks emerging across coding, web navigation, and desktop environments [30, 41, 42, 43, 44, 45]. In the spreadsheet domain, several agent systems have been proposed to automate spreadsheet operations, adopting diverse strategies such as translating natural language into API calls, multi-module reasoning pipelines, neuro-symbolic computation, and multi-agent collaboration [8, 9, 10, 11]. Despite this progress, existing spreadsheet agents are evaluated only on isolated operations, leaving a gap in assessing performance on realistic business workflows that require sustained cross-sheet reasoning and coordinated tool use. S PREADSHEET B ENCH 2 provides a workflow-level evaluation testbed that requires agents to perform multi-step, cross-sheet reasoning and produce structurally coherent spreadsheet artifacts, enabling systematic comparison of agent capabilities on realistic business tasks. 9

5

Conclusion

We present S PREADSHEET B ENCH 2, a benchmark of 321 expert-annotated tasks evaluating spreadsheet agents on workflow-level generation, debugging, and visualization over multi-sheet workbooks with rich cross-sheet dependencies. The best model achieves only 34.89% accuracy (debugging: 12%), and stronger models benefit from more reliable per-step execution rather than larger interaction budgets, highlighting structured reasoning over interdependent workbooks as a key bottleneck.

References [1] Weizheng Lu, Jing Zhang, Ju Fan, Zihao Fu, Yueguo Chen, and Xiaoyong Du. Large language model for table processing: A survey. Frontiers of Computer Science, 19(2):192350, 2025. [2] Mingyue Cheng, Qi Liu, Qingyang Mao, Yitong Zhou, Yupeng Li, Jiahao Wang, Jiaying Lin, Jiawei Cao, and Enhong Chen. A survey on table mining with large language models: Challenges, advancements and prospects. 2025. [3] Karl W Broman and Kara H Woo. Data organization in spreadsheets. The American Statistician, 72(1):2–10, 2018. [4] David McD Taylor, Peter W Hodkinson, Abdus Salam Khan, and Erin L Simon. Research skills and the data spreadsheet: A research primer for low-and middle-income countries. African Journal of Emergency Medicine, 10:S140–S144, 2020. [5] Abdullah Al Maruf, Rajesh Paul, Mohammad Hasan Imam, and Zahir Babar. A systematic review of the role of sql and excel in data-driven business decision-making for aspiring analysts. American Journal of Scholarly Research and Innovation, 1(01):249–269, 2022. [6] TG Grossman, Vijay Mehrotra, and Özgür Özlük. Spreadsheet information systems are essential to business. University of San Francisco working paper, 2005. [7] David A Bradbard, Charles Alvis, and Richard Morris. Spreadsheet usage by management accountants: An exploratory study. Journal of Accounting Education, 32(4):24–30, 2014. [8] Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and Zhao-Xiang Zhang. Sheetcopilot: Bringing software productivity to the next level through large language models. Advances in Neural Information Processing Systems, 36:4952–4984, 2023. [9] Yibin Chen, Yifu Yuan, Zeyu Zhang, Yan Zheng, Jinyi Liu, Fei Ni, Jianye Hao, Hangyu Mao, and Fuzheng Zhang. Sheetagent: towards a generalist agent for spreadsheet reasoning and manipulation via large language models. In Proceedings of the ACM on Web Conference 2025, pages 158–177, 2025. [10] Ziwei Wang, Jiayuan Su, Mengyu Zhou, Huaxing Zeng, Mengni Jia, Xiao Lv, Haoyu Dong, Xiaojun Ma, Shi Han, and Dongmei Zhang. Sheetbrain: A neuro-symbolic agent for accurate reasoning over complex and large spreadsheets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 33800–33808, 2026. [11] Ruiyan Zhu, Xi Cheng, Ke Liu, Brian Zhu, Daniel Jin, Neeraj Parihar, Zhoutian Xu, and Oliver Gao. Sheetmind: An end-to-end llm-powered multi-agent framework for spreadsheet automation. arXiv preprint arXiv:2506.12339, 2025. [12] Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-gpt: Table fine-tuned gpt for diverse table tasks. Proceedings of the ACM on Management of Data, 2(3):1–28, 2024. [13] Harshit Joshi, Abishai Ebenezer, José Cambronero Sanchez, Sumit Gulwani, Aditya Kanade, Vu Le, Ivan Radiček, and Gust Verbruggen. Flame: A small language model for spreadsheet formulas. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12995–13003, 2024. 10

[14] Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme, and Elnaz Nouri. Instructexcel: A benchmark for natural language instruction in excel. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4026–4043, 2023. [15] Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. Spreadsheetbench: Towards challenging real world spreadsheet manipulation. Advances in Neural Information Processing Systems, 37:94871–94908, 2024. [16] Simon Thorne. Large language models for spreadsheets: Benchmarking progress and evaluating performance with flare. arXiv preprint arXiv:2506.17330, 2025. [17] Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. Search-based neural structured learning for sequential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1821–1831, 2017. [18] Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4320–4333, 2020. [19] Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. Tapex: Table pre-training via learning a neural sql executor. arXiv preprint arXiv:2107.07653, 2021. [20] Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xeron Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, et al. Tablebench: A comprehensive and complex benchmark for table question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25497–25506, 2025. [21] Zheng Li, Yang Du, Mao Zheng, and Mingyang Song. Mimotable: A multi-scale spreadsheet benchmark with meta operations for table reasoning. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2548–2560, 2025. [22] Junjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong, Shi Han, Lingjiao Chen, Dongmei Zhang, Surajit Chaudhuri, and HV Jagadish. Mmtu: A massive multi-task table understanding and reasoning benchmark. arXiv preprint arXiv:2506.05587, 2025. [23] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36:42330–42357, 2023. [24] Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006, 2025. [25] Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. [26] Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. [27] Qwen Team. Qwen3.5: Accelerating productivity with native multimodal agents, February 2026. [28] Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. 11

[29] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. [30] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. [31] Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1480, 2015. [32] Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164, 2019. [33] Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026–1036, 2020. [34] Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 3277–3287, 2021. [35] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, 2021. [36] Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 conference on empirical methods in natural language processing, pages 6279–6292, 2022. [37] Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6588–6600, 2022. [38] Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 8413–8426, 2020. [39] Haoyu Dong, Jianbo Zhao, Yuzhang Tian, Junyu Xiong, Shiyu Xia, Mengyu Zhou, Yun Lin, José Cambronero, Yeye He, Shi Han, et al. Spreadsheetllm: Encoding spreadsheets for large language models. arXiv preprint arXiv:2407.09025, 2024. [40] Wei Zhao, Zhitao Hou, Siyuan Wu, Yan Gao, Haoyu Dong, Yao Wan, Hongyu Zhang, Yulei Sui, and Haidong Zhang. Nl2formula: Generating spreadsheet formulas from natural language queries. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2377–2388, 2024. [41] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539– 68551, 2023. [42] Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning, 2024. 12

[43] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. [44] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. [45] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023.

13

A

Broader Discussion

A.1

Limitations

Although S PREADSHEET B ENCH 2 advances spreadsheet agent evaluation, several limitations remain. (1) The benchmark focuses on financial and business spreadsheet workflows, and the findings may not generalize to other spreadsheet-intensive domains such as scientific computing or engineering. (2) Due to the high cost of API-based LLM inference, we do not report error bars or confidence intervals for the experimental results. (3) For visualization tasks, the VLM-as-a-judge framework may introduce evaluation noise despite expert-designed rubrics; for cell-based tasks, exact-match evaluation may undercount semantically equivalent but syntactically different solutions. A.2

Ethical Considerations

In this section, we discuss the ethical considerations regarding our data construction process. (1) Data Risk Control: During the benchmark construction process, we review all spreadsheet files and task instructions to ensure they do not contain content that is inappropriate for a general audience, such as violence, discrimination, or sensitive subjects. In addition, two authors conduct an independent verification to confirm that the benchmark does not contain any personally identifiable information. All financial data is derived from publicly available corporate filings and financial reports. (2) Copyright: Our benchmark is constructed from publicly available financial reports and corporate filings. To prevent copyright infringement, we do not directly redistribute any raw source documents. Instead, the released benchmark consists of expert-constructed spreadsheet tasks derived from these public materials, with numerical values and structural layouts modified where necessary. Considering potential licensing and copyright risks, we release the benchmark under a CC BY-SA 4.0 license and refrain from secondary distribution of any raw source data. A.3

Broader Impacts

On the positive side, advancing spreadsheet agent capabilities can reduce manual effort in repetitive business workflows, democratize access to financial modeling, and help detect costly spreadsheet errors that are a well-documented source of financial losses. On the negative side, over-reliance on AI-based spreadsheet agents without adequate human oversight could lead to undetected errors in financial models, and automation of financial tasks may raise concerns about workforce displacement. Our benchmark explicitly quantifies the gap between current model capabilities and real-world requirements, with the best model achieving only 34.89% overall accuracy, discouraging premature deployment and encouraging further research on reliability. A.4

License

The S PREADSHEET B ENCH 2 dataset is released under the CC BY-SA 4.0 license, which permits sharing and adaptation, including commercial use, with appropriate credit and under the same license. The evaluation code is released under the MIT License. All third-party assets used in this work are properly cited and their respective licenses are respected.

B

Details of Data

B.1

Data Sources

The financial data underlying S PREADSHEET B ENCH 2 is drawn from a diverse set of publicly available sources. These include primary regulatory filings from Bseindia 7 , Bloomberg 8 , Screener.in 9 , and the NYU Stern (Damodaran) dataset 10 . For sources with redistribution restrictions, we release only derived task metadata and evaluation artifacts when permitted, while separately documenting access requirements and licensing constraints. Together, these sources provide broad coverage of 7 https://www.bseindia.com/corporates/ann 8 https://www.bloomberg.com/ 9 https://www.screener.in/ 10 https://pages.stern.nyu.edu/~adamodar/New_Home_Page/data.html

14

real-world business data across multiple geographies and industry sectors, ensuring the diversity and authenticity of the benchmark tasks. All source materials are publicly available at the time of collection, and no proprietary or restricted-access data is used in the construction of the benchmark. B.2

Annotation Effort

The construction process required substantial expert time. Across the four task categories, the total annotation effort exceeded 1,500 hours: Financial Modeling tasks required around 400 hours, Debugging tasks around 500 hours, Visualization tasks around 380 hours, and Template tasks around 200 hours. These figures do not include additional quality assurance rounds, such as having selected tasks performed by human solvers to verify solvability, conducted both internally and by our data provider partners. B.3

Visualization Task Rubric Examples

We evaluate visualization tasks using a VLM-as-a-judge approach with hand-crafted rubrics that assess two complementary aspects: Format Compliance (chart type, axis configuration, color scheme) and Data Correctness (value ranges, data labels). Table 5 shows the complete rubric for a football field valuation chart (Task 129), illustrating the granularity of our per-criterion evaluation. Table 5: Rubric example for Task 129: Football Field Valuation Chart. Each criterion is independently verified by the VLM judge. #

Criterion

Category

1

The chart is a horizontal bar chart (football field valuation chart) showing valuation ranges The chart title reads “Project Mars – Standalone Valuation Summary” The methodologies appear top to bottom in this order: DCF (Based on Management Forecast), Precedent Transactions, Mgmt Projections 2026E, Mgmt Projections 2027E, Analyst Target Price Range, 52-Week Trading Range The X-axis starts at $15.00 The X-axis ends at $65.00 The X-axis displays gridlines at $5.00 intervals The Analyst Target Price Range bar is displayed in light blue The 52-Week Trading Range bar is displayed in light blue The DCF (Based on Management Forecast) bar is displayed in dark blue The Precedent Transactions bar is displayed in dark blue The Mgmt Projections 2026E bar is displayed in dark blue The Mgmt Projections 2027E bar is displayed in dark blue

Format Compliance

2 3

4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

The DCF bar starts at 41.89 and ends at 51.56 on the X-axis The data labels on the DCF bar read “41.89” and “51.56” The Precedent Transactions bar starts at 38.53 and ends at 46.93 The data labels on the Precedent Transactions bar read “38.53” and “46.93” The Mgmt Projections 2026E bar starts at 20.9 and ends at 37.99 The data labels on the Mgmt Projections 2026E bar read “20.9” and “37.99” The Mgmt Projections 2027E bar starts at 36.25 and ends at 58.69 The data labels on the Mgmt Projections 2027E bar read “36.25” and “58.69” The Analyst Target Price Range bar starts at 33.72 and ends at 46.48 The data labels on the Analyst Target Price Range bar read “33.72” and “46.48” The 52-Week Trading Range bar starts at 38.65 and ends at 44.4 The data labels on the 52-Week Trading Range bar read “38.65” and “44.4”

15

Format Compliance Format Compliance

Format Compliance Format Compliance Format Compliance Format Compliance Format Compliance Format Compliance Format Compliance Format Compliance Format Compliance Data Correctness Data Correctness Data Correctness Data Correctness Data Correctness Data Correctness Data Correctness Data Correctness Data Correctness Data Correctness Data Correctness Data Correctness

B.4

Debugging Task Types

Table 6 presents the taxonomy of spreadsheet error types used in the Debugging task. These ten categories were identified through iterative annotation of real-world spreadsheet errors encountered in financial modeling and accounting practice. The taxonomy spans a range of failure modes, from low-level syntactic issues such as formula errors and incorrect sign assignments, to higher-level structural problems including cross-sheet reference failures and unit mismatches. Each debugging instance contains one or more injected errors drawn from this taxonomy, and the agent is required to localize and correct all errors to restore the spreadsheet to its intended state. Table 6: Taxonomy of spreadsheet error types in the Debugging task. Error Type

Definition

Double Counting

A value is aggregated multiple times within a computation, resulting in systematic overestimation.

Embedded Hardcodes

Constant numeric values are hardcoded into formulas instead of being referenced from input cells, limiting correctness and adaptability.

Errors

Formulas that result in error messages such as #REF!, #NUM!, or similar, indicating invalid references or computations leading to unintended outputs. Formatting rules are inconsistently applied across semantically equivalent cells, reducing interpretability. Averages are computed over incorrect ranges, leading to biased or misrepresented summary statistics.

Inconsistent Color Incorrect Average Cross-Sheet References

Inter-sheet references are missing, broken, or incorrectly specified, causing dependency failures in multi-sheet models.

Incorrect index-match

Lookup functions are misconfigured due to incorrect range alignment, match settings, or indexing logic. Numerical signs are incorrectly assigned, reversing the intended meaning of computed values. Improper use of relative and absolute cell references causes formulas to break when copied or propagated.

Incorrect Sign Relative vs. Absolute References Unit Mismatch

Inconsistent or incompatible units are combined within computations, resulting in scale or magnitude errors.

C

Details of Experiments

C.1

Environment.

Each task instance is executed inside a sandboxed Docker container built on Python 3.11 (Debian Bullseye). The container is pre-installed with common data-science libraries (NumPy, Pandas, Matplotlib, openpyxl) and LibreOffice with its Python UNO bridge, enabling both programmatic and application-level spreadsheet manipulation. No network access is provided during execution, ensuring that the agent must rely solely on its own reasoning and the provided tools. C.2

Tools.

We conduct our experiments within the SWE-Agent framework. To rigorously evaluate the intrinsic capabilities of the model, we restrict the available toolset to three fundamental utilities: bash, view_xlsx, and submit. • bash: Executes arbitrary shell commands, providing the agent with general-purpose programmatic access to the operating system. Common usage includes file operations (ls, cat, cp, mv), running Python scripts via python3 for data manipulation and formula computation, and invoking standard Unix utilities for text processing (grep, awk, sed). • view_xlsx: A read-only utility for inspecting .xlsx spreadsheet files. The agent must specify one of two modes: list, which enumerates all sheet names in the workbook, or content, which 16

displays the contents of a specific sheet. The content mode optionally accepts a row range to limit the output (e.g., rows 1–50), and exposes both the original formulas and their evaluated values. • submit: Finalizes the agent’s solution and submits the modified spreadsheet for evaluation. The agent is expected to invoke this tool only after all edits have been completed and verified. C.3

Prompt.

The agent receives a two-part prompt at each episode. The system prompt (Figure 7) establishes the agent’s role and interaction protocol, while the instance prompt supplies the task-specific context, including available tools, a recommended workflow, and the concrete instructions with input/output file paths. We use two instance prompt variants: one for Template, Financial Modeling, and Debugging tasks (Figure 8), and a separate one for Visualization tasks (Figure 9) that additionally specifies the visualization-oriented objective and restricts the library set. Template variables (shown in angle brackets) are populated per task instance. System Prompt You are a helpful Spreadsheet Automation Engineer that interacts with a computer shell to solve data tasks. You operate in a REPL (Read-Eval-Print Loop) environment where you must issue exactly ONE command at a time.

Figure 7: System prompt used for all task instances. Instance Prompt Important • When completing spreadsheet tasks, strictly avoid altering any cells that already contain values unless explicitly instructed. Modify only the cells that are required for the task. • You need to complete the instructions and ensure that the original formatting is preserved as much as possible. Tools You are provided with three tools: bash, view_xlsx and submit. You must use these tools to complete the target task. You can only call ONE tool at a time per response. Environment You can complete the corresponding spreadsheet tasks using the following Python environment: Pandas, openpyxl, NumPy, and LibreOffice. Recommended Workflow 1. Inspect: Use view_xlsx to understand the structure and contents of the spreadsheet. 2. Plan: Decide how to manipulate the data using openpyxl or LibreOffice. 3. Implement: Create a Python script that performs the edit. 4. Execute: Run the script via python3. 5. Verify: Check the output file exists and ensure correctness. Do not combine verify and submit commands. 6. Submit: When verification is successful, run submit. Task Instructions You need to process a spreadsheet file based on specific instructions. Instruction: ⟨instruction⟩ Input File: ⟨spreadsheet_path⟩ Output Path: ⟨output_path⟩

Figure 8: Instance prompt template for Template, Financial Modeling, and Debugging tasks. Instance Prompt (Visualization) Important • When completing spreadsheet tasks, strictly avoid altering any cells that already contain values unless explicitly instructed. Modify only the cells that are required for the task. • You need to complete the instructions and ensure that the original formatting is preserved as much as possible. Tools You are provided with three tools: bash, view_xlsx and submit. You must use these tools to complete the target task.

17

Environment You can complete spreadsheet visualization tasks using the following Python environment: openpyxl and xlsxwriter. Your task scope only includes visualization-related operations, specifically chart creation and pivot table construction. Task Objective Your goal is to create clear, professional, and interpretable visualizations from the given spreadsheet. This includes: selecting appropriate chart or pivot table types based on the data semantics; ensuring the visualization accurately reflects the underlying data; and producing outputs that would be considered presentation-ready in a professional or analytical context. Recommended Workflow 1. Inspect: Use view_xlsx to understand the structure and contents of the spreadsheet. 2. Plan: Determine whether the task requires a chart or a pivot table, choose the visualization type, map data fields to visual elements, and consider design choices that improve clarity. 3. Implement: Create a Python script that performs the visualization. 4. Execute: Run the script via python3. 5. Verify: Check the output file exists and ensure correctness. Do not combine verify and submit commands. 6. Submit: When verification is successful, run submit. Task Instructions You need to process a spreadsheet file based on specific instructions. Instruction: ⟨instruction⟩ Input File: ⟨spreadsheet_path⟩ Output Path: ⟨output_path⟩

Figure 9: Instance prompt template for Visualization tasks. &ODXGH2SXV

3HUFHQWDJHRIHUURUKLWWDVNV

7HPSODWH 



)LQDQFLDO0RGHO

(UURUUDWH2SXV0LQL0D[





'HEXJJLQJ

(UURUUDWH2SXV0LQL0D[



(UURUUDWH2SXV0LQL0D[

 



 

  

0LQL0D[0

)L[HG  HUURU

 

3DVVHG HQGWRHQG







(QFRXQWHUHG HUURU







)L[HG  HUURU

(QFRXQWHUHG HUURU

3DVVHG HQGWRHQG

(QFRXQWHUHG HUURU

)L[HG  HUURU

3DVVHG HQGWRHQG

Figure 10: Error-recovery analysis on Template, Financial Model, and Debugging tasks. For each task where the agent encounters at least one error, we track three stages: whether an error occurred (baseline), whether the agent fixed at least one error (per-error fix rate), and whether the task still passed end-to-end evaluation (end-to-end pass rate). MiniMax can fix individual errors but rarely produces a correct final output for error-affected tasks.

D

Additional Experimental Analysis

D.1

Error-Recovery Analysis

When an agent makes a mistake during execution (e.g., a runtime exception or a wrong intermediate result), can it fix the mistake and still finish the task correctly? Figure 10 answers this question by tracking three stages for every task that contains at least one error. The x-axis shows: 1. Encountered error: the starting point (100%), representing all tasks where the agent hit at least one error. 2. Fixed ≥1 error: of those tasks, how many did the agent manage to fix at least one error in. 3. Passed end-to-end: of those tasks, how many still produced a fully correct final output. 18

The key finding is the drop from stage 2 to stage 3. MiniMax M2.5 can fix individual errors reasonably often (72% on Template, 47% on Financial Model, 34% on Debugging), but its final pass rate collapses (to 7%, 9%, and 3%). In other words, MiniMax patches the immediate problem, but the rest of its solution is already too far off track to recover. Opus 4.6 shows a much smaller drop—for instance, 50% → 45% on Financial Model—meaning that when Opus fixes an error, it usually still gets the final answer right. Each panel also notes the error rate: the share of all tasks in that category where at least one error occurred. MiniMax hits errors in 62–88% of tasks, compared to 16–67% for Opus, which means MiniMax not only recovers worse but also needs to recover far more often. D.2

Fine-Grained Evaluation of Visualization Tasks

As described in Section 2.4, Visualization tasks are scored using a vision-language model (VLM) against a hand-crafted rubric. Each rubric criterion falls into one of two dimensions: • Data Correctness: whether the generated chart accurately reflects the underlying spreadsheet data (e.g., correct data series, values, and axis ranges). • Format Compliance: whether the chart meets formatting specifications such as chart type, axis labels, legend placement, color assignments, and layout. Figure 11 reports per-model scores on these two dimensions. A consistent pattern emerges: every model scores higher on Format Compliance than on Data Correctness. This means current models are relatively good at following explicit style instructions (e.g., “use a bar chart with blue bars”) but weaker at understanding which data to extract from complex spreadsheets and how to map it onto the chart. The gap between the two dimensions is smallest for Claude Opus 4.6 (82.0 vs. 72.7) and largest for MiniMax M2.5 (64.7 vs. 31.4), showing that stronger models pull ahead primarily by improving data handling, not formatting. Claude Opus 4.6 achieves the highest Overall score (76.2), followed by GPT-5.2 (70.4) and Gemini 3.1 Pro (69.0). Notably, Gemini 3.1 Pro has the highest Format Compliance (83.3) of any model, but its lower Data Correctness (60.7) limits its Overall ranking. At the other end, MiniMax M2.5 trails on both dimensions, with a Data Correctness of only 31.4—less than half that of the top model—confirming that extracting and correctly plotting data remains the primary bottleneck for weaker models. D.3

Failure Taxonomy

To systematically understand why agents fail, we manually inspected a sample of unsuccessful trajectories from Claude Opus 4.6 and developed a taxonomy of six recurring failure modes, summarized in Table 7. The categories are designed to be mutually exclusive at the primary-cause level: each failed trajectory is assigned to the single category that best describes its root cause. Task Misunderstanding and Insufficient Inspection capture failures in the comprehension stage, where the agent either misinterprets the goal or skips critical context in the spreadsheet. Wrong Target Selection and Format/Output Error capture execution-stage failures, where the agent understands the task but applies changes to incorrect locations or produces malformed output. Turn Limit Exceeded reflects planning inefficiency, and Other subsumes rare or ambiguous cases that do not fit the preceding categories. As discussed in Section 3, the dominant failure modes are Insufficient Inspection and Wrong Target Selection, indicating that accurately locating and understanding target cells within complex spreadsheet structures remains a core challenge. D.4

Failure Cases

To complement the quantitative failure taxonomy in Table 7, we present six representative failure cases drawn from Claude Opus 4.6’s failed trajectories—two per task category (Debugging, Financial Modeling, Visualization). These cases illustrate common reasoning breakdowns that lead to partial or complete task failure. 19

76.2

Claude Opus 4.6

82.0

72.7 70.4

GPT-5.2

65.6

78.6

69.0

Gemini 3.1 Pro

83.3

60.7 65.1

GLM-5

80.9

55.5 60.1

Kimi K2.5

76.2

50.4 58.2

Deepseek V3.2

69.8

51.4 55.0

Qwen3.5-397B-A17B

67.9

47.2 44.0

MiniMax M2.5

64.7

31.4

0

20

40

60

Data Correctness Format Compliance Overall

80

Score (%)

Figure 11: Rubric-based evaluation scores for Visualization tasks across two dimensions: Data Correctness (whether the chart accurately reflects the spreadsheet data) and Format Compliance (adherence to chart type, axis labels, colors, and layout specifications). The Overall score is a weighted combination of both. Models are sorted by Overall score. Table 7: Descriptions of failure mode categories. Category

Definition

Task Misunderstanding

The agent misinterprets the user’s instructions and pursues an entirely incorrect objective.

Insufficient Inspection

The agent fails to sufficiently explore the spreadsheet prior to making edits, thereby missing critical context. The agent correctly identifies the required action but applies it to the incorrect cells or worksheets.

Wrong Target Selection Turn Limit Exceeded

The agent fails to complete the task before exceeding the maximum permitted number of dialogue turns.

Format/Output Error

The agent derives the correct values but applies an incorrect format, or otherwise fails to generate a valid output file.

Other

There was some other problem that prevented the agent from resolving this issue.

D.4.1

Debugging Cases

Case 1. As shown in Figure 12, the agent correctly identified all three double-counting bugs through cross-sheet formula tracing, yet only fixed two of them. The unfixed bug—two balance sheet line items referencing identical source cells—was structurally unambiguous, but the agent grouped it with genuinely ambiguous patterns and chose not to act on it. This shows that correct diagnosis does not guarantee complete repair: the agent applied an overly conservative filter that treated a clear structural error as uncertain, leaving a known bug unresolved at submission time. Case 2. As shown in Figure 13, the agent correctly recognized that the task objective was to fix color-coding inconsistencies, but gradually drifted into a general spreadsheet audit. It applied formula corrections that were never requested while skipping the required color-only fixes. Worse, for cell C90, the agent changed the cell value from −0.3 to 0.3 instead of merely recoloring the font—confusing a formatting inconsistency with a data error and introducing a destructive modification. This case illustrates a failure to stay within the task’s specified scope: the agent expanded its edits beyond what the instruction asked for, and the out-of-scope changes were not only unnecessary but harmful.

20

Case 1: Double Counting (Debugging) Task Description Audit and fix a 13-sheet LBO/DCF model for Monro, Inc. (NASDAQ: MNRO). The task requires finding and fixing double-counting bugs—components counted multiple times in aggregate formulas across interconnected sheets (Control, Summary, Assumptions, Income Statement, Balance Sheet, Cash Flow Statement, etc.). Agent Reasoning Over 17 inspection steps, the agent correctly identified three bugs: Bug A (Balance Sheet Row 46): Total Liabilities double-counts Current Portion of LT Debt. Bug B (Cash Flow Row 39): Net Cash Flow double-counts Net Income. Bug C (Balance Sheet Rows 31 & 39): Both “Curr. Port. Of Cap. Leases” and “Capital Leases” reference identical cells from the PP&E sheet, causing the same balance to appear in both current and long-term liabilities. Key Decision Point (Step 18: fix script creation) The agent fixed Bugs A and B correctly but explicitly deprioritized Bug C: “I’ll focus on fixing the clear double-counting issues first and leave these averaging patterns as-is unless they’re definitively wrong.” This conflated an unambiguous bug—two distinct balance sheet line items pulling from the exact same cell—with genuinely ambiguous patterns (averaging inconsistencies, rolling windows). Outcome Modification accuracy: Unfixed Bug C propagation: Failure mode:

0.8974 Cell H48 — gold: 1094.14, agent: 1259.34 Insufficient Inspection

Figure 12: Insufficient inspection case of Debugging task. Case 2: Inconsistent Color Coding (Debugging) Task Description Fix font color inconsistencies in a 13-sheet Atlassian DCF model. Convention: green = cross-sheet links, blue = hardcoded inputs, black = formulas/calculations. Identify cells where font color does not match content type and correct the color coding. Agent Reasoning The agent spent 17 steps inspecting sheets and 4 steps analyzing colors via Python scripts. It correctly noted the color convention but then overrode its own understanding: “I should focus on fixing those issues. . . I’ll be conservative and address only the clear violations.” The agent applied 13 fixes—mostly formula/logic corrections that were not required—while systematically skipping the actual color-only fixes. Key Decision Point (Steps 18–20: analysis phase) The cells the agent decided were “not clear violations” (D41/D47/D53, C62:H62, C90:H90) are precisely the cells the gold answer requires. Worse, for cell C90, the agent changed the value from −0.3 to 0.3 instead of merely recoloring the font to blue—a destructive error. Outcome Modification accuracy: Destructive change: Failure mode:

0.0 C90 value altered (−0.3 → 0.3) instead of font color change Task Misunderstanding

Figure 13: Task misunderstanding case of Debugging task. D.4.2

Financial Modeling Cases

Case 3. As shown in Figure 14, the agent constructed the Debt/Equity ratio formula by copying the numerator pattern from an adjacent Debt/Total Capitalization formula, without checking whether 21

the two ratios use the same definition of “Debt.” The adjacent formula sums both long-term and short-term borrowings, but Debt/Equity conventionally uses only long-term borrowings. The agent provided detailed reasoning for more complex subtasks (e.g., WACC calculation) but treated this formula as a trivial copy operation with no explicit reasoning trace. This shows that the agent relied on surface-level pattern matching from a neighboring cell rather than verifying the formula against the financial definitions used elsewhere in the workbook. Case 4. As shown in Figure 15, the agent generated all formulas from abstract financial knowledge (e.g., “Book Value per Share = Total Equity / Shares Outstanding”) without verifying whether the referenced rows and columns match the workbook’s actual layout. As a result, nearly every formula was conceptually reasonable but structurally wrong for this particular banking model, yielding only ∼4% cell accuracy. Additionally, by filling previously-empty cells that other formulas depended on being zero, the agent broke pre-existing formulas and caused cascading errors. This case represents a more severe version of the pattern in Case 3: instead of copying from a nearby cell, the agent bypassed the workbook entirely and relied on general domain knowledge to construct formulas, without grounding them in the spreadsheet’s actual structure. Case 3: Debt/Equity Ratio (Financial Modeling) Task Description Complete a Procter & Gamble financial model (9 sheets). Among 8 subtasks: calculate Debt/Equity ratio for all periods in the Ratio_Analysis sheet (Row 25). Agent Reasoning The agent observed an adjacent formula for Debt/Total Capitalization (Row 26): =(F95+F90)/(F87+F95+F90) (sums long-term + short-term borrowings) It copied this numerator pattern directly into the Debt/Equity formula without any explicit reasoning—in contrast to the detailed reasoning it articulated for WACC. Key Decision Point (Step 12: script creation) Agent formula: =(F95+F90)/F103 vs. Gold formula: =F90/F103 The agent included short-term borrowings (Row 95) in the numerator by analogy with the adjacent ratio, but Debt/Equity conventionally uses only long-term borrowings. Outcome Modification accuracy: Output vs. gold: Failure mode:

0.8824 Cell L37: 0.4591 vs. 0.2265 (exactly ∼2×, confirming doubled numerator) Wrong Target Selection

Figure 14: Wrong target selection case of Financial Modeling task.

22

Case 4: Banking Model (Financial Modeling) Task Description Complete a multi-sheet banking model (“Project Banking”) by filling six groups of empty cells across Balance Sheet (Check row, Book Value per Share, Shareholders’ Equity w/o dividends), Macro sheet (Deposits-to-GDP, Competition 6 Market Share), and Assumptions-BS (Total Funding Growth). Agent Reasoning The agent derived formulas from general financial knowledge rather than examining how similar rows in the same workbook compute their values: “Check row: Total Assets − Total Liabilities & Owner’s Equity. . . Book Value per Share: Total Equity divided by Shares outstanding, so row 26 / row 32.” Key Decision Point (Step 10: script creation) All six formula groups were generated from abstract financial definitions without verifying against the workbook’s local conventions. The formulas were “textbook correct” but the specific cell references (row numbers, column ranges) did not match this particular banking model’s structure. Outcome Modification accuracy: Collateral damage: Failure mode:

0.0411 (≈3 / 73 cells correct) Cell E29 returned #VALUE! (expected: 0.0236) Insufficient Inspection + Wrong Target Selection

Figure 15: Insufficient inspection and wrong target selection case of Financial Modeling task. D.4.3

Visualization Cases

Case 5. As shown in Figure 16, the agent encountered a genuine tool limitation—openpyxl does not natively support clustered-stacked charts—and chose an elaborate dual-axis workaround that produced overlapping rather than clustered bars. The agent itself anticipated this risk (“the bars could overlap”) but then dismissed it with incorrect reasoning. More critically, all three verification attempts returned negative or empty results, yet the agent stated that “the chart XML confirms all 4 series are present”—directly contradicting its own tool output. This case shows two compounding problems: the agent selected a workaround it could not verify, and when verification failed, it fabricated a positive result rather than reconsidering its approach. Case 6. As shown in Figure 17, the agent correctly determined that no available Python library supports sunburst charts and explicitly considered two viable alternatives: matplotlib embedding and a nested doughnut chart approximation. It rejected both—reasoning that “the task specifically asks for a sunburst, so I need to build the real thing”—and instead attempted raw XML injection into the XLSX archive using the Office 2016+ Extended Chart namespace. The evaluation system reported “no chart objects,” resulting in a complete failure. A nested DoughnutChart, which openpyxl fully supports, would have visually approximated the sunburst hierarchy and likely earned partial credit. This case shows that the agent prioritized literal compliance with the task specification over a pragmatic approximation that its tools could actually produce and verify, ultimately achieving a worse outcome than the alternative it rejected.

23

Case 5: Clustered-Stacked Chart (Visualization) Task Description Create a stacked column chart comparing sales for Type A and Type B across Q1–Q4 for 2022 and 2023. Columns should be grouped by year (clustered), with types stacked within each column. Required chart title: “Sales Analysis.” Agent Reasoning The agent correctly identified that openpyxl does not natively support clustered-stacked charts and chose a dual-axis workaround: two separate BarChart objects combined via chart1 += chart2. It even anticipated the risk: “The bars could overlap rather than cluster properly. . . Actually, the dual-axis approach does work for clustered stacked charts in Excel.” Key Decision Point (Steps 6–7: verification hallucination) Three verification attempts all failed: view_xlsx only shows cell data; the openpyxl API only exposes the primary chart’s series; an XML regex for <c:barChart> returned 0 matches (namespace mismatch). Despite this, the agent stated: “The chart XML confirms all 4 series are present. . . creating the clustered stacked effect.” — a direct hallucination contradicting the actual tool output. Outcome VLM score: Key failures: Failure mode:

0.1304 (3 / 23 criteria passed) Wrong title, columns overlap instead of clustering, data values incorrect Format/Output Error

Figure 16: Format/Output error case of Visualization task. Case 6: Sunburst Chart (Visualization) Task Description Create a sunburst chart titled “Expense Breakdown” showing a 3-level hierarchy (department → category → subcategory) of expense data, sized by spend amount. Required colors: Orange for IT, Blue for HR, Grey for Sales. All labels must be fully contained within their segments. Agent Reasoning The agent probed available tools and correctly determined that neither openpyxl nor xlsxwriter supports sunburst charts. It explicitly considered two viable alternatives: • Matplotlib image embedding — rejected (“no built-in sunburst support”). • Nested DoughnutChart approximation — rejected (“the task specifically asks for a sunburst, so I need to build the real thing”). Key Decision Point (Steps 5–6: rejecting viable alternatives) The agent spent 4 steps building a raw XML injection pipeline: extracting the XLSX as a ZIP, creating chartEx1.xml with the Office 2016+ Extended Chart namespace (cx:chart), adding relationship files and content type registrations, and re-zipping. Verification confirmed ZIP integrity and file presence but never tested whether the chart was recognizable as a chart object. Outcome Evaluation result: Missed opportunity: Failure mode:

“XLSX has no chart objects to export” — complete failure A nested DoughnutChart (openpyxl-supported) could have earned partial credit Format/Output Error

Figure 17: Format/Output error case of Visualization task.

24

E

Data Examples

Figure 18: Example of Debugging tasks.

25

Figure 19: Example of Financial Modeling tasks.

26

Instruction Complete the financial model based on the provided assumptions. Ensure that all formulas, references, and calculations are correct.

Input Spreadsheet

Ground Truth Spreadsheet

Figure 20: Example of Template tasks.

27

Instruction Create a football field valuation chart titled 'Project Mars - Standalone Valuation Summary'. Display horizontal bars showing valuation ranges for each methodology. List methodologies vertically on the Y-axis in this order from top to bottom: DCF (Based on Management Forecast), Precedent Transactions (10.7x - 13.0x Management 2025A EBITDA), Mgmt Projections 2026E (Based on 9.0x - 15.0x Management Projection), Mgmt Projections 2027E (Based on 8.5x - 14.5x Management Projection), Analyst Target Price Range, 52-Week Trading Range. Use light blue bars for Analyst Target Price Range and 52-Week Trading Range, and dark blue bars for all other methodologies. The X-axis should show share price from $15.00 to $65.00 with gridlines every $5.00. Add data labels on each bar showing the low and high values of the range

Input Spreadsheet

Ground Truth Spreadsheet Project Mars Standalone Valuation Summary DCF…

$41.89

Precedent Transactions…

$38.53

Mgmt Projections 2026E… $20.90

$37.99

Mgmt Projections 2027E… Analyst Target Price Range

$51.56 $46.93

$36.25 $33.72

52-Week Trading Range

$58.69 $46.48

$38.65 $44.40

$15.00 $20.00 $25.00 $30.00 $35.00 $40.00 $45.00 $50.00 $55.00 $60.00 $65.00

Figure 21: Example of Visualization tasks.

28

Related documents

• MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents#652393
• MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents#650940
• MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents#363286
• LawCompass: Navigating from Legal QA to Multi-Agent Deep Research with Grounded Evidence#1155962
• SheetCompass: Hierarchical Relation Graphs for Agentic Spreadsheet Reasoning#609618