FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows

FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows Bo Deng1,2∗ , Kang Zhou2 , Lifan Guo2 , Chongyang Tao1† , Xuanren Chen1 , Chenggang Xie1 , Renzhao Liang1 , Feng Chen2 , Chi Zhang2 1

2

Beihang University Qwen DianJin Team, Alibaba Cloud Computing

arXiv:2608.06144v1 [cs.AI] 6 Aug 2026

Abstract Most agent benchmarks evaluate tasks independently and cannot measure whether experience from one task helps with later tasks. Existing self-evolution benchmarks do not jointly cover professional workflows, open-ended deliverables, and multiaspect evaluation. We introduce FinEvo-Bench, a longitudinal benchmark with 120 real-case-grounded tasks, 20 business scenes across six financial domains. Institution-provided professional procedures define the required operations and constraints. Eligible institution-provided and publicly documented cases supply the task facts. Each scene contains six related but substantively distinct cases that share a professional procedure and a manually reviewed rubric for task quality and financial compliance. We compare four self-evolving agent scaffolds using the same Qwen3.7-Max backbone and three independently shuffled, globally interleaved task streams. Paired non-evolving controls estimate each scaffold’s self-evolution gain from retained experience, while an independent Claude Code scoring agent backed by Claude Opus 4.6 evaluates all outputs. Letta achieves the highest evolved score (91.65) and fewest compliance issues (0.09 per task); Codex achieves the largest self-evolution gain (+19.37). Across scaffolds, the evolving condition raises scores by 9.33–19.37 points and reduces compliance issues by 0.12–0.44 per task. Paired score gains at within-scene ranks 4–6 exceed those at ranks 1–3 by 6.10–8.70 points. In Claude Code, skill-only evolution produces higher task quality and fewer compliance issues than memory-only and combined memory–skill evolution. Across all four scaffolds, rubric feedback also yields higher scores and fewer compliance issues than reference-answer feedback. FinEvo-Bench measures both professional performance and self-evolution ability: how effectively an agent turns prior experience into later improvement.

1

Introduction

Self-evolving agents retain completed interactions as memories, skills, or other persistent state for use in later work (Shinn et al. 2023; Zhao et al. 2024; Wang et al. 2023; Zhang et al. 2026). Their evaluation must test whether experience from one task becomes reusable knowledge for subsequent tasks, not simply whether the agent completes the task at hand. Most agent benchmarks evaluate each task independently. They measure current execution capability but do not ∗ This work was conducted during an internship at Alibaba Cloud Computing. † Corresponding author.

Benchmark

OpenMultiCrossaspect task Domain ended Evolution Workflow Artifact Evaluation

LifelongAgent Bench SEA-Eval Evo-Memory StuLife Frontier-Eng

✓

FinEvo-Bench (Ours)

✓

✓ ✓ ✓

✓ ✓

✓

✓ ✓ ✓ ✓

✓

✓

✓

Table 1: Representative self-evolution benchmarks. observe how experience accumulates or transfers. Recent self-evolution benchmarks use sequential task streams, situated environments, or iterative artifact optimization to study this process (Zheng et al. 2025; Jiang et al. 2026; Wei et al. 2025; Cai et al. 2025; Chi et al. 2026). Table 1 compares representative benchmarks; Section 2 covers the broader literature. Here, a domain workflow groups tasks under a shared professional process, an openended artifact admits multiple valid outputs, and multi-aspect evaluation goes beyond terminal success. LifelongAgentBench, SEA-Eval, and Evo-Memory test ordered-task reuse; StuLife studies persistent situated adaptation; Frontier-Eng measures iterative artifact improvement. Prior work covers these properties separately but does not combine them in recurring professional workflows. Financial cases share procedures, analytical checks, and compliance requirements while differing in their files, risks, and conclusions. Credit reviews, claim analyses, insurance advice, and investment research all recur across clients or institutions: lessons about evidence checking, analytical sequencing, and professional boundaries can transfer, but every new case still requires case-specific calculations and judgment. Financial workflows therefore test whether an agent can reuse professional procedures without copying case-specific solutions. Recent financial benchmarks evaluate professional artifacts (Jin et al. 2026; Kundurthy et al. 2026), but score cases independently rather than testing whether retained experience improves later related work. Final artifact quality, domain-specific compliance, and im-

provement from retained experience answer different questions. A scaffold may achieve a high final score because its backbone is already strong, or show a large gain while remaining weaker in absolute terms. We report evolved performance and paired within-scaffold gains separately: the former measures final capability; the latter, self-evolution ability. FinEvo-Bench addresses this setting. We define selfevolution as a cross-task process: an agent extracts signals from completed tasks and feedback, updates its persistent state, and applies that state to later tasks. FinEvo-Bench contains 20 business scenes across six financial domains, with six multi-file tasks per scene and 120 tasks in total. Tasks within a scene are related but substantively distinct and are grounded in eligible institution-provided and publicly documented cases. Before release, direct identifiers and sensitive fields are removed or replaced when necessary. Reference professional procedures validated in practice define the required inputs, main steps, expected deliverables, and professional constraints. Two domain experts construct each scenelevel rubric and review it against all six tasks. On the 120 deliverables from one complete main-experiment run, the automated rubric judge achieves high absolute agreement with a financial expert (ICC(A, 1) = 0.95; 95% CI: [0.93, 0.97]). The evaluation uses three independently shuffled, globally interleaved streams rather than contiguous scene blocks, requiring each scaffold to retain and retrieve relevant experience amid unrelated intervening tasks. Each evolving run is paired with a state-reset control, allowing us to estimate its self-evolution gain from retained experience. All four scaffolds outperform their state-reset controls on the same three shuffled streams: scores increase by 9.33– 19.37 points and compliance issues decrease by 0.12–0.44 per task. Letta has the highest evolved score (91.65), while Codex has the largest self-evolution gain (+19.37). For every scaffold, mean gains over within-scene ranks 4–6 exceed those over ranks 1–3 by 6.10–8.70 points. Additional diagnostics show that skill-only evolution performs best among Claude Code’s experience carriers, reaching 93.71 points and 0.05 compliance issues per task. Rubric feedback exceeds reference-answer feedback by 3.95–7.93 score points and reduces compliance issues for all scaffolds. Every measured capability dimension improves, with the largest gains in report quality. Cross-scene score differences on a fivescene sample remain small and mixed in direction, while scene isolation triggers 0.01–0.05 more compliance issues per task. Our contributions are: • FinEvo-Bench provides 120 real-case-grounded, multifile tasks with open-ended outputs scored for quality and financial compliance. • A paired protocol uses interleaved streams and state-reset controls to measure cross-task self-evolution gains. • Shared-backbone experiments compare four scaffolds, experience carriers, feedback forms, capability dimensions, and cross-scene execution.

2

Related Work

Self-evolution. Self-evolving agents use task outcomes to improve later behavior. They may retain reflections or memories (Shinn et al. 2023; Zhao et al. 2024), compile reusable skills or playbooks (Wang et al. 2023; Zhang et al. 2026), or adapt workflows and model policies (Zhang et al. 2024, 2025; Wang et al. 2025). FinEvo-Bench does not assume a particular mechanism. It compares frameworks with different persistent representations and uses paired non-evolving controls to isolate gains attributable to retained experience. Longitudinal evaluation of evolving agents. Conventional web, computer, and software-engineering benchmarks evaluate tasks independently and do not preserve task-derived experience across episodes (Zhou et al. 2024; Xie et al. 2024b; Jimenez et al. 2024). Recent benchmarks instead expose task sequences or persistent memory so that adaptation beyond isolated episodes can be assessed. LifelongAgentBench constructs skill-dependent interactive tasks (Zheng et al. 2025), while SEA-Eval evaluates correlated and orthogonal streams through success and token trajectories (Jiang et al. 2026). Evo-Memory and EvoMemBench organize their protocols around memory retrieval, update, and reuse; SEAGym separates update evidence from held-out transfer and replay assessment (Wei et al. 2025; Wang et al. 2026; Zheng et al. 2026). Table 1 summarizes the benchmarks closest to our design; EvoMemBench and SEAGym provide complementary memory- and transfer-centered protocols. FinEvo-Bench combines recurring real-case-grounded financial workflows, open-ended professional deliverables, and expert-derived rubric evaluation within a globally interleaved stream. Financial benchmarks. Financial benchmarks cover numerical and document reasoning (Chen et al. 2021; Zhu et al. 2021; Chen et al. 2022; Islam et al. 2023), broad capability suites (Xie et al. 2023, 2024a), and professional analysis, research, tool-use, or spreadsheet tasks (BenYoash et al. 2025; Bigeard et al. 2025; Choi et al. 2025; Jin et al. 2026; Zhu et al. 2026; Kundurthy et al. 2026). FinRpt’s multidimensional evaluation and BlueFin’s granular criteria accommodate professional artifacts with multiple valid forms, but their instances are still scored independently. FinEvo-Bench interleaves related but distinct cases, isolates retained-experience gains, and separately reports quality and compliance.

3 3.1

FinEvo-Bench

Benchmark Scope and Design

FinEvo-Bench contains 20 business scenes across the six financial domains shown in Figure 1. Tasks cover record intake, calculation, cross-file verification, risk assessment, client advice, and report writing. Each scene contains six substantively distinct cases that share a professional procedure. The 120 tasks contain 775 input files, averaging 6.46 per task (range: 2–11), and request open-ended reports, assessments, or recommendations. Outputs may differ in form but must be supported by the supplied evidence and satisfy the relevant business and compliance

Figure 1: Overview of FinEvo-Bench. Left: 20 business scenes across six financial domains, comprising 120 tasks. Right: an example Stock Technical Analysis task with its natural-language request and six input files. requirements; scene-level rubrics evaluate these properties without requiring lexical similarity to a reference answer.

3.2

Scene Definition and Task Construction

We construct each scene s from three sources: an institutionprovided scene description Ds , a reference professional procedure Ps validated in practice, and a candidate pool Cs drawn from institution-provided and publicly documented cases. The scene description defines the business problem. The reference procedure specifies the required input files, main steps, expected deliverable, and professional constraints. Individual cases supply the facts, data, and judgment conditions. We review source permissions and release conditions before admitting a case to the eligible pool Cseligible . When necessary, we remove or replace direct identifiers and sensitive fields while preserving data relationships, numerical logic, chronology, and decision conditions. A domain expert consolidates these sources into a scene specification: Ss = Φ(Ds , Ps , Cseligible ) = (Os , Es , As , Ys , Ks ).

(1)

Here, Os is the business objective and scope; Es , the required inputs; As , the professional operations and checks; Ys , the expected deliverable; and Ks , the compliance requirements and professional boundaries. After defining a scene, experts select six cases from its eligible pool. The selection follows the scene’s business characteristics and professional procedure. Cases must differ substantively in their business situations, input files, analytical focus, judgment conditions, or conclusions; cases with only superficial differences are excluded from the same scene. Table 2 gives four examples. Each selected case becomes a task with a natural-language request, input files, task metadata, and a reference answer. Domain experts review the final tasks in two stages. First, they assess the scene and task settings. They then verify that the input files are complete and reflect actual business conditions, and that the reference answer has correct calculations,

Scene

Representative differences across the cases

Financial Statement Analysis Claim Payout Calculation

Healthy operations; cyclical downturn; divergence between earnings and cash flow; reporting red flags; multiple distress signals Standard calculation; substantial expense disallowance; ineligible hospitalization; deductible threshold; data anomaly; repeated claims reaching the coverage limit Single-Fund High-performing equity fund; persistently Diagnosis underperforming fund; index fund; pure bond fund; mixed bond fund Personalized Client-information update; risk reminder; Client Outreach product maturity and rollover; event invitation; routine review; outreach with incomplete client information

Table 2: Examples of six-case design for selected scenes. analysis, and conclusions. Appendix A gives a construction example with task files, reference answer, rubric, and review.

3.3

Scene-Level Rubrics and Quality Control

The six tasks in a scene share a 100-point rubric derived from the scene’s professional procedure. The rubric checks whether an agent uses the input files correctly, completes the necessary analysis, and reaches supported conclusions. It does not require the output to match the reference answer in structure or wording. The rubric also checks financial compliance, including fabricated or unsupported data and terms, definitive claims based on insufficient information, decisions beyond the agent’s professional role, and guarantees about credit approval, claim outcomes, or investment returns. Two domain experts construct each rubric. One drafts the criteria, point allocations, and grading rules. The other applies the rubric to all six tasks to check case applicability, coverage of required business steps, and consistency with the input files and refer-

ence answers. They resolve any omission or conflict before using the rubric to score the six tasks. Section 4.2 separately validates the automated application of these rubrics against financial-expert scoring.

4 4.1

Experiments

Experimental Setup

We evaluate four self-evolving agent scaffolds on the 120 tasks and 20 business scenes introduced in Section 3. All scaffolds follow the same longitudinal protocol. Figure 2 summarizes benchmark construction and this protocol. Self-Evolving Agent Scaffolds. The four scaffolds retain experience in different forms. Claude Code and Codex can distill task experience into reusable skills, project-scoped memory, and global memory. Letta maintains editable memory blocks for each agent. Its memory-management tools can insert, replace, or rewrite content, and every model call receives the full contents of all core memory blocks. GenericAgent distills each completed task into a reusable Markdown experience file. A prompt-resident title-only L0 index guides selective loading of full files (Liang et al. 2026). Longitudinal Evaluation Protocol. We independently shuffle the 120 tasks three times to form three globally interleaved streams. Each evolving run starts without benchmarkderived experience and processes tasks sequentially in stream order. Each task follows an execute–score-and-feedback– reflect-and-consolidate cycle (Shinn et al. 2023; Zhao et al. 2024; Zhang et al. 2026). After the scaffold produces an answer, a separate Claude Code scoring agent backed by Claude Opus 4.6 applies the scene rubric and returns the rubric feedback defined below. The scaffold reflects on this feedback and may store the resulting experience. We then close the session, removing raw conversation history while preserving stored experience. The paired non-evolving condition resets agent state before every task. Within each run, both conditions share the task order, backbone, decoding configuration, and scoring procedure; only retention of prior-task feedback differs. Feedback Visibility and Rubric Isolation. The scaffold never receives the complete scene rubric or the judge’s full item-level record. After finalizing a task, it receives rubricbased feedback summarizing the problems in the current deliverable and their corresponding reasons. The complete evaluation information used by the judge to compute the score is not passed into reflection, while the same scoring procedure applies to both conditions. This design provides concrete feedback for reflection while limiting access to the complete evaluation specification, rather than exposing the full rubric for direct optimization. Because tasks in a scene share a procedure but differ in facts, calculations, and conclusions, later improvement requires generalizing reusable lessons from the current task’s problems and applying them to a new case rather than reproducing an earlier answer. Appendix C gives one complete task example: its deliverable, score, feedback, and retained experience.

Evaluation Metrics. We report four metrics. (1) Task quality is the mean scene-specific rubric score on a 0–100 scale. (2) Financial compliance is the mean number of triggered compliance issues per task. (3) Self-evolution ability is measured by the paired score gain and reduction in compliance evo issues relative to the non-evolving condition. Let Sa,k,i and noevo Sa,k,i denote the scores of scaffold a in run k on task i, and evo noevo let Ca,k,i and Ca,k,i denote the corresponding numbers of compliance issues. For K = 3 runs and N = 120 tasks, K

∆Scorea =

N

1 X X evo noevo Sa,k,i − Sa,k,i , KN i=1

(2)

k=1 K

∆Comp.a =

N

1 X X noevo evo . Ca,k,i − Ca,k,i KN i=1

(3)

k=1

(4) Agent-side cost is the mean number of tokens consumed during task execution and post-evaluation reflection. We report token counts in units of 104 per task. Total tokens combine execution and reflection tokens but exclude the independent rubric judge. For longitudinal evolution, we rank each scene’s six tasks by their order in the global stream, so rank r follows r − 1 same-scene tasks. With K = 3 runs, M = 20 scenes, and score Sa,k,s,r , the mean paired gain is ∆Scorea,r =

K M
1 X X evo noevo Sa,k,s,r − Sa,k,s,r . KM s=1

(4)

k=1

We define ∆Comp.a,r analogously, using the compliancenoevo evo issue difference Ca,k,s,r − Ca,k,s,r . Table 3 summarizes ranks 1–3 as Early and ranks 4–6 as Late, with score gains 3

= ∆Scoreearly a

1X ∆Scorea,r , 3 r=1

1 = ∆Scorelate a

6 X

3 r=4

∆Scorea,r .

(5) (6)

Early and Late compliance reductions use the same rank averages of ∆Comp.a,r . Ranks are induced independently in each run rather than assigned to cases in advance. We mark experience activation whenever stored experience enters the execution context. Claude Code, Codex, and GenericAgent retrieve it selectively, whereas Letta injects all core memory blocks automatically. Letta’s 120/120 count therefore denotes an always-on interface, not selective retrieval. We report activation counts and scores for activated and non-activated subsets. Implementation Details. All four scaffolds use Qwen3.7Max with a 1M-token context window, a maximum output length of 64K tokens, greedy decoding, and temperature zero. The evaluated Claude Code scaffold also uses Qwen3.7-Max; only the independent Claude Code scoring agent is backed by Claude Opus 4.6. We run each full-benchmark configuration three times and report the mean. The cross-scene diagnostic uses the single sampled 30-task stream described below. Appendix B gives the three-stage pipeline code and prompts.

4.2

Human Validation of Rubric-Based Evaluation

The Claude Code rubric judge and a financial expert independently score the 120 deliverables from one complete main-experiment run using the same scene rubrics. Under a two-way mixed-effects absolute-agreement model, the judge achieves ICC(A, 1) = 0.95 (95% CI: [0.93, 0.97]). ICC here measures agreement in absolute scores, not only consistency in ranking the deliverables. The judge’s mean and maximum absolute differences from expert scores are 1.6 and 5 points, respectively, on the 0–100 scale, supporting its use for the full evaluation.

4.3

Main Results

Overall Performance and Efficiency. Table 3 reports evolved performance and cost, paired gains over controls, and Early/Late gains. Across the three independently shuffled task streams, retained experience yields positive self-evolution gains for all four scaffolds: relative to the paired non-evolving controls, mean scores rise by 9.33–19.37 points and mean compliance issues fall by 0.12–0.44 per task. Averaged across the three runs, Letta has the highest evolved score (91.65) and fewest compliance issues (0.09 per task), while Codex has the largest self-evolution gain (+19.37). GenericAgent has the lowest evolved score (83.34) and smallest gain (+9.33), despite its lowest token cost. No single scaffold is best in absolute performance, self-evolution gain, and cost. Letta and GenericAgent use fewer reflection tokens than Claude Code and Codex, but their execution costs differ sharply. Letta uses 17.87 × 104 reflection tokens and 32.56 × 104 execution tokens per task because every core memory block enters each model call. GenericAgent uses only 10.21 × 104 and 11.57 × 104 , respectively, because it writes compact experience files and loads them selectively. Its total cost is less than one-third of Claude Code’s. Agentside cost depends on both reflection and how stored experience enters the execution context.

Evolution gain (score points)

Figure 2: FinEvo-Bench construction and cross-task self-evolution. (A) Dataset construction. (B) Experimental setup.

Later ranks Codex +8.50 Letta +8.70 Claude +7.24

Early ranks

25 20

Generic +6.10

15 10 5 0

1

2

3

4

5

6

Occurrence rank within the same scene

Figure 3: Self-evolution score gain by within-scene occurrence rank, averaged over three runs. Final scores and self-evolution gains produce different rankings: Letta ranks first by evolved score, whereas Codex ranks first by gain. GenericAgent ranks last on both measures but has the lowest agent-side cost. Longitudinal Evolution. Despite fluctuations across individual ranks, Figure 3 shows an overall upward trend: every scaffold gains 6.10–8.70 points more over ranks 4–6 than over ranks 1–3. Codex has the highest late gain (23.62), while Letta has the largest early-to-late increase (8.70). Compliance reductions also increase by 0.02–0.10 issues per task (Table 3), supporting gradual experience accumulation. Because later within-scene ranks also occur later globally, the larger gains can reflect both same-scene and broader crosstask experience. Late gains exceed early gains in each of the three independently shuffled streams, so the trend is not driven by one task order. Each evolving run shares its permutation with the

Evolved performance

Overall gain

Agent-side cost (104 tokens/task)

Longitudinal evolution Score gain

Agent scaffold Score ↑ Claude Code Codex Letta GenericAgent

89.47 91.17 91.65 83.34

Comp. ↓ ∆Score ↑ ∆Comp. ↑ Exec. ↓ Reflect. ↓ 0.11 0.11 0.09 0.34

+17.89 +19.37 +17.82 +9.33

0.44 0.44 0.39 0.12

16.31 20.53 32.56 11.57

60.19 48.22 17.87 10.21

Comp. reduction

Total ↓ Early ↑ Late ↑ Early ↑ 76.50 68.75 50.43 21.78

14.28 15.12 13.47 6.28

21.52 23.62 22.17 12.38

0.43 0.41 0.35 0.07

Late ↑ 0.45 0.47 0.43 0.17

Table 3: Performance, cost, and longitudinal evolution across four agent scaffolds. Agent

Activated Act. score Non-act.

Claude Code Codex Letta GenericAgent

87/120 102/120 120/120 71/120

91.82 92.42 91.65 87.04

Gap

83.29 +8.53 84.11 +8.31 – – 77.98 +9.06

Table 4: Experience utilization during evolution.

∆Act. ↑ Agent scaffold ∆Score ↑ ∆Comp. ↓ (tasks/run) Claude Code Codex GenericAgent Letta

+6.38 +7.22 +3.95 +7.93

−0.07 −0.14 −0.10 −0.06

+4 +6 +9 –

Table 6: Rubric feedback versus reference-answer feedback.

Cost (10 tokens/task) 4

Setting No evolution Fixed expert skill Full evolution Memory only Skill only

Score ↑ Comp. ↓ Exec. ↓ 71.58 86.67 89.47 90.42 93.71

0.55 0.13 0.11 0.09 0.05

14.12 15.92 16.31 17.94 17.53

Reflect. ↓ – – 60.19 26.18 44.03

Table 5: Memory and skill carrier analysis on Claude Code. paired control, while different cases occupy the ranks across runs. Experience Utilization. In Table 4, Gap is the activatedsubset score minus the non-activated-subset score. Letta loads all core memory blocks on every task. Codex and Claude Code selectively use skills or project/global memories on 102 and 87 tasks, respectively, while GenericAgent loads a full experience file on 71 tasks. GenericAgent’s lower rate may reflect the limited information in its title-only L0 index for identifying relevant files. For the three selective-retrieval scaffolds, activated tasks score 8.31–9.06 points higher than non-activated tasks. The non-activated subsets score 77.98–84.11, still 3.97–12.31 points above the corresponding state-reset averages. This suggests that, on some tasks, the scaffolds consider their internal knowledge and task-local files sufficient and therefore do not activate stored experience.

4.4

Discussion

Memory, Skills, and Their Combination. In Claude Code, we compare memory-only, skill-only, and unrestricted memory–skill evolution under the same full-benchmark protocol. No evolution and a fixed expert skill derived from each scene’s reference workflow serve as references. Costs are reported in 104 tokens per task; reference settings omit reflection.

Skill-only performs best, scoring 93.71 with 0.05 compliance issues per task (Table 5). On this benchmark, the result favors skills for recurring procedures such as evidence checks, analytical steps, report structures, and compliance constraints. Memory-only remains competitive and improves substantially over no evolution, but it trails skill-only. The combined memory–skill setting improves neither quality nor compliance. Execution traces show that reflection updates both stores, but later tasks often load only one; incomplete loading may partly explain the absence of an additional gain. Execution costs remain similar across the three evolving settings, but reflection costs vary substantially: memory-only is lowest at 26.18, skill-only uses 44.03, and the unrestricted combination uses 60.19. Skill-only obtains the best outcomes with less reflection overhead than the combination. The fixed expert skill outperforms no evolution but trails all dynamically updated carriers, supporting continued updates beyond an initial professional procedure. Rubric Feedback vs. Reference Answers. Holding all other settings fixed, we compare rubric feedback with a complete reference answer. Each delta is the rubric-feedback condition minus the reference-answer condition: positive ∆Score and ∆Act. favor rubric feedback, while negative ∆Comp. means fewer issues. ∆Act. is the mean loggedactivation difference per 120-task run. Table 6 shows that rubric feedback raises scores by 3.95– 7.93 points, reduces compliance issues by 0.06–0.14, and yields 4–9 more activations for selective-retrieval scaffolds. Letta is omitted from the activation comparison because its interface loads memory on every task. A reference answer gives one valid solution but may not identify what the current response is missing or separate case-specific choices from reusable procedures. Rubric feedback instead identifies omitted evidence, incomplete analysis, and compliance failures in the current response.

Agent scaffold

∆Score

∆Comp.

15

Claude Code Codex GenericAgent Letta

−0.84 +0.46 +1.63 −1.00

+0.04 +0.01 +0.02 +0.05

Claude Code 13.17 12.01 13.77 18.17 6.58 Codex 13.98 14.15 13.92 17.11 6.74 10 Letta 15.21 11.69 12.81 15.88 5.64 5 GenericAgent 6.64

6.62

5.83

6.85

1.93

e n n rt is atio nalys clusio Repo plianc m r n A o m Co Inf Co

Gain (percentage points)

20

0

Figure 4: Percentage-point self-evolution gains across five normalized dimensions, averaged over 120 tasks and three runs. Capability Dimensions. We map every rubric item to one of five dimensions: information and evidence use, analysis and calculation, conclusions and recommendations, report quality, and financial compliance. For a quality dimension d, let Ji,d contain the rubric items assigned to d for task i, with awarded points pa,k,i,j and maximum points pmax i,j . Its normalized score rate is P pa,k,i,j j∈J Ra,k,i,d = P i,d max , (7) j∈Ji,d pi,j ∆Ra,d =

K N
100 X X evo noevo Ra,k,i,d − Ra,k,i,d . KN i=1

(8)

Table 7: Scene-isolated execution relative to mixed-scene execution over five sampled scenes (30 tasks).

runs extract each scene’s six tasks in their relative order from that stream and use a fresh agent state for each scene. The conditions use the same tasks and within-scene order; only experience from intervening scenes differs. Let Smixed , Cmixed and Ssingle , Csingle be the mean score and compliance-issue count over the 30 mixed and isolated outputs, respectively. We report ∆Score = Ssingle − Smixed and ∆Comp. = Csingle − Cmixed ; positive values favor isolation in score but mixed execution in compliance. Table 7 shows small score differences with no common direction. Scene isolation lowers Claude Code and Letta by 0.84 and 1.00 points, but raises Codex and GenericAgent by 0.46 and 1.63 points. Letta’s always-on memory may benefit from general experience accumulated across scenes, whereas GenericAgent’s title-only index may retrieve more precisely from a homogeneous scene store. Scene-isolated execution also triggers 0.01–0.05 more compliance issues per task for every scaffold, consistent with cross-scene transfer of general practices such as evidence checking, cautious wording, and avoiding decisions beyond the agent’s role. Score effects, however, remain mixed in this five-scene sample.

k=1

For compliance, let ca,k,i be the number of triggered issues and Gs(i) the total number of compliance issues defined for task i’s scene. We report  K N  cevo 100 X X cnoevo a,k,i a,k,i comp ∆Ra = − . (9) KN Gs(i) Gs(i) i=1 k=1

Quality gains subtract the non-evolving score rate from the evolving score rate. Compliance gain reverses the subtraction, so positive values always indicate fewer normalized issue triggers. All five gains are reported in percentage points. Every scaffold improves across all five dimensions (Figure 4). Report quality has the largest quality gain for each scaffold (6.85–18.17 percentage points), while normalized compliance gains range from 1.93 to 6.74 points. The comparatively large report-quality gains suggest that structure, coverage, and expression transfer more readily than evidence use, analysis, and conclusions, which remain tied to casespecific facts. GenericAgent improves least in every dimension, consistent with its lower overall gain and less frequent activation. Cross-Scene Interference. Interleaved scenes may create retrieval competition (Wei et al. 2025; Wang et al. 2026). We randomly sample five scenes (six tasks each) and shuffle their 30 tasks once into a mixed stream. Scene-isolated

5

Conclusion and Limitations

FinEvo-Bench evaluates four self-evolving agent scaffolds on 120 real-case-grounded financial tasks. The automated rubric judge agrees closely with financial-expert scoring on 120 outputs (ICC(A, 1) = 0.95). Across three shuffled, globally interleaved streams, all four scaffolds outperform their paired state-reset controls: scores rise by 9.33–19.37 points and compliance issues fall by 0.12–0.44 per task. Letta records the highest evolved score (91.65), whereas Codex records the largest self-evolution gain (+19.37); GenericAgent has the lowest cost. Skill-only evolution performs best among Claude Code’s carriers, rubric feedback outperforms reference answers, and all five capability dimensions improve. Cross-scene score differences remain small and mixed, while scene isolation triggers 0.01–0.05 more compliance issues per task. FinEvo-Bench covers 20 scenes in six financial domains, uses one backbone, and studies only non-parametric evolution. Results may therefore differ across professions, backbones, and parameter-updating agents. The carrier comparison covers only Claude Code; the cross-scene diagnostic uses five scenes and one ordering. Future work should expand these settings.

References BenYoash, N.; Brief, M.; Ovadia, O.; Shenderovitz, G.; Mishaeli, M.; Lemberg, R.; and Sheetrit, E. 2025. SECQUE: A Benchmark for Evaluating Real-World Financial Analysis Capabilities. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics. Bigeard, A.; Nashold, L.; Krishnan, R.; and Wu, S. 2025. Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks. arXiv preprint arXiv:2508.00828. Cai, Y.; Hao, Y.; Zhou, J.; et al. 2025. Building Self-Evolving Agents via Experience-Driven Lifelong Learning: A Framework and Benchmark. arXiv preprint arXiv:2508.19005. Chen, Z.; Chen, W.; Smiley, C.; et al. 2021. FinQA: A Dataset of Numerical Reasoning over Financial Data. In Proceedings of EMNLP. Chen, Z.; Li, S.; Smiley, C.; Ma, Z.; Shah, S.; and Wang, W. Y. 2022. ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering. In Proceedings of EMNLP. Chi, Y.; Hong, D.; Jiang, D.; et al. 2026. Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization. arXiv preprint arXiv:2604.12290. Choi, C.; Kwon, J.; Lopez-Lira, A.; et al. 2025. FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering. arXiv preprint arXiv:2508.14052. Islam, P.; Kannappan, A.; Kiela, D.; Qian, R.; Scherrer, N.; and Vidgen, B. 2023. FinanceBench: A New Benchmark for Financial Question Answering. arXiv preprint arXiv:2311.11944. Jiang, S.; Ma, L.; Hong, Z.; et al. 2026. SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment. arXiv preprint arXiv:2604.08988. Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In International Conference on Learning Representations. Jin, S.; Li, S.; Zhang, S.; and Yan, R. 2026. FinRpt: Dataset, Evaluation System and LLM-based Multi-agent Framework for Equity Research Report Generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 507–515. Kundurthy, S.; Na, C.; Moraine, C.; et al. 2026. BlueFin: Benchmarking LLM Agents on Financial Spreadsheets. arXiv preprint arXiv:2605.30907. Liang, J.; Han, J.; Li, W.; Wang, X.; Zhang, Z.; Jiang, Z.; Liao, Y.; Li, T.; Huang, Y.; Shen, H.; Wu, H.; Guo, F.; Wang, K.; Hong, Z.; Lu, Z.; Ma, L.; Jiang, S.; and Xiao, Y. 2026. GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization. arXiv preprint arXiv:2604.17091. Shinn, N.; Cassano, F.; Berman, E.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems.

Wang, G.; Xie, Y.; Jiang, Y.; Mandlekar, A.; Xiao, C.; Zhu, Y.; Fan, L.; and Anandkumar, A. 2023. Voyager: An OpenEnded Embodied Agent with Large Language Models. arXiv preprint arXiv:2305.16291. Wang, Y.; Zhang, Z.; Chi, M.; et al. 2026. EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective. arXiv preprint arXiv:2605.18421. Wang, Z.; Wang, K.; Wang, Q.; et al. 2025. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv preprint arXiv:2504.20073. Wei, T.; Sachdeva, N.; Coleman, B.; et al. 2025. EvoMemory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory. arXiv preprint arXiv:2511.20857. Xie, Q.; Han, W.; Chen, Z.; et al. 2024a. FinBen: A Holistic Financial Benchmark for Large Language Models. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track. Xie, Q.; Han, W.; Zhang, X.; Lai, Y.; Peng, M.; LopezLira, A.; and Huang, J. 2023. PIXIU: A Comprehensive Benchmark, Instruction Dataset and Large Language Model for Finance. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track. Xie, T.; Zhang, D.; Chen, J.; et al. 2024b. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems. Zhang, J.; Xiang, J.; Yu, Z.; et al. 2025. AFlow: Automating Agentic Workflow Generation. In International Conference on Learning Representations. Zhang, Q.; Hu, C.; Upasani, S.; et al. 2026. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. In International Conference on Learning Representations. Zhang, S.; Zhang, J.; Liu, J.; Song, L.; Wang, C.; Krishna, R.; and Wu, Q. 2024. Offline Training of Language Model Agents with Functions as Learnable Weights. In International Conference on Machine Learning. Zhao, A.; Huang, D.; Xu, Q.; Lin, M.; Liu, Y.-J.; and Huang, G. 2024. ExpeL: LLM Agents Are Experiential Learners. In AAAI Conference on Artificial Intelligence. Zheng, C.; Xue, C.; Liang, B.; Yang, J.; and Zhang, C. 2026. SEAGym: An Evaluation Environment for Self-Evolving LLM Agents. arXiv preprint arXiv:2606.17546. Zheng, J.; Cai, X.; Li, Q.; et al. 2025. LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners. arXiv preprint arXiv:2505.11942. Zhou, S.; Xu, F. F.; Zhu, H.; et al. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. In International Conference on Learning Representations. Zhu, F.; Lei, W.; Huang, Y.; et al. 2021. TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance. In Proceedings of ACL-IJCNLP. Zhu, J.; Tian, Y.; Li, B.; Wu, K.; Liang, Z.; Li, J.; Zhang, X.; Guo, L.; Chen, F.; Liu, Y.; and Zhang, C. 2026. FinMCPBench: Benchmarking LLM Agents for Real-World Financial

Tool Use under the Model Context Protocol. arXiv preprint arXiv:2603.24943.

A

Benchmark Construction: A Worked Financial-Statement-Analysis Scene

This appendix documents the construction process from Section 3 for the Financial Statement Analysis scene and Task 3. It traces the scene sources and six-task design to the input files, reference answer, scoring rubric, and review process. During evaluation, the agent receives only the natural-language request and input files.

A.1

Scene Sources, Specification, and Six-Task Construction

Construction sources. Each scene s uses the three sources defined in Section 3: an institution-provided scene description Ds , a reference professional procedure Ps validated in practice, and a candidate pool Cs drawn from institution-provided and publicly documented cases. Table 8 lists their roles in task construction. Source

Information extracted

Role in construction

Scene description Ds

Business setting, professional roles, objective, and scope Required input files, professional operations and checks, expected output, and professional constraints Concrete facts, cross-file relations, risk patterns, judgment conditions, and possible conclusions

Defines the business problem addressed by the scene

Reference procedure Ps Candidate cases Cs

Specifies how the work is performed and reviewed Provide the facts and data used to construct the six tasks

Table 8: Construction sources and their roles in the worked scene. We review the source permissions and release conditions of each candidate case before admitting it to the eligible pool Cseligible . Scene specification. A domain expert consolidates the scene description, reference procedure, and eligible cases into the five-part specification defined in Section 3. Table 9 gives the result for Financial Statement Analysis. Element

Meaning

Instantiation in the worked scene

Os

Objective and scope

Es

Required input files

As

Operations and checks

Ys

Expected deliverable

Ks

Constraints

Assess corporate financial health for credit due diligence, post-loan review, large-credit assessment, or risk warning, without replacing the final credit decision Financial statements, statement notes, audit reports, company information, and industry comparisons provided with the task Verify data quality; calculate and interpret financial indicators; analyze profitability, assets, liabilities, cash flow, and DuPont drivers; scan warning signals; reconcile statements; form a supported conclusion A structured financial-health report with calculations, findings supported by source data, a risk rating, and credit or monitoring recommendations Use only supplied data; mark unavailable information; distinguish facts from inference; avoid unsupported assurances or guarantees

Table 9: Extracted specification for the Financial Statement Analysis scene. The reference procedure determines the work covered by the tasks and scoring rubric. Closely related steps may be combined, and optional operations apply only when the task requires them. Table 10 maps the procedure to scoring requirements.

Procedure stage

Professional requirement

Input and quality Confirm the entity, period, accounting basis, review available statements and notes, audit opinion, and material changes in policy or consolidation scope. Profitability and Calculate profitability ratios and distinguish earnings quality accounting profit from cash realization and recurring operating performance. Asset and liability Examine receivables, inventory, special assets, quality debt structure, liquidity, and maturity matching. Cash-flow analysis DuPont decomposition Warning review Valuation when applicable Cross-statement reconciliation Synthesis and delivery

Task and scoring requirement

Rubric coverage

Identify the input files provided, mark unavailable information, and report any non-standard audit opinion.

Data quality: 8

Calculate and interpret the required ratios, explain changes, and compare them with the supplied industry data. Support findings with the relevant statements and notes and explain their business implications. Analyze operating, investing, and financing cash Cover all three cash-flow categories, flows; calculate free cash flow; interpret their operating-cash-flow quality, free cash flow, and joint pattern. the combined pattern. Decompose ROE and attribute changes to Assess the decomposition, change attribution, margin, turnover, and leverage. and sustainability of the main driver. Scan reporting warning signals and check Check the defined warning signals, possible scene-specific risk conditions. reporting problems, and the eight risk conditions. Add relative or absolute valuation only when the Apply valuation criteria only to tasks that business request requires it. require valuation. Verify consistency across the balance sheet, Check the defined relations and explain any income statement, cash-flow statement, and unresolved discrepancy. notes. Produce a health assessment, prioritized risks, Assess the conclusion, quantitative support, quantitative support, monitoring actions, and a professional boundaries, and report quality. professional report.

Profitability: 15 Asset quality: 12 Cash flow: 13 DuPont: 10 Warning review: 15 Not universally scored Reconciliation: 5 Conclusions: report: 10

12;

Table 10: Reference procedure and corresponding scoring requirements. The procedure is not shown to evaluated agents. Six-task construction. Experts select six cases from the eligible pool based on the scene’s business characteristics and professional procedure. The cases differ in company condition, input files, analytical focus, judgment conditions, or conclusions; cases with only superficial differences are excluded. Table 11 lists the six Financial Statement Analysis cases. Task Business setting 1 2 3

4

5

6

Input files

Main analytical focus

Healthy mature manufac- Complete three-year state- Full-process baseline: profitability, solvency, cash turer seeking a working- ments, audit report, notes, flow, DuPont analysis, reconciliation, and credit capital facility and industry benchmark recommendation Cyclical manufacturer Complete statements with Separate firm-specific deterioration from an industry under post-loan review selected notes and sector downturn; assess efficiency and cash-flow pressure benchmark Fast-growing materials Complete statements, au- Assess earnings quality, collection risk, company requesting a dit report, revenue and re- leverage-driven ROE, and credit capacity despite larger credit line ceivable notes, related-party headline growth information, non-recurring items, and benchmark Listed manufacturer Complete statements plus Detect multiple warning signals, verify under financial-reporting cash, construction-in- cross-statement consistency, and assess the level of scrutiny progress, other-receivable, risk and related-party notes Trading company with re- Complete statements, Assess losses, leverage, debt service, negative cent delinquency and se- non-standard audit report, operating cash flow, and multiple risk conditions vere deterioration borrowing, receivable, contingent-liability, and benchmark files Private manufacturer One-year income statement Limit the analysis to the available input files, avoid with severely incomplete and closing balance sheet imputation, identify unavailable analyses, and request reporting plus company context follow-up files

Files Difficulty 10

Easy

9

Medium

10

Medium–hard

10

Hard

9

Hard

3

Easy–medium

Table 11: The six cases used for Financial Statement Analysis. File counts refer to the input files released with each task. Each selected case becomes a task with a natural-language request, input files, task metadata, and a reviewed reference answer.

All six tasks follow the same professional procedure, but their facts and conclusions come from their own input files.

A.2

Complete Worked Task: Task 3

Task 3 concerns Ruiheng New Materials and its request for a CNY 120 million credit line. The agent receives the following request: User request. “Ruiheng New Materials is applying for a CNY 120 million credit line. It has been growing quickly: revenue has increased for three consecutive years and profit is also rising. However, its cash flow appears problematic, and operating cash flow was negative last year. Please analyze its financial condition and assess whether the available financial information supports the requested credit amount.” Table 12 lists the components of Task 3 and their visibility. The agent does not receive the reference procedure, task metadata, reference answer, or scoring rubric. Component

Contents and purpose

Visible to agent

User request Input files

Business question, requested amount, and initial concern Ten files containing company information, financial statements, notes, an audit report, and comparison data Variant type, difficulty, and dataset identifiers One reviewed analysis and conclusion Criteria, score bands, and professional and compliance checks used by the judge

Yes Yes

Task metadata Reference answer Scoring rubric

No No No

Table 12: Task 3 components and their visibility to the agent. The task includes ten input files. Table 13 maps each file group to the required analysis and scoring criteria. Input-file group

Files

Relevant information and required analysis

Rubric links

Business context

company_overview.md

P9-1–P9-4

Core statements

balance_sheet.md; income_statement.md; cashflow_statement.md

Audit report

audit_report.md

Statement notes

notes_revenue.md; notes_receivables.md; notes_related_party.md; notes_non_recurring.md

Comparison data

industry_benchmark.md

Provides the requested amount and purpose, banking relationship, growth history, and management explanation. Revenue rises by 25% while operating cash flow falls to CNY −80 million and net receivables rise by 75%; the report must calculate, reconcile, and interpret the divergence. Provides the audit opinion and the emphasis on receivable collection, which must be considered together with the statements and notes. Receivables older than one year reach 34.07%, and non-recurring gains equal 38.18% of net profit; the analysis must consider customer concentration, related-party transactions, allowances, and recurring profit. Industry profitability, earnings-quality, solvency, efficiency, and growth values provide the comparison basis for judging whether headline performance is sustainable.

P2-1–P2-3; P3-1; P4-1– P4-4; P5-1–P5-3; P8-1– P8-2 P1-2–P1-3; P3-1; P6-1 P2-2–P2-3; P3-1; P6-1– P6-3

P2-4; P3-4; P5-3; P9-3

Table 13: Task 3 input-file-to-criterion mapping. A complete response identifies available and unavailable input files; analyzes profitability, earnings quality, assets, liabilities, cash flow, and DuPont drivers; reviews financial warning signals; reconciles the statements; and gives a quantitatively supported rating and credit recommendation within the stated professional scope. The reviewed reference answer rates the company as Concern. It recommends against the full CNY 120 million request and proposes a CNY 60 million one-year facility with collateral, staged disbursement, and monitoring conditions. The cited reasons are negative operating cash flow, rapidly growing and aging receivables, the material contribution of non-recurring gains to profit, and leverage-driven ROE. Other recommendations receive credit if they are quantitatively supported, address the material risks, and remain within professional scope.

A.3

Scene-Level Rubric and Quality Control

Rubric derivation. The six tasks share a 100-point rubric derived from the professional procedure. It checks whether an agent uses the input files correctly, completes the necessary analysis, and reaches supported conclusions. Outputs need not match the reference answer in structure or wording.

Criterion group

Points

Data confirmation and quality assessment Profitability and earnings quality Asset quality and liability structure Cash-flow analysis DuPont analysis Financial warning and reporting-risk review Cross-statement reconciliation Integrated assessment and conclusions Report quality

8 15 12 13 10 15 5 12 10

Total

100

Table 14: Rubric allocation for the Financial Statement Analysis scene. Table 15 gives representative scoring rules. Applicability depends on the task input files and business request. A required analysis cannot be marked non-applicable when the necessary information is available. Numerical checks use the source data and defined formula, allowing a rounding tolerance below 0.5 percentage points or a clearly stated valid alternative convention. Criterion

Full credit

P2-2 Earnings qual- The deliverable calculates cash ity (5) collection, profit-to-cash conversion, and recurring-profit share and interprets them using the defined thresholds. P3-1 Receivables (4) The deliverable analyzes turnover or aging, allowance adequacy, and customer concentration using the available notes.

Partial or zero credit

Non-applicable rule

Two measures: 3; one measure or measures without interpretation: 1; no earnings-quality analysis: 0.

Applicable when the required statement values are present.

Material but incomplete analysis: 2; no receivables analysis: 0.

Full credit only when the package provides no receivables detail and the deliverable correctly states that the analysis is unavailable. P5-2 Change attribu- The deliverable attributes changes in Abbreviated attribution: 2; no attribution: Full credit when only one petion (4) ROE using chain substitution or an 0. riod is supplied and change atequivalent complete method. tribution is impossible. P9-4 Decision sup- The deliverable gives an actionable credit Generic recommendation: 2; no Full credit only when the busiport (3) or monitoring recommendation supported recommendation: 0. ness request genuinely does by the task input files and avoids not call for a decision recomunsupported assurance. mendation.

Table 15: Representative criterion-level scoring rules. The rubric also checks for fabricated or unsupported information, conclusions that exceed the available input files, decisions beyond the agent’s professional role, and unsupported assurances about credit approval or investment returns. ID

Check

Type

Observable condition in the final deliverable

RED-1 Fabricated financial data

Data use and compli- Introduces a financial value, ratio, or audit opinion absent from the task input ance files. RED-2 Omitted core analysis Critical task quality Omits at least two of profitability analysis, cash-flow analysis, and warning review. RED-3 Skipped or missed risk Critical professional Omits the R1–R8 review or misses a condition explicitly supported by the checks quality task input files. RED-4 Severe-risk misclassifica- Critical professional Gives a favorable rating despite clear high-risk information and provides no tion quality corresponding warning. RED-5 Unsupported imputation Data use and compli- Fills missing core financial data by unsupported estimation instead of marking ance the data unavailable. RED-6 Missing scope statement Professional scope Does not state that the report supports analysis but does not constitute a final credit approval decision. RED-7 Prohibited assurance Financial compliance Guarantees approval or returns, or issues an unsupported directive to invest or lend.

Table 16: Professional and financial-compliance checks in the scene rubric. Quality control. Domain experts first review the scene and task settings, the completeness of the input files, and the correctness of the reference answers. One expert then drafts the rubric from the scene specification and professional procedure. A second

applies it to all six tasks, checking case applicability, coverage of required business steps, and consistency with the input files and reference answers. The experts correct and re-review any omission or conflict. Table 17 lists these checks. Stage

Review target

Checks

Source screening

Candidate cases

Source permissions and release conditions

Scene definition Task construction Rubric drafting Rubric review

Action when revision is needed

Exclude cases that do not meet the release conditions (Os , Es , As , Ys , Ks ) Consistency with the scene description, professional Revise the scene speciprocedure, and cases fication before selecting tasks Request, input files, metadata, Reasonable task setting, complete task package, correct Correct the task and reand reference answer calculations and conclusions, and substantive variation peat the review across cases Criteria and grading rules Coverage of the required business steps, clear score Revise the criterion or bands, and appropriate non-applicable conditions grading rule Rubric applied to all six tasks Applicability to each case and consistency with the Revise the affected task or input files and reference answers rubric and review it again

Table 17: Review steps for task construction and scoring. At evaluation time, the judge receives the agent’s final output and the scoring rubric, but not the execution trajectory or reference answer. The criteria assess only information visible in the final output. Complete criterion inventory. Tables 18 and 19 list all 32 scored criteria: 27 task criteria and five report-quality criteria. The P7 prefix is omitted because valuation is optional and does not apply to all six tasks. ID

Criterion

P1-1 Data completeness P1-2 Audit opinion P1-3 Policy and scope changes P2-1 Core profitability metrics P2-2 Earnings quality P2-3 Profitability drivers P2-4 Industry comparison

Full-credit condition

Pts.

The deliverable identifies the available statements, notes, and audit report and marks missing items without unsupported imputation. The deliverable correctly identifies the audit opinion and prominently reports any non-standard opinion; if no audit report is supplied, it states that the opinion is unverified. The deliverable checks material accounting-policy and consolidation-scope changes and explains their effect when the relevant information is available.

3

Correctly calculate gross margin, core operating margin, net margin, ROE, and other defined core indicators from the supplied data. Calculate and interpret cash collection, profit-to-cash conversion, and recurring-profit share using the defined thresholds. Attribute margin changes and analyze material expense-ratio movements rather than merely listing values. Compare the core profitability indicators with the supplied industry benchmark or explicitly state that comparison data are unavailable.

4

P3-1 Receivables

3 2

5 3 3

Analyze turnover or aging, allowance adequacy, and customer concentration using the available notes; flag an over-one-year share above 30%. P3-2 Inventory Analyze inventory turnover, impairment allowance, and consistency with revenue using the available information. P3-3 Goodwill and special assets Evaluate goodwill or long-running construction in progress and associated impairment or capitalization risk when applicable. P3-4 Debt structure and solvency Analyze interest-bearing debt, maturity structure, liquidity ratios, and interest coverage.

4

P4-1 Three cash-flow categories Analyze operating, investing, and financing cash flow and identify the material movements in each. P4-2 Operating-cash-flow quality Reconcile operating cash flow with reported profit and connect the result to earnings-quality measures. P4-3 Free cash flow Correctly calculate free cash flow and interpret its trend. P4-4 Joint cash-flow pattern Identify and explain the combined operating, investing, and financing cash-flow pattern and its credit implications.

4 4

P5-1 Three-factor DuPont P5-2 Change attribution P5-3 Driver sustainability

4 4 2

Correctly decompose ROE into net margin, asset turnover, and equity multiplier. Use chain substitution or an equivalent method to attribute changes in ROE. Identify the dominant ROE driver and assess whether it is sustainable.

Table 18: Complete scene-level criterion inventory, Part I: data quality through DuPont analysis (58 points).

3 2 3

2 3

ID

Criterion

Full-credit condition

P6-1

Warning-signal scan

P6-2

Reporting-risk review

P6-3

Risk conditions R1–R8

The deliverable reports checks of at least six defined warning types and assigns a severity supported by the task input files. The deliverable checks income, cost, profit, and cash-flow information for signs of aggressive reporting or manipulation. The deliverable reports checks of all eight defined risk conditions and surfaces any triggered condition in the report summary.

P8-1 P8-2

Cross-statement reconcilia- Verify the defined relations among cash, profit, equity movements, and statement totals. tion Discrepancy handling Confirm consistency or identify, localize, and qualify any unresolved discrepancy.

2

P9-1 P9-2 P9-3 P9-4

Financial-health rating Strengths and risks Quantitative support Decision support

Give a clear rating whose severity is consistent with the preceding analysis. Summarize concrete, case-specific strengths and risks rather than generic observations. Support qualitative conclusions with values, changes, periods, and relevant comparisons. Give an actionable credit or monitoring recommendation supported by the task input files and avoid unsupported assurance.

3 3 3 3

Present the main findings and numerical results clearly; tables may be used when appropriate. Cover at least seven of the eight required analytical components. Section titles and ordering may vary. Mark reporting periods and sources for central data and conclusions. State that the report is for decision support and does not constitute a final credit approval decision. Use only values supported by the task input files.

2 3

FMT-1 Clear presentation FMT-2 Analytical coverage FMT-3 Period and source labels FMT-4 Scope statement FMT-5 Data fidelity

Pts. 5 5 5 3

2 1 2

Table 19: Complete scene-level criterion inventory, Part II: warning review, reconciliation, synthesis, and report quality (42 points).

B

Three-Stage Evaluation Pipeline and Prompt Templates

This appendix documents the three-stage harness used in the longitudinal evaluation. Stage 1 executes a task, Stage 2 scores the resulting deliverable, and Stage 3 returns the feedback to the evaluated agent for reflection. Stages 1 and 3 run one of the four evaluated agent frameworks—Claude Code, Codex, Letta, or GenericAgent—whereas Stage 2 always uses the same Claude Code judge. For each task, Stage 3 resumes the exact conversation created in Stage 1.

B.1

Pipeline Overview

Table 20 summarizes the executor, workspace, and output of each stage. The task instruction and user request are concatenated into the Stage 1 message sent to the agent; they are not exposed as files in the agent workspace. Stage

Function

Executor

Visible workspace

1

Task execution

2

Rubric scoring

3

Feedback reflection

One of four evaluated agent input/, containing only the files frameworks supplied with the task Fixed Claude Code judge input/, RUBRIC.md, and model_result.md The same agent framework in The Stage 1 workspace, with the exact Stage 1 conversation judge.md added beside input/

Primary output model_result.md judge.md Updated agent state

Table 20: Roles, workspace visibility, and outputs in the three-stage evaluation pipeline. Algorithm B.1: Three-stage longitudinal evaluation Input: State: 1 2 3 4 5 6 7

ordered task stream (τ1 , . . . , τN ); one of the four evaluated frameworks a; task rubrics; fixed Claude Code judge J agent state Sa accumulated across the task stream for each task τi in stream order do Create input/; concatenate the task instruction and user request as message pi , without adding prompt files to the workspace. Start conversation Ci with a, send pi , and save the Markdown deliverable as model_result.md. In an isolated workspace containing input/, RUBRIC.md, and model_result.md, invoke J and save its complete result as the single file judge.md. Add judge.md beside input/ in the Stage 1 workspace. Resume the exact conversation Ci , send the Stage 3 prompt, and allow a to update Sa . end for

B.2

Stage 1: Task Execution

Stage 1 Prompt Panel — Execute the Current Task Claude Code, Codex, Letta, or GenericAgent A task conversation that will be resumed unchanged in Stage 3 The task instruction and user request are concatenated as text before the message is sent input/, containing only the input files required by the task model_result.md, a Markdown document generated by the agent

Executor Conversation Message construction Visible workspace Required output

Prompt template Complete the task described below using the files in input/. Write the final deliverable as model_result.md in the current directory. After the file has been written, reply only “done.” [Task instruction] [User request]

Figure 5: Stage 1 prompt and workspace state. The two text components are sent in one message and do not appear as workspace files.

B.3

Stage 2: Fixed Claude Code Scoring

Stage 2 Prompt Panel — Score with the Fixed Claude Code Judge Fixed Claude Code judge input/; RUBRIC.md; model_result.md model_result.md RUBRIC.md judge.md, the single scoring and feedback file

Executor Visible workspace Scoring target Scoring standard Required output

Prompt template You are a strict evaluator. The current workspace contains the original task inputs under input/, the scoring rubric in RUBRIC.md, and the agent deliverable in model_result.md. Evaluate model_result.md strictly according to RUBRIC.md. Check every scoring item and support each score and deduction with evidence from the deliverable and, when needed, the task inputs. Write the complete scoring result and feedback to judge.md. After the file has been written, reply only “done.”

Figure 6: Stage 2 prompt and workspace state. The fixed Claude Code judge produces one scoring and feedback file.

B.4

Stage 3: Feedback Reflection in the Same Conversation

Stage 3 Prompt Panel — Reflect on Feedback The same one of Claude Code, Codex, Letta, or GenericAgent used in Stage 1 The exact Stage 1 task conversation, resumed rather than restarted The original workspace, with judge.md added beside input/ judge.md, containing the Stage 2 score and feedback The agent may update its state and accumulate experience

Executor Conversation Visible workspace New information Result

Prompt template The file judge.md contains feedback on your previous response. This is feedback for you; you may update your state and accumulate experience based on it. After you have finished, reply only “done.”

Figure 7: Stage 3 prompt and workspace state. Feedback is returned to the exact conversation used for Stage 1.

C

Complete Example of Feedback-Driven Experience Consolidation

This appendix follows one Task 04 trace from the Financial Statement Analysis scene. The task asked the evaluated Codex agent to prepare a risk-oriented analysis of an anonymized biomedical company’s 2021–2023 consolidated statements, audit

reports, notes, and industry reference data. The trace contains the Stage 1 deliverable, the fixed judge’s score and feedback, and the changes made to the agent’s two persistent experience carriers during Stage 3. Table 21 maps the archived artifacts to the three-stage protocol in Appendix B. Stage

Archived artifact

Content used in this example

Result

1

answer.txt

369-line Markdown report

2

judge.txt

3

before_MEMORY.txt memory_CHANGE.txt before_skill.txt SKILL_CHANGE.txt

The complete financial-analysis deliverable generated from the task files Section scores, deductions, violation count, and textual feedback Persistent feedback-oriented experience before the task and the recorded update Reusable financial-analysis procedure before the task and the recorded update

3

and and

88/100; 0 violations 6 change blocks 9 change blocks

Table 21: Artifacts in the complete Task 04 trace. The violation count is reported separately from the report’s internal financialrisk findings.

C.1

Stage 1 Deliverable

The agent produced a complete financial-analysis report rather than a short answer. Its eight sections covered data quality, profitability, asset quality, cash flow, DuPont analysis, financial warnings and possible statement manipulation, three-statement reconciliation, and an overall credit conclusion. Table 22 preserves the full report structure and the principal conclusion of each section. No.

Report section

Content and principal conclusion

1

Data confirmation and quality

2

Profitability

3

Asset quality

4

Cash flow

5

DuPont analysis

6

Financial warning and manipulation checks Three-statement reconciliation Overall assessment

Checked the three statements, audit opinions, key audit matters, and missing supporting materials. Calculated core profit and profitability ratios; concluded that margin improvement was offset by rising expense ratios. Examined receivables, inventory, related-party balances, goodwill, construction in progress, and cash holdings. Compared operating, investing, and financing cash flows; identified negative 2023 operating cash flow and increased financing dependence. Verified the three-factor identity and attributed the direction of ROE growth mainly to higher leverage. Examined revenue, cost, asset, and cash-flow signals and identified concentrated risk in assets and cash flow. Reconciled statement balances and identified an unexplained 2023 retained-earnings difference. Assigned a high-risk rating, recommended deferring new credit, and listed eight follow-up documents.

7 8

Table 22: Structure of the Stage 1 deliverable. The deliverable was substantively strong but contained six localized omissions or analytical errors. The excerpts below are translated from the archived output and retain the values used by the judge. Selected evidence from the Stage 1 deliverable Profitability table Profit quality Receivables Inventory ROE attribution

“Gross margin: 42.00% (2023), 40.02% (2022), and 40.00% (2021).” The report explicitly calculated the cash collection ratio and net cash ratio, but did not calculate core profit divided by total profit. The report analyzed turnover days and receivables growth against revenue growth, then noted possible year-end collection arrangements. The report compared turnover days with the industry value and discussed possible understatement or incomplete accounting. The report verified the three-factor product and stated that leverage was the main driver, but gave no numerical contribution for each factor.

The report also contained substantial case-specific analysis that received full credit. Table 23 records the main quantitative findings supporting its final disposition.

Dimension

Quantitative evidence in the deliverable

Report conclusion

Profit and cash flow

2023 core-profit margin was 10.79%; operating cash flow was −3.60; free cash flow was −10.10; financing cash inflow was 26.60. Cash/total assets was 48.57%, interest-bearing debt/total assets was 44.00%, other receivables/net assets was 22.1%, and construction in progress/total assets was 10.29%. 2023 net profit was 12.00, whereas retained earnings increased by only 6.00 and no dividend distribution was reported. The report assigned an overall D (high-risk) rating and listed eight follow-up documents.

Accounting profit remained positive, but internal cash generation weakened and financing dependence increased. High cash coexisted with high debt, related-party balances, and long-running construction projects.

Asset and leverage risk

Statement reconciliation Final disposition

The unexplained 6.00 difference required additional supporting material. Defer new credit and strengthen monitoring of existing exposure until the major uncertainties are resolved.

Table 23: Principal quantitative findings and final disposition in the Stage 1 deliverable. Monetary amounts are in CNY 100 million.

C.2

Stage 2 Score and Feedback

The fixed judge assigned 88 out of 100 points and recorded no violations. Table 24 gives the complete section-level score. All lost points came from six two-point deductions in profitability, asset quality, and DuPont analysis; the other five sections received full credit. Rubric section

Score

Maximum

Deduction

Data confirmation and quality assessment Profitability analysis Asset-quality analysis Cash-flow analysis DuPont analysis Financial warning and manipulation identification Three-statement reconciliation Overall assessment and conclusion Report quality

8 9 8 13 8 15 5 12 10

8 15 12 13 10 15 5 12 10

0 6 4 0 2 0 0 0 0

Total

88

100

12

Table 24: Complete Stage 2 score for Task 04. Violation count: 0. Table 25 reproduces all six feedback items. The feedback identifies the problem and its evidence, but does not expose the complete rubric or a reference answer. No.

Rubric item

1

Core profitability metric

2 3 4 5 6

Judge feedback

The 2022 gross margin was reported as 40.02%; the correct value is 41.00%, beyond the 0.5-percentage-point tolerance. Profit-quality assessment Only the cash collection ratio and net cash ratio were explicitly calculated. The report omitted core profit divided by total profit and its interpretation. Profitability-driver de- The report listed changes in gross margin and expense ratios but did not systematically composition attribute them, for example through a quantity–price decomposition. Receivables analysis The report omitted allowance sufficiency and customer concentration, and did not mark the missing receivables-aging note as unavailable. Inventory analysis The report omitted inventory-impairment analysis and the relation between inventory and revenue, and did not mark the missing inventory detail as unavailable. Change attribution The report verified the three-factor ROE product and gave a directional interpretation, but did not use chain substitution or an equivalent method to quantify each factor’s contribution.

Deduction 2 2 2 2 2 2

Table 25: Complete textual feedback returned for Task 04. The numerical and coverage implications of the feedback are shown in Table 26. These values are computed directly from the Stage 1 report and make explicit what the next execution should verify or add.

Feedback target

Evidence in the submitted report

Required correction or extension

Gross-margin arithmetic

2022 revenue was 115.05 and cost was 67.88, but the table reported 40.02%. Core profit was 14.89, 12.27, and 9.82; total profit was 16.00, 13.25, and 10.67 for 2023–2021. The report observed gross margin of 40.00%, 40.02%, and 42.00% and rising selling-expense ratios.

Recalculate (115.05 − 67.88)/115.05 = 41.00%.

Core-profit share Margin drivers Receivables coverage

Turnover days were 34.8, 36.0, and 37.2 versus an industry value of 65 days.

Inventory coverage

Turnover days were 44.5, 44.4, and 46.8 versus an industry value of 120 days.

ROE attribution

The report verified the identity and gave only the direction of the three drivers.

Report 93.06%, 92.60%, and 92.03%, respectively, and interpret their stability. Separate price, volume, product-mix, and cost effects when supported; otherwise state which decomposition inputs are unavailable. Add allowance-sufficiency and customer-concentration checks and label the unavailable aging note. Add impairment adequacy; compare inventory growth with revenue growth (16.67% vs. 20.0% in 2023; 20.0% vs. 22.0% in 2022); label unavailable detail. A consistent 2021–2023 chain substitution gives +0.38, +0.59, and +1.30 percentage points from net margin, turnover, and leverage; the sum, +2.26 points, equals the ROE change.

Table 26: Detailed evidence and corrections corresponding to the six Stage 2 feedback items.

C.3

Stage 3 Memory Update

Stage 3 resumed the same conversation used to generate the report. The agent converted the six feedback items into persistent, task-general guidance rather than storing company-specific figures or conclusions. The memory update retained the prior guidance and inserted the new lessons at six locations: the profitability block, the asset-quality block, the DuPont block, two repeated application summaries, and the memory descriptor. Memory Before and After Task 04 Existing guidance already covered stepwise core-profit arithmetic, ROIC consistency, missing-data labels, DuPont identity checks, cash-flow-pattern checks, goodwill and construction-in-progress analysis, statement reconciliation, and qualitative ratings.

Before

Added feedback

after

(1) Recalculate gross margin exactly; (2) always compute all three profit-quality ratios, including core profit/total profit; (3) use a systematic quantity–price or equivalent attribution for gross-margin changes; (4) assess receivables allowances and customer concentration and label unavailable aging data; (5) assess inventory impairment and inventory–revenue matching and label unavailable detail; and (6) quantify ROE-factor contributions with chain substitution.

Updated application summary Check percentage calculations explicitly; cover all three profit-quality ratios; perform systematic gross-margin attribution; complete allowance, concentration, impairment, and matching checks; label unavailable notes; and quantify each ROE driver rather than reporting direction alone.

Figure 8: The Task 04 memory update. The retained content abstracts from the current company and records reusable failureprevention guidance.

C.4

Stage 3 Skill Update

The skill update translated the same feedback into executable additions to the financial-analysis procedure. Nine change blocks modified the calculation checklist, driver-decomposition procedure, asset-quality checklist, DuPont procedure, three deduction lists and summary tables, and the final report-quality checklist. Table 27 shows the operational before–after differences.

Procedure block

Before Task 04

Added after Task 04

Calculation validation

Validate core-profit arithmetic and reconcile the result with operating profit.

Gross-margin attribution

Discuss raw-material prices, product mix, pricing power, competition, and cost pass-through.

Receivables

Compare receivables growth, revenue growth, aging, allowance practice, and related-party collection periods. Compare growth and turnover with revenue and industry values; include inventory-impairment analysis. Check that the three-factor product equals ROE and that the contributions sum to the total change.

Recalculate gross margin as (revenue − cost)/revenue and explicitly compute the cash collection ratio, net cash ratio, and core profit/total profit. Require a systematic decomposition, including quantity and price contributions when supported, rather than a list of directional changes. Require allowance-sufficiency and top-customer-concentration analyses; mark unavailable aging-note data explicitly. Require an explicit inventory–revenue matching test, impairment sufficiency, and an unavailable-data label when inventory details are absent. Require numerical chain-substitution contributions for net margin, asset turnover, and the equity multiplier; a product check and directional conclusion alone are insufficient. Add the six Task 04 failure modes and their two-point deductions under profitability, asset quality, and DuPont analysis. Add explicit checks for percentage arithmetic, all three profit-quality ratios, systematic margin attribution, receivables and inventory coverage, and quantitative ROE attribution.

Inventory ROE attribution

Section-level deduction lists

List prior calculation, completeness, and attribution failure modes.

Consolidated quality check- Check data completeness, accounting consistency, list cash flow, financial risk, reconciliation, conclusions, and reporting quality.

Table 27: Operational changes made to the reusable financial-analysis skill after Task 04.

C.5

Selected Final Persisted State

The preceding tables describe the edits. This subsection instead shows selected content as it appears in the persistent state after reflection. The excerpts are translated from the archived memory and skill files, with formatting normalized for the appendix. They include only the blocks materially changed by Task 04. Final Memory Entry The feedback memory retained earlier lessons and incorporated the new findings into the existing profitability, asset-quality, and DuPont blocks. The final entry is identified as financial-analysis-report-standards; the selected post-update text is reproduced below. Final Memory Entry — Selected Post-Update Content Profitability calculations Profit-quality coverage Profitability attribution Receivables Inventory ROE attribution

Gross margin must be recomputed as (revenue − cost)/revenue. Percentage values must be checked explicitly rather than inferred from a trend table. Explicitly calculate all three measures: cash collection ratio, net cash conversion ratio, and core profit/total profit. Missing one measure leaves the earnings-quality assessment incomplete. A change in gross margin requires systematic attribution. Separate price, volume, product mix, raw-material cost, and cost pass-through effects when the inputs permit; otherwise mark the unavailable inputs. Cover turnover efficiency, allowance sufficiency, and customer concentration. If the aging note or customer detail is absent, label it unavailable rather than silently omitting the analysis. Cover turnover efficiency, impairment allowance, and inventory growth versus revenue growth. If inventory detail is absent, label it unavailable. Use chain substitution or an equivalent method to quantify the contributions of net margin, asset turnover, and the equity multiplier. A product check and directional explanation alone are insufficient.

Figure 9: Selected final content of the feedback memory after Task 04. Final Skill: Calculation and Coverage Rules The skill incorporated the feedback at the point of execution rather than keeping it only as a retrospective note. The first affected block now requires the agent to complete the following checks while drafting the profitability and asset-quality sections.

Final Skill Excerpt 1 — Profitability and Asset Quality Calculation validation Profit-quality metrics Gross-margin drivers Receivables checklist Inventory checklist

For each year, recompute gross margin from revenue and cost; reconcile core profit with operating profit; and verify that summary-table values equal the detailed calculations. Calculate cash collection ratio = cash collected/revenue, net cash conversion = operating cash flow/net profit, and core-profit share = core profit/total profit. Interpret all three. Do not stop at year-to-year changes. Quantify price, volume, mix, raw-material cost, and cost pass-through effects when supported, and assess whether the identified drivers are sustainable. (1) Turnover or aging; (2) allowance adequacy and industry comparison; (3) top-customer concentration; (4) related-party collection; and (5) an explicit unavailable-data label for a missing aging note. (1) Inventory growth versus revenue growth; (2) turnover versus the industry; (3) impairment allowance and adequacy; and (4) an explicit unavailable-data label for missing inventory detail.

Figure 10: Selected final profitability and asset-quality instructions in the updated skill. Final Skill: Quantitative ROE Attribution The final DuPont block specifies the calculation rather than merely requiring “deeper attribution.” Let mt , ut , and et denote net margin, asset turnover, and the equity multiplier in period t. For a base period 0 and current period 1, the skill now contains the following chain-substitution procedure: Final Skill Excerpt 2 — Chain-Substitution Procedure ∆margin = (m1 − m0 )u0 e0 , ∆turnover = m1 (u1 − u0 )e0 , ∆leverage = m1 u1 (e1 − e0 ), ∆ROE = ∆margin + ∆turnover + ∆leverage . Required validation. Compute each contribution numerically, verify that their sum equals the observed ROE change, and recheck the inputs if the identity does not close. Report both the contribution values and their business interpretation.

Figure 11: Final chain-substitution instructions added to the updated skill. Final Skill: Embedded Failure Catalogue The updated skill also embeds the six lessons in its section-level failure catalogue. This makes the feedback available during both execution and final checking, rather than leaving it only in the task history. Skill section

Final failure condition added after Task 04

Deduction

Profitability Profitability Profitability

Gross-margin error exceeds 0.5 percentage points. Any of the three profit-quality indicators is missing, including core-profit share. Gross-margin attribution remains a numerical comparison without systematic driver analysis. Receivables omit allowance sufficiency, customer concentration, or an unavailable-data label for missing aging information. Inventory omits impairment adequacy, inventory–revenue matching, or an unavailable-data label for missing detail. ROE change is described qualitatively without numerical chain-substitution contributions.

2 points 2 points 2 points

Asset quality Asset quality DuPont

2 points 2 points 2 points

Table 28: Selected final failure conditions embedded in the updated skill. Final Skill: Pre-Delivery Checks The final quality-control block converts all six feedback items into reusable cross-task checks without generalizing company-specific facts or exposing the full rubric. Before saving a deliverable, the agent verifies every item in Table 29.

No.

Final post-update check

1 2 3 4

Recompute gross margin and other percentage indicators directly from their source values. Confirm that the cash collection ratio, net cash conversion ratio, and core-profit share are all calculated and interpreted. Confirm that gross-margin changes receive systematic driver attribution rather than a directional description alone. Confirm that receivables cover allowance adequacy and customer concentration, with missing aging data labeled unavailable. Confirm that inventory covers impairment adequacy and growth relative to revenue, with missing detail labeled unavailable. Confirm that ROE contributions are numerical and sum exactly to the observed ROE change.

5 6

Table 29: Selected final pre-delivery checks added after Task 04.

Related documents

• Towards Expert Financial QA via Self-Improving RAG#1063994
• Low Reasoning Effort Is Enough for Routine Office Work by Language-Model Agents#1131631
• Exploratory Interview Study with Financial Agents#464589
• NATURAL LANGUAGE UNDERSTANDING MODELS FOR PERSONALIZED FINANCIAL SERVICES#119188
• AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios#800039