SciDiagramEdit: Learning to Edit Scientific Diagrams from Paper Revisions Yasheng Sun1 , Zezi Zeng2 , Yifan Yang2,* , Chong Luo2 , Wenyi Wang1 , Ziwei Liu3 , Jürgen Schmidhuber1 1 King Abdullah University of Science and Technology 2 Microsoft Research 3 Nanyang Technological University [email protected] [email protected] * Corresponding author
70
Organize as two-stage.
best-so-far score per iteration Autofigure-Edit
n
arXiv:2607.15272v1 [cs.CL] 16 Jul 2026
Vectorized Assets > Layout_Plan > Svg_Render > Aesthetic_Verifier > _... > Svg_Render Trace 1 Trace 2
Human Revision Repository
... Trace N
Update Agentic Learning
Success Rate (%)
65 60 55 50
4) Element-wise verify untargeted elements stayed unchanged
2) Math super/subscript typesetting via dx-reset and tspan 3) Strict self-review + double check replacement
45
1) When adding elements: scale comparably + reuse existing rasters
40 35 0
1
2
3
4
5
6
7
8
Evolution Iteration
Figure 1: Overview of SciDiagramEdit. Left: the Human Revision Repository, naturally occurring before/after figure pairs mined from arXiv paper revisions, where each pair encodes an edit the original authors performed during manuscript refinement. Middle: Agentic Learning via Skill Evolution — an SVG-editing agent edits the source figure, and execution traces are distilled into an evolving skill specification that drives future editing behaviour. Right: success rate over evolution iterations on a held-out validation split. The best-so-far skill snapshot rises monotonically; callouts illustrate representative learned skill rules.
Abstract
adopt agentic learning via skill evolution: an agentic proposer continually refines the agent’s skill specification from execution traces over multiple epochs. Experiments show that the agent learns meaningful editing skills from paper revisions and substantially improves edit quality over strong single-pass baselines.
Editing the figures in a research paper is a routine and time-consuming part of everyday research practice: authors relabel components, rearrange panels, and restyle visuals as they revise their manuscripts. Automating this editing workflow under a natural-language instruction, however, is challenging, because a scientific figure is a dense infographic in which heterogeneous visual elements such as schematics, plots, photos, captions, and arrows are composed under a tight visual grammar to advance a specific argument. To address this, we present SciDiagramEdit, a benchmark and skill-evolution framework that learns from natural paper revisions and operates on the figure’s editable vector source, where users can inspect and coedit individual primitives alongside the agent. Our benchmark mines before/after figure pairs from arXiv version histories, each grounded in the authors’ own revision intent. To accommodate the diversity of editing instructions, we
1
Introduction
Scientific figures often determine whether a paper’s contribution is grasped at a glance, and researchers refine them through many cycles of revision before camera-ready submission: relabelling components, rearranging panels, restyling visuals. These edits are typically driven by natural-language feedback from coauthors and reviewers, making instructiondriven figure editing one of the most common figure-related operations in everyday research practice. Existing automated tools, however, target diagram generation from text (Zhu et al., 2026b,a; 1
9
10
Huang et al., 2026) and its evaluation (Liao et al., 2026; Chang et al., 2025), offering little support for the editing regime. Automating instruction-driven editing of scientific figures presents a central challenge: a scientific figure is a dense infographic in which heterogeneous visual elements such as schematics, plots, photos, captions, and arrows are composed under a tight visual grammar to advance a specific argument. Editing such a figure therefore demands localised, compositional operations on its primitives that combine many sub-skills, making one-shot hand-engineered scaffolds brittle across the long tail of edit types. Existing commercial text-to-image editors such as Nano Banana Pro (Google DeepMind, 2025) and GPT-Image2 (OpenAI, 2025) can re-render a complex figure from a natural-language prompt, but they operate at the raster level in a single pass and expose no compositional handle for the targeted local operations a researcher actually performs during revision, such as adding or removing a sub-panel, relabelling a specific component, or re-routing arrows. Natural paper revisions, which capture genuine authorwritten edit intent at scale, remain a largely untapped source of supervision for scientific-figure editing. To address this challenge, we present SciDiagramEdit, a skill-evolution framework that learns to edit scientific diagrams from natural paper revisions. Our key insight is that author-drawn figure revisions encode the implicit visual grammar of scientific communication, and that an agent can distil this grammar by contrasting each of its edits against the author-drawn target. This insight plays out in two parts. First, the publication record itself already contains a long trail of natural editing pairs: between two arXiv versions of the same paper, the same figure is often re-rendered by the authors. Mining these pairs yields a benchmark of before/after figures against which a rubric-grounded judge scores an agent’s edit by directly comparing it to the author-drawn target. Second, because the space of editing instructions is too diverse for any fixed scaffold to anticipate, we adopt agentic learning via skill evolution: by contrasting each rollout against the author-drawn target, an agentic proposer continually distils the drawing patterns observed in those targets into the agent’s skill specification over multiple epochs, lifting edit accuracy without requiring a hand-engineered pipeline. To implement this loop, the editor operates directly on
the figure’s editable vector source, which has the additional benefit of keeping every primitive open for users to inspect and co-edit alongside the agent. Our contributions are threefold: 1) We introduce S CI D IAGRAM E DIT, an editing benchmark of 364 before/after figure pairs mined from real arXiv paper revisions across 23 subjects, annotated with 2,628 atomic editing claims that capture the authors’ own revision intent. 2) We build a selfimproving scientific figure-editing agent that learns from paper revisions via skill evolution, distilling execution traces and the authors’ demonstrations into a portable skill specification. 3) Experiments show that our agent learns meaningful editing skills from paper revisions and substantially improves edit quality over strong single-pass baselines.
2
Related Work
2.1
Scientific Diagram Generation
A first line of work generates a scientific diagram from a textual description. Recent systems include AUTO F IGURE (Zhu et al., 2026b), PAPER BANANA (Zhu et al., 2026a), and S CI F IG (Huang et al., 2026), which build on general SVG generation backbones such as S TARV ECTOR (Rodriguez et al., 2025), LLM4SVG (Xing et al., 2025), and O MNI SVG (Yang et al., 2025). A parallel line evaluates such systems: AIB ENCH (Liao et al., 2026) and S RID B ENCH (Chang et al., 2025) adopt rubric-based protocols whose checklist style inspires the judge we use for the editing setting. All of these systems and benchmarks address diagrams produced from scratch; none takes an existing diagram plus an editing instruction as input. 2.2
Scientific Diagram Editing
The work closest to ours is AUTO F IGURE E DIT (Lin et al., 2026), a one-shot pipeline that converts a raw illustration into an editable SVG but does not itself learn how to edit given an instruction. C HAT 2SVG (Wu et al., 2025) performs iterative instruction-driven editing on generic SVGs. In the raster domain, I NSTRUCT P IX 2P IX (Brooks et al., 2023) and commercial editors such as Nano Banana Pro (Google DeepMind, 2025) and GPT-Image2 (OpenAI, 2025), together with native-unified VLMs such as S ENSE N OVA -U1 (Diao et al., 2026), re-render the image in a single pass and expose no addressable vector primitives for targeted local edits. Among editing benchmarks, SVGE DITB ENCH V2 (Nishina and Matsui, 2025) and SVG E 2
Benchmark
Task
Domain
Source
#
Scientific-illustration evaluation AIBench (Liao et al., 2026) SridBench (Chang et al., 2025)
Eval Eval
Sci illustration Sci illustration
Papers, top venues 2025 Papers, 13 disciplines
300 1,120
Scientific-illustration generation SVGEditBench V2 (Nishina and Matsui, 2025) AutoFigure (Zhu et al., 2026b) AutoFigure-Edit (Lin et al., 2026) PaperBananaBench (Zhu et al., 2026a) SciDiagramEdit (ours)
Edit Gen Gen Gen Edit
Generic SVG (emojis) Sci illustration Sci illustration Methodology diag. Sci illustration
Emoji icons, GPT-4o prompts Papers, surveys, blogs Papers, AutoFigure subset Papers, NeurIPS 2025 Papers, arXiv revisions
1,683 3,300 200 292 364
Table 1: Benchmarks adjacent to our setting, all spanning 2024–2026. Task: Gen = text→figure generation; Eval = evaluation of generated figures; Edit = instruction-grounded edit of an existing figure. Only S CI D IAGRAM E DIT combines instruction-driven editing with scientific diagrams sourced from real author-written paper revisions.
NIUS (Chen et al., 2025) evaluate instruction-based SVG editing on generic content, with neither targeting scientific diagrams.
2.3
natural-language-instructed edits, but it operates on generic emoji SVGs rather than scientific diagrams. Our benchmark is the only one combining (i) the editing regime, (ii) scientific-diagram domain, (iii) paired before/after figures, and (iv) naturally occurring author-written instructions, with paper revisions as the data source.
Self-Improving LLM Agents
A line of work improves LLM-agent behaviour through natural-language feedback over execution traces, varying in the unit of evolution. T EXT G RAD (Yuksekgonul et al., 2024) operates at the level of textual gradients through a compound AI system; VOYAGER (Wang et al., 2023), GEPA (Agrawal et al., 2025), T RACE 2S KILL (Ni et al., 2026), and E VO S KILL (Alzubi et al., 2026) evolve prompts and skill artifacts; and M ETA H ARNESS (Lee et al., 2026) evolves the surrounding coding harness. We adopt the same patch-andmerge philosophy and specialise it to scientificdiagram editing, where the unit of evolution is a skill specification consumed by an SVG-editing agent operating on editable vector primitives.
3
Distribution and Diversity. Figure 2 (a) breaks down edits into four semantic groups: Content (relabeling text), Structure (adding, removing, moving, re-laying-out, or re-routing elements), Visual (restyling colours, fonts, strokes), and Misc. Content-level renaming (R ELABEL, 22.8%) and structural additions (A DD ELEMENT, 21.4%) jointly account for 44% of edits, while connection rewiring is the rarest operation (4.2%). Figure 2 (b) summarises the source-paper distribution: papers span 23 arXiv primary subjects, with the four largest categories (CS .LG, CS .CL, CS .CV, CS .RO) accounting for 73.6% of samples and a long tail of 15 further categories adding disciplinary breadth. Sample years (Figure 2 c) skew toward recent revisions, with 47% drawn from 2023–2024.
Dataset Curation
We construct SciDiagramEdit, a benchmark of 364 natural before/after figure-edit pairs mined from arXiv paper revisions. Each pair places the same figure across two arXiv versions (vold and vnew ) of a paper, providing a naturally occurring edit grounded in real research practice.
Annotation. Scientific figures combine schematic elements that are naturally vectorisable (arrows, frames, text labels, connectors) with raster regions (photos, complex plots, dense sub-schematics) that resist lossless vectorisation. To enable instruction-driven editing at the primitive level, we decompose each figure into these two layers via the pipeline of AUTO F IGURE -E DIT (Lin et al., 2026). (i) A region segmenter identifies the raster panels and returns a set of bounding boxes covering them. (ii) An LLM writes an SVG template that vectorises the surrounding schematic elements as native SVG primitives, leaving a placeholder slot at each bounding box. (iii) The
Relation to Prior Benchmarks. Table 1 situates S CI D IAGRAM E DIT relative to adjacent benchmarks. Among scientific-illustration benchmarks (FigureBench, PaperBananaBench, AIBench, SridBench), all target generation or evaluation of figures produced from scratch and lack paired before/after data. Among instruction-based SVG editing benchmarks (SVGEditBench V2, AutoFigureEdit), only SVGEditBench V2 provides paired 3
Relabel Add element Restyle Layout Remove element Move Other Connection
Structure
Visual
Misc
22.8% 21.4%
Other cs.LG cs.HC eess.IV 9.6% 19.9% cs.AI cs.IR 7.6%
14.1% 12.7% 10.2% 9.2% 5.4% 4.2% 0
10
20
Share of edits (%)
(b) arXiv subjects
cs.RO
16.7%
18.7% cs.CL
18.3% cs.CV
30
30
Share of samples (%)
(a) Edit types
Content
(c) Publication years
24.3% 22.7% 19.1% 20 15.9% 15 13.9% 25
10 4.0%
5 0
2020 2021 2022 2023 2024 2025
Year
Figure 2: Dataset statistics. (a) Edit-type distribution over 2,628 atomic claims across the 364 samples, grouped into Content / Structure / Visual / Misc semantic categories. (b) Source papers span 23 arXiv primary categories; the four machine-learning categories (CS .LG, CS .CL, CS .CV, CS .RO) jointly cover 73.6% of samples, with the remainder distributed across a long tail. (c) Publication years skew toward recent revisions.
Architecture Overview. Our architecture (Figure 3) centres on the Editor E, an agentic system that uses code-level tools to perform the various edits on the SVG figure under a skill specification S. To progressively improve the Editor’s ability across instructions, we further introduce a Judge J that evaluates the Editor’s current performance, and a Coach C that reads past traces and Judge verdicts to upgrade the Editor by emitting patches to S. Iterating this loop refines S across training steps.
original raster crop of each panel is inserted back into its placeholder as an embedded <image> node. Formally, we represent each processed figure as a pair F = (V, A),
A = {ak }m k=1 ,
(1)
where V is the SVG source (vectorised primitives plus <image> reference nodes) and A is the set of m embedded raster panels indexed by placeholder. Beyond the figure pair itself, each sample is annotated at curation time with a natural-language editing instruction and a per-sample checklist Q =
n
(qj , Cj , a⋆j ) j=1 ,
4.2
The Editor is an agentic system, realised as a codewriting subprocess with file-system and Python tool access. Given (F in , I, S), it issues a sequence of tool calls (at , ot )Tt=1 that manipulate the SVG, yielding an edited figure F out and an execution trace τ = (at , ot )Tt=1 . The Editor loads the toplevel SKILL.md verbatim into its system prompt and reads any referenced workflows/ or tools/ file on demand via its file-system tool, so context cost is independent of the skill directory’s size.
(2)
where qj is a natural-language question grounded in the instruction (e.g., “Was the panel-(C) caption added below the rightmost block?”), Cj is the admissible-answer set (binary {YES, NO} or a multi-choice list), and a⋆j ∈ Cj is the ground-truth answer. Across the 364 samples we collect 2,628 such questions in total (Figure 2 a). The questions are designed to cover both surface-level instructionfollowing (whether each named change appears in the output) and the figure’s overall logical coherence (whether the edit preserves the diagram’s reasoning structure).
4
Methodology
4.1
Problem Formulation
Scientific Diagram Editor E
Skill Specification. Formally, S is a finite map from relative paths to file contents, |S| S = (pi , ci ) i=1 , pi ∈ R, ci ∈ Σ∗ , (3) where R is the set of admissible relative paths (a top-level SKILL.md plus optional workflows/ and tools/ subdirectories), and Σ∗ denotes arbitrary unicode content. Following recent work on agentic skills (Alzubi et al., 2026; Ni et al., 2026), we treat S as a learnable artefact and evolve it from execution traces rather than authoring it by hand.
Given an editable scientific figure F in = (V in , Ain ) and a natural-language editing instruction I, the goal is to produce a figure that satisfies I while preserving figure content unrelated to the instruction.
Tool Primitives. To save the Editor from reinventing common utilities and to shorten its ex4
Reorganize from a three-row vertical layout into a two-column horizontal layout, with the right column holding Element Attributes / Image Annotation / Textual Choices top-to-bottom.
ℰ 𝐹
𝒮
()
> Layout_Plan > Svg_Render > Aesthetic_Verifier > ...
𝑟$%&
𝒞 𝑟
𝒫
𝒯
𝐹 *+,
𝐹 -%.
Multi-Choice Question 𝑞
Expected Answer 𝑎 ∗
Judge
"Element Attributes" panel moved?
Bottom-left → top-right
✅
Purple "Image Annotation" moved?
Right-middle
✅
⋯
𝑟&%'
⋯
⋯
Figure 3: Overview of the SciDiagramEdit training loop on a representative sample. Given an input figure F in and a natural-language editing instruction, the Editor E uses the current skill specification S to produce an edited figure F out and an execution trace τ . The Judge J scores the output along two axes: an aesthetic preference raes from a pairwise comparison against the author’s revision F ref , and a semantic faithfulness rsem from a per-sample multiple-choice checklist. The Coach C consumes the trace, the scores, and F ref as a demonstration, emitting a patch that updates S. The loop iterates over training minibatches from Dtrain with validation gating on Dval .
ploration, we expose several optional CLI primitives that it can call on demand: render-svg proInput Diagram vides headless-Chromium SVG rendering with full SVG/CSS/font support, layout-lint runs deterType equation here. ministic geometric checks on the rendered output, and gen-icon synthesises small raster icons of realworld objects (e.g. a sensor, a server rack) that SVG Evolve primitives cannot naturally draw. When and how Agent Instruction to invoke these primitives is encoded in S.
queried with (F out , F in , F ref , I, qj ) and returns a predicted answer âj ∈ Cj for each question, yielding the verification accuracy
target’s visual polish.
where ⪰ denotes “judged at least as polished as” under the comparison.
rsem =
n X 1 Which ⊮is more âj =aesthetic a⋆j ∈ and [0, comprehensible? 1]. (4) n j=1
Aesthetic Preference. Pointwise VLM scoring of a single figure’s aesthetic quality is known to Expected Revision be noisy, whereas pairwise comparison between two figures yields a more reliable preference sig4.3 Aesthetic-Aware Judge JSkill nal (Wang Generated Coachet al., 2025). We therefore evaluate aesThe Judge is a vision-language model that assignsRevision thetics by asking Updatethe VLM to compare the Editor’s a scalar score r ∈ [0, 1] to each edit, combining output F out directly against the author’s revision a semantic-faithfulness term that quantifies how F ref in randomised order to control position bias, well the edit realises the changes specified in the yielding a binary preference h i instruction with an aesthetic-preference term that out ref r = ⊮ F ⪰ F , (5) aes gates the score on whether the output matches the
Semantic Faithfulness. We assess instructionfollowing, factual, and logical fidelity at subinstruction granularity through the per-sample checklist Q of Eq. 2 rather than free-form scoring: constraining the VLM to a small admissible answer set raises answer-level precision and yields an interpretable per-question breakdown, in line with the question-answering protocol used by recent academic-illustration evaluators (Liao et al., 2026). Given an edited figure F out , the VLM is
Composite Score. An edit is only useful in practice if it is both semantically faithful and visually presentable. We therefore gate verification accuracy on aesthetic preference, rather than averaging the two components: r = raes · rsem ,
(6)
so an edit must reach the target’s aesthetic threshold before any verification credit accrues to it. 5
Generate
4.4
Demonstration-Aware Coach C
rules, capturing the compositional and stylistic conventions the author followed rather than samplespecific fixes.
The Coach progressively improves the Editor’s ability by editing the skill specification S in light of past Editor trajectories. We realise it as a codingagent subprocess with file-system and shell access, dropped into a workspace that exposes the current skill directory S, the per-sample trajectory bundles, the author’s revised figure F ref , and a feedback-history file H summarising prior accept/reject decisions so previously rejected ideas are not re-litigated.
where each edit π = (ACTION, p, OP, c) specifies an action (CREATE _ FILE or EDIT _ FILE), a target path p, an operator OP drawn from standard filemodification primitives (CREATE, REWRITE, AP PEND , INSERT _ AFTER , REPLACE , DELETE ), and new content c. The schema lets a single edit introduce a new workflows/ file rather than append to SKILL.md, keeping the always-loaded core concise as expertise accumulates.
Experiment Setup
5.2
(8)
Quantitative Results
Comparison with Baselines. The raster regenerators set a high bar on the aesthetic axis: on UniPercept’s IAA and ISTA, both GPT-I MAGE -1.5 and GPT-I MAGE -2 score strongly, even surpassing the author’s revised figure (Target), consistent with their position as state-of-the-art closed-source image editors. On the semantic axis, however, Ours attains the highest score on the semantic checklist, surpassing both raster regenerators and all singlepass AUTO F IGURE -E DIT backbones, and is essentially tied with GPT-I MAGE -2 on the instructionfollowing win rate. On the aesthetic axis, Ours is the strongest vector-aware method and nearly matches Target on both IAA and ISTA, though it
where the argmax ranges over candidates seen during training and r̄(S; D) := E(F in ,I,F ref )∼D J (E(F in , I; S), F in , F ref , I) is the expected per-sample score.1 Beyond Trace and Score. The trace τ and score r tell the Coach what the Editor attempted and how well it fared, but neither shows what the edit should look like. The author’s revised figure F ref supplies this missing demonstration: comparing F ref against the Editor’s output F out lets the Coach distil the systematic gaps into generalisable skill 1
5.1
Comparison Approaches. We compare against two families of approaches drawn from Figure 4. (i) AUTO F IGURE -E DIT-style Editors (Lin et al., 2026): the LLM is shown the input SVG source and the editing instruction and emits the full edited SVG in a single pass. We evaluate three frontier LLM backbones (GPT-5.3, GPT-5.4, GPT-5.5). (ii) Raster Regenerators: GPT-I MAGE -1.5 and GPT-I MAGE -2 (OpenAI, 2025), which accept the figure and instruction as a raster pair and re-render an edited raster. We include GPT-I MAGE -2, the strongest closed-source image editor available, as our demanding raster-regime reference.
Training Loop. The Coach is invoked iteratively on minibatches drawn from Dtrain and each candidate is scored on a held-out validation split Dval . We maintain a top-K frontier of the best-scoring skills seen so far and propose new candidates from frontier entries in round-robin order; once training terminates, the highest-scoring entry of the frontier is returned: S
Experiments
Evaluation Protocol. We measure the quality of the agent’s edits along two axes. Semantic Faithfulness is scored by the per-sample checklist success rate rsem and a blind-pairwise instructionfollowing win rate. Aesthetic Quality is scored by U NI P ERCEPT’s Image Aesthetic Assessment (IAA) and Image Structure and Texture Assessment (ISTA) on a 0–100 scale (Liao et al., 2026), and a blind-pairwise aesthetic win rate raes . Both pairwise comparisons use the author’s revised figure as the reference. All scores are computed on the test split of the S CI D IAGRAM E DIT corpus (§3), split 2:1:3 into Dtrain , Dval , Dtest ; skill evolution runs for two epochs on Dtrain , with candidates accepted only when they do not regress the running validation score on Dval .
Patch Operator. Let B ⊂ Dtrain be a minibatch of M training samples, with τ , r the per-sample traces and judge scores produced by the Editor on B. The Coach outputs a patch of at most L edits, P = C S, B, τ , r, H , (7)
S ⋆ = arg max r̄(S; Dval ) ,
5
Full pseudocode appears as Algorithm 1 in the appendix.
6
Semantic
Method
Aesthetic
Success Rate ↑
Win Rate ↑
IAA ↑
ISTA ↑
Win Rate ↑
GPT-I MAGE -1.5 (OpenAI, 2025) GPT-I MAGE -2 (OpenAI, 2025)
0.516 0.882
0.347 0.778
50.77 49.60
46.60 45.78
0.349 0.637
AUTO F IGURE -E DIT (GPT-5.3) (Lin et al., 2026) AUTO F IGURE -E DIT (GPT-5.4) (Lin et al., 2026) AUTO F IGURE -E DIT (GPT-5.5) (Lin et al., 2026)
0.828 0.823 0.844
0.528 0.520 0.573
45.30 45.08 44.87
41.24 41.35 41.37
0.389 0.317 0.400
Ours
0.932
0.756
47.93
45.58
0.515
Target (GT)
—
—
48.25
46.49
—
Table 2: Quantitative comparison on the test set. Both win rates are blind-pairwise versus the author’s revised figure. Semantic: checklist success rate and instruction-following win rate. Aesthetic: UniPercept’s Image Aesthetic Assessment (IAA) and Image Structure and Texture Assessment (ISTA) on a 0–100 scale (Liao et al., 2026), and aesthetic win rate. Higher is better; Target (GT) reports UniPercept on the author’s revision. w/o skill
Model GPT-5.1 GPT-5.3 GPT-5.4 GPT-5.5
w/ evolved skill
w/ transferred skill
Semantic ↑
Aesthetic ↑
Semantic ↑
Aesthetic ↑
Semantic ↑
Aesthetic ↑
0.224 0.659 0.700 0.745
0.168 0.285 0.360 0.466
0.252 +0.028 0.713 +0.054 0.724 +0.024 0.756 +0.011
0.205 +0.037 0.358 +0.073 0.436 +0.076 0.515 +0.049
0.257 +0.033 0.706 +0.047 0.718 +0.018 —
0.199 +0.031 0.366 +0.081 0.425 +0.065 —
Table 3: Effect of skill across backbones. Each column group reports semantic and aesthetic win rates under one skill setting: w/o skill, w/ evolved skill, and w/ transferred skill (the skill evolved with GPT-5.5 reused here). Superscripts on the w/ skill columns show the change (∆) relative to the w/o skill baseline in the same row. Reorganize from 2-column vertical to 2-row horizontal layout split by dashed line. Top row: "1. Scene Graph Generation" (image + Task Prompt → LMM → scene graph). Bottom row: "2. Response Generation" (image + Scene Graph → LMM → Response). Swap example to books/mouse/laptop; add curved arrow from scene graph (row 1) into bottom row's input.
Input
Target
AutoFigure-Edit (GPT-5.3)
GPT-Image-2
AutoFigure-Edit (GPT-5.4)
GPT-Image-1.5
AutoFigure-Edit (GPT-5.5)
OURS
Figure 4: Qualitative comparison on a representative editing instance. The panels present the input figure, the author’s target revision, and the outputs of baseline editors (AUTO F IGURE -E DIT with GPT-5.3, GPT-5.4, GPT-5.5; GPT-I MAGE -1.5; GPT-I MAGE -2) alongside our method (Ours). Reorganize from 2-column vertical to 2-row horizontal layout, split by a dashed line, with underlined headings. Row 1 (Scene Graph Generation): green Task Prompt panel → LMM → scene graph panel. Row 2 (Response Generation): green Scene Graph+Task Prompt panel → LMM → Response box. Replace frisbee-man example with books/mouse/laptop on a table; scene-graph nodes become "pink mouse → on → table" + "books → on → laptop". Add curved arrow from row 1 scene graph into row 2 input.
trails the raster regenerators on the blind pairwise aesthetic gate.
the skill evolved on GPT-5.5 fixed and apply it to each backbone in turn. The transferred skill yields gains on both axes, with the aesthetic axis benefiting more. It is also comparable to each backbone’s self-evolved skill (w/ evolved skill); we conjecture that GPT-5.5’s greater capacity lets it explore a
Effect of Skill Across Backbones. Table 3 tests whether the procedural knowledge captured by the evolved skill transfers across backbones. We hold 7
Replace waveform+spectrogram with a single blue waveform; relabel center "+" → "Normalizing Flow"; insert "CNN" before percentages; swap 3D plot ↔ mannequins; recolor connectors to dark blue.
Input wo/ skills
Target w/ skills Aesthetic-Composition Gate triggered → per-item visual self-check → passed blind A/B.
Figure 5: Qualitative skill ablation on a representative editing instance. Left: the input figure (top) and the author’s Reflow 1×4 panels into 2×2 grid (LS/BS top, FA/EM bottom) and drop the per-thumbnail % labels. revised target (bottom); right: our method’s outputs without the evolved skill (top) and with it (bottom). Input
wider space of skill rules during evolution, producing rules that generalise well to weaker backbones.
O URS vs.
5.3
Table 4: We report the win rate of O URS against each baseline on 30 instances sampled from Dtest , with five participants under a blinded protocol.
GPT-I MAGE -2 AUTO F IGURE -E DIT (GPT-5.5)
Qualitative Results
Aesthetic
Semantic
0.54 0.59 0.63 wo/ skills 0.68
Comparison Against Baselines. Figure 4Target shows our qualitative comparison on a scene-graph figure 5.4 User Study that the author rearranged from a 2-column vertical skills Beyond the automated scores, wew/run a human layout into a 2-row horizontal layout with a dashed separator and a curved arrow leading from the row- pairwise study against the two strongest baselines, 1 scene graph back into the row-2 input. GPT- GPT-I MAGE -2 and AUTO F IGURE -E DIT (GPT5.5). We sample 30 instances uniformly from Dtest I MAGE -2 trades logical fidelity for reflow visual→polish: Tighten viewBox after structural no leftover blank bands → passed blind A/B. and recruit five volunteer participants who each rate the rendering is clean, but the curved arrow lands on the wrong target panel. The AUTO F IGURE - O URS against each baseline along two axes, aesthetic and instruction following, in separate rounds. E DIT variants make the opposite trade, getting the Each trial shows the input figure F in , the naturalsemantic content of the instruction right but at the language instruction, and two anonymised outputs cost of aesthetic quality, with the new connector in randomised left/right order; the author’s revision crossing the surrounding label text. Ours navigates F ref is withheld. Table 4 reports the win rate of this trade-off: the curved arrow lands on the correct O URS against each baseline along each axis. target panel, and the connector does not cross the 2 surrounding label text.
6
We presented S CI D IAGRAM E DIT, a skill-evolution framework for editing scientific diagrams that operates on the figure’s editable vector source and acquires a library of editing rules from execution traces and author-drawn targets, so its outputs are SVG the user can continue editing, not a flattened raster. Alongside the framework, we release a benchmark mined from arXiv paper revisions, where each before/after pair is grounded in the authors’ own revision intent. On this benchmark, the agent matches the strongest closed-source raster editor on semantic faithfulness while keeping the figure’s structure editable, and its learned skill further transfers across backbones.
Effect of the Evolved Skill. Figure 5 compares our method’s outputs without (wo/ skills, top right) and with (w/ skills, bottom right) the evolved skill on a single editing instance. Without the skill, the inserted CNN box and the percentage labels are crammed against each other and the panel reads as visually cramped. With the skill in place, the Aesthetic-Composition Gate rule (annotated in orange) triggers a per-item visual self-check that rebalances the inter-element spacing into a more breathable composition. 2
Conclusion
See the appendix for additional visual examples.
8
Limitations
Potential for misuse. Like any figure-editing tool, our system could in principle be used to misrepresent experimental visuals. Its intended use is to support legitimate authoring during revision, where the author retains editorial authority and every model-suggested modification is left as inspectable SVG primitives rather than baked into pixels.
Our benchmark focuses on editing instructions whose intent is stated explicitly: each edit clause is spelled out in the natural-language prompt. We have not yet explored edits that require multi-step reasoning over the figure’s underlying argument, nor the harder regime where the agent must infer authorial intent from the surrounding paper context rather than from the prompt itself. In the same spirit, the skill-evolution loop in this work is still externally orchestrated; a more product-grade pipeline would self-refine autonomously, weighing research taste alongside the explicit metric signal and progressively reducing human supervision in the loop. Finally, the experiments we report operate at a relatively modest scale, with a few hundred training pairs and a few dozen evolution steps. Scaling either axis substantially, by mining many more revision pairs or running the loop for many more iterations, may surface emergent agent behaviours and more abstract skill rules than the current compute budget affords.
AI assistants. The GPT-5. X and C LAUDE models are integral components of our method. Separately, we used an AI assistant in two humansupervised roles. First, during dataset curation, as a co-author of the natural-language editing instruction, with every released annotation humanconfirmed. Second, for prose polishing of the paper itself, namely paraphrasing and grammar refinement of author-written passages, without it suggesting new technical content.
References Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, and 1 others. 2025. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457.
Ethical Considerations Data sources and licensing. The benchmark is mined from publicly available arXiv preprints. The underlying papers and figures retain their original licences and we use them under fair use for noncommercial research. We will release the curated instruction–revision pairs and our atomic-claim annotations under CC BY-NC 4.0 for research use only; the source figures themselves are not redistributed and must be fetched from arXiv under each paper’s original licence.
Salaheddin Alzubi, Noah Provenzano, Jaydon Bingham, Weiyuan Chen, and Tu Vu. 2026. Evoskill: Automated skill discovery for multi-agent systems. arXiv preprint arXiv:2603.02766. Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18392–18402. Yifan Chang, Yukang Feng, Jianwen Sun, Jiaxin Ai, Chuanhao Li, S Kevin Zhou, and Kaipeng Zhang. 2025. Sridbench: Benchmark of scientific research illustration drawing of image generation model. arXiv preprint arXiv:2505.22126.
Content and privacy. All figures in the benchmark are technical scientific diagrams such as plots, scene graphs, and network or pipeline diagrams. During curation we manually inspected every pair and did not encounter personally identifying information beyond the author bylines already present in arXiv metadata, which we do not include in the model inputs. We also did not encounter offensive content.
Siqi Chen, Xinyu Dong, Haolei Xu, Xingyu Wu, Fei Tang, Hang Zhang, Yuchen Yan, Linjuan Wu, Wenqi Zhang, Guiyang Hou, and 1 others. 2025. Svgenius: Benchmarking llms in svg understanding, editing and generation. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 13289– 13296.
Reproducibility. Because the framework relies on closed-weight GPT-5. X backbones, trainingtime access is gated by API cost. We release the benchmark and the skill specification produced by the evolution loop so that future work can reuse the distilled rules without re-running training.
Haiwen Diao, Penghao Wu, Hanming Deng, Jiahao Wang, Shihao Bai, Silei Wu, Weichen Fan, Wenjie Ye, Wenwen Tong, Xiangyu Fan, and 1 others. 2026. Sensenova-u1: Unifying multimodal understanding and generation with neo-unify architecture. arXiv preprint arXiv:2605.12500.
9
Google DeepMind. 2025. Nano banana pro (Gemini image). Image generation and editing model.
Ximing Xing, Juncheng Hu, Guotao Liang, Jing Zhang, Dong Xu, and Qian Yu. 2025. Empowering llms to understand and generate complex vector graphics. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19487–19497.
Siyuan Huang, Yutong Gao, Juyang Bai, Yifan Zhou, Zi Yin, Xinxin Liu, Rama Chellappa, Chun Pong Lau, Sayan Nag, Cheng Peng, and 1 others. 2026. Scifig: Towards automating scientific figure generation. arXiv preprint arXiv:2601.04390.
Yiying Yang, Wei Cheng, Sijin Chen, Xianfang Zeng, Fukun Yin, Jiaxu Zhang, Liao Wang, Gang Yu, Xingjun Ma, and Yu-Gang Jiang. 2025. Omnisvg: A unified scalable vector graphics generation model. arXiv preprint arXiv:2504.06263.
Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. 2026. Metaharness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052.
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. Textgrad: Automatic "differentiation" via text. arXiv preprint arXiv:2406.07496.
Zhaohe Liao, Kaixun Jiang, Zhihang Liu, Yujie Wei, Junqiu Yu, Quanhao Li, Hong-Tao Yu, Pandeng Li, Yuzheng Wang, Zhen Xing, and 1 others. 2026. Aibench: Evaluating visual-logical consistency in academic illustration generation. arXiv preprint arXiv:2603.28068.
Dawei Zhu, Rui Meng, Yale Song, Xiyu Wei, Sujian Li, Tomas Pfister, and Jinsung Yoon. 2026a. Paperbanana: Automating academic illustration for ai scientists. arXiv preprint arXiv:2601.23265.
Zhen Lin, Qiujie Xie, Minjun Zhu, Shichen Li, Qiyao Sun, Enhao Gu, Yiran Ding, Ke Sun, Fang Guo, Panzhong Lu, and 1 others. 2026. Autofigure-edit: Generating editable scientific illustration. arXiv preprint arXiv:2603.06674.
Minjun Zhu, Zhen Lin, Yixuan Weng, Panzhong Lu, Qiujie Xie, Yifan Wei, Sifan Liu, QiYao Sun, and Yue Zhang. 2026b. Autofigure: Generating and refining publication-ready scientific illustrations. In The Fourteenth International Conference on Learning Representations.
Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. 2026. Trace2skill: Distill trajectory-local lessons into transferable agent skills. arXiv preprint arXiv:2603.25158.
Appendix
Kunato Nishina and Yusuke Matsui. 2025. Svgeditbench v2: A benchmark for instruction-based svg editing. arXiv preprint arXiv:2502.19453.
A
Additional Technical Details
A.1
Annotation Interface
Raw paired-figure mining from arXiv revisions is noisy: two versions of the same figure often differ only in a re-rendered raster or a font swap with no semantic intent, and a small fraction are wholesale replacements that share no structure with the original. We keep only the meaningful middle band, in which the author has added a panel, renamed a label, re-routed an arrow, or re-laid out a block of elements while retaining enough of the original scaffold for the change to read as an edit rather than a redraw. Even within this band the raw visual diff seldom makes the author’s intent explicit on its own: the same surface change can encode different revision intents, and disambiguating which is meant requires reading the surrounding paper context. A useful editing instruction therefore has to be authored, not just extracted, which is what Figure 6 is designed to support. The interface in Figure 6 covers the instructionannotation stage of curation. The before/after figure pair is rendered at the top of the panel. The curator first applies two filtering decisions: a binary keep/drop verdict on whether the pair is suitable for the benchmark, and a binary accurate/inaccurate verdict on whether the candidate instruction
OpenAI. 2025. GPT-Image: Image generation and editing. Image generation and editing model. Juan A Rodriguez, Abhay Puri, Shubham Agarwal, Issam H Laradji, Pau Rodriguez, Sai Rajeswar, David Vazquez, Christopher Pal, and Marco Pedersoli. 2025. Starvector: Generating scalable vector graphics code from images and text. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 16175–16186. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. Yibin Wang, Zhimin Li, Yuhang Zang, Yujie Zhou, Jiazi Bu, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. 2025. Pref-grpo: Pairwise preference rewardbased grpo for stable text-to-image reinforcement learning. arXiv preprint arXiv:2508.20751. Ronghuan Wu, Wanchao Su, and Jing Liao. 2025. Chat2svg: Vector graphics generation with large language models and image diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 23690–23700.
10
Figure 6: Gradio interface used to annotate the natural-language editing instruction for each S CI D IAGRAM E DIT pair. The before/after figure pair is shown side-by-side, followed by binary verdicts on whether to keep the pair and whether the candidate instruction is accurate, an editable instruction field, and a chat panel for iteratively refining the instruction with a vision LLM.
faithfully describes the visual edit. The candidate instruction itself is shown in an editable text field that the curator can revise directly. Below it, a chat panel backed by a vision LLM, GPT-5.5, lets the curator iteratively refine the instruction by describing what is missing or wrong; the assistant returns a revised draft that the curator can accept, edit fur-
ther, or discard. The loop typically converges in two to three rounds. The Save button commits the final instruction together with the curator’s tags; the Reset button reverts to the last saved version. Two parallel panels share this layout but are not shown for brevity: one filters SVG vectorisations whose rendering drifts noticeably from the source 11
Algorithm 1 Skill-Evolution Training Loop
raster, since the AUTO F IGURE -E DIT vectoriser of §3 occasionally produces SVG that diverges visually from the original figure and we drop those samples; the other authors the per-sample checklist of Eq. 2. All editing instructions and checklist questions are written in English; the source figures are drawn from English-language arXiv preprints. A.2
Require: Splits Dtrain , Dval ; initial skill S0 ; Editor E, Judge J , Coach C; total steps T ; minibatch size M ; edit budget L; frontier size K Ensure: Best-so-far skill S ⋆ 1: s0 ← r̄(S0 ; Dval ) 2: Initialise frontier F ← {(S0 , s0 )}, H ← ∅ 3: for t = 1, . . . , T do 4: (S, s) ← next entry of F in round-robin order ▷ round-robin parent 5: Sample minibatch B = in ref M {(Fi , Ii , Fi )}i=1 ∼ Dtrain 6: for i = 1, . . . , M do 7: (Fiout , τi ) ← E(Fiin , Ii ; S) 8: ri ← J (Fiout , Fiin , Firef , Ii ) 9: end for 10: P ← C(S, B, τ , r, H) ▷ |P| ≤ L 11: S̃ ← apply P to S 12: s̃ ← r̄(S̃; Dval ) 13: if |F| < K then ▷ room in the frontier 14: F ← F ∪ {(S̃, s̃)} 15: H ← H ∪ {(P, ACCEPT, s̃)} 16: else if s̃ > min(S,s′ )∈F s′ then ▷ beats current worst 17: F ← F \ {arg min(S,s′ )∈F s′ } ∪ {(S̃, s̃)} 18: H ← H ∪ {(P, ACCEPT, s̃)} 19: else 20: H ← H ∪ {(P, REJECT, s̃)} 21: end if 22: end for 23: (S ⋆ , s⋆ ) ← arg max(S,s′ )∈F s′ 24: return S ⋆
Skill Evolution
Training Loop. Algorithm 1 formalises the skillevolution loop sketched in §4.4. The loop maintains a top-K frontier F that stores the highestscoring skills seen so far, in the spirit of the Paretofrontier search of Alzubi et al. (2026). Each iteration picks a parent from F in round-robin order, runs the Editor on a fresh training minibatch with that skill, lets the Judge score the resulting outputs, and asks the Coach to write a patch from the traces, scores, and the running feedback history H. The patched skill is re-evaluated on Dval and admitted to F whenever the frontier still has room or its score strictly exceeds the frontier’s worst entry, in which case that worst entry is evicted; otherwise the candidate is rejected. Either way the accept/reject verdict is appended to H so the Coach does not re-propose a rejected idea. Settings. Table 5 lists the concrete configuration used to instantiate this loop in our main experiments, and a few choices there are worth motivating. The Editor uses a GPT-5. X backbone while the Coach and Judge use C LAUDE O PUS 4.7: the cross-family pairing keeps the Editor from gaming a same-family Judge’s idiosyncratic preferences, Claude Opus’s code competence suits the Coach’s role of authoring skill-rule patches, and the capacity gap between a stronger Coach and a weaker Editor lets procedural knowledge distilled by the Coach lift students of varying capacity, including GPT-5.1, 5.3, and 5.4. The search itself is multifrontier: evolution maintains a top-K=3 frontier of candidate skills rather than a single best-so-far, since a frontier of one plateaued quickly in early experiments while multi-frontier search consistently escaped local optima. The reward governing admission to that frontier is the multiplicative composite r = raes ·rsem from Eq. 6 rather than an average; because vector-aware editing tends to under-perform on aesthetics, multiplying rather than averaging raises the bar there, gating out semantically faithful edits whose aesthetics still fall short and steering the loop to focus on that axis. Skill updates are
applied as patches over the prior skill rather than as full rewrites, which empirically avoids forgetting rules the Coach had previously introduced for unrelated failure modes. Compute Resources. All model inference is performed through hosted APIs; we do not train any model weights and do not use any GPU compute. The Editor calls GPT-5. X through the C ODEX CLI, and the Coach and Judge call C LAUDE O PUS 4.7 through the C LAUDE C ODE CLI. Orchestration, SVG rendering, and trajectory bookkeeping run on a single CPU-only workstation. Across all reported experiments in this paper, spanning four Editor backbones together with the skill-evolution, transfer, and ablation runs and the development cycles that preceded them, total API spend was on the 12
Component
Value
Models Editor E (student) Editor backbones (transfer) Coach C (analyst) Judge J (vision)
GPT-5.5 GPT-5.1/5.3/5.4/5.5 C LAUDE O PUS 4.7 (1M ctx) C LAUDE O PUS 4.7
Evolution loop Frontier size Frontier strategy Epochs Batch size Max analyst rounds per step Edit budget per skill update Skill update mode Initial skill scaffold
3 (multi-frontier) round-robin parent 2 8 3 8 patches patch (vs. rewrite) basic tool usage and guide
Reward Composite r (Eq. 6) Semantic rsem Aesthetic raes
raes · rsem checklist accuracy ∈ [0, 1] binary, pairwise vs. target
Inference Coach rewrite max tokens Coach parallelism Editor rollout parallelism Editor call timeout
64,000 4 workers 8 workers 600 s
in randomised left/right order: one pairs O URS against GPT-I MAGE -2 and one pairs O URS against AUTO F IGURE -E DIT (GPT-5.5); for each trial the participant answers “Which output is more visually polished?”. The semantic round repeats the same per-instance setup with the question “Which output more faithfully realises the instruction?”. Each trial displays the input figure F in , the natural-language editing instruction, and the two anonymised outputs; the author’s revision F ref is withheld throughout, and no “tie” option is provided. Table 4 reports the proportion of votes in favour of O URS per baseline and per axis, computed over the resulting 150 votes per cell.
Further Analysis
B.1
Typical Learned Skills
To give a concrete sense of what the skill-evolution loop actually produces, we first show one evolved skill at work, then describe the broader library that the final skill pack S ⋆ contains. Figure 7 shows one such skill in action. The wo/ skills output gets the panel reflow right but the SVG viewBox still encloses the original 1×4 footprint, so visible bands surround the new content and the aesthetic gate fails. With the evolved skill, the rule “tighten the viewBox after any structural reflow” fires and the bands disappear. Beyond this single rule, the pack is not a single, general-purpose prompt: the agent instead accumulates a structured library of trigger-plus-rule entries, each targeting a specific recurring failure mode the Coach observed in Editor traces. On disk the pack has the following layout:
Table 5: Settings used for the skill-evolution loop and the results reported in §5.
order of US $20,000 at standard rate-card pricing. We use headless C HROMIUM for SVG → PNG rendering, the C ODEX CLI for the Editor, the C LAUDE C ODE CLI for the agentic Coach, and the official anthropic SDK for the Judge. A.3
B
User Study Protocol
The user study reported in §5.4 was conducted with five volunteer participants from the authors’ institution; all participants are graduate students in computer science / electrical engineering and participated without compensation. Before each session, participants were informed verbally that their anonymised preferences would be aggregated and reported in a research paper, and they consented before proceeding. The protocol collects only anonymous A/B preferences over already-public scientific figures, does not record any personally identifying information, and was determined to be exempt from formal IRB review by our institution on that basis. Each participant judges every one of the 30 sampled instances against each of the two baselines on each of the two axes, giving 30 × 2 × 2 = 120 forced-choice trials per participant and 600 trials across the five participants. The trials are presented in two rounds. In the aesthetic round, for each instance the participant sees two trials
SKILL/ |-- SKILL.md `-- workflows/ |-- safe-string-replace.md |-- math-notation.md |-- post-restructure-compactness.md |-- deliverable.md |-- workspace.md |-- available_cli_tools_call_via_bash.md `-- ...
The top-level SKILL.md groups rules by their trigger; longer sub-routines are factored out into individual files under workflows/. Below we highlight five representative entries that span the abstraction range: a top-level verification loop, a visualfidelity rule, two concrete edit-execution idioms covering string replacement and math typesetting, and a structural-edit verification rule whose entry doubles as a meta-comment on the typical shape 13
w/ skills Aesthetic-Composition Gate triggered → per-item visual self-check → passed blind A/B.
Reflow 1×4 panels into 2×2 grid (LS/BS top, FA/EM bottom) and drop the per-thumbnail % labels. Input
wo/ skills
Target w/ skills
Tighten viewBox after structural reflow → no leftover blank bands → passed blind A/B.
Figure 7: Effect of an evolved skill. The instruction reflows a 1×4 panel grid into a 2×2 layout and drops the per-thumbnail percentage labels. Without the skill, the Editor leaves leftover blank bands around the new content; with the skill, the learned rule “tighten the viewBox after structural reflow” eliminates them and the output passes the blind aesthetic A/B against the author’s target.
committed rules take. Two patterns recur across these entries. The verification rules cover distinct scopes, with a whole-image check after every edit and a separate scoped check after structural edits, rather than collapsing into a single “check everything” rule. The concrete edit idioms in turn target SVG-specific traps, such as overlapping string replacements, <tspan> ordering, and viewBox tightening, that a generic “be careful” instruction would never surface. Render-and-Verify Loop
lished version of the input. Anchor a new element’s size to a comparable existing element’s bounding box, and reuse the source’s representation (the existing <image> reference, or the matching <g> subtree anchored by a distinctive fill colour) when the instruction names a known icon. Surface correctness is necessary but not sufficient. Anchored String Replace
Trigger. Edits that swap two labels (A↔B) or rewrite a label that recurs across symmetric panels (e.g., MAML vs. O URS, before vs. after). Failure. A naive Path.read_text() → .replace() → write_text() idiom has two sharp edges: .replace(A,B) followed by .replace(B,A) undoes the first edit, and a bare .replace(label,new) silently rewrites the label in both panels when only one was named. Rule. For two-way swaps, route both sides through a @@hash@@ placeholder pass. For symmetric-panel relabels, anchor each .replace() with the element’s x=". . . " coordinate (or any other unique neighbouring context) so the rewrite is panel-scoped instead of file-wide.
V ERIFICATION
Trigger. Every editing task, before writing summary.txt. Failure. A single “looks done” judgment from the Editor is the most common path to a failed eval: shipped outputs either miss an instruction clause or carry an aesthetic regression the Editor did not notice in the SVG source. Rule. Always render output.png and pass it through a strict vision-LLM reviewer that returns a JSON ship/revise verdict alongside an enumerated list of missing clauses and aesthetic issues; if the verdict is revise, fix the listed items and re-render before writing the summary. Never accept a single optimistic self-check. Match the Visual Register
E DIT EXECUTION
Math-Notation Typesetting
V ISUAL FIDELITY
Trigger. Sub/superscripts on the same symbol (e.g., πtAlice , (un)p Iu ) or an inline math legend listing several notations. Failure. Sequential <tspan baseline-shift> calls flow the subscript after the superscript rather than under it; Unicode super/subscript codepoints render at the wrong scale and break the surrounding font register. Rule. Emit real <tspan> elements with dx="-N em" to back-shift the subscript under the superscript, and rewrite any Unicode super/subscripts into the same tspan idiom.
V ISUAL FIDELITY
Trigger. Edits that add a new element (icon, label, panel, callout) or rewrite a notation that recurs across the figure. Failure. Technically-correct additions still lose the blind A/B when they do not visually belong: under-sized so they read as decoration rather than as a diagram element, or drawn as crude SVG primitives where the source uses polished raster or vector icons. Rule. Aim for an output that could plausibly be the pub-
14
Layered Structural Review
Demonstration-Aware Analyst (with F ref )
V ERIFICATION
Trigger. Instruction contains any of delete, remove, swap, replace . . . with, move . . . into, restructure, reorganize. Failure. The strict whole-image reviewer judges the output plausible and returns ship, but the blind A/B against F ref still fails on four category-specific side-effects the holistic pass misses: duplicated content where a move was requested, blank regions left after a deletion, shrunken replacements, and wrong-coloured replacements. Rule. Chain a scoped second vision-LLM pass (workflows/post-restructure-compactness.md) that audits only those four side-effect categories under a JSON schema; treat each non-empty category as a separate revise signal even when the first pass already shipped.
Each sample folder is described to the analyst as containing input.png (rendered input), output.png (rendered output), target.svg (ground-truth SVG authored as the edit), and target.png (rendered ground truth). The prompt then instructs: 1. Read both PNGs input.png, output.png, and target.png: “your Read tool supports images; see visually what changed and what didn’t.” 2. Examine target.svg source: “how did the groundtruth author solve the same edit?” 3. Decompose what makes target.svg into skill rules.
Figure 8 shows the resulting contrast. With the demonstration in hand, the Coach can ground each patch on the author’s resolution and produce rules that downstream Editor runs realise more reliably, as the marked details show. As an empirical observation across our runs, the two variants also tend to commit qualitatively different kinds of rules. Without a direct aesthetic signal, the demonstration-free Coach has no easy way to tell which parts of the output read well and which do not, so its proposed rules lean toward process-level patterns such as verification loops, restructuring checks, and panellevel reorganisation. With the demonstration available, the Coach can compare the Editor’s output to the author’s resolution at the pixel level, and the rules it proposes become more sensitive to rendering specifics such as math typesetting, colour, and stroke choices. In effect, the demonstration lowers the evolution difficulty by letting the Coach read concrete improvements off the author’s solution rather than infer them from trajectories alone.
Why this is the typical shape. The Coach commits the specific reviewer-shipping pattern (whole-image reviewer + structural-edit triggers + four named side-effects) rather than the abstract principle “check layout after structural edits.” Across iterations, committed rules follow this shape: a precise trigger paired with a scoped sub-routine, not a general exhortation.
B.2
Demonstration-Aware Coach Ablation
A central design choice for the Coach is whether to expose the author’s revised figure F ref as a worked demonstration, or to rely on the trajectories alone. We ablate this choice by comparing a demonstration-aware Coach that sees F ref against a demonstration-free Coach that does not, with everything else about the loop held fixed. Both variants are C LAUDE C ODE subprocesses with filesystem access to each sample folder; the analyst uses its Bash and Read tools to inspect trajectories and writes proposed patches to analyst_response.json. They differ only in whether the author’s target files are in the workspace: the demonstration-aware variant places target.png and target.svg alongside input.png/output.png so the analyst can Read them as multimodal input, while the demonstrationfree variant omits the targets, leaving only the input/output renders and trajectory transcripts. Their system prompts also differ:
B.3
Additional Qualitative Examples
Figures 9, 10, and 11 extend the qualitative comparison in §5.3 with more editing instances spanning different figure types and edit complexities. Across them the same trade-off recurs. GPT-I MAGE -2 keeps individual elements visually polished but tends to lose fidelity on the embedded text and symbols, so labels, superscripts, and tabular numbers drift from the source even when the surrounding layout looks plausible. Because it regenerates the whole figure as a raster, it also imposes its own aesthetic prior on parts the instruction does not target, restyling colours, fonts, and line weights that the author left untouched. The strongest AUTO F IGURE E DIT variant, GPT-5.5, preserves the text and symbols correctly because it edits the SVG source, but the visual register is noticeably weaker: new shapes
Demonstration-Free Analyst “You will be given MULTIPLE failed agent trajectories from a single minibatch and the current skill document. Your job is to identify the most important COMMON failure patterns across the batch and propose a concise set of skill edits.” The analyst has filesystem access to input.png and output.png for each sample via its Read tool; only the author’s target files (target.png, target.svg) are withheld from the workspace.
15
Restructure 2-row → 3-panel (A/B/C) layout: remove top headers and right-side abnormal-tray columns; add per-row captions ("Normal Images" / "Few Labeled Images" / "Normal Images"); rename "Part Seg" → "Part Segmentation"; add new panel C with 4 stacked photo + part-segmentation pairs on yellow background.
Input
Target
wo / demo-aware
w / demo-aware
Merge the lower-branch "Evol-Instruct" document icon + OpenAI circle into a single green rounded tile (white swirl logo + "Instruction Generation Template" text); reroute the input arrow + right brace to this tile. Simplify "evolved dataset round 1" by removing its two inner sub-panels (keep only the pill). Enlarge the OpenAI circle on the upper "re-generate output" branch.
Input
Target
wo / demo-aware
w / demo-aware
Replace small "FM" with wide "Factorization Machine Layer" box. In both User-/Item-Net columns, remove the stacked-MLPs+bidirectional e vector inside the dashed box; relocate e_u/e_i outside (bottom-left/right) and route two MLPs from each into the Graph Representation Learning box and the ⊕. Add labels m_u/m_i at ⊕ inputs and 𝒢_u/𝒢_i inside graph boxes. Widen figure to landscape.
Input
Target
wo / demo-aware
w / demo-aware
Add "Real Image" label with dashed leader to bottom rabbit thumbnail. Remove dotted border around the bottom binarization sub-panel (step-function icon + "1 0 1 0" sit free). Move "Binarization" label from above-right to below the step-function group.
Input
Target
wo / demo-aware
w / demo-aware
Drop "Task:" prefix and add volcano icon to top heading. Replace document icon with stacked-folders icon. Lowercase all "S1/S2/S3" → "s1/s2/s3" (and "E1"→"e1"). Highlight panel (A) border red-dashed; thin out panel (C) border + un-bold its caption. Lighten panel (B) middle node and bottom-tree e1 box to pale blue.
Input
Target
wo / demo-aware
w / demo-aware
Figure 8: Qualitative ablation of the demonstration-aware Coach (§4.4). Each row presents an editing instruction with its input figure, the author’s revised target, the Editor’s output under a skill evolved without access to the author’s revision (F ref withheld from the Coach; wo/ demo-aware), and the Editor’s output under our full demonstrationaware Coach (w/ demo-aware). Orange arrows mark places where the demonstration-aware variant realises an instruction detail that the demonstration-free variant misses.
16
Add 3 bold sans-serif headers above top edge: 'Stage 1' over left video-frame thumbnails; 'Stage 2' over Summarization Prompt+LLM+Summary; 'Stage 3' over Prediction Prompt+LLM+Reason/Confidence. Same height; layout grows taller. Else unchanged.
Input
AutoFigure-Edit (GPT-5.3)
Target
GPT-Image-2
AutoFigure-Edit (GPT-5.4)
AutoFigure-Edit (GPT-5.5)
GPT-Image-1.5
OURS
Reorganize from 1×4 vertical column → 2×2 grid: Lung Size (top-L), Bone Size (top-R), Fat Amount (bottom-L), Emphysema (bottom-R). Remove all percentage labels above thumbnails. Keep thumbnails, arrows, endpoint labels, and panel titles unchanged.
Input
AutoFigure-Edit (GPT-5.3)
AutoFigure-Edit (GPT-5.4)
Target
AutoFigure-Edit (GPT-5.5)
GPT-Image-2
OURS
GPT-Image-1.5
Delete 'θ_0' + ⊕ node between PINN x̂ and ⊖ (upper branch); arrow from x̂ runs straight up into ⊖ (y still enters ⊖ from left). Histogram legend: green 'y − x̂ − θ_0' → 'y − x̂'; blue EBM unchanged. Else unchanged.
Input
AutoFigure-Edit (GPT-5.3)
GPT-Image-1.5 Target
AutoFigure-Edit (GPT-5.5)
AutoFigure-Edit (GPT-5.4)
OURS
GPT-Image-2
Figure 9: Additional qualitative examples. Orange arrows mark notable discrepancies between methods.
17
3-col layout: L=ATOMIC 2020+Triple Template+Relation Categories; C=Data Prep (KG-To-Text→Verbalization)→MLM Pretraining (BERT+RoBERTa)→Eval (TCR+COPA+BCOPA-CE); R=new GLUCOSE+Connectives (Causes/Enables/Motivates/Results in/Causes-Enables)→arrow to Verbalization.
Input
Target
AutoFigure-Edit (GPT-5.3)
AutoFigure-Edit (GPT-5.4)
GPT-Image-2
GPT-Image-1.5
AutoFigure-Edit (GPT-5.5)
OURS
Add dashed 'Tool Collection' around left col. Merge Multi-Agent+Solution Annotation as 'Golden Solution Annotation'. Init→Initial (Query, Annotation); Refined Results→Refined Annotation. Move 'Final Output' under Refined Annotation. GPT-3.5 on outgoing arrow.
Input
Target
AutoFigure-Edit (GPT-5.3)
GPT-Image-2
AutoFigure-Edit (GPT-5.4)
AutoFigure-Edit (GPT-5.5)
GPT-Image-1.5
OURS
Add 4 serif labels above orange internal nodes: 'Fruit' (Apple/Orange/Banana parent, above-left); 'Vegetable' (veg subtree, above-right); 'Green Leaf' (Kale/Spinach/Lettuce parent, above-left); 'Root Vegetable' (Carrot/Beetroot parent, above-right).
Input
AutoFigure-Edit (GPT-5.3)
Target
GPT-Image-2
AutoFigure-Edit (GPT-5.4)
AutoFigure-Edit (GPT-5.5)
GPT-Image-1.5
OURS
Rename table 'Secret Word Embedding Table T'; reorder words. New Unwatermarked/Watermarked passages (troops/Kuma/jungle); Watermarked has highlighted insertions. Right: 'Cosine+Top-K₁ Selection' set, 'Top-K₂ Semantic Filtering' set. Add step '7' on I→Watermarked arrow.
Input
Target
AutoFigure-Edit (GPT-5.3)
AutoFigure-Edit (GPT-5.4)
GPT-Image-2
AutoFigure-Edit (GPT-5.5)
GPT-Image-1.5
OURS
Figure 10: Additional qualitative examples. Orange arrows mark notable discrepancies between methods.
18
Replace 4-box "System Demo" with 2-row layout (User Input ŷ_t / Response R_t with speech bubbles). MA-LLM: add "Memory" to History/State labels, reorder flow. Rename "Big Eyes"→"Big eyes", "Relevance Parameter"→"Vector", arrow "Translate"→"Edit".
Input
AutoFigure-Edit (GPT-5.3)
Target
GPT-Image-2
AutoFigure-Edit (GPT-5.4)
GPT-Image-1.5
AutoFigure-Edit (GPT-5.5)
OURS
Drop "Dental engine" outer box. Restructure inputs as 2-cube dashed group + Down-sample(x4) → Coarse seg/det. Merge Global/Local crop → single Crop block + 3cube dashed group → Refinement/Detection. Rename Merge→"Merge & Zero fill".
Input
Target
AutoFigure-Edit (GPT-5.3)
AutoFigure-Edit (GPT-5.4)
GPT-Image-2
AutoFigure-Edit (GPT-5.5)
GPT-Image-1.5
OURS
Restructure 2-col + bottom captions → 3-col with vertical side captions (blue rotated CCW left, orange rotated CW right). Move narrow black column from farleft to center.
Input
AutoFigure-Edit (GPT-5.3)
Target
GPT-Image-2
AutoFigure-Edit (GPT-5.4)
AutoFigure-Edit (GPT-5.5)
GPT-Image-1.5
OURS
Replace Profile/User → Request/Display connectors with single vertical double-headed arrow from User's feet to E-commerce panel top, centered between Request and Display.
Input
AutoFigure-Edit (GPT-5.3)
AutoFigure-Edit (GPT-5.4)
GPT-Image-1.5
Target
GPT-Image-2
AutoFigure-Edit (GPT-5.5)
OURS
Figure 11: Additional qualitative examples. Orange arrows mark notable discrepancies between methods.
19
In all three panels, prepend a dark-gray "BOS" node to each active (solid-arrow) chain and shorten that chain to four white circles so the total node count stays the same. Add a sequence-index superscript to every node: 𝑦 ! in the top panel; 𝑦 ! (f-row) and 𝑧 " (g-row) in the middle panel; 𝑦 " (frow) and 𝑧 " (g-row) in the bottom panel. Render each lower panel's inactive (dashed) chain — its circles, labels, dashed arrows, and left-margin "f"/"g" letter — entirely in light gray, keeping five white circles and no BOS prefix. Leave the blue dashed diagonals 𝑑" . . 𝑑# unchanged.
Input
Target
OURS
Restructure into a horizontal left-to-right flow: 𝑆 → 𝜑 stack → Relevance → green bar → Select MAS → AM+Init. Replace the inline "Relevancy scores 𝑟𝑗,𝑐 " label with a new red box "Relevance 𝑟𝑗,𝑐 " right of the 𝜑 stack, with arrows from each 𝜑𝑗 and from the Random Subset feeding in and one output to the green bar. Shrink the Random Subset oval and place it between 𝑆 and the new Relevance box. Remove the Input Attribution and Decoder ovals; move AM + Partial Initialization to the right of Select MAS. Rename bottom-of-stack 𝜑𝑗 to 𝜑𝐽 .
Input
Target
OURS
Lowercase titles to "teacher network" / "student network" (drop "(fixed)"); strip "ResBlock" from all 8 rounded rectangles. Left: replace "augmented sample" with a rotated 90° CCW "updated learning materials" label spanning the face images. Center: add "same architecture" with yellow doubleheaded arrow between the words. Right: rename "MSE" to "mean squared error loss"; remove "self distillation..." and the diagonal "CrossEntropy(𝑦′ , 𝑦𝑖 )" arrow; add purple "softmax" box right of 𝑞𝑠 with arrows 𝑞𝑠 → softmax → "cross entropy loss".
Input
Target
OURS
Figure 12: Three editing instances where our model captures the explicit instruction but misses an implicit edit the author also makes. First, nodes are renumbered as requested, but stale nodes that the renumbering leaves behind are not removed. Second, the explicit edit lands but an already-present connection between elements is dropped. Third, a new frame is added but is not rotated to match the surrounding visual style.
are drawn as crude primitives, panel positions sit off-grid, and the added elements do not blend with the source figure’s style. O URS keeps the same SVG structure as AUTO F IGURE -E DIT while matching the surrounding visual register closely enough that the output reads as a continuation of the original figure rather than a sketch on top of it. B.4
Case Study of Imperfect Outputs
Figure 12 highlights three representative instances where our model captures the explicit edit but misses an implicit one that the author also makes. These misses point to limitations the current framework does not yet address, such as reasoning over the figure’s underlying argument and the kind of aesthetic taste a human reader brings to revision. 20