TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

arXiv:2608.16765v1 [cs.CV] 17 Aug 2026

TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation Haoran Wang∗

Chaofan Ma∗

a.museum@sjtu.edu.cn Shanghai Jiao Tong University

chaofanma@sjtu.edu.cn Shanghai Jiao Tong University

Ran Yi†

Lizhuang Ma‡

ranyi@sjtu.edu.cn Shanghai Jiao Tong University

ma-lz@cs.sjtu.edu.cn Shanghai Jiao Tong University

Figure 1: Representative TRACE-Bench cases. Green/red tags indicate satisfied/failed requirements; red boxes localize failures.

Abstract

structural complexity is quantified by the number of operator slots. Building on this formulation, we construct TRACE-Bench, comprising approximately 1,600 evaluation cases across slot counts 1–8, built from 631 formula templates and around 4,000 reference images spanning diverse artistic styles and real-world subjects. The formula structure directly drives an operator-aligned evaluation protocol for per-capability scoring and a diagnostic tree analysis for recursive failure localization. Evaluating 9 leading models reveals insights invisible to holistic scoring: the primary bottleneck lies in disentanglement (𝑔) and attribute binding (⊕) rather than scene-level composition (𝐶), with even the best model scoring only 0.74 on attribute fidelity. Project page: https://amuseum-whr.github.io/TraceBench

Despite recent advances in unified multimodal models for multireference image generation, existing benchmarks remain organized around predefined task types (e.g., “subject composition”), which are ill-suited to this combinatorial setting and lead to fragmented coverage, uncontrolled complexity, and little diagnostic value. Recognizing that diverse multi-reference tasks share a common set of atomic operations, we adopt a capability-oriented perspective and formalize four operators: Anchor (𝑓 ), Disentangle (𝑔), Apply (⊕), and Compose (𝐶). Any multi-reference prompt can then be represented as a compositional formula over these operators, whose ∗ Equal contribution. † Project lead. ‡ Corresponding author.

1

1

Table 1: Operator-aligned evaluation dimensions.

Introduction

Text-to-image generation [5, 44] has achieved remarkable success, yet text alone is often insufficient to convey precise visual details. This limitation has motivated reference-based image generation, which allows users to ground outputs in user-provided visual content. A particularly challenging extension is multi-reference generation, where models must jointly condition on multiple visual elements—a capability essential to real-world workflows such as virtual try-on, group photo composition, and multi-source creative design. Recent models, both proprietary (GPT-Image-1.5 [36], Nano Banana 2 [41]) and open-source (OmniGen2 [56], Emu3.5 [3], QwenImage-Edit [55]), have demonstrated strong capabilities in following instructions that combine entities, attributes, and styles from different sources. On the evaluation side, while existing benchmarks have extensively addressed text-to-image alignment [10, 13] and single-image editing [58, 72], evaluation of multi-reference generation remains in its early stages. Recent efforts like MultiBanana [37], MICONBench [57], and MacroBench [2] have pioneered this direction. However, while these works cover more complex reference settings, they do not fundamentally rethink the underlying evaluation structure. Following the design of earlier generation and editing benchmarks, they still organize test cases around predefined task types (e.g., “object composition”). In the combinatorial setting of multi-reference generation, this task-oriented organization exposes three critical limitations. (1) Incomplete coverage: predefined task categories cannot scale to the full combinatorial space of practical multi-reference usage. (2) No failure diagnosis: holistic task-level scoring cannot pinpoint which specific capability is responsible for a failure; for example, if a model fails to generate a person wearing a referenced outfit, a single score cannot reveal whether the failure stems from misidentifying the person, incorrectly extracting the outfit, or wrongly binding the outfit to the target. (3) Uncontrolled complexity: without a unifying structure that formally characterizes each case, it is difficult to systematically control or compare structural complexity across scenarios. These limitations motivate us to rethink the evaluation of multireference generation from a capability-oriented perspective. Our key observation is that seemingly diverse multi-reference generation tasks share a common set of atomic operations. Consider a prompt such as “Generate a scene containing the blue car from [Image 1] and a coffee cup decorated with the floral pattern worn by the woman on the right in [Image 2], with the cup placed on a table to the lower right of the car.” As illustrated in Fig. 2, fulfilling this prompt requires the model to first anchor the intended blue car from a visually cluttered reference image while preserving its identity-defining characteristics; disentangle the floral pattern worn by the woman on the right, separating this transferable attribute from its original carrier; apply the disentangled pattern to a new carrier (the coffee cup), binding the extracted attribute to a target instance; and finally compose the anchored car together with the modified cup into a coherent scene, with the cup placed on a table to the lower right of the car. We formalize these as four capability operators: Anchor (𝑓 ), Disentangle (𝑔), Apply (⊕), and Compose (𝐶). This capability-oriented formulation naturally overcomes the three limitations of task-oriented evaluation identified

Operator

Dimension

Criteria

𝑓 : Anchor

Identity

𝑔 : Disentangle ⊕ : Apply

Attribute Fidelity Binding

𝐶 : Compose

Composition

Existence; appearance consistency with the reference Presence; consistency with the reference source Carrier integrity; attribute exclusivity; natural integration Coexistence; relation satisfaction; spatial coherence; no duplication or leakage

above. (1) Any complex multi-reference prompt can be expressed as a compositional formula over these operators, elegantly covering the infinite combinatorial space of real-world usage without needing ad-hoc task labels. (2) This formulation enables operatoraligned evaluation: rather than assigning a holistic score, we can precisely diagnose which capability (e.g., identity preservation in 𝑓 or attribute exclusivity in ⊕) caused a failure. (3) The referenceconditioned structural complexity of any test case can be rigorously quantified and systematically controlled by the number of operator slots in its underlying formula. Moreover, common applications such as virtual try-on and group photo layout emerge naturally as specific instantiations of this compositional framework, demonstrating its expressiveness and practical coverage. Building on this formulation, we construct TRACE-Bench, a capability-oriented benchmark for multi-reference image generation. TRACE-Bench comprises approximately 1,600 cases built from 631 formula templates involving around 4,000 reference images, with slot counts ranging from 1 to 8 for systematic control over structural complexity. We collect reference images from multiple complementary sources spanning diverse artistic styles and real-world subjects, and apply structured tagging to extract entities and attributes in a fine-grained hierarchy for formula-driven prompt construction. Each sampled formula template is realized as a natural-language prompt by a vision-language model (VLM), and paired with an operator-aligned evaluation checklist scored by a VLM judge. As summarized in Table 1, this checklist associates each capability with a corresponding evaluation dimension and a set of fine-grained criteria. Fig. 1 shows representative benchmark cases and their operator-aligned evaluation results. Beyond case-level scoring, TRACE-Bench further supports diagnostic tree analysis, which recursively decomposes complex failure cases into simpler sub-cases to identify the responsible source of failure. We evaluate 9 leading proprietary and open-source models on TRACE-Bench. Our operator-level analysis yields two key insights not captured by conventional holistic scoring. First, the primary bottleneck in current models lies in attribute disentanglement (𝑔) and attribute binding (⊕) rather than scene-level composition (𝐶), indicating that precise reference transfer remains substantially harder than plausible scene arrangement. Second, anchor difficulty is driven more by the number of entities in the reference image than by formula slot count, suggesting that reference-image clutter, not task structure, is the dominant source of error. Our contributions are summarized as follows: 2

Figure 2: A multi-reference request in TRACE-Bench is progressively resolved through Anchor, Disentangle, Apply, and Compose. • We propose a capability-oriented formulation that decomposes reference-based image generation into four atomic operators, with a compositional formula for systematically characterizing diverse multi-reference settings. • We construct TRACE-Bench, a benchmark of approximately 1,600 cases with slot-based complexity control (slot 1–8), built from 631 formula templates and around 4,000 reference images. • We design an operator-aligned evaluation protocol and a diagnostic tree analysis method that enable fine-grained, per-capability assessment and failure localization. • We benchmark 9 leading models and reveal that the primary bottleneck lies in attribute disentanglement and attribute binding rather than scene-level composition, among other insights invisible to holistic scoring.

GPT-Image-1.5 [36] and the Nano Banana series [7, 40, 41] demonstrate strong multi-reference capabilities. On the open-source side, OmniGen2 [56], BAGEL [4], and Emu3.5 [3] adopt unified architectures that jointly handle understanding and generation; FLUX.1 Kontext [1], Qwen-Image-Edit [55], and FireRed Image Edit [48] support instruction-based generation and editing conditioned on reference images; DeepGen [50] and UniReason [51] emphasize lightweight and reasoning-centered unification of generation and editing. As these models grow increasingly capable, how to rigorously evaluate their multi-reference abilities at a fine-grained capability level remains an open challenge.

2.2

Multi-Reference Generation Benchmarks

Extensive benchmarks already exist for text-to-image alignment [10, 12, 13, 23] and single-image editing [15, 58, 72], yet evaluation of multi-reference generation remains nascent. MultiBanana [37] scales evaluation to 8 reference images and introduces difficulty factors such as domain mismatch and rare concepts. MICON-Bench [57] defines six compositional tasks for multi-image context generation. OmniContext [56] introduces 8 task categories for in-context generation, though its scope is limited to relatively simple subject-centric compositions. MacroBench [2] provides 4,000 samples across four task dimensions with up to 10 references. Despite expanding the scope and scale of multi-reference evaluation, these benchmarks share a common design philosophy: organizing test cases around predefined task types or surface-level difficulty factors, and assessing results with holistic scores or coarse metrics such as FID [11] and CLIP similarity [39]. Although recent evaluation practices have advanced towards VLM-as-a-judge protocols [18] and structured checklist questions [23, 54], the underlying benchmark design still lacks two critical properties: it cannot systematically control evaluation complexity across structurally different cases, and it cannot localize failures to specific capability dimensions or distinguish standalone weaknesses from cross-reference interference. TRACE-Bench addresses both gaps by decomposing multi-reference generation into four atomic capability operators and using their compositional structure as the

2 Related Work 2.1 Reference-Based Image Generation Early text-to-image models [42, 44] generate images purely from text prompts, offering limited control over fine-grained visual details. To address this, reference-based methods enable users to condition generation on visual examples. DreamBooth [46] and Textual Inversion [9] fine-tune or learn embeddings from a small set of reference images to capture subject identity. ControlNet [70] and IP-Adapter [67] introduce auxiliary conditioning branches that accept spatial or semantic signals from a single reference image without fine-tuning. While these methods achieve strong results in single-reference settings, they are not inherently designed for multi-reference generation, where inputs from several images must be jointly processed. Early multi-subject methods such as Custom Diffusion [19] and FastComposer [60] extend personalization to multiple concepts but still require per-subject optimization or rely on localized attention mechanisms tied to a fixed set of subjects. More recently, unified multimodal models have emerged that natively support interleaved image-text inputs, enabling flexible multi-reference generation within a single forward pass. Proprietary systems such as 3

3.3

unified basis for benchmark construction, operator-aligned evaluation, and diagnostic failure analysis.

3 TRACE-Bench 3.1 Overview Evaluating multi-reference image generation requires both diverse test cases and a structured representation of what each case demands. TRACE-Bench therefore couples capability-oriented benchmark construction with operator-aligned evaluation. Sec. 3.2 introduces the four core operators, and Sec. 3.3 defines the compositional formula that structures each prompt. Secs. 3.4 and 3.5 then detail benchmark construction and operator-aligned evaluation.

3.2

Formula Composition

The four operators of Sec. 3.2 give us the atomic vocabulary, but describing how a real case combines them purely in natural language leaves the underlying structure implicit: which references are involved, how they interact, and how difficult the overall case is are all buried inside free-form text. We therefore represent each prompt’s reference-conditioned part as a compositional formula over these operators. Making this structure symbolic turns each case into a shared backbone that the rest of TRACE-Bench builds on: its operator terms can be systematically enumerated to form a template space of diverse cases, the number of reference-dependent terms provides a controllable measure of structural complexity, and each operator instance seeds an evaluation question aligned with the corresponding capability. The formula only captures referenceconditioned content, namely which entities are anchored, which attributes are disentangled, where they are applied, and how the resulting contents are composed. Text-only descriptions that do not depend on any reference stay in natural language. We organize the formula from local to global with three levels: entity expressions (single target objects) → scene expressions (compositions of entities) → the complete prompt formula (entire reference-conditioned structure). Entity Expressions. An entity expression 𝐸 describes a target subject in the generated image. It may be an anchored reference entity 𝑓 , a text-specified carrier modified by a disentangled attribute 𝑇𝑒 ⊕𝑔, or an existing entity expression further augmented with additional attributes, written as 𝐸 ⊕ 𝑔. Thus, the entity level answers what each generated subject is and which reference-derived attributes are bound to it. Scene Expressions. A scene expression 𝑆 composes multiple entity expressions through 𝐶 (𝐸 1, 𝐸 2, . . . , 𝐸𝑛 ), optionally together with a relation term. The relation may be specified by text,𝑇rel , or extracted from a reference image, 𝑔rel . When a referenced relation applies only to a subset of entities, we represent that subset as a nested subscene such as 𝐶 (𝐸𝑖 , 𝐸 𝑗 ) ⊕ 𝑔rel , and then compose it into the larger scene. The scene level therefore answers how the target objects coexist and interact. Complete Prompt Formulas. A complete prompt formula 𝐹 further augments the scene expression 𝑆 with optional global reference conditions, such as style, lighting, layout, or color tone, denoted by
𝑔global . For example, 𝐹 = 𝐶 𝐶 (𝑓1, 𝑇𝑒 ⊕𝑔1 ) ⊕𝑔rel, 𝑓2 ⊕𝑔2 ⊕𝑔3 ⊕𝑔global . Here, the inner composition groups two entities (𝑓1 and 𝑇𝑒 ⊕ 𝑔1 ) under 𝑔rel , the outer composition combines this sub-scene with another attribute-modified entity (𝑓2 ⊕ 𝑔2 ⊕ 𝑔3 ), and the final global term applies a scene-level reference condition. We use this formula as the canonical structure for benchmark construction and evaluation. For readability, Fig. 4 also presents the same formula as a noun-based expression while preserving the underlying structure.

Capability Decomposition

A key difficulty in benchmarking multi-reference image generation is that holistic scores can obscure fine-grained task failures [10], while general perceptual quality can diverge from source-conditioned validity [45], making it hard to identify whether a failure comes from poor image generation or incorrect use of the references. To trace a model’s reference ability more explicitly, we decompose it into four core capabilities: Anchor, Disentangle, Apply, and Compose. Let 𝐼 denote a reference image, 𝑒 an entity in a reference image, E a referenced entity set, and 𝑎 a referenced attribute. We use 𝑇𝑒 to denote an entity specified in the text prompt. • Anchor 𝑓 (𝐼, 𝑒): locating a specific entity 𝑒 in reference image 𝐼 and preserving its identity-defining visual information in the generated image. For example, 𝑓 (𝐼 1, person) denotes the person in Image 1 as the target entity to preserve. • Disentangle 𝑔(𝐼, E, 𝑎): extracting a referenced attribute 𝑎 from entity set E in image 𝐼 , while decoupling it from irrelevant properties. Here, E may contain a single entity or multiple entities, depending on the attribute type. For example, in Case 1 of Fig. 4, 𝑔(𝐼 1, {robe}, pattern) denotes extracting only the decorative pattern on the robe in Image 1, while discarding the robe’s shape and the identity of the camel wizard wearing it. • Apply ⊕: binding a disentangled attribute to a designated entity. The designated entity may be either an anchored entity 𝑓 (·) from reference images or an entity specified in the text prompt. For example, 𝑇𝑒 ⊕ 𝑔1 denotes applying the extracted attribute 𝑔1 to the text-described entity 𝑇𝑒 (e.g., applying the running pose of the man from Image 1 to a robot). • Compose 𝐶 (·): arranging multiple referenced or text-specified contents into a coherent scene, optionally under additional relational constraints. For example, 𝐶 (𝑓1, 𝑇𝑒 ⊕ 𝑔1 ) ⊕ 𝑔rel denotes composing the anchored entity 𝑓1 with a text-described entity 𝑇𝑒 modified by 𝑔1 , while further enforcing a referenced relation 𝑔rel = 𝑔(𝐼 3, {𝑒𝑖 , 𝑒 𝑗 }, 𝑎 rel ) between them. If the desired relation is specified in the text prompt rather than referenced from an image, we denote it by 𝑇rel .

3.4

Benchmark Construction

The remaining question is how to instantiate many diverse formulas into concrete benchmark prompts. To this end, we design a structured construction pipeline that combines multi-source image curation, structured tagging, balanced sampling, formula-template sampling, and prompt generation. Fig. 3 illustrates the overall pipeline, and we describe its stages below.

These four operators form the atomic capability space of multireference generation, but a real prompt typically nests several of them at once. We next organize such nested structure into a compositional formula. 4

Figure 3: Overview of the benchmark construction pipeline. Candidate images are first collected and filtered from multiple sources, then annotated through structured tagging. The tagged pool is then balanced through source-wise sampling and augmented with synthetic data. It is subsequently used for formula-template sampling and prompt construction. Image Collection and Filtering. We collect candidate reference images from three complementary sources: Danbooru2025 [49], a large-scale anime and illustration dataset with rich stylistic diversity and well-defined character designs; LAION-2B-en-Aesthetics [20], a subset of LAION-5B covering diverse internet image-text data; and cc12m-4mp-realistic [35], a human-focused subset of Conceptual Captions that strengthens coverage of real human subjects. We then apply source-specific filtering: for Danbooru, we retain only images with score > 26.15 that satisfy the safety filter; for LAION, we retain only samples with aesthetic > 6.5 and similarity > 30. After filtering, we sample approximately 50,000 candidate images spanning diverse artistic styles and visual themes. Structured Tagging. Each image is annotated by a Gemini-2.5pro-based tagging pipeline that summarizes foreground entities and their associated attributes in a structured form. Entities are assigned category labels from a predefined ontology, including Human, Animal, Object, Food, Clothing, Transportation, Structure, and Text. Attributes are organized into four layers—Appearance, Form, Dynamics, and Global—each further divided into finer-grained subcategories. In addition, each entity is associated with a grounding phrase for localizing it in the image, and each attribute is accompanied by a short textual description to facilitate downstream prompt construction. Balanced Sampling. Since category and attribute distributions differ substantially across sources, we perform source-wise balanced sampling to improve long-tail coverage. For each source, each candidate image is represented by a feature vector x𝑖 derived from the tagging results. We then greedily select samples according to 𝑖 ★ = arg max x𝑖⊤ w (𝑡 ) , 𝑖

𝑤 𝑗(𝑡 ) =

1 1 + 𝑐 𝑗(𝑡 )

,

approximately 4,000 images (about 8% of the candidate pool), after which full manual image-quality inspection retains 3,839 images. We further augment the pool with about 200 synthetic samples generated by Nano Banana Pro to supplement rare cases. Structural Complexity Control. Each benchmark prompt is associated with a compositional formula, and we use its slot count as a controllable measure of structural complexity: slot(𝐹 ) := |𝑓 | + |𝑔|,

(2)

where |𝑓 | and |𝑔| count anchored and disentangled terms in 𝐹 , respectively; slot count measures formula structure rather than fully determining case difficulty. TRACE-Bench covers slots 1–8, from simple single-reference to highly compositional cases. For example, formula 𝐶 (𝑓1, 𝑇𝑒 ⊕ 𝑔1 ⊕ 𝑔2 ) ⊕ 𝑔global is a slot-4 case. Template Sampling. We first construct each benchmark instance as a formula template composed of the operators defined above. In the standard case, both 𝑓 and 𝑔 are sampled from the structured tagging results. To better cover practical applications, we further introduce two special designs in which an anchored instance 𝑓 is used as an attribute-like reference term 𝑔: attachment reference (𝑔attach ), where a referenced instance serves as an attachable component of another entity, and IP-style reference (𝑔ip ), where the holistic design identity is transferred to another entity. Across all slot levels, we sample from the template space under controlled distributions. Prompt Generation. Each sampled template is paired with tagged reference images and fed into a VLM (Gemini-2.5-Pro) through a customized prompting interface, which realizes it as a naturallanguage prompt. We require that the resulting prompt describe a coherent scene, clearly bind each reference to a specific target, and use every referenced image at least once. For quality control, we first use GPT-5.4 to filter out 4.3% of the constructed prompts. We then manually inspect the remaining cases and remove another 9%. For each benchmark prompt, we additionally construct a text-only

(1)

where c (𝑡 ) denotes the accumulated feature counts of the selected subset at iteration 𝑡. This helps to yield a more balanced reference pool while preserving visual diversity. Balanced sampling selects 5

Figure 4: Representative operator-aligned evaluation examples from TRACE-Bench. Each row shows one benchmark case with its references, brief prompt, compositional formula, readable expression, generated result, and operator-level analysis. The examples illustrate how formula terms are mapped to capability-specific checks and how failures such as identity mismatch, duplication, and leakage-induced mis-binding can be localized. counterpart in which all image-referenced descriptions are replaced by textual ones, providing a no-reference baseline. Benchmark Statistics. For the general benchmark, we construct 180 cases for each slot level, and further include several applicationspecific cases. In total, the benchmark contains approximately 1,600 cases, built from 631 distinct formula templates and involving around 4,000 reference images.

3.5

Case-level scores are then obtained by aggregating the normalized scores across all operator instances in the case. Diagnostic Tree Analysis. Diagnostic-tree decomposition reverses the local-to-global formula construction introduced in Sec. 3.3. Starting from the complete formula at the root, we progressively remove full-prompt-level terms to recover the underlying scene expression, separate the scene into entity expressions, and simplify multi-attribute bindings until each leaf contains a single anchored entity 𝑓 or an atomic attribute transfer 𝑇𝑒 ⊕ 𝑔. Whenever referenceconditioned content is removed, it is replaced with a corresponding text-only description to preserve the original prompt context. The resulting sub-cases form a tree. Evaluating and comparing its nodes allows us to localize the source of a failure observed at the root, as illustrated by the decomposition of a slot-4 formula in Fig. 5. The complete construction rules are detailed in Appendix C.2.

Evaluation Protocol

Operator-Aligned Question Generation. Our formula-based construction explicitly grounds each referenced term to its source image, enabling evaluation questions to be derived automatically from the formula structure. Rather than assigning a single holistic score to the generated result, we decompose evaluation into operator-aligned question sets following Table 1. Specifically, anchor (f) evaluates entity existence and consistency; disentangle (g) evaluates attribute existence and consistency; apply (⊕) evaluates binding correctness and integration quality; and compose (C) evaluates compositional coherence and the absence of anomalies such as duplication or leakage. VLM-Based Judging. A representative evaluation example is shown in Fig. 4. For each benchmark sample, we assess referenced entities with 𝑓 , referenced attributes with 𝑔, attribute application with ⊕, and scene composition with 𝐶. All questions are scored in a binary manner by a VLM judge (Gemini-2.5-Pro), which receives the reference images, the generated image, and the operator-aligned question set, and outputs a pass/fail decision for each question. Evaluation Metrics. For each operator instance, we instantiate a set of fine-grained evaluation questions that assess complementary aspects of the same capability. Their scores are normalized such that the aggregate contribution of each operator instance equals 1.

Figure 5: Representative diagnostic tree analysis case. 6

Table 2: Overall results on TRACE-Bench averaged over slots 1–8. Avg. denotes the mean of the four operator-aligned metrics. Best results are in bold and second-best results are highlighted with a gray background. Model

Anchor (f)

Disentangle (g)

Apply (⊕)

Compose (C)

Avg.

CLIP Sim

GPT-Image-1.5 Nano Banana Nano Banana 2 Nano Banana Pro

0.7649 0.7650 0.7724 0.7488

0.6890 0.6786 0.7384 0.7148

0.7541 0.7631 0.7989 0.7869

0.9259 0.8975 0.9100 0.9214

0.8118 0.7981 0.8205 0.8172

0.2969 0.2867 0.2944 0.2962

Emu3.5 FireRed Image Edit 1.1 [6, 48] Qwen-Image-Edit-2509 Qwen-Image-Edit-2511 OmniGen2 [56]

0.6587 0.6348 0.5210 0.6009 0.5635

0.4982 0.4703 0.3755 0.4097 0.3719

0.5434 0.4258 0.3282 0.3742 0.3195

0.7871 0.7218 0.7627 0.7776 0.7070

0.6561 0.5889 0.5483 0.5858 0.5348

0.2917 0.2603 0.2742 0.2758 0.2693

4 Experiments 4.1 Experimental Setup

duplication. In the Emu3.5 case, the referenced woman, cat, hat, visual effects, and background are all present in the scene, but the referenced relation and attribute application both fail because of source leakage. These examples show that TRACE-Bench can localize specific failure modes rather than collapsing them into a single overall judgment.

Baselines. We compare against 9 representative baselines, including 4 proprietary models and 5 open-source models. The proprietary models are GPT-Image-1.5 [36], Nano Banana [7], Nano Banana Pro [40], and Nano Banana 2 [41]. The open-source models are Emu3.5 [3], FireRed Image Edit 1.1 [6, 48], Qwen-Image-Edit [38, 55] with two released versions (2509 and 2511), and OmniGen2 [56]. Evaluation Setup. Unless otherwise specified, all reported scores are computed on the full benchmark. Operator-aligned evaluation uses Gemini-2.5-Pro as the VLM judge. We retain a text-only prompt for each case and additionally report text-image similarity computed by CLIP ViT-L/14 [39] as a supplementary metric.

4.2

Overall Evaluation Figure 6: Anchor performance versus template slot count (left) and reference-image entity count (right).

Using the operator-aligned checklist in Sec. 3.5, we evaluate all baselines on TRACE-Bench. Table 2 reports the overall results. Proprietary models consistently outperform open-source baselines, with Nano Banana 2 achieving the best average score. However, the task is still far from solved. Since each operator score is normalized to an ideal value of 1, even the best model remains well below saturation: 0.7724 on anchor, 0.7384 on disentangle, 0.7989 on apply, and 0.9100 on compose. The largest gaps appear on disentangle and apply, namely 𝑔 and ⊕. Even the strongest models remain far below 1 on these dimensions, showing that correct attribute extraction and target assignment are still the main bottlenecks. Composition is relatively stronger: GPT-Image-1.5 reaches 0.9259 on 𝐶. Anchor is also more stable, but the best score is still only 0.7724. Among open-source models, Emu3.5 performs best overall. QwenImage-Edit-2511 improves over Qwen-Image-Edit-2509 on all four metrics. Still, all open-source baselines remain substantially behind the leading proprietary systems, especially on 𝑔 and ⊕. CLIP similarity follows a similar ranking trend, but it is less sensitive to whether the referenced content is transferred to the correct target. Qualitative Case Analysis. Fig. 4 shows two representative cases from TRACE-Bench: one generated by Nano Banana and the other by Emu3.5. These examples illustrate the value of such fine-grained evaluation. In the Nano Banana case, the generated image appears plausible overall, but the analysis reveals an identity mismatch for the referenced woman and a composition error caused by hat

Anchor Under Different Difficulty Factors. To illustrate how different sources of difficulty can affect a specific capability, we take anchor as an example and compare its performance by slot level and by the number of entities in the reference image. As shown in Fig. 6, anchor performance varies only weakly across slot levels, but declines more clearly as the reference image contains more entities. This suggests that, for anchor, reference-image complexity is a more direct source of difficulty than slot count alone. More broadly, it highlights the value of operator-level analysis in TRACE-Bench: even when two cases have similar overall structural complexity, they may differ substantially in the difficulty of a specific capability.

4.3

Application-Oriented Analysis

During formula-template sampling and prompt construction, we observe that many common applications can be naturally expressed within our compositional framework. Rather than defining them as separate task types, we treat them as particular instantiations of the same core operators. Figure 7 illustrates two representative examples: virtual try-on and group-photo layout. Virtual Try-On. Virtual try-on binds one or more clothing-related references to a target person: 𝑓 (person) ⊕ 𝑔attach,1 ⊕ 𝑔attach,2 ⊕ · · · . 7

(3)

Figure 7: Formula abstractions and representative results for virtual try-on (left) and group-photo layout (right). This pattern covers different numbers and types of garments. In the left example of Fig. 7, the successful result preserves both the target person and the referenced clothing, whereas the failed result transfers incorrect garment attributes. Group Photo Layout. Group-photo layout composes multiple anchored subjects under a shared layout reference: 𝐶 (𝑓1, 𝑓2, . . . , 𝑓𝑛 ) ⊕ 𝑔layout .

step at which it passes and attribute the failure to the referenceconditioned component removed at that step. Instances whose outcomes remain unchanged throughout the tree are categorized as Stable Success or Persistent Failure. As shown in Table 3, joint-composition interference is the dominant localized source for 𝑓 , 𝑔, and ⊕. This indicates that Emu3.5 often preserves isolated reference content but loses it when multiple reference-conditioned entities are composed. In contrast, failures in 𝐶 are most frequently localized to global-reference interference (29.0%), suggesting that global style or scene constraints are a major source of compositional disruption. Overall, most localized failures arise from interactions introduced at higher compositional levels rather than from persistent failure on isolated reference units.

(4)

The same pattern extends to different group sizes and spatial arrangements. As shown on the right, satisfying the shared layout may cause identity loss or subject duplication. Table 3: Distribution of diagnostic outcomes across 200 cases. Outcome / Source

𝑓

𝑔

⊕

𝐶

Stable Success 27.9 27.6 38.9 45.2 Persistent Failure 4.7 13.2 7.3 12.9 Global Reference 14.0 10.5 9.1 29.0 Joint Composition 48.8 43.4 41.1 9.7 Relation / Attribute 4.7 5.3 3.6 3.2

4.4

Overall

5

33.7 9.8 13.7 38.5 4.4

Conclusion

We presented TRACE-Bench, a capability-oriented benchmark for multi-reference image generation. Rather than organizing evaluation around predefined task types, we decompose multi-reference generation into four atomic operators (Anchor, Disentangle, Apply, Compose). Their compositional structure serves as the unified basis for benchmark construction, operator-aligned evaluation, and diagnostic tree analysis. Evaluation of 9 leading models reveals that the primary bottleneck lies in attribute disentanglement (𝑔) and binding (⊕) rather than scene-level composition (𝐶), and that diagnostic tree analysis can effectively separate cross-reference interference from standalone capability deficits. In the future, we hope the capability-oriented formulation can serve not only as an evaluation tool but also as a guide for targeted model improvement.

Diagnostic Tree Analysis

In Fig. 1, the Qwen-Image-Edit-2511 example fails on the woman and the vintage car. To better understand these errors, we further analyze this case with the diagnostic tree in Fig. 5, where we decompose the original formula into simpler sub-cases and evaluate the model on each node. Two distinct patterns emerge. For 𝑓2 , the woman is correct in 𝑁 1 and 𝑁 2𝑏 , but becomes inconsistent in 𝑁 0 . This indicates that 𝑓2 itself is not the problem; rather, identity information is lost when the global style constraint is introduced. For 𝑓1 , the vintage car already changes in 𝑁 2𝑎 , showing that this anchor is intrinsically harder. Yet it is preserved in 𝑁 1 , suggesting that jointly referencing the interacting subject 𝑓2 can reinforce its identity. This example shows that the diagnostic tree can distinguish two failure sources within the same case: style-induced interference for 𝑓2 , and intrinsic anchor difficulty for 𝑓1 , which is partially alleviated under composition. Aggregate Diagnostic Patterns. We construct diagnostic trees for 200 Emu3.5 cases, generate an image at every node, and score each node using the operator-aligned evaluation. For each operator instance that fails at the root, we identify the first decomposition

Acknowledgments This work was supported by the National Natural Science Foundation of China (No. 62302297, 72192821, 62472282, 62272447, 62472285), the Fundamental Research Funds for the Central Universities (project number: YG2023QNA35), YuCaiKe [2023] Project Number: 231111310300.

References [1] Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. 2025. FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space. arXiv preprint arXiv:2506.15742 (2025). 8

[22] Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. 2022. Language-driven Semantic Segmentation. In International Conference on Learning Representations (ICLR). [23] Ouxiang Li, Yuan Wang, Xinting Hu, Huijuan Huang, Rui Chen, Jiarong Ou, Xin Tao, Pengfei Wan, Xiaojuan Qi, and Fuli Feng. 2026. Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?. In International Conference on Learning Representations (ICLR). [24] Jinxiang Liu, Chen Ju, Chaofan Ma, Yanfeng Wang, Yu Wang, and Ya Zhang. 2023. Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation. arXiv preprint arXiv:2307.13236 (2023). [25] Jinxiang Liu, Yu Wang, Chen Ju, Chaofan Ma, Ya Zhang, and Weidi Xie. 2024. Annotation-Free Audio-Visual Segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). [26] Wentao Liu, Chaofan Ma, Yuhuan Yang, Weidi Xie, and Ya Zhang. 2022. Transforming the Interactive Segmentation for Medical Imaging. In Medical Image Computing and Computer Assisted Intervention (MICCAI). [27] Chaofan Ma, Zhenjie Mao, Yuhuan Yang, Fanqin Zeng, Yue Shi, Yingjie Zhou, Xiaofeng Cao, and Jiangchao Yao. 2026. Reason, Then Re-reason: Cross-view Revisiting Improves Spatial Reasoning. In International Conference on Machine Learning (ICML). [28] Chaofan Ma, Qisen Xu, Xiangfeng Wang, Bo Jin, Xiaoyun Zhang, Yanfeng Wang, and Ya Zhang. 2021. Boundary-Aware Supervoxel-Level Iteratively Refined Interactive 3D Image Segmentation With Multi-Agent Reinforcement Learning. IEEE Transactions on Medical Imaging (2021). [29] Chaofan Ma, Yuhuan Yang, Chen Ju, Yue Shi, Ya Zhang, and Yanfeng Wang. 2025. FreeSegDiff: Annotation-free Saliency Segmentation with Diffusion Models. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). [30] Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Jinxiang Liu, Yu Wang, Ya Zhang, and Yanfeng Wang. 2023. DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery. arXiv preprint arXiv:2303.09813 (2023). [31] Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Ya Zhang, and Yanfeng Wang. 2023. AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation. In Advances in Neural Information Processing Systems (NeurIPS). [32] Chaofan Ma, Yuhuan Yang, Yanfeng Wang, Ya Zhang, and Weidi Xie. 2022. Openvocabulary Semantic Segmentation with Frozen Vision-Language Models. In British Machine Vision Conference (BMVC). [33] Zhenjie Mao, Yuhuan Yang, Chaofan Ma, Dongsheng Jiang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. 2025. SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation. In Advances in Neural Information Processing Systems (NeurIPS). [34] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In European Conference on Computer Vision (ECCV). [35] Open Diffusion AI. 2025. cc12m-4mp-realistic. https://huggingface.co/datasets/ opendiffusionai/cc12m-4mp-realistic. [36] OpenAI. 2025. GPT Image 1.5 Model. https://developers.openai.com/api/docs/ models/gpt-image-1.5. [37] Yuta Oshima, Daiki Miyake, Kohsei Matsutani, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo, and Hiroki Furuta. 2026. MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [38] Qwen Team. 2025. Qwen-Image. https://github.com/QwenLM/Qwen-Image. [39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning. [40] Naina Raisinghani. 2025. Introducing Nano Banana Pro. https://blog.google/ innovation-and-ai/products/nano-banana-pro/. [41] Naina Raisinghani. 2026. Nano Banana 2: Combining Pro capabilities with lightning-fast speed. https://blog.google/innovation-and-ai/technology/ai/nanobanana-2/. [42] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125 (2022). [43] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feichtenhofer. 2025. SAM 2: Segment Anything in Images and Videos. In International Conference on Learning Representations (ICLR). [44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2] Zhekai Chen, Yuqing Wang, Manyuan Zhang, and Xihui Liu. 2026. MACRO: Advancing Multi-Reference Image Generation with Structured Long-Context Data. arXiv preprint arXiv:2603.25319 (2026). [3] Yufeng Cui, Honghao Chen, Haoge Deng, Xu Huang, Xinghang Li, Jirong Liu, Yang Liu, Zhuoyan Luo, Jinsheng Wang, Wenxuan Wang, Yueze Wang, Chengyuan Wang, Fan Zhang, Yingli Zhao, Ting Pan, Xianduo Li, Zecheng Hao, Wenxuan Ma, Zhuo Chen, Yulong Ao, Tiejun Huang, Zhongyuan Wang, and Xinlong Wang. 2025. Emu3.5: Native Multimodal Models are World Learners. arXiv preprint arXiv:2510.26583 (2025). [4] Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. 2025. Emerging Properties in Unified Multimodal Pretraining. arXiv preprint arXiv:2505.14683 (2025). [5] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. 2024. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. In Proceedings of the 41st International Conference on Machine Learning. [6] FireRedTeam. 2026. FireRed-Image-Edit. https://github.com/FireRedTeam/ FireRed-Image-Edit. [7] Alisa Fortin, Guillaume Vernade, Kat Kampf, and Ammaar Reshi. 2025. Introducing Gemini 2.5 Flash Image, our state-of-the-art image model. https: //developers.googleblog.com/en/introducing-gemini-2-5-flash-image/. [8] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander J. Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. 2023. DataComp: In search of the next generation of multimodal datasets. In Advances in Neural Information Processing Systems (NeurIPS). [9] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. 2023. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. In Proceedings of the International Conference on Learning Representations (ICLR). [10] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. 2023. GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment. In Advances in Neural Information Processing Systems (NeurIPS). [11] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems. [12] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. 2024. ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment. arXiv preprint arXiv:2403.05135 (2024). [13] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2023. T2ICompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation. In Advances in Neural Information Processing Systems (NeurIPS). [14] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. 2024. VBench: Comprehensive Benchmark Suite for Video Generative Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [15] Zhangqi Jiang, Zheng Sun, Xianfang Zeng, Yufeng Yang, Xuanyang Zhang, Yongliang Wu, Wei Cheng, Gang Yu, Xu Yang, and Bihan Wen. 2026. GEditBench v2: A Human-Aligned Benchmark for General Image Editing. arXiv preprint arXiv:2603.28547 (2026). [16] Chen Ju, Haicheng Wang, Jinxiang Liu, Chaofan Ma, Ya Zhang, Peisen Zhao, Jianlong Chang, and Qi Tian. 2023. Constraint and Union for Partially-Supervised Temporal Sentence Grounding. arXiv preprint arXiv:2302.09850 (2023). [17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. 2023. Segment Anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). [18] Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. 2024. VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [19] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. 2023. Multi-Concept Customization of Text-to-Image Diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [20] LAION eV. 2025. laion2B-en-aesthetic. https://huggingface.co/datasets/laion/ laion2B-en-aesthetic. [21] Jie Lei, Tamara L. Berg, and Mohit Bansal. 2021. Detecting Moments and Highlights in Videos via Natural Language Queries. In Advances in Neural Information Processing Systems (NeurIPS). 9

[64] Yuhuan Yang, Chaofan Ma, Zhenjie Mao, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. 2025. MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition. In International Conference on Machine Learning (ICML). [65] Yuhuan Yang, Chaofan Ma, Jiangchao Yao, Zhun Zhong, Ya Zhang, and Yanfeng Wang. 2024. ReMamber: Referring Image Segmentation with Mamba Twister. In European Conference on Computer Vision (ECCV). [66] Yuhuan Yang, Xianwei Zhuang, Yuxuan Cai, Chaofan Ma, Shuai Bai, Jiangchao Yao, Ya Zhang, Junyang Lin, and Yanfeng Wang. 2026. GenMask: Adapting DiT for Segmentation via Direct Mask Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [67] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. 2023. IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models. arXiv preprint arXiv:2308.06721 (2023). [68] Ziyu Ye, Chen Ju, Chaofan Ma, and Xiaoyun Zhang. 2021. Unsupervised Domain Adaption via Similarity-Based Prototypes for Cross-Modality Segmentation. In MICCAI Workshop on Domain Adaptation and Representation Transfer (DART). [69] Fei Zhang, Tianfei Zhou, Boyang Li, Hao He, Chaofan Ma, Tianjiao Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. 2023. Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation. In Advances in Neural Information Processing Systems (NeurIPS). [70] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). [71] Tianjiao Zhang, Chaofan Ma, and Yanfeng Wang. 2024. Tracking the Rareness of Diseases: Improving Long-Tail Medical Detection with a Calibrated Diffusion Model. Electronics (2024). [72] Xiangyu Zhao, Peiyuan Zhang, Kexian Tang, Xiaorong Zhu, Hao Li, Wenhao Chai, Zicheng Zhang, Renqiu Xia, Guangtao Zhai, Junchi Yan, Hua Yang, Xue Yang, and Haodong Duan. 2025. Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing. In Advances in Neural Information Processing Systems (NeurIPS).

[45] Dingyi Rong, Yue Shi, Chaofan Ma, Jiezhang Cao, Zongrui Wang, Zeyu Zhang, Yao Mu, Guangtao Zhai, and Ning Liu. 2026. H2R-Bench: Benchmarking Humanto-Robot Manipulation Video Generation in World Models. arXiv preprint arXiv:2608.13049 (2026). [46] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [47] Yue Shi, Dingyi Rong, Chang Chen, Chaofan Ma, Bingbing Ni, and Wenjun Zhang. 2025. DARF: Depth-Aware Generalizable Neural Radiance Field. Displays (2025). [48] Super Intelligence Team, Changhao Qiao, Chao Hui, Chen Li, Cunzheng Wang, Dejia Song, Jiale Zhang, Jing Li, Qiang Xiang, Runqi Wang, Shuang Sun, Wei Zhu, Xu Tang, Yao Hu, Yibo Chen, Yuhao Huang, Yuxuan Duan, Zhiyi Chen, and Ziyuan Guo. 2026. FireRed-Image-Edit-1.0 Technical Report. arXiv preprint arXiv:2602.13344 (2026). [49] trojblue. 2025. Danbooru 2025 Metadata. https://huggingface.co/datasets/ trojblue/danbooru2025-metadata. [50] Dianyi Wang, Ruihang Li, Feng Han, Chaofan Ma, Wei Song, Siyuan Wang, Yibin Wang, Yi Xin, Hongjian Liu, Zhixiong Zhang, Shengyuan Ding, Tianhang Wang, Zhenglin Cheng, Tao Lin, Cheng Jin, Kaicheng Yu, Jingjing Chen, Wenjie Wang, Zhongyu Wei, and Jiaqi Wang. 2026. DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing. arXiv preprint arXiv:2602.12205 (2026). [51] Dianyi Wang, Chaofan Ma, Feng Han, Size Wu, Wei Song, Yibin Wang, Zhixiong Zhang, Tianhang Wang, Siyuan Wang, Zhongyu Wei, and Jiaqi Wang. 2026. UniReason 1.0: A Unified Reasoning Framework for World Knowledge Aligned Image Generation and Editing. arXiv preprint arXiv:2602.02437 (2026). [52] Haicheng Wang, Chen Ju, Weixiong Lin, Chaofan Ma, Shuai Xiao, Ya Zhang, and Yanfeng Wang. 2025. Contrast-Unity for Partially-Supervised Temporal Sentence Grounding. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). [53] Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. 2022. CRIS: CLIP-Driven Referring Image Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [54] Xinyu Wei, Jinrui Zhang, Zeqing Wang, Hongyang Wei, Zhen Guo, Bairui Li, and Lei Zhang. 2025. TIIF-Bench: How Does Your T2I Model Follow Your Instructions? arXiv preprint arXiv:2506.02161 (2025). [55] Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng, Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang, Yongqiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. 2025. Qwen-Image Technical Report. arXiv preprint arXiv:2508.02324 (2025). [56] Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng, Jiahao Wang, Kun Luo, Bo Zhang, Defu Lian, Xinlong Wang, Zhongyuan Wang, Tiejun Huang, and Zheng Liu. 2025. OmniGen2: Towards InstructionAligned Multimodal Generation. arXiv preprint arXiv:2506.18871 (2025). [57] Mingrui Wu, Hang Liu, Jiayi Ji, Xiaoshuai Sun, and Rongrong Ji. 2026. MICONBench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [58] Yongliang Wu, Zonghui Li, Xinting Hu, Xinyu Ye, Xianfang Zeng, Gang Yu, Wenbo Zhu, Bernt Schiele, Ming-Hsuan Yang, and Xu Yang. 2025. KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models. In Advances in Neural Information Processing Systems (NeurIPS). [59] Dayu Xia, Yue Shi, Yao Mu, Huiting Ji, Chaofan Ma, Yingjie Zhou, Hua Chen, Yang Liu, Jiezhang Cao, and Guangtao Zhai. 2026. RoboProcessBench: Benchmarking Process-Aware Understanding in Vision-Language Robotic Manipulation. arXiv preprint arXiv:2606.13040 (2026). [60] Guangxuan Xiao, Tianwei Yin, William T. Freeman, Frédo Durand, and Song Han. 2025. FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention. International Journal of Computer Vision (2025). [61] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. 2023. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. In Advances in Neural Information Processing Systems (NeurIPS). [62] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. 2022. GroupViT: Semantic Segmentation Emerges From Text Supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [63] Yuhuan Yang, Chaofan Ma, Chen Ju, Fei Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. 2024. Multi-modal Prototypes for Open-World Semantic Segmentation. International Journal of Computer Vision (2024).

A

Task Formulation and Symbolic Representation Details A.1 Comparison with Existing Benchmarks Compared with existing benchmarks, TRACE-Bench differs in three main aspects, as summarized in Table 4. Capability-oriented decomposition. A key difference lies in the principle used to organize the benchmark. Existing benchmarks are mostly decomposed by task categories, such as reference-count settings, predefined subtasks, or long-context task types. In contrast, TRACE-Bench is decomposed by atomic capabilities, namely Anchor (𝑓 ), Disentangle (𝑔), Apply (⊕), and Compose (𝐶). This capability-oriented design allows different generation scenarios to be analyzed within a shared capability space, rather than being treated as isolated task categories. In this sense, TRACE-Bench differs not only in benchmark split, but also in the basis on which benchmark cases are constructed. Compositional case construction with aligned evaluation. A second key difference is the explicit connection between case construction and evaluation. Existing benchmarks typically define cases through task templates, subtasks, or task-specific settings, while the evaluation protocol is designed as a separate layer. In contrast, TRACE-Bench constructs cases from compositional formulas and uses the same underlying structure to define evaluation targets. During construction, the formula specifies how referenceconditioned contents are combined into a benchmark instance. During evaluation, the same structure determines the operator targets and the corresponding checklist items. As a result, benchmark construction and evaluation are explicitly aligned, which makes the overall pipeline more coherent and also makes the meaning of each evaluation item easier to interpret. Richer reference content and harder grounding. A third difference lies in the richness of the reference content and the 10

Figure 8: Benchmark statistics after construction. The left panel shows the distributions of semantic categories and controllable attributes, and the right panel shows the composition of image sources. difficulty of grounding it correctly. Our candidate image pool is designed to contain more diverse and information-rich images, and the tagging system extracts more comprehensive referenceable content from each image. As a result, a single image may provide multiple usable reference contents, rather than serving only as a source for one object or one simple global attribute. This makes both Anchor and Disentangle more challenging: the intended reference may need to be distinguished through language-aligned, attributebased, or prototype representations [22, 31, 32, 62, 63, 69] and then grounded from referring descriptions in multi-object scenes [33, 53, 65], with TRACE-Bench further including same-class distractors. The examples in Fig. 2 illustrate this point: in Steps 1 and 2, the correct reference cannot be identified by a simple noun phrase alone, but instead requires more specific grounding. For this reason, Appendix B.4 further provides detailed prompt realization rules for such cases.

appearance of human or humanoid entities, including length, curliness, bangs, braids, and overall styling. Facial Features refers to visually recognizable facial characteristics such as beard, makeup, eye shape, nose shape, or other salient facial details. Shape captures the overall geometric or morphological form of an entity, especially for objects, creatures, or clothing silhouettes. Dynamics attributes capture transient states or motion-related properties. Action/Pose describes the body configuration or ongoing motion of an instance, such as running, sitting, raising one hand, or leaning forward. Expression is mainly used for humans or humanoid characters, and captures facial states such as smiling, frowning, surprise, or anger. Orientation/Position records how the instance is oriented or spatially placed, such as facing left, side view, front-facing, or lying on a surface. Motion Effect captures visible motion-related effects or dynamic cues, such as splashing water, flying sparks, motion trails, or magical glow produced during an action. Finally, global attributes describe image-level properties that are not naturally attached to a single foreground instance. Style refers to the overall rendering style of the image, such as oil painting, anime, watercolor, or realistic photography. Lighting describes global illumination conditions, such as backlighting, warm indoor light, or strong contrast. Camera/Viewpoint captures the overall photographic perspective, such as close-up, top-down view, side shot, or wide-angle composition. Color Tone describes the overall palette or grading, such as warm-toned, low-saturation, or bluedominant. Layout/Composition refers to the global arrangement of major scene elements. Inter-instance Relation describes explicit relations among multiple instances, such as holding, standing beside, hugging, or facing each other. Overall, this taxonomy is designed to balance transferability, perceptual salience, and annotation stability, while remaining aligned with the operator-based formulation used throughout the benchmark.

B Benchmark Construction Details B.1 Attribute Taxonomy and Tagging Criteria Our attribute taxonomy is organized into four levels: Appearance, Form, Dynamics, and Global. A key design consideration is that human and humanoid entities require substantially finer-grained annotation than ordinary objects. Both in practical applications and in perceptual evaluation, users are typically more sensitive to identity and appearance errors on humans than on other categories. As a result, our taxonomy includes several tags that are especially important for human-centered references, such as hairstyle, facial features, and expression. Appearance attributes describe visible local appearance details of an instance. Specifically, Pattern/Texture refers to repeated or local surface appearance, such as floral prints, stripes, embroidery, or decorative motifs. Material describes what the surface appears to be made of, such as metal, glass, wood, fur, or knitted fabric. Damage/Wear captures visible aging or usage traces, such as scratches, rust, cracks, folds, or worn edges. Font/Text Style is used when the visual identity of text itself is important, including letterform style, stroke shape, and decorative typography. Form attributes describe relatively stable structural or morphologyrelated properties of an instance. Hairstyle describes hair-related

B.2

Structured Tagging Format

Based on the attribute taxonomy above, we organize the tagging result of each reference image into a structured representation with three top-level fields: ent_list, background, and global_tag. Listing 1 shows the overall schema. 11

Benchmark

Split by

Case basis

Eval. alignment

Hard grounding

OmniContext MultiBanana MICON-Bench MacroBench TRACE-Bench (ours)

Tasks Tasks Tasks Tasks Capabilities

Subtasks Reference-count tasks Task templates Long-context tasks Compositional formulas

✗ ✗ ✗ ✓ ✓

✗ ✗ ✓ ✗ ✓

Table 4: Comparison with existing benchmarks. “Split by” indicates the primary principle used to decompose the benchmark. “Case basis” summarizes the main basis used to construct individual benchmark cases. “Eval. alignment” indicates whether the evaluation protocol is explicitly aligned with the benchmark construction logic. “Hard grounding” indicates whether identifying the intended reference content often requires detailed localization descriptions rather than simple noun phrases.

attributes” format. The field background records visually salient background content that is useful for later prompt construction but is not treated as a foreground instance. The field global_tag stores image-level attributes, including salient relations among foreground instances when these relations are useful for later case construction. This structured format follows a simple principle: we first identify meaningful foreground instances, and then attach transferable attributes to them. In this way, the grounding of each attribute remains explicit, and the resulting representation is easier to use in later formula construction. In addition to ordinary fine-grained attributes, the structured format also supports two special transferable types, namely 𝑔attach and 𝑔ip . We place them in this section because they are represented in a more holistic way than standard local tags. 𝑔attach is used for attachment-like transferable content, such as clothing, accessories, or other attached components whose identity should be preserved as a whole during transfer. For example, when transferring the T-shirt worn by a man in Image 1 to a woman in the target image, the clothing item should remain intact, rather than being decomposed into a few isolated local attributes. 𝑔ip is used for holistic IP-style transfer. In such cases, the transferable content is not a single local attribute, but the overall design language of an instance. For example, when generating “a hat in the style of the rabbit police officer in Image 1,” the reference signal includes the characteristic silhouette, color scheme, and iconic motifs of the original design. Overall, this structured tagging format serves as an intermediate representation between raw reference images and later formula instantiation. It preserves explicit grounding at the instance level, while also retaining background information and global attributes when needed.

Listing 1: Schema of the structured tagging format. {

}

"ent_list": [ { "category": "...", "ent_desc": "...", "tag_dict": { "1_Appearance": { "1.1_pattern_texture": ["..."], "1.2_material": ["..."], "1.3_damage_wear": ["..."], "1.4_font": ["..."] }, "2_Form": { "2.1_hairstyle": ["..."], "2.2_facial_features": ["..."], "2.3_shape": ["..."] }, "3_Dynamics": { "3.1_action_pose": ["..."], "3.2_expression": ["..."], "3.3_orientation": ["..."], "3.4_motion_effect": ["..."] } } } ], "background": "...", "global_tag": { "4.1_style": ["..."], "4.2_lighting": ["..."], "4.3_camera_viewpoint": ["..."], "4.4_layout_sequence": ["..."], "4.5_color_tone": ["..."], "4.6_inter_instance_relation": ["..."] }

B.3

The field ent_list contains the foreground instances selected from the image. Each instance is represented by a groundingoriented description ent_desc, a coarse category label category, and a nested attribute dictionary tag_dict. In tag_dict, attributes are grouped into three levels: appearance attributes, form attributes, and dynamics attributes. This design keeps each transferable attribute explicitly attached to the instance it belongs to. Although our formulation separates Anchor and Disentangle, attribute extraction in real images is still naturally instance-based. We therefore represent each image using an “instance + attached

Template Construction and Sampling Strategy

This section provides additional details on how template construction is built upon the local-to-global formula levels introduced in the main paper. Local-to-global formula levels. At the entity level, the formula describes individual reference-conditioned targets. At the scene level, multiple targets are composed through 𝐶, optionally together with relational terms. At the full-prompt level, the scene expression may be further augmented with a global reference term such as 12

Slot

Example formula templates

𝑇𝑒 ⊕ 𝑔;

1

𝐶 ( 𝑓 , 𝑇𝑒 ⊕ 𝑔) ;

2 3

𝑓 ⊕ 𝑔 ⊕ 𝑔;

4

𝐶 (𝑇𝑒 ⊕ 𝑔, 𝑓 , 𝑓 ) ⊕ 𝑔global ;

5

𝑓 ⊕ 𝑔;

𝐶 ( 𝑓 , 𝑇𝑒 ⊕ 𝑔) ⊕ 𝑔global ;

𝐶 (𝑇𝑒 ⊕ 𝑔, 𝑓 , 𝑓 , 𝑓 ) ⊕ 𝑔global ;

𝑓 𝑇𝑒 ⊕ 𝑔 ⊕ 𝑔;

𝐶 (𝑓 , 𝑇𝑒 ⊕ 𝑔, 𝑇𝑒 ⊕ 𝑔)

𝐶 (𝑇𝑒 ⊕ 𝑔, 𝑇𝑒 ⊕ 𝑔, 𝑓 ) ⊕ 𝑔global ;

𝐶 (𝑓 , 𝑓 ⊕ 𝑔) ⊕ 𝑔global

𝐶 (𝐶 (𝑇𝑒 ⊕ 𝑔, 𝑓 ) ⊕ 𝑔rel , 𝑓 ) ⊕ 𝑔global ;

𝐶 (𝐶 ( 𝑓 , 𝑓 ) ⊕ 𝑔rel , 𝑇𝑒 ⊕ 𝑔 ⊕ 𝑔) ⊕ 𝑔global ;

6

𝐶 (𝑓 , 𝑓 )

𝐶 (𝑓 , 𝑓 ) ⊕ 𝑔global ;

7

𝐶 (𝐶 ( 𝑓 , 𝑓 , 𝑓 ) ⊕ 𝑔rel , 𝑓 ⊕ 𝑔) ⊕ 𝑔global ;

8

𝐶 (𝐶 (𝑇𝑒 ⊕ 𝑔, 𝑓 ⊕ 𝑔) ⊕ 𝑔rel , 𝑇𝑒 ⊕ 𝑔, 𝑓 ⊕ 𝑔) ⊕ 𝑔global ;

𝐶 (𝐶 (𝑇𝑒 ⊕ 𝑔, 𝑇𝑒 ⊕ 𝑔) ⊕ 𝑔rel , 𝑓 ) ⊕ 𝑔global

𝐶 (𝑇𝑒 ⊕ 𝑔, 𝑓 , 𝑓 ⊕ 𝑔 ⊕ 𝑔) ⊕ 𝑔global

𝐶 (𝐶 (𝑇𝑒 ⊕ 𝑔, 𝑓 , 𝑓 ) ⊕ 𝑔rel , 𝑓 ⊕ 𝑔) ⊕ 𝑔global 𝐶 (𝐶 (𝑓 , 𝑓 ) ⊕ 𝑔rel , 𝑇𝑒 ⊕ 𝑔, 𝑇𝑒 ⊕ 𝑔, 𝑓 ) ⊕ 𝑔global ⊕ 𝑔global

Table 5: Example formula templates across different slot levels.

𝑔global , yielding the final compositional formula used to instantiate a benchmark prompt. Scope of the formula. The proposed formula is not intended to represent the full natural-language prompt. Instead, it only describes the combination structure of reference content. We assume that the text-to-image backbone can already handle ordinary textual instructions reasonably well, and therefore focus only on the part of the prompt that involves reference-conditioned content. As a result, pure text-only modifications are not explicitly included in the formula unless they participate in a reference-dependent operation through 𝑔. For example, ordinary editing instructions such as changing a color are treated as textual modifications rather than part of the formula structure. Structural constraint. In practice, the formula structure is deliberately kept simple. The scene representation uses at most two nested levels of the composition operator 𝐶. The inner level is mainly used to express reference-grounded relational composition, such as cases involving 𝑔rel , while the outer level is used to form the complete scene expression. Text-specified relations are omitted from the formula unless they are necessary for disambiguation. As illustrated by the formula below, this assumption is sufficient for the vast majority of benchmark cases while keeping the template space interpretable and manageable.

  𝐹 = 𝐶 𝐶 (𝑓1, 𝑇𝑒 ⊕ 𝑔1 ) ⊕ 𝑔rel, 𝑓2 ⊕ 𝑔2 ⊕ 𝑔3 ⊕𝑔global . | {z }

Figure 9: A Cacao-pod Case

both simple reference transfer cases and more structured multireference compositions involving relational constraints and global reference attributes. Representative templates. To give a concrete picture of the resulting distribution, Table 5 lists some example formula templates for slots 1–8 after template construction and sampling. As the slot number increases, the dominant patterns gradually shift from simple anchored entities or single-attribute transfer to more complex compositions of the same small set of atomic operators.

B.4

(5)

inner composition

|

{z

Prompt Realization Format

Given a compositional formula and its associated reference images, we further convert them into a natural-language reference prompt for image generation. This step is particularly important in our setting because many images in the reference pool are visually complex, and the source objects for Anchor or Disentangle often require relatively detailed grounding descriptions. At the same time, we avoid using overly artificial placeholders or highly specialized prompt markup, since such forms may be unnatural for image generation models. We therefore adopt a natural-language prompt realization format that keeps the prompt fluent while preserving explicit source-target grounding. Our realization format contains two parts. The first paragraph describes the target scene in natural language, where each referenceconditioned target object is introduced as a readable referring expression such as (’man_A’). The second paragraph explicitly specifies the source-target assignments for all reference operations, including the reference source image, the source content to be extracted, and the target object in the realized prompt. In this way, the main prompt remains natural, while the reference mapping

}

outer composition

Template construction and sampling. Based on the above representation, we sample valid templates across all slot levels under controlled distributions. We explicitly control the proportions of templates containing different numbers of 𝑔rel and 𝑔global terms, preventing the sampled cases from collapsing to a few repeated flat structures while maintaining diverse relational and global-reference patterns. Design goal. This design serves two purposes. First, it increases reference-conditioned structural complexity with slot number in a controlled and interpretable way. Second, it prevents the benchmark from being dominated by a narrow set of repeated formula patterns, especially at larger slots. As a result, the final benchmark includes 13

remains explicit. A real example is shown below (corresponding to Fig. 9).

the generated questions easier to trace back to the original prompt representation.

Listing 2: Example of natural-language prompt realization.

C Evaluation and Diagnostic Details C.1 Reliability of the Evaluation Protocol

Generate a new scene. In a whimsical fantasy bakery, a special enchanted confection is displayed on a velvet cushion. This item is a unique ('cacao_pod_A'), which looks like a real cacao pod but has been magically altered.

To assess whether our evaluation depends on the choice of VLM judge, we conduct a human audit involving Gemini-2.5-Pro (G25P), Gemini-3-Pro (G3P), GPT-5.1, and GPT-5.4. We sample 200 benchmark cases and generate each case with both Nano Banana 2 and Emu3.5, yielding 400 outputs. For each output, human annotators answer the same binary checklist questions used by the VLM judges. We aggregate the checklist decisions associated with each operator and normalize the resulting operator-level scores. Pearson and Spearman measure linear and rank correlation, respectively, between VLM and human operator scores; MAE measures their normalized score difference, while agreement is the percentage of individual checklist decisions that match the human annotations. As shown in Table 7, all four VLM judges achieve 85.4–88.4% checklist-level agreement with human annotations and exhibit broadly comparable operator-level correlations and errors. This indicates that the checklist-based evaluation is not tied to a single judge. We use G25P for full-benchmark evaluation because it provides a practical trade-off between reliability and evaluation cost. The ensemble further improves Pearson correlation and agreement, providing a higher-confidence option when additional evaluation cost is acceptable.

('cacao_pod_A') references [the cacao pod being cut in the man's hands in Image 1]. ('cacao_pod_A') references [the fairy in the jar in Image 2].

B.5

Checklist Construction from Structured Prompts

Our evaluation checklist is generated from the structured information preserved during prompt realization. When converting a symbolic case into a natural-language reference prompt, we retain the corresponding source–target mappings and operator-level structure. This intermediate representation is then passed to an LLM, which produces checklist questions aligned with the evaluation target of each operator. The main design principle is that each operator should be evaluated through multiple binary questions rather than a single scalar or holistic judgment. This is necessary because operator-level success is often not atomic. For example, an output may contain the correct target object but fail to match the referenced source faithfully, or an attribute may be transferred but bound to the wrong carrier. A single binary judgment would be too coarse to distinguish such cases, while fully open-ended or non-binary judgments may introduce additional bias and reduce consistency. We therefore decompose the evaluation of each operator into several binary questions, each focusing on one concrete aspect of correctness. This principle is applied consistently to all four operators, namely 𝑓 , 𝑔, ⊕, and 𝐶. For an anchored entity 𝑓 , the checklist may separately verify whether the target exists and whether it matches the referenced source entity. For a disentangled attribute 𝑔, the checklist may separately ask whether the intended attribute appears on the target and whether it is faithful to the reference image. For ⊕, multiple questions are used to check whether the transferred attribute is present, whether it is correctly bound to the intended carrier, whether the carrier itself remains coherent, and whether the transferred content resembles the referenced source. For 𝐶, the checklist likewise separates coexistence, relation satisfaction, and structural coherence into different binary checks. In this way, operator-level failures can be localized more precisely instead of being collapsed into a single judgment. Each generated question also records the corresponding target part in the structured representation. This design makes the checklist easy to trace back to the original formula and reference prompt, and also supports later grouping and aggregation by operator type, target entity, or failure mode. As a result, the checklist is both fine-grained enough to capture diverse operator-level errors and structured enough to support systematic analysis. For completeness, we show the full checklist for the cacao-pod case (Fig. 9) below. Here, the Target field records the corresponding operator target or target part in the formula structure, which makes

C.2

Complete Diagnostic Tree Rules

The diagnostic tree is used to localize the source of failure in a complex multi-reference case. Starting from the full formula at the root node, we recursively simplify the case into a set of easier sub-cases and compare the model behavior across nodes. General principle. Each child node should preserve the same overall scene as the root case, while reducing part of the referenceconditioned complexity. Removed reference content is not simply deleted; when necessary, it is downgraded to an ordinary text-only description so that the scene context remains comparable across nodes. Rule 1: Global-reference stripping. If the formula contains one or more 𝑔global terms, we remove them one by one. This rule is used to diagnose whether global references, such as style or scenelevel constraints, interfere with lower-level anchor or attribute fidelity. Rule 2: Composition flattening. If the formula contains a composition operator 𝐶 (·), we flatten it into simpler branches. One child node keeps one reference-conditioned branch, while the remaining branches are downgraded to text-only scene descriptions. This rule is especially useful for identifying whether a failure only appears under joint composition. In particular, when multiple anchors cooccur in the same scene, composition flattening can isolate one anchor at a time while retaining the rest of the scene as ordinary textual context. Rule 3: Relation simplification. If a node contains an explicit interaction term, the relation is simplified before removing the participating branches themselves. In the simplified child node, the original interaction is converted into a text-only relation description 14

Op.

Target

Binary question

Case formula: 𝐶 ( 𝑓1 ⊕ 𝑔1 ) Target object: (’cacao_pod_A’)

𝑓

𝑓1

𝑓 𝑔 𝑔 𝑔 𝑔 ⊕ ⊕ ⊕ ⊕ 𝐶 𝐶 𝐶 𝐶 𝐶

𝑓1 𝑔1 , IP on cacao_pod_A 𝑔1 , IP on cacao_pod_A 𝑔1 , IP on cacao_pod_A 𝑔1 , IP on cacao_pod_A ⊕, 𝑓1 ⊕ 𝑔1 , carrier ⊕, 𝑓1 ⊕ 𝑔1 , fit ⊕, 𝑓1 ⊕ 𝑔1 , fit ⊕, 𝑓1 ⊕ 𝑔1 , exclusivity 𝐶 , spatial 𝐶 , relation 𝐶 , duplication 𝐶 , leakage 𝐶 , leakage

Does a food item that is clearly a cacao pod, retaining its rugby-ball shape and vertically grooved surface, exist in the generated image? Does (’cacao_pod_A’) match the referenced cacao pod in terms of its core identity as a cacao pod? Does (’cacao_pod_A’) incorporate the overall fairy-like design, including wings, pose, and magical effects? Are recognizable fairy-design cues present in the overall design of (’cacao_pod_A’)? Is the color scheme and motif of the referenced fairy transferred to the overall design of (’cacao_pod_A’)? Is the overall fairy design integrated with (’cacao_pod_A’) while preserving its identity as a cacao pod? After the fusion, is (’cacao_pod_A’) still clearly recognizable as itself and structurally intact? Is the transferred IP clearly visible on (’cacao_pod_A’) and naturally integrated? Does the fusion remain visually coherent and physically plausible on (’cacao_pod_A’)? Is the transferred IP confined to the intended scope (only on 𝑓1 ), without leaking to other entities or the background? Is the scene composition spatially coherent and physically plausible? Does the generated image satisfy the intended scene relation? Does any prompted entity appear more times than intended in the generated image? Does any unintended source content appear in the generated image beyond the referenced cacao-pod content? Does any unintended source content appear in the generated image beyond the intended fairy-design transfer on (’cacao_pod_A’)?

Table 6: Full checklist example for the cacao-pod case. The Target field records the corresponding operator target or target part in the formula structure, which supports tracing and later grouping.

Table 7: Alignment between VLM judges and human annotations on outputs from 200 sampled benchmark cases. The ensemble averages the four VLM judges. Judge

Pear.↑ Spear.↑ MAE↓ Agree.↑

G3P G25P GPT-5.1 GPT-5.4 Ensemble

0.608 0.554 0.569 0.580 0.662

0.613 0.537 0.558 0.542 0.604

0.152 0.173 0.162 0.156 0.153

86.8% 85.4% 88.1% 88.4% 88.6%

while the main scene context is preserved. This helps distinguish failures caused by relation grounding from failures caused by entity appearance or attribute transfer. Rule 4: Attribute removal. For expressions of the form 𝐸 ⊕ 𝑔, 𝐸 ⊕ 𝑔ip , or 𝐸 ⊕ 𝑔attach , we may remove the added attribute while preserving the carrier entity 𝐸. This rule is used to determine whether the failure comes from the carrier anchor itself or from the added attribute transfer. Stopping criterion. The decomposition stops when the remaining node contains only a single informative reference-conditioned unit, or when further simplification would no longer help isolate a more specific source of failure. In practice, leaf nodes usually correspond to a single anchor, a single attribute transfer, or a minimally composed scene. Node-to-question mapping. Each diagnostic node is evaluated only with the checklist items that correspond to the retained operator targets in that node. Therefore, the diagnostic tree is not only a formula decomposition, but also an evaluation decomposition. The root node uses the full checklist of the original case, while each child node uses the subset of questions that remains relevant after simplification. Figure 10 shows a formula-level example of diagnostic tree decomposition. Starting from the full case at 𝑁 0 , we first apply Rule 1

Figure 10: A formula-level example of diagnostic tree decomposition.

to strip the global reference term 𝑔global , yielding 𝑁 1 . We then apply Rule 2 to flatten the outer composition, which separates the left relational subscene 𝑁 2𝑎 from the right attribute-transfer branch 𝑁 2𝑏 . On the left branch, Rule 3 removes the relation term 𝑔rel , and Rule 2 is applied again to flatten the remaining inner composition into two simpler nodes, 𝑁 4𝑎 and 𝑁 4𝑏 . On the right branch, Rule 4 15

Figure 11: Cumulative pass rate over relative diagnostic-tree depth on 200 Emu3.5 cases.

removes one or more transferred attributes, producing the simplified nodes 𝑁 3𝑏1 , 𝑁 3𝑏2 , and 𝑁 3𝑏3 . This example illustrates how the complete rule set recursively reduces a complex formula into a set of simpler diagnostic branches while preserving the same overall scene context.

C.3 Quantitative Validation of Diagnostic Trees To validate diagnostic reliability, we construct diagnostic trees for 200 Emu3.5 cases and generate an image at every node. A VLM evaluates each node, after which the rules above localize the source of each operator-level failure. Independently, human annotators inspect the same trees and identify the node at which each failure originates. The automatic and human localizations agree in 82.6% of cases. Table 3 in the main paper summarizes the localized failure sources. Here, we additionally examine where requirements become solvable along the simplification process. For each root-to-leaf path, we normalize node position as relative depth, with 0 denoting the original complete case and 1 the maximally simplified node. For each operator requirement, we record the earliest depth at which its evaluation changes from failure to success. The cumulative pass rate at a given depth is the proportion of requirements that are already successful at the root or first become successful by that point. Requirements that remain unsuccessful at every node are treated as persistent failures and therefore do not enter the cumulative count. As shown in Fig. 11, the cumulative pass rate rises consistently with relative depth for all four operators. Thus, many requirements that fail in the complete case become solvable only after interfering reference-conditioned components are removed. Conversely, the endpoints remain below 100% because some requirements persistently fail even in the simplest diagnostic nodes. Together with the source distribution in Table 3, this result shows that many observed failures arise from interactions introduced by more complex formula structure, while a smaller subset reflects difficulty intrinsic to the isolated reference unit.

Figure 12: A diagnostic tree example for a complex multireference case. Starting from the full case at 𝑁 0 , the tree is expanded by sequentially applying global-reference stripping, composition flattening, relation simplification, and attribute removal. The resulting branches isolate different potential sources of failure in a structured way.

16

C.4

Additional Diagnostic Tree Examples

decline is substantially sharper for open-source models, whereas the leading proprietary models remain comparatively stable as more reference-conditioned elements are introduced. The trend is not strictly monotonic for every model because slot count does not fully determine empirical difficulty, which also depends on content-level factors such as reference clutter, entity composition, and attribute granularity.

Figure 12 shows an additional diagnostic tree example, corresponding to the third-from-last row in Fig. 15. In the root node 𝑁 0 , GPT-Image-1.5 shows two visible problems. First, the chess-pieceinspired head shape of (horse_A) does not appear at all. Second, the hat on (bear_A) is present, but its color does not faithfully match the referenced hat. The diagnostic tree helps determine whether these errors are caused by the same underlying source. For the horse branch, the chess-piece-inspired head shape never appears from the root node down to 𝑁 3𝑎2 . This indicates that GPTImage-1.5 does not reliably realize this shape-related reference requirement itself. In other words, the disentangle-and-apply process for this structural attribute is already failing even after the case is simplified, rather than the error being introduced only by additional scene complexity. The bear branch exhibits a different pattern. In 𝑁 0 , the bear is generated, but the transferred hat attribute is not faithful to the reference, since its color is mismatched. In 𝑁 2𝑏 , however, the failure changes form: the bear itself is not generated, so the error is no longer only about hat fidelity, but about the carrier entity collapsing altogether. By contrast, the hat becomes much more accurate in 𝑁 1 and 𝑁 3𝑏1 . This suggests that the model is not uniformly incapable of realizing the hat transfer; instead, the corresponding content is preserved unstably, and the failure mode changes across compositional contexts. Overall, this example shows that the diagnostic tree can distinguish between two qualitatively different situations: a reference requirement that is consistently not realized at all, and a reference requirement whose behavior is unstable, with the observed error shifting between attribute mismatch and carrier-level failure.

D.2

Cross-Model Qualitative Comparisons

Figure 15 provides additional cross-model qualitative comparisons on several representative cases from TRACE-Bench. Several consistent patterns can be observed. First, under high-complexity settings such as slot-8 cases, open-source models are often able to retain multiple reference-conditioned contents simultaneously, whereas closed-source models more often drop part of the reference information and instead fall back to generic text-to-image generation. Second, among the open-source models, Emu3.5 tends to preserve more reference content overall, while FireRed occasionally produces unusual blurring artifacts in slot-8 cases. We also observe that Qwen-Image-Edit-2509 and Qwen-Image-Edit-2511 generate highly similar outputs in certain cases, suggesting closely related generation behavior. Finally, across closed-source models, the GPT and Gemini families exhibit noticeably different image-generation tendencies and stylistic preferences, even when given the same reference prompt.

D.3

Performance across Attribute Subtypes

Since the disentangle operator covers a diverse set of attribute types, an overall 𝑔 score may hide important differences across sub-capabilities. We therefore further analyze model performance by attribute subtype, as shown in Fig. 14.

Figure 13: Overall operator-aligned performance across slot levels.

D Additional Experimental Results D.1 Performance across Slot Levels

Figure 14: Breakdown of disentangle performance across fine-grained attribute subtypes. Each axis corresponds to one subtype in the tagging taxonomy.

Figure 13 reports average operator-aligned performance from slot 1 to slot 8. Performance generally declines as slot count increases, supporting its use as a controllable measure of formula structure. The 17

Benchmark / application task

Typical requirement

TRACE-Bench abstraction

MultiBanana: X Objects MultiBanana: X–1 Objects + Local MultiBanana: X–1 Objects + Global MultiBanana: X–1 Objects + Background

multiple referenced objects objects + one local attribute reference objects + one global reference objects + one background reference

𝐶 ( 𝑓1 , 𝑓2 , . . . , 𝑓𝑋 ) 𝐶 ( 𝑓1 , . . . , 𝑓𝑋 −1 ⊕ 𝑔local ) 𝐶 ( 𝑓1 , . . . , 𝑓𝑋 −1 ) ⊕ 𝑔global 𝐶 ( 𝑓1 , . . . , 𝑓𝑋 −1 ) ⊕ 𝑔background

MICON: Object Composition MICON: Spatial Composition MICON: Attribute Disentanglement MICON: Component Transfer MICON: FG/BG Composition MICON: Story Generation

combine multiple referenced instances compose multiple instances with relation transfer one disentangled attribute transfer an attachable component combine foreground and background references multi-entity narrative scene generation

𝐶 ( 𝑓1 , 𝑓2 , . . . , 𝑓𝑛 ) 𝐶 ( 𝑓1 , 𝑓2 , . . . , 𝑓𝑛 ) ⊕ 𝑇rel 𝑇𝑒 ⊕ 𝑔 or 𝑓 ⊕ 𝑔 𝑇𝑒 ⊕ 𝑔attach or 𝑓 ⊕ 𝑔attach 𝐶 ( 𝑓1 , . . . , 𝑓𝑛 ) ⊕ 𝑔background 𝐶 ( 𝑓1 , . . . , 𝑓𝑛 ) ⊕ 𝑇rel

Application: Virtual Try-On Application: Group Photo Layout Application: Novel View Synthesis Application: IP-style Reference Application: Stylization

transfer garments or accessories to a target person compose multiple subjects under a layout constraint preserve the same subject under a changed viewpoint transfer overall design language to a target carrier preserve scene content under a global style transfer

𝑓 (person) ⊕ 𝑔attach,1 ⊕ 𝑔attach,2 ⊕ · · · 𝐶 ( 𝑓1 , 𝑓2 , . . . , 𝑓𝑛 ) ⊕ 𝑔layout 𝑓 ⊕ 𝑔view 𝑇𝑒 ⊕ 𝑔ip or 𝑓 ⊕ 𝑔ip 𝐶 ( 𝑓1 , . . . , 𝑓𝑛 ) ⊕ 𝑔style

Table 8: Examples of mapping benchmark-defined and application-oriented tasks into the TRACE-Bench formula space.

D.4

More Application-Oriented Formula Abstractions

initial diagnosis [27, 28, 61]. For benchmark construction, detectorverifiable properties such as object co-occurrence, position, count, and color [10] could provide scalable checks for newly sampled formula templates. Future releases could treat rare concepts and underrepresented combinations of operators and attributes as explicit difficulty axes and expand them through generation with quality filtering [37, 71]; they could also reduce annotation costs through scalable multimodal data curation and composition of compatible labeled resources [8, 25]. Beyond the current image setting, a video extension could ground reference evidence to query-conditioned temporal moments and derive operator-level supervision from partial temporal annotations and progressive pseudo-label refinement [16, 21, 52]. Promptable masks could provide localized reference tracks across frames, while efficient spatiotemporal adaptation could support longer reference sequences [43, 64]. Its evaluation could combine intermediate state, motion, contact, and temporalorder checks with video-specific dimensions such as subject consistency, motion smoothness, and temporal flickering, rather than relying on final-frame quality alone [14, 59]. A separate cross-view variant could instantiate 𝑔view through radiance-field view synthesis and depth-aware generalizable rendering, testing whether reference identity and geometry remain consistent across viewpoint changes [34, 47]. Together, these directions would preserve the central principle of TRACE-Bench by ensuring that each added capability remains explicit in case construction and independently diagnosable during evaluation.

To further illustrate the practical coverage of our formulation, Table 8 maps representative task categories from existing benchmarks, together with several common application-oriented settings, into the TRACE-Bench formula space. The key point is that diverse and realistic task categories can be expressed within the same formula space by assigning different types of reference-conditioned content to the same compositional structure. In this way, our abstraction is not limited to benchmark-specific categories, but can also cover a broad range of practical generation settings within an operator-aligned framework. Moreover, by combining formula-level sampling with instance- and attribute-level sampling, our framework also has the potential to support large-scale generation of diverse multi-reference tasks in a systematic way.

E

Potential Extensions

TRACE-Bench currently focuses on diagnosing multi-reference image generation, while its operator formulation also suggests several concrete extensions that preserve the alignment between case construction, operator targets, and diagnostic questions. At the model level, Anchor could combine decoupled image-prompt attention with query-conditioned localization to retain reference-specific evidence while suppressing salient distractors [24, 67]. Disentangle and Apply could combine feature-level comparison and prototype memories with localized attention that reduces identity mixing when several references must be bound to distinct targets [26, 60, 68]. Future benchmark annotations could further include promptable region masks for individual operator instances, optionally bootstrapped with diffusion-derived pseudo-masks or direct mask generation [17, 29, 30, 66]; these localized targets would allow diagnostic child cases to provide structured feedback for reward-guided model updates, iterative refinement, or evidence-based revisiting of an 18

Figure 15: Additional cross-model qualitative examples from TRACE-Bench. Each row presents one benchmark case, including the reference images, the realized natural-language reference prompt, and outputs from multiple representative models. These examples highlight recurring differences in reference retention, compositional fidelity, and stylistic tendencies across models, especially under complex multi-reference settings.

19

Related documents

• TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation#625322
• TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation#623049
• Vision as Unified Multimodal Generation#351909
• Boogu-Image-0.1: Boosting Open-Source Unified Multimodal Understanding and Generation#375987
• XFeat Revisited: Reproducibility and Evaluation of a Lightweight Image Matcher#441817