Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing Feng Wang1 *, Canmiao Fu2 , Zhipeng Huang2 , Chen Li2 , Jing LYU2 , Ge Li1 1 Peking University 2 WeChat Vision, Tencent Inc.
arXiv:2607.08497v1 [cs.CV] 9 Jul 2026
Abstract
1. Introduction Unified multimodal models have recently shown that a single architecture can jointly perform vision-language understanding and image generation and editing. Recent approaches [6, 7, 12, 22, 34, 39] consolidate perception and generation within a shared parameter space and achieve strong performance across diverse tasks. These systems typically treat multimodal interaction as autoregressive prediction over a unified token sequence.
Recent unified multimodal models demonstrate that a single architecture can jointly perform vision/language understanding and image generation/editing. However, these monolithic designs rely on repeatedly feeding all historical visual and textual inputs into a shared context window, limiting scalability in long-horizon multimodal dialogue due to visual token explosion and unreliable cross-turn visual referencing. In this work, we propose a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant visual episodes during reasoning. The agent consists of a Perceptual Abstraction Engine for structured visual abstraction, a Cognitive Retrieval Engine for cross-turn memory retrieval, and a Multimodal Executive Controller for autonomous task inference and action planning. To address the lack of turn-level retrieval supervision in existing multimodal dialogue datasets, we further develop a Unified Scenario Engine that programmatically generates structured multiturn conversations with fine-grained retrieval annotations, enabling reinforcement learning to optimize perceptual abstraction and retrieval policies. We additionally construct a long-horizon visual-dialogue benchmark and stratify it by difficulty to evaluate episodic visual recall. Extensive experiments show that our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2%, while nearly halving per-turn inference time (23.1s → 12.7s). We further present the Cognitive-structured Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure that integrates persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving. These results suggest that structured memory and modular decision-making provide a more scalable and efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling. Our code, dataset, and project page (caseclose.github.io/cma-harness) will all be released.
While effective for short-context interaction, this unified paradigm exhibits structural limitations in long-horizon multimodal dialogue. In practical image-text conversations, users frequently reference images introduced many turns earlier, request iterative modifications, or switch between understanding and generation without explicit task indicators. Figure 1 shows one such session, spanning 20 turns and four distinct topics. Under such settings, repeatedly injecting all historical visual tokens into the context window leads to two major issues. First, visual tokens are substantially more expensive than text tokens; as dialogue length increases, token usage grows rapidly and crowds out the context budget available for reasoning. Second, reliance on implicit attention over extended contexts weakens cross-turn visual referencing, resulting in retrieval errors and semantic drift. These challenges indicate that parameter scaling alone is insufficient for sustained multimodal interaction. Indeed, a strong unified model (BAGEL [7]) retrieves the correct visual episode in fewer than 3% of the hardest turns of our benchmark (Sec. 5.3), despite excelling at single-turn tasks. Recent agent-based approaches attempt to address parts of this problem. Memory-augmented video agents [8, 42] maintain episodic buffers for long video reasoning, and multiagent generation systems [30, 36] coordinate specialized modules for image creation and editing. However, these methods either operate on a single input stream (video) or a single task family (image creation), and none simultaneously addresses cross-turn visual retrieval, mixed-task orchestration, and long-horizon context management within a unified dialogue framework. Text-memory agents [23, 41] externalize long-term state into semantic memory banks, but their
* Work done during an internship at WeChat Vision, Tencent Inc.
1
Q: Generate an image of a large stargazing facility situated on a mountain peak at dusk. The main dome is completely closed, and the sky is painted in shades of orange and purple..
Q: Now update the scene to nighttime. The dome should be open, revealing a massive telescope pointing towards the stars. The interior of the dome is illuminated by a subtle red light.
Keep the exact same composition, telescope position, and starry night sky, but change the interior illumination of the dome from red to a cool blue light.
Looking at the two images of the open dome, what is the specific difference in the lighting between them?
The specific difference is the color of the interior illumination inside the dome. In the first image, the dome is lit with a warm red light, whereas in the second image, the interior is illuminated by a cool blue light. The change also affects the color of the light spilling onto the rocky peak directly beneath the dome.
Generate an image of a sterile, brightly lit cleanroom testing facility. High-tech robotic arms are meticulously inspecting green circuit boards on a conveyor belt.
Let's shift focus. Create an image showing a microscopic view of healthy biological cells. They should have a bright green fluorescent stain, clearly visible under magnification lens.
Modify this microscopic view to show infected cells. Keep the exact same arrangement and cell shapes, but change the fluorescent stain to a deep red color.
Add a human technician wearing a full blue protective cleanroom suit standing next to the robotic arms, monitoring a digital tablet.
In the healthy green-stained cell image we generated first, are there any visible parasites or foreign organisms present?
Look at the image of the red-stained infected cells and the image of the cleanroom with the human technician. Do both of these environments feature scientific or technical instruments?
No, there are no visible parasites or foreign organisms present in the healthy green-stained cell image. The cells appear uniform and regular, with consistent shapes and structures throughout. Each cell displays the same bright green fluorescent staining pattern with clearly defined cell boundaries and central nuclei, which is characteristic of healthy, uninfected cells. There are no irregular structures, unusual organisms, or signs of parasitic infection visible in that image - it shows exactly what was requested: healthy biological cells.
No, both environments do not feature visible scientific or technical instruments in the same way. The first image (the red-stained cells) is a micrograph, likely taken with a scanning electron microscope. While the image itself is a product of a scientific instrument, the instrument is not visible within the frame; you are looking directly at the biological subject (the cells). The second image (the cleanroom) clearly features technical instruments and machinery. You can see robotic arms assembling circuit boards on a conveyor belt, and the technician is holding a tablet computer.
Going all the way back to our very first image of the closed stargazing dome at dusk, please change the sky to feature a vibrant and dramatic meteor shower.
Let's move to space. Generate an image of a large communications spacecraft orbiting high above the Earth, with expansive golden solar panels fully deployed catching the sunlight.
Thinking way back to the image of the open observatory dome that had the cool blue interior lighting, what was the large instrument inside pointing towards? Based on the image, the large telescope inside the observatory is pointing upwards towards the night sky, specifically aimed at the stars and the Milky Way galaxy visible in the background. Let's combine some of our previous concepts. Take the large telescope from the blue-lit observatory, and the spacecraft orbiting the Earth. Create a surreal composite image with the telescope pointing at the spacecraft with the gold panel in space.
Remember the initial microscopic view of the healthy cells with the bright green stain? I'd like you to change that stain color from green to a vivid purple.
Figure 1. A multi-turn multimodal dialogue produced by our Cognitive-structured Multimodal Agent, spanning 20 turns across four topics The specific difference is the color of the interior illumination inside the dome. In the first image, the dome is lit with a warm red (stargazing dome, biological cells, cleanroom facility, spacecraft). The agent autonomously handles interleaved understanding, generation, light, whereas in the second image, the interior isfrom illuminatedearlier by a cool blueturns. light. and editing tasks while accurately retrieving and referencing visual episodes The change also affects the color of the light spilling onto the rocky peak directly beneath the dome.
purely textual abstractions discard the fine-grained visual detail needed to disambiguate near-duplicate images (Sec. 5.6).
By retrieving only relevant visual episodes, the agent avoids repeatedly feeding all historical images into the model, significantly reducing visual token overhead while maintaining long-range consistency. Decoupling the executive controller (MEC) from the retrieval engine (CoRE) further lets each module be trained, replaced, or scaled independently, so that model capacity can be allocated where it matters most for the performance–latency trade-off.
Building a long-horizon multimodal agent therefore requires explicit memory management and autonomous decision-making: rather than treating images as persistent tokens, visual information should be externalized into structured memory units and selectively reactivated on demand. As Fig. 2 shows, we propose a Cognitive-structured Multimodal Agent that decouples perception, episodic memory, retrieval reasoning, and executive control. Incoming images are first processed by a Perceptual Abstraction Engine (PAE), which produces structured semantic abstractions including descriptive captions, attribute tags, and compact thumbnails. These representations are stored in an external Episodic Visual Memory (EVM), allowing visual content to persist without occupying the dialogue context window. Given the evolving dialogue state, a Cognitive Retrieval Engine (CoRE) performs cross-modal reasoning to retrieve relevant visual episodes. A Multimodal Executive Controller (MEC) integrates dialogue context and retrieved memory, autonomously infers task intent (understanding, generation, editing, composition, or pure chat), and plans the final action.
However, developing such memory-aware agents introduces a new challenge: existing multimodal dialogue datasets rarely provide explicit supervision for cross-turn visual retrieval. Most datasets [32, 38] focus on singleturn grounding or short-context reasoning, and therefore lack fine-grained annotations indicating which historical visual episode should be recalled at each dialogue turn. To address this limitation, we introduce a Unified Scenario Engine that programmatically generates structured multiturn multimodal conversations. The engine simulates diverse visual scenes and dialogue trajectories, automatically producing fine-grained turn-level retrieval annotations that specify which visual episode is relevant to each query. Using these data, we apply a staged SFT+RL pipeline: the Cognitive Retrieval Engine is first trained to retrieve, and the Perceptual 2
Abstraction Engine is then optimized against a frozen retriever, so that memory representations are rewarded directly by downstream retrieval success rather than by captioning similarity. To systematically evaluate long-horizon visual memory and agent behavior, we additionally construct the Multi-turn Context Agent Benchmark (M2CA-Bench), consisting of 100 sessions of 20 turns each (2,000 annotated turns) with tasks randomly interleaved within each session. The benchmark stratifies turns into four difficulty levels according to temporal span, topic-shift frequency, multi-image interaction, and referential ambiguity, and injects hard negatives— high-similarity confounder images and no-retrieval-needed queries—to prevent shortcut learning. Unlike existing multimodal benchmarks that focus on single-turn reasoning, it specifically targets episodic memory retrieval in sustained multimodal interaction. Extensive experiments show that our 8B agent reaches 91.4% retrieval accuracy over 20-turn English sessions (and 89.6% in Chinese), surpassing the strongest 32B agent baseline by +8.2% while nearly halving per-turn inference time (23.1s → 12.7s). The margin widens with dialogue horizon, growing from +9.6% on the Full subset to +18.4% on the Hard subset, while our approach maintains substantially lower visual token usage. Beyond the benchmark-optimized agent, we further instantiate the same cognitive structure as CMA-Harness, which combines persistent multimodal memory with web search, image generation, editing, composition, and OpenAI-compatible serving. These results indicate that structured memory and modular decision-making provide a more scalable foundation for multimodal agents than monolithic parameter scaling. To summarize, our main contributions are as follows: (1) We propose a Cognitive-structured Multimodal Agent with explicit episodic visual memory for long-horizon dialogue that enables autonomous task inference. (2) We develop a Unified Scenario Engine that programmatically generates structured multimodal conversations with turn-level retrieval supervision. (3) We construct a benchmark specifically designed to evaluate cross-turn visual retrieval in multimodal interaction. (4) We introduce reinforcement learning-based optimization techniques for perceptual abstraction and retrieval, enabling an 8B agent to surpass unified 32B baselines in long-range multimodal dialogue. (5) We present CMAHarness, a tool-augmented deployment of the same architecture that extends the memory-centric agent to open-ended multimodal workflows.
driven autonomous agents capable of deliberative reasoning and tool execution [18, 26]. To address context-window limits, recent agents incorporate explicit memory mechanisms [14, 23, 41] that externalize information into episodic and semantic memory banks [13, 42], mitigating the cost of attending to long histories. In multimodal domains, VLM agents [15, 20, 21, 31] act as controllers that orchestrate visual tools for long-form tasks such as video understanding [8, 35]. Single & Multi-agents. While single-agent systems [9, 36] excel at routing sub-tasks to expert tools with self-correction, multi-agent frameworks [10, 16, 25, 30] distribute complex goals—such as creative generation—across specialized roles using programmable, collaborative protocols. However, existing visual agents typically operate within short contexts or restrict their scope to isolated domains. Crucially, our work introduces the first long-horizon, multi-agent architecture that seamlessly unifies pure text dialogue, multimodal understanding, image generation, and image editing within a single conversational framework. By leveraging an external episodic visual memory and a decoupled retrieval engine, our system maintains cross-turn intent and visual consistency without the exponential token costs of standard context scaling.
2.2. Multimodal Models Foundational VLMs such as the Qwen-VL [3] family have progressed from contrastive representation learning to instruction-following assistants capable of complex reasoning. Unified multimodal models [7, 27] consolidate perception and generation into a shared parameter space, either via token-based autoregression over interleaved text and visual tokens [6, 7, 34] or via hybrid architectures that blend autoregression with continuous diffusion or rectified flow [22, 39]. In parallel, diffusion-based frameworks like InstructPix2Pix [5] and TalkPhoto [11] enable instructiondriven editing, with recent methods addressing error accumulation in multi-turn iterative editing [44]. Despite their elegance, scaling the context window is insufficient for long-horizon interactions: benchmarks such as MMNeedle [32] and Visual Haystacks [38] reveal that implicit attention over massive visual histories leads to severe retrieval degradation and semantic drift, and repeatedly encoding visual tokens in unified architectures is computationally exorbitant. By externalizing visual history into structured episodes, our framework bypasses the inherent token limits of unified models, effectively maintaining reliable, long-range visual grounding across interleaved understanding, generation, and editing tasks.
2. Related Work 2.1. Agents LLM & VLM Agents. Large Language Models [1, 2, 29, 40] and Vision-Language Models [4, 17, 19, 24, 33, 45] have 3
Figure 2. End-to-end pipeline of the proposed Cognitive-structured Multimodal Agent. Each incoming image passes through the Perceptual Abstraction Engine (PAE), which compresses it into a structured entry (tags, description, thumbnail) written to the Episodic Visual Memory (EVM). Given the current query and dialogue state, the Cognitive Retrieval Engine (CoRE) selects only the visual episodes relevant to this turn. The Multimodal Executive Controller (MEC) integrates the retrieved memory with the dialogue context and dispatches the request to understanding, generation, editing, composition, or pure chat, keeping per-turn visual-token consumption bounded regardless of dialogue length.
3. Methods
3.2. Cognitive-structured Multimodal Agent
3.1. Problem Formulation
Relying on a single monolithic model to compress memory, select context, and execute tasks results in conflicting objectives and unbounded visual-token growth. This limitation is confirmed empirically by the near-zero retrieval accuracy of unified baselines on the hardest turns (Sec. 5.3). We therefore decompose the pipeline into the Cognitivestructured Multimodal Agent framework (Fig. 2), which assigns each sub-problem to a dedicated specialist, making every component independently trainable and upgradable: • Perceptual Abstraction Engine (PAE): abstracts each incoming image into a structured memory entry, including semantic tags, textual description, and a compressed thumbnail, which is stored in the EVM. • Cognitive Retrieval Engine (CoRE): retrieves indices of task-relevant visual episodes from EVM given the current query. • Multimodal Executive Controller (MEC): performs task classification (generate, edit, understand, or pure chat), orchestrates PAE and CoRE accordingly, and produces the final system output. Orchestration loop. Here, an orchestration loop denotes a single interaction iteration with the language model (i.e., one dialogue turn with ChatGPT). Within each loop, MEC (1) classifies the incoming request to determine its task type (generation/editing versus understanding/chat); (2) for tasks requiring visual context, queries CoRE for relevant memory
We formalize long-horizon multimodal dialogue as an interaction trajectory: T
D = {(ui , xi , yi )}i=1 , where ui denotes the user text input at turn i, xi the optional visual input, and yi the system response. The dialogue history before turn i is H<i = {(uj , xj , yj )}i−1 j=1 , and the historical image set is I<i = {xj }i−1 . j=1 A naive approach feeds the full history into a multimodal model, yi = f (ui , xi , H<i ), which suffers from three limitations as i grows: (i) prohibitive visual-token growth, (ii) contextual interference from irrelevant images, and (iii) unstable reasoning over long histories. While text history carries essential semantic continuity, only a small subset of past images is relevant at each turn—a fundamental asymmetry overlooked by existing unified models. We reformulate the problem as visual-context selection, learning a policy π that selects a minimal relevant image subset and predicts from it: Ii∗ = π(ui , H<i ) ⊆ I<i , yi = f ui , Text(H<i ), Ii∗ . Our objective is to learn π that minimizes |Ii∗ | while preserving downstream task performance. The Cognitive-structured Multimodal Agent framework (Sec. 3.2) implements π via coordinated specialized agents. 4
Table 1. Retrieval difficulty taxonomy.
entries and fuses them with the dialogue history and current instruction into a unified context; and (3) dispatches accordingly—rewriting the context into a refined, modelready prompt for the image-generation/execution model (e.g., Qwen-Image-Edit [37]) for generation/editing, or directly generating a text response from the unified context for understanding/chat. To limit computational cost, only the retrieved subset of historical images is forwarded to the executor in each orchestration loop. This bounds the number of visual tokens processed per loop and nearly halves per-turn inference time (a 1.8× speedup) compared to all-context baselines (see Sec. 5.5).
Accurate retrieval of relevant historical images is critical for downstream performance. We jointly optimize PAE (memory construction) and CoRE (memory retrieval) for retrieval success, rather than standalone captioning quality. Training data is exclusively drawn from the Unified Scenario Engine pool (Sec. 3.4), while the evaluation benchmark M2CABench is strictly held out. Supervised fine-tuning of the retriever. Each SFT sample pairs memory entries (tags, descriptions, thumbnails) with a user query; the target is the ground-truth retrieval set. We use 300 dialogue sessions (20 turns each), split 90:10 for training and validation. Reinforcement learning for the retriever (DAPO). We apply DAPO [43] to further optimize retrieval policies. The reward is the difficulty-weighted Jaccard similarity between predicted and ground-truth retrieval sets: |R̂ ∩ R∗ | |R̂ ∪ R∗ |
Characteristic Patterns
Typical Tags
easy
Same-topic sequential generation or editing. High-similarity variants; simple topic switch.
—
medium
3.3. Optimizing memory construction and retrieval
r=
Difficulty
hard
Cross-topic retrieval; long-range callback (≥ 8 turns).
very hard
Multi-image comparison; fusion edits; ambiguous references.
topic switch cross topic long term mem
multi image ambiguous ref
loop, ensuring memory representations are optimized for end-task utility.
3.4. Training Data & Benchmark Construction Existing long-horizon multimodal dialogue datasets lack turn-level retrieval supervision. We therefore build a Unified Scenario Engine that programmatically generates structured 20-turn sessions with dense retrieval annotations (Fig. 3). The pipeline operates as a closed loop: (1) A humansimulation module (Gemini) samples a topic, generates a query, its ground-truth retrieval set, and a difficulty tag; (2) A zero-shot multi-agent system (Qwen-VL + Qwen-ImageEdit) produces a candidate answer; (3) A GT-justify mechanism verifies retrieval correctness; mismatches are corrected; (4) The verified answer is fed back to Gemini to generate the next turn. Table 2 shows the data splits of the training data and M2CA-Bench. Training and evaluation sessions are produced with disjoint random seeds and non-overlapping topics; we denote the held-out evaluation set as Multi-turn Context Agent Benchmark (M2CA-Bench). Scenario representation and topic ontology. Each session is structured as:
· wd ,
where R̂ is the prediction, R∗ the ground truth, and wd = 1.2 for hard/very hard turns (1.0 otherwise). By convention we set r = 1 when R∗ = R̂ = ∅ (correctly abstaining from retrieval on a negative sample) and r = 0 when exactly one of R∗ , R̂ is empty. Soft length penalties discourage malformed outputs. RL samples are filtered to emphasize (i) incorrect baseline predictions, (ii) hard/very hard turns, and (iii) negative samples requiring no retrieval, mitigating shortcut learning. Reinforcement learning for the memory writer (DAPO). Rather than optimizing traditional captioning metrics (CIDEr, BERTScore), we directly reward memory entries based on retrieval effectiveness. Specifically, CoRE is frozen, and retrieval accuracy on memory entries produced by PAE serves as the reward:
S = {(ui , τi , Ri∗ , di , fi )}Ti=1 , where ui is the user input, τi the task type, Ri∗ the groundtruth retrieval set, di the difficulty, and fi encodes retrieval challenge tags. We curate 55 topics across eight domains (commercial, industrial, educational, public service, hospitality, natural landscape, scientific, and space); each topic defines generate, edit, cross-reference-edit, and understand prompts to ensure compositionality. Gemini [28] synthesizes dialogue flows with structural inductive biases (e.g., “at least 3 long-range callbacks with span ≥ 8 turns”), ensuring compositional and controllable conversations.
r = Jaccard(R̂, R∗ ) − Lformat . Lformat represents the format punishment. This closes the 5
Figure 3. Closed-loop Unified Scenario Engine for structured multi-turn scenario construction. A Gemini-based user simulator samples a topic and emits the next query together with its ground-truth retrieval set and difficulty tag; a zero-shot multi-agent system (Qwen-VL + Qwen-Image-Edit) produces the candidate answer; a GT-justify verification step checks retrieval correctness and corrects mismatches before the verified answer is fed back to the simulator for the next turn. The resulting sessions are split into disjoint training and held-out (M2CA-Bench) partitions by random seed and topic.
Table 2. Data splits derived from the Unified Scenario Engine. Split
Scale
Turns
Usage
SFT train SFT val RL M2CA-Bench
270 sessions × 20 turns 30 sessions × 20 turns Filtered hard subset 100 sessions × 20 turns
∼5,400 ∼600 varies 2,000
Retrieval SFT Validation DAPO training Evaluation
composition use Qwen-Image-Edit; inference runs 50 denoising steps with a classifier-free guidance scale of 4.0. Full module interfaces, prompt formats, and data-construction configurations are provided in Appendix A and Appendix B.
4. CMA-Harness: A Tool-Augmented Engineering Deployment Framework
Retrieval difficulty and hard-negative design. Turns are stratified into four difficulty levels based on topic shift, temporal span, multi-image interaction, and ambiguity (Table 1). To prevent shortcut learning, we include (i) high-similarity confounders (near-duplicate images differing subtly) and (ii) negative retrieval samples (semantic negatives requiring no retrieval, and structural negatives prompting new generation). Held-out benchmark. The held-out benchmark consists of 100 sessions × 20 turns, generated with disjoint seeds and topics. Annotation fidelity is ensured via LLM validation and manual review. Evaluation metrics are detailed in Sec. 5.2.
We introduce CMA-Harness, a tool-augmented deployment of the same cognitive-structured architecture in Fig. 2. Rather than replacing the Cognitive-structured Multimodal Agent framework, it extends the response-and-tool-usage branch while preserving the division of labor among PAE, EVM, CoRE, and MEC. The benchmark decomposition already works well when the user’s intent is grounded in dialogue-local visual history; practical workflows, however, often require information beyond the current session—real product images, logos, names, news events, screenshots, or web pages—and benefit from remembering user preferences and project context across conversations. CMA-Harness therefore keeps the memory-centric structure, assigns tool selection and invocation to MEC as the executive controller, expands the response module with external and deterministic tools, and strengthens the memory layer for persistent multi-session interaction. We defer full deployment details— including the 17-tool MEC action space, the multi-scope persistent memory schema, and the interactive tool-augmented execution loop—to Appendix C.
3.5. Implementation Details PAE writes each image as a structured JSON memory entry (thumbnail + tags + description), while CoRE operates over aligned thumbnails and memory entries and returns an image-index list. MEC classifies each turn into one of five modes: generation, editing, composition, understanding, or pure-chat, and dispatches accordingly. Text-to-image generation uses Qwen-Image, and image editing and multi-image 6
5. Experiments
image-retrieval annotations, following the difficulty definition in Table 1. Evaluation subsets. We report three subsets of increasing difficulty: Full (all 20 turns), Medium (turns 11–20, emphasizing longer context), and Hard (the very hard subset within turns 11–20), which measure how performance degrades as retrieval distance grows. Metrics. We report two complementary measures: Context Usage Accuracy, the turn-level exact-match accuracy of the retrieved image index; and Generation Quality, a per-turn 0–10 score from Gemini-3-Pro assessing faithfulness and visual correctness. Evaluation fairness. All methods use identical scenario sessions, turn annotations, retrieval targets, and post-processing rules; only the context-utilization mechanism differs, so performance gaps reflect context modeling rather than pipeline implementation details.
Our goal is to evaluate whether the proposed Cognitivestructured Multimodal Agent can effectively utilize longhorizon visual context while maintaining strong generation capability. To this end, we first describe the model setup (Sec. 5.1) and evaluation criteria (Sec. 5.2). We then compare our agent with unified and modular baselines on M2CA-Bench to measure context usage accuracy in multiturn multimodal dialogues (Sec. 5.3) and analyze how improved retrieval translates into downstream generation quality (Sec. 5.4). Finally, we examine the contribution of SFT/RL training (Sec. 5.5) and ablate memory representations (Sec. 5.6); a comparison against a concurrent agent framework is provided in Appendix D.
5.1. Experimental Setup Our system consists of three specialized vision-language modules (PAE, CoRE, and MEC) together with EVM. PAE performs visual abstraction by summarizing images into tags, description and thumbnails stored in the memory bank. CoRE conducts reasoning over the dialogue context and retrieves the relevant images from the memory bank. Finally, MEC integrates the retrieved visual context with the current dialogue state to perform task planning and generation. Unless otherwise specified, all three modules are instantiated using Qwen3-VL-8B [4] as the backbone model. Image generation and editing are performed using Qwen-ImageEdit [37]. We compare our approach against three representative baselines, listed below alongside our full model: • Unified Model. A unified multimodal model (BAGEL [7]) that handles the entire multi-turn dialogue within a single model. • Agent Baseline-8B/32B (All-Context). A directprompting baseline that concatenates the full history (all prior images and texts) into Qwen3-VL-8B/32B-Instruct for comprehension, then feeds its output as the conditional prompt—together with all historical images—to Qwen-Image-Edit for generation. As this paradigm has no explicit retrieval mechanism, we prompt Qwen3-VL to emit target image indices only during retrieval evaluation, omitting this step in generation experiments. • Multi-Agent Baseline. A modular architecture where PAE, CoRE, and MEC are instantiated with Qwen3-VL8B/32B-Instruct but trained without our long-horizon multimodal dialogue optimization. • Ours. Our full agent trained on Unified Scenario Enginegenerated long-horizon dialogues, optimized with SFT and RL to improve visual abstraction and retrieval accuracy.
5.3. Context Usage Accuracy Table 3 reports turn-level retrieval accuracy, from which several observations emerge. Unified models struggle with long-horizon retrieval: BAGEL, lacking an explicit memory mechanism, reaches only 23.3% (EN) and 17.8% (CN) on Full and collapses to below 3% on Hard, showing that unified architectures cannot isolate relevant visual context once the history grows long. Structured memory filtering improves context selection: the Multi-Agent baseline consistently beats the All-Context baseline, with the 8B VLM gaining +2.9% (EN) / +5.8% (CN) on Full and larger margins on Medium (+8.2%/+11.6%), confirming that filtering irrelevant memories is essential in long conversations. Task-specific training beats parameter scaling: enlarging the Multi-Agent baseline from 8B to 32B yields only modest gains (+1.4% EN, +2.0% CN on Full), whereas our SFT+RL-optimized 8B model surpasses the 32B baseline by +8.2% (EN) and +5.1% (CN). Advantages increase with difficulty: the gap over the 8B Multi-Agent baseline widens from +9.6% on Full to +14.2% on Medium and +18.4% on Hard, and our Full-to-Hard drop is far smaller (9.4% vs. 18.2%), indicating stronger robustness in long-horizon visual recall.
5.4. Generation Quality Table 4 reports generation quality scored by Gemini-3-Pro. Our method achieves the highest overall scores (8.49 EN / 8.53 CN), and the method ranking closely mirrors retrieval accuracy, indicating that correct visual-context retrieval remains the primary bottleneck in long-horizon multimodal generation. At the task level, Generate is least sensitive to retrieval (R∗ = ∅; most agents score above 7.1), Edit is the hardest (typically 6.1–7.5, requiring both accurate source retrieval and faithful instruction adherence; ours is best at 7.44 CN and competitive at 7.31 EN), and Under-
5.2. Evaluation Protocol and Metrics We evaluate all methods on M2CA-Bench, which contains structured multi-turn multimodal dialogues with explicit 7
Table 3. Pipeline retrieval results on M2CA-Bench. English
Method Unified Model (BAGEL) Agent Baseline-8B (All-Context) Agent Baseline-32B (All-Context) Multi-Agent Baseline-8B Multi-Agent Baseline-32B Ours (8B)
Chinese
Full
Medium
Hard
Full
Medium
Hard
23.3% 78.9% 81.9% 81.8% 83.2% 91.4%
5.6% 67.0% 72.1% 75.2% 79.4% 89.4%
1.9% 59.5% 62.0% 63.6% 72.1% 82.0%
17.8% 76.7% 80.3% 82.5% 84.5% 89.6%
3.2% 66.0% 72.5% 77.6% 80.0% 85.6%
2.3% 60.8% 71.4% 77.8% 79.0% 79.0%
Table 4. Gemini generation quality on M2CA-Bench. Image Generation and Understanding/Chat are abbreviated as Gen. and U./C., respectively. Following the five task modes defined in Sec. 3.5, composition is folded into Edit and pure chat is folded into U./C. English
Method Unified Model (BAGEL) Agent Baseline-8B (All-Context) Agent Baseline-32B (All-Context) Multi-Agent Baseline-8B Multi-Agent Baseline-32B Ours (8B)
Chinese
Overall
Gen.
Edit
U./C.
Overall
Gen.
Edit
U./C.
4.97 7.89 7.93 8.07 8.24 8.49
5.11 8.85 9.14 9.04 8.97 9.26
4.09 6.71 6.38 7.33 7.49 7.31
5.77 8.05 8.19 7.75 8.22 8.87
4.77 7.10 7.45 7.63 7.91 8.53
5.68 7.86 8.31 8.35 9.24 9.38
3.39 6.13 6.25 6.19 7.03 7.44
4.97 7.15 7.53 8.03 7.34 8.56
Table 5. Ablation on the staged training pipeline for memory construction and retrieval. Turn-level exact-match retrieval accuracy on the English M2CA-Bench; Full / Medium / Hard follow the subset definitions in Sec. 5.2. Method
Full
Medium
Hard
Multi-Agent Baseline-8B + SFT (CoRE) + RL (CoRE) + RL (PAE)
81.8% 86.6% 90.1% 91.4%
75.2% 83.7% 88.3% 89.4%
63.6% 70.9% 77.4% 82.0%
(12.7s vs. 23.1s) while also improving generation quality (8.77 vs. 7.93) over the Agent Baseline-32B.
5.6. Ablation on Memory Representations Text-only LLM memories provide useful long-term semantic state, but they are not sufficient for fine-grained visual recall. We evaluate MemoryLLM-style and ReasoningBank-style text memories by supplying agents with textual abstractions of images and asking them to retrieve the relevant historical visual episode. Both text-only strategies degrade sharply on the Hard subset (24.4% and 39.0%, respectively), where queries often require cross-topic retrieval, long-range callbacks, or subtle visual discrimination. In contrast, our full EVM retains compressed thumbnails together with textual metadata and reaches 82.0% (Table 8, Appendix D).
stand/Chat benefits most from retrieval (ours reaches 8.87 EN, +0.65 over Multi-Agent Baseline-32B). Comparing the Agent Baseline-8B (All-Context) with our method, a roughly 12% retrieval gain translates into quality improvements of +0.60 (EN) and +1.43 (CN)—a strong multiplier effect that propagates through the entire generation pipeline.
6. Conclusion
5.5. Training Stage Analysis
We presented a cognitive-structured multimodal agent that externalizes visual history into episodic memory and selectively retrieves relevant episodes for long-horizon dialogue. With the M2CA-Bench benchmark and a staged SFT+RL pipeline, our 8B agent reaches 91.4% English retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving inference time. The same decomposition instantiates CMA-Harness, a deployed tool-augmented assistant with persistent memory, web access, generation/editing/composition, and OpenAIcompatible serving. Future work includes joint optimization, open-domain extension, and richer modalities.
Our training pipeline consists of three stages; Table 5 reports Hard-subset retrieval accuracy after each. Stage 1 (SFT on CoRE) improves Hard accuracy by 7.3%, showing that supervised fine-tuning helps the retriever extract relevant visual memories from structured dialogue data. Stage 2 (RL on CoRE) adds a further 6.5%, refining long-horizon memory selection beyond supervised signals. Stage 3 (RL on PAE) adds another 4.6% by enhancing visual abstraction so that the most informative representations are stored, confirming that a carefully staged SFT+RL combination unlocks longhorizon multimodal context usage. Our decoupled architecture is also more efficient (Table 6, Appendix D): by delegating memory filtering to PAE and CoRE, the agent processes only retrieved memories rather than the full context, nearly halving per-turn inference time 8
A. Extended Implementation Details
stable input-output contracts across training, evaluation, and deployment.
This appendix expands Sec. 3.5 with the full module interfaces and data-construction configurations omitted from the main text. Structured memory format. For every incoming or generated image, PAE writes a structured memory entry containing a compact thumbnail, semantic tags, and a naturallanguage description. The memory-writing prompt requires valid JSON with fields such as tags and description; this format makes the stored episode both machine-readable and interpretable. In practice, tags provide high-recall lexical anchors, while the description preserves spatial, relational, and stylistic cues that are difficult to capture with short labels alone. Multimodal retrieval interface. CoRE receives a list of candidate thumbnails aligned with candidate indices, textual memory entries containing tags and descriptions, and the current user query. It outputs an image-index list, with an empty list indicating that no historical image is needed. This interface is shared by SFT, RL, and evaluation, ensuring that the model is optimized under the same input-output contract used at inference time. Executive controller modes. MEC classifies each turn into generation, editing, composition, understanding, or purechat modes. For generation, it rewrites short user requests into detailed visual prompts. For editing and composition, it grounds the instruction in retrieved images and emits a precise editing prompt that specifies the target region or object while preserving unrelated content. For understanding and chat, it answers directly using the retrieved visual evidence, dialogue history, and memory metadata. Generation and editing backbones. Text-to-image generation is handled by Qwen-Image, while image editing and multi-image composition use Qwen-Image-Edit. Unless otherwise stated, inference uses 50 denoising steps and a classifier-free guidance scale of 4.0. During data construction, we use a faster Qwen-Image-Edit-2511-Lightning configuration with four denoising steps to reduce generation cost; final evaluation uses the standard high-quality pipeline. Prompt interfaces. The module prompts are intentionally format-constrained so that memory writing, retrieval, task classification, prompt rewriting, and visual answering expose stable input-output contracts across SFT, RL, evaluation, and deployment. We provide the full prompt templates in Appendix B.
PAE Prompt (Image → Memory) <image> Please analyze this image in detail and generate complete image features + description in English. The output must be JSON containing the following fields: tags (tag list, at least 5 tags, array format), description (detailed description, at least 100 words). Please ensure the output is valid JSON. CoRE Prompt (Retrieval) Candidate image list (aligned with the order in ‘images’): - rank 0: <image> - rank 1: <image> ... Memory bank information (including tags + description): {[0] | tags: sunset, sports car, highway | description: A sleek black ...} {[1] | tags: beach, ocean, sunset | description: A panoramic view ...} ... User query: {q} Please select the most relevant image ID list based on the user query and memory bank information. If no historical image is needed, output an empty list. MEC Prompt (Task Classification) You are a multimodal task classification expert. Given the user input and current state, categorize the task into one of the following five types: generate, edit, composite, understand, pure chat. Current state: User text = {user input}; Image status = {image info}. Output requirement: Return JSON only, with no explanation: {"task type": "generate" | "edit" | "composite" | "understand" | "pure chat"}.
B. Prompt Templates We provide the core prompt templates used by the agent modules to clarify the module interfaces and support reproducibility. The wording is intentionally simple and formatconstrained, so that memory writing, retrieval, task classification, prompt rewriting, and visual answering expose 9
tion, uploaded images, and prior turns. New visual inputs are first persisted as memory assets and then asynchronously abstracted by a PAE-like card extractor into structured tags, captions, palettes, and thumbnails. These entries are written into EVM and later retrieved by CoRE when the user refers to previous visual context. The central difference lies in the operational scope of MEC. In the original benchmark pipeline, MEC mainly decides whether the task requires understanding, retrieval, generation, or editing, and then prepares the prompt for the downstream multimodal model. In CMA-Harness, MEC becomes a tool-calling executive: it still performs task decomposition and multimodal context fusion, but it can additionally call external and deterministic tools before producing the final answer. This design keeps the cognitive interpretation of Fig. 2. PAE remains responsible for visual compression, including tags, captions, and thumbnails. EVM remains the longhorizon visual memory substrate. CoRE remains the retrieval module that selects relevant memory for the current turn. MEC remains the main reasoning module, but its action space is enlarged from a small set of generation/editing choices to a structured set of callable tools. As a result, CMA-Harness is best understood as a harness around Cognitive-structured Multimodal Agent: it provides service interfaces, tool schemas, persistent storage, and execution policies around the same cognitive modules.
MEC Prompt (Generate Mode Rewriting) You are a prompt rewriting assistant for text-to-image generation models. Convert a short user instruction into a detailed visual prompt. Identify the main subject, expand visual attributes such as color, shape, size, and appearance, add reasonable scene or environment details if not specified, optionally include viewpoint or composition, and keep the user’s original intent unchanged. Output only the final rewritten prompt. User instruction: {instruction}. MEC Prompt (Edit Mode Rewriting) You are a multimodal prompt rewriting assistant for image editing models. You will receive an image, a user editing instruction, and optional dialogue context. Identify the object or region to modify using the image, clearly describe the requested modification, preserve the user’s intent, explicitly indicate that all other parts of the image should remain unchanged, and output only one concise editing prompt. Image editing instruction: {instruction}.
C.2. MEC-Driven Action Space Expansion CMA-Harness exposes a registry of 17 tools to MEC. Each tool is described by a name, natural-language capability description, JSON schema, concurrency-safety flag, progress message, and asynchronous execution function. The registry is converted into an OpenAI-style function-calling interface, so MEC can reason about the user’s goal, decide whether tools are necessary, choose the appropriate tool, fill its arguments, observe the result, and continue the loop if another action is needed. The tools extend the original response-andtool-usage block in Fig. 2: generation and editing tools include image generation, image editing, multi-image composition, deterministic collage construction, background removal, watermark removal, cropping, and text overlay; understanding and memory tools include image inspection, best-image selection, and image retrieval; external-information tools include web search, batch search, web-image search, web-page fetching, and image fetching. The key design choice is that tool access is governed by an MEC action policy rather than exposed as an undifferentiated toolbox. MEC first decides whether a tool is needed at all; many requests are better answered directly without external execution. When the requested output is a purely imagined scene, MEC can call text-to-image generation. When the output must contain a specific real-world visual entity, such as a logo, product, screenshot, artwork,
MEC Prompt (Understanding / Chat) Text-only mode: Current user instruction = {user instruction}; task type = pure chat. Please answer directly. Contextual mode: <image> <image> ...; dialogue history; retrieved memories with tags and descriptions; current user instruction = {user instruction}; task type = mmu. Please reason carefully based on the instruction and reference images, and provide a detailed answer.
C. CMA-Harness Deployment Details This appendix expands the deployment overview presented in Sec. 4. We describe how the core Cognitive-structured Multimodal Agent is wrapped into a service-facing harness, the MEC-driven tool registry, the flexible persistent memory system, and the interactive tool-augmented execution loop.
C.1. From Core Agent System to Deployed Harness CMA-Harness follows the same information flow as Cognitive-structured Multimodal Agent. User interactions first enter a dialogue state that contains the current instruc10
or named person’s face, MEC is instructed not to rely on a verbal description, because text-only generation or editing will often hallucinate the identity. Instead, it first obtains real pixels through memory retrieval or web-image fetching, and then routes the task to composition or editing with those image identifiers as visual references. CMA-Harness also separates semantic composition from deterministic layout. Multi-image diffusion composition is used when several visual references must be fused into a new scene. In contrast, poster, infographic, timeline, comparison, and many-item showcase tasks often require faithful faces, logos, captions, dates, or CJK text. For these cases, MEC can choose a PIL-based collage tool that preserves each input by pixel-level placement and renders text deterministically, optionally followed by a lightweight beautification pass. This division makes the enlarged MEC action space practical: MEC can route each request to the operation that best preserves what the user cares about, instead of forcing all visual tasks through a single generative or editing model. Tool execution is constrained by the same executive role assigned to MEC. Consecutive read-only operations, such as web search, web fetch, image inspection, and memory retrieval, can be executed concurrently to reduce latency. Mutating or GPU-heavy operations, such as image generation, editing, composition, cropping, and text rendering, are serialized to preserve memory consistency and avoid resource contention. The loop tracks candidate images produced during a turn and can use an image-selection tool to choose the best final deliverable, keeping intermediate drafts from being over-exposed to the user. To prevent tool use from degenerating into uncontrolled loops, CMA-Harness adds simple controller-level constraints. If MEC repeatedly issues the same tool call with the same arguments, the system prompts it to revise its plan instead of executing the duplicate call indefinitely. Each user turn also has a maximum number of tool rounds; when this budget is exhausted, MEC summarizes the current progress and returns a graceful partial response. These mechanisms keep tool invocation under MEC control while still allowing multi-step workflows.
At the visual level, each image is stored as a three-part asset. The original image is preserved for future editing, composition, inspection, and download; a compact thumbnail is used for efficient retrieval and UI preview; and a JSON image card stores semantic tags, a short description, color palette, aspect ratio, file paths, and lifecycle metadata. The lifecycle fields are important in deployed workflows: parent-child links track edited variants, an “is current” flag records the active deliverable, user feedback marks preferred or rejected outputs, and selected images can be promoted into a reusable gallery. This turns PAE’s output from a transient caption into a structured memory object that can be updated and reused across turns. Beyond visual assets, CMA-Harness maintains userscope textual memories. These memories are stored as typed Markdown entries, including user preferences, feedback, project facts, and external references, with an automatically refreshed memory index. At prompt construction time, the memory manager assembles session summaries, user memories, and the visual memory index into a compact context block for MEC. This allows MEC to condition tool selection and response generation not only on the current dialogue, but also on persistent user intent and project-level constraints. Retrieval is adaptive rather than uniform. Obvious references such as “the latest image”, “the first image”, or explicit image indices are resolved by deterministic rules without invoking a large model. When many candidates exist, a lightweight text prefilter narrows the candidate set using tags, descriptions, palette, recency, and feedback. The final selection uses multimodal retrieval over both image-card metadata and thumbnails, allowing CoRE to disambiguate references that are difficult to capture textually, such as “the blue one” or “the busier layout”. Full-resolution pixels are loaded only after a target image is selected and a downstream tool requires them. The memory layer is also maintained asynchronously so that memory quality can improve without blocking the foreground interaction. When a new image is uploaded, generated, edited, fetched, or composed, CMA-Harness immediately stores the original image and thumbnail and can return the result to the user at once. A background extractor then analyzes the thumbnail and enriches the corresponding image card with semantic tags, a concise description, and color-palette information. This separates latency-sensitive interaction from slower memory annotation: the user does not wait for full visual abstraction, but subsequent turns can still benefit from richer retrieval cues. Long conversations are handled similarly. Rather than replaying an unbounded transcript into MEC, CMA-Harness compacts older turns into structured summaries once a session exceeds token or tool-call thresholds, while keeping the most recent turns verbatim. The summary preserves highlevel goals, established preferences, generated image iden-
C.3. Flexible Persistent Multimodal Memory CMA-Harness extends EVM from a session-local visual memory bank into a flexible persistent memory system. The Cognitive-structured Multimodal Agent focuses on controlled memory construction and retrieval within a fixed dialogue, whereas deployment requires memory to support multiple users, multiple sessions, reusable visual references, long transcripts, generated variants, fetched assets, and user feedback. We therefore organize memory into several scopes: session memory for the current interaction, user-level memory for durable preferences and project facts, gallery memory for reusable visual assets, and compact summaries for longrunning conversations. 11
Table 6. Ablation study showing the efficiency and quality gains from the proposed PAE +CoRE modules.
tifiers, open follow-ups, and the current active state. Thus, long-term context remains available to the controller without continuously increasing the prompt size. Together, these mechanisms make EVM in CMA-Harness more flexible than the benchmark memory bank: it is multi-scope, multitype, lifecycle-aware, and persistent, yet still preserves the token-efficient retrieval principle of the original Cognitivestructured Multimodal Agent.
Method
Full
Hard
Quality
Runtime (s)
Agent-Baseline-32B PAE +CoRE + Qwen3VL-32B
81.9% 92.0%
62.0% 83.5%
7.93 8.77
23.1 12.7
Table 7. Multi-turn generation and editing quality scores (0–10) on English M2CA-Bench. Sessions are split into early and late stages to highlight robustness against long-horizon semantic drift.
C.4. Interactive Tool-Augmented Agent Execution
Method
Finally, CMA-Harness turns the cognitive pipeline into an interactive execution loop. For each user turn, the system reconstructs the current memory context, rebuilds the system prompt with the latest memory blocks and tool schemas, lets MEC reason over the task, and executes any selected tools. Tool results are appended back into the transcript and then fed to MEC for the next decision, so the controller can iteratively search, retrieve, inspect, generate, edit, or compose until it has enough information to answer. This event-driven design keeps the foreground interaction responsive: intermediate reasoning, tool steps, tool results, images, and final text can be surfaced progressively while the same underlying MEC loop remains responsible for the complete decision process.
GenArtist Ours (8B Agent)
Generation Quality
Editing Quality
Early
Late
Early
Late
6.77 9.51
4.80 9.06
5.29 7.74
4.03 7.13
Table 8. Ablation on M2CA-Bench comparing text-only LLM memory systems against our full EVM. Retrieval accuracy (%) is reported across difficulty subsets. Memory Strategy
Full
Medium
Hard
MemoryLLM-style memory ReasoningBank-style memory
51.7% 68.2%
47.2% 51.8%
24.4% 39.0%
Ours (Full EVM)
91.4%
89.4%
82.0%
D. Additional Experimental Analysis Efficiency analysis. Table 6 compares our decoupled design with the Agent Baseline-32B, which processes the entire multimodal history in a single model. By delegating memory filtering to PAE and CoRE, the agent retrieves only relevant visual memories instead of the full context, reducing the visual tokens fed into the generation module. As a result, per-turn inference time is nearly halved (12.7s vs. 23.1s) while generation quality also increases (8.77 vs. 7.93). Comparison with concurrent agent frameworks. We further compare our agent with GenArtist [36], a concurrent multimodal agent framework for unified image generation and editing. To expose long-horizon drift, we split English M2CA-Bench sessions into early (turns 1–10) and late (turns 11–20) stages and separately evaluate generation and editing quality. Table 7 shows that GenArtist degrades substantially in late turns, while our agent remains stable. This result supports the central claim that tool orchestration alone is insufficient: without explicit episodic visual memory and retrieval, multi-turn generation and editing suffer from context confusion as sessions grow. Memory-representation ablation. Table 8 details the memory-representation ablation summarized in Sec. 5.6, comparing text-only LLM memory systems against our full EVM across difficulty subsets.
The deployed instance uses Qwen3.5-122B-A10B as the MEC planner, Qwen-Image-2512 for text-to-image generation, and Qwen-Image-Edit-2511 for editing and multiimage composition. These model choices are implementation instantiations of the modules rather than changes to the architecture. To make the agent practical for multi-step use, the planner and image backbones are kept as persistent components instead of being reloaded for every request. The generation and editing paths also include automatic prompt engineering: generation prompts are expanded toward richer visual descriptions, while editing prompts emphasize locality, object grounding, and preservation of unrelated regions. The interaction loop also preserves the agent’s stateful multimodal semantics across turns. Uploaded or produced images are stored as addressable memory assets, tool outputs are recorded as part of the dialogue state, and the current deliverable is tracked so follow-up instructions such as “make it brighter” or “use the previous poster” can be resolved without reattaching every historical image. Lightweight safeguards, including duplicate-call detection, per-turn toolround limits, and graceful partial summaries, keep the loop bounded while still allowing long multi-step workflows. Together, these choices turn the original Cognitive-structured Multimodal Agent design into a practical CMA-Harness: the same PAE–EVM–CoRE–MEC structure, but with a broader MEC action space, stronger persistent memory, and interactive execution for open-ended multimodal workflows.
E. Additional Qualitative Examples We include additional long-horizon dialogue examples to illustrate how episodic visual memory supports mixed generation, editing, composition, and visual question answering. 12
One example begins with a Mars habitat, later edits it with a dust storm, and then correctly answers questions about earlier visual details such as solar panels. A second example performs environmental transformations such as converting a sunny street scene into a rainy night version while preserving spatial layout, then recalls distant details from a separate savanna scene. A third example spans indoor scenes, logistics environments, wildlife landscapes, and Mars habitats, requiring object insertion, cross-scene composition, and detailed visual comparison. Across these examples, the agent demonstrates long-range visual recall, topic switching without memory loss, multi-image composition, and grounded multimodal reasoning. We first contrast our method with the all-context baseline (Fig. 4). In the all-context baseline, lack of filtering causes hallucination: the agent over-relies on the first image and incorrectly propagates its content into subsequent generations. In contrast, our staged training ensures accurate retrieval and, coupled with MEC’s prompt rewriting, produces outputs that adhere closely to user instructions across multi-turn interactions.
[10] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for a multi-agent collaborative framework. In The twelfth international conference on learning representations, 2023. [11] Yujie Hu et al. Talkphoto: A versatile training-free conversational assistant for intelligent image editing. arXiv preprint arXiv:2601.01915, 2026. [12] Zhipeng Huang, Shaobin Zhuang, Canmiao Fu, Binxin Yang, Ying Zhang, Chong Sun, Zhizheng Zhang, Yali Wang, Chen Li, and Zheng-Jun Zha. Wegen: A unified model for interactive multimodal generation as we chat. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 23679–23689, 2025. [13] Hanqi Jiang et al. Synapse: Empowering llm agents with episodic-semantic memory via spreading activation. arXiv preprint arXiv:2601.02744, 2026. [14] Hongbo Jin, Qingyuan Wang, Wenhao Zhang, Yang Liu, and Sijie Cheng. Videomem: Enhancing ultra-long video understanding via adaptive memory management. arXiv preprint arXiv:2512.04540, 2025. [15] Somnath Kumar, Yash Gadhia, Tanuja Ganu, and Akshay Nambi. Mmctagent: Multi-modal critical thinking agent framework for complex visual reasoning. arXiv preprint arXiv:2405.18358, 2024. [16] Guohao Li, Hasan Abed Al Abbas Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for” mind” exploration of large language model society. In Advances in Neural Information Processing Systems, 2023. [17] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023. [18] Pengxiang Li et al. Iterative trajectory exploration for multimodal agents. In Advances in Neural Information Processing Systems, 2025. [19] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. [20] Jiaqi Liu, Kaiwen Xiong, Peng Xia, Yiyang Zhou, Haonian Ji, Lu Feng, Siwei Han, Mingyu Ding, and Huaxiu Yao. Agent0vl: Exploring self-evolving agent for tool-integrated visionlanguage reasoning. arXiv preprint arXiv:2511.19900, 2025. [21] Lin Long, Yichen He, Wentao Ye, Yiyuan Pan, Yuan Lin, Hang Li, Junbo Zhao, and Wei Li. Seeing, listening, remembering, and reasoning: A multimodal agent with long-term memory. arXiv preprint arXiv:2508.09736, 2025. [22] Yiyang Ma et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. [23] Siru Ouyang, Jun Yan, I Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T Le, Samira Daruki, Xiangru Tang, et al. Reasoningbank: Scaling agent self-evolving with reasoning memory. arXiv preprint arXiv:2509.25140, 2025.
References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. [3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. 2023. [4] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. [5] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. [6] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. [7] Chaorui Deng et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025. [8] Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding. In European Conference on Computer Vision, pages 75–92. Springer, 2024. [9] Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaojian Ma, Tao Yuan, Yue Fan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, and Qing Li. Multi-modal agent tuning: Building a vlm-driven agent for efficient tool usage. arXiv preprint arXiv:2412.15606, 2024.
13
Qwen Baseline
Create an image of the African savanna at noon. A pride of lions is resting under a large acacia tree.
Ours
Create an image of the African savanna at noon. A pride of lions is resting under a large acacia tree.
Qwen Baseline
Create a variation of that factory scene where the lights are dimmed and the assembly line is completely paused for the night.
Ours
Create a variation of that factory scene where the lights are dimmed and the assembly line is completely paused for the night.
Qwen Baseline
Compare the active factory image with the paused night version. What are the main differences in lighting?
Ours
Compare the active factory image with the paused night version. What are the main differences in lighting?
Generate another one with the exact same composition, but change the time to sunset and have all the lions sleeping soundly.
In the first noon image, add a safari jeep in the background observing the lions.
Show a modern car manufacturing line. Robot arms are assembling silver sedan chassis under bright factory lights.
Generate another one with the exact same composition, but change the time to sunset and have all the lions sleeping soundly.
In the first noon image, add a safari jeep in the background observing the lions.
Show a modern car manufacturing line. Robot arms are assembling silver sedan chassis under bright factory lights.
Take the active assembly line image and change the silver cars to bright red ones
Generate a view of a coastal breakwater with huge concrete blocks. Stormy waves are crashing against them.
Looking back at the very first image of the savanna at noon, are the lions standing or lying down?
Answer: In the first image of the African savanna at noon, all the lions in the pride are lying down in the shade of the acacia tree.
Take the active assembly line image and change the silver cars to bright red ones
Generate a view of a coastal breakwater with huge concrete blocks. Stormy waves are crashing against them.
Looking back at the very first image of the savanna at noon, are the lions standing or lying down?
Answer: Based on the image rank_1, the lions are lying down. They appear to be resting on the ground in the foreground.
Answer: Based on a comparison of the two factory images, the main differences in lighting are: Overall Brightness and Color Temperature: The active factory scene is brightly lit with cool-toned, white light from numerous overhead fixtures. In contrast, the paused night version has significantly dimmed lights, creating a much darker environment with a warmer, bluish-green color cast. Shadows and Reflections: The bright lights in the first image create sharp, distinct shadows under the car chassis and robotic arms. The floor also shows clear, bright reflections of the ceiling lights. In the second image, the shadows are softer and less defined due to the lower light levels, and the reflections on the polished floor are muted and diffuse.
Answer: Based on a comparison of the two images, here are the main differences in lighting: Overall Brightness: The first image is brightly lit, simulating a fully active daytime shift. The second image is significantly darker and dimmer, creating a "night mode" or low-light atmosphere. Color Temperature: The active factory scene has a crisp, cool white light typical of industrial LED fixtures. The "paused" version has a heavy blue tint cast over the entire scene, which is often used to depict nighttime or a dormant state. Intensity of Reflections: In the first image, the overhead lights create sharp, bright reflections on the metallic car body and the polished floor. In the second image, these reflections are much softer and more subdued. The Welding Sparks: The most distinct lighting difference is the presence of bright, warm-colored sparks flying from the robot's arm in the first image. In the second image, the robot is inactive, and these sparks are completely absent, leaving that area dark.
Generate a brand new image of a peaceful urban park with a pond.
Create a surreal fusion by combining the sunset sky from the sleeping lions image with the stormy waves from the coastal scene.
Generate a brand new image of a peaceful urban park with a pond.
Create a surreal fusion by combining the sunset sky from the sleeping lions image with the stormy waves from the coastal scene.
Dialogue 1
Create a realistic view of a habitat on the surface of the red planet, showing modular pods connected by tunnels under a pink sky.
Dialogue 1
Change the season of that forest scene to late autumn, with orange and brown leaves covering the ground.
Compare the visibility between the clear red planet image and the stormy one. How much of the sky is visible in the first one?
Dialogue 1
I want to start fresh. Create a brand new image of a quiet reading area in a library with wooden tables.
Create a fusion image by placing the modular pods from the red planet scene into the pine forest clearing.
Dialogue 1
Figure 4. Qualitative comparison between the baseline and our method on multi-turn visual understanding, generation, and editing.
Zoom in on the robotic arms in the manufacturing line image to focus on the soldering tip.
Update the red planet scene to show a heavy dust storm obscuring the background, making the visibility low.
Now show me a brightly lit aisle in a grocery store stocked with colorful cereal boxes on both sides.
Add a student reading a book at one of the tables in the library image.
Answer: In the first (clear) image, ~100% of the sky region is visible and transparent enough to see the sun and distant terrain — meaning the full sky dome is optically accessible, though colored by Martian atmospheric physics.
Generate an image of a peaceful pine forest clearing in summer with sunlight filtering through the canopy.
Show a high-tech manufacturing line where robotic arms are assembling green circuit boards.
In the stormy red planet image, can you see any astronauts walking outside? Or is it deserted?
In the original grocery aisle image, what color are the floor tiles?
Answer: Light gray, with a glossy, reflective finish.
Going back to the very first image of the modular pods, are there any solar panels visible on the structures?
Answer: In the stormy red planet, there is one astronaut visible, walking outside — specifically, in the foreground, on the right side, facing the viewer.
Take the stormy red planet image and add a small rover driving near the pods.
Answer: Yes — in the very first image , solar panels are clearly visible.
Generate a scene of an airport security checkpoint with plastic trays on rollers.
Is there a metal detector arch visible in the airport security scene?
Answer: Yes — in the airport security checkpoint image, multiple metal detector arches are clearly visible.
Figure 5. Example Dialogue 1. A long-horizon multimodal interaction containing image generation, editing, topic switching, and visual question answering.
[24] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023.
et al. Chatdev: Communicative agents for software development. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 15174–15186, 2024.
[25] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong,
[26] Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda,
14
Dialogue 2
Now change the lighting to a dramatic sunset, casting long shadows, but keep the lions and tree exactly where they are.
Dialogue 2
Modify that street scene to look like a rainy night. Add streetlights reflecting on the wet pavement, but keep the street layout the same.
Compare the sunny morning street image with the rainy night version. Are the cars parked in the same positions in both?
Dialogue 2
Generate a brand new scene of a dense green forest with a deer, completely starting from scratch and ignoring previous contexts.
Take the marble statue from the museum gallery image and place it directly onto the red dusty surface of that Mars outpost scene we created earlier.
Dialogue 2
Create a realistic image of a vast African savanna during the day with a pride of lions resting under an acacia tree.
Add a sleeping cat to one of the leather armchairs in that library scene.
Do the deer in the forest image and the cat in the library image appear to be looking in the same direction?
Generate a photo of a quiet residential street in a modern city on a sunny morning, with parked cars and green trees lining the sidewalk.
Let's switch to a sci-fi scene. Show me a human outpost on the surface of Mars with red dust and a large glass habitat dome. Answer: Yes — the cars are parked in the exact same positions in both images. The rainy night version is a direct environmental transformation of the sunny morning scene, preserving all spatial elements including car placement, building positions, tree locations, and sidewalk layout.
I want to see the interior of an art museum gallery featuring classical marble statues and paintings on the walls.
Think back to the image of the savanna at sunset. What color was the sky in that specific version?
Answer: No, the deer and the cat are not looking in the same direction. The deer faces the viewer directly, while the cat faces leftward.
Looking back at the Mars outpost image, is there any flowing water or a river visible on the surface?
Answer: The sky was predominantly orange and red, with significant accents of purple and deep magenta, creating a vivid, cinematic sunset palette.
Create a new image of the African savanna with lions under an acacia tree, very similar composition to the earlier one, but with a stormy overcast sky.
Answer: No, there is no flowing water or river visible in the Mars outpost image. .
Create a cozy reading area in an old library with high wooden shelves and leather armchairs.
Recall the sunny residential street image. Were there any pedestrians walking on the sidewalk?
Answer: No—there were no pedestrians on the sidewalk in the sunny residential street image. The scene was explicitly described as "quiet" and "calm," with "no pedestrians or moving vehicles" present, emphasizing the stillness of the urban neighborhood. The focus was on the parked cars, modern architecture, and natural elements like trees and sunlight, without any human activity visible.
Dialogue 3
Create an image of a vintage watch shop facade with dark wood trim and a display of antique clocks.
Dialogue 3
Create a similar savanna scene with the acacia tree, but show the elephants marching towards a waterhole.
Dialogue 3
Start fresh and generate a vibrant underwater coral reef scene teeming with small colorful fish.
Dialogue 3
Figure 6. Example Dialogue 2. A dialogue emphasizing environmental transformations, long-range recall, and stable visual consistency.
Generate a quiet Victorian-style residential street with brick houses and a cobblestone road.
Show a wide angle view of a modern logistics warehouse interior filled with tall metal shelving and cardboard boxes.
In the warehouse image, add a bright orange autonomous forklift carrying a pallet in the center aisle.
Generate a sci-fi scene of a Mars outpost with several dome habitats on the red dusty surface.
Change the time of day to sunset for the marching elephants scene, casting long shadows.
Thinking back to the very first image of the watch shop, was there a door handle visible on the entrance?
I want an image of the African savanna with a large acacia tree and a herd of elephants resting in the shade.
Answer: The first image did not show the entrance door or handle—it was a tight composition of the window and its ornate wood trim.
Make it look like it just rained on that Victorian street, adding puddles on the cobblestones.
In the underwater reef image you just made, is there a scuba diver swimming near the bottom?
Create a picture of a misty pine forest with a narrow dirt path winding through it.
Compare the vintage watch shop from the beginning with the warehouse interior. Which one appears to have more artificial lighting sources?
Answer: No, the underwater coral reef image does not contain a scuba diver—or any human presence at all.
Take the stormy red planet image and add a small rover driving near the pods.
Answer: The warehouse interior has more artificial lighting sources.
Take the orange forklift from the warehouse image and place it onto the surface of the Mars outpost scene.
Compare the terrain of the savanna where the elephants were resting with this pine forest path. Which ground looks drier?
Answer: The savanna ground (elephants resting scene) looks significantly drier than the pine forest path. The savanna shows signs of prolonged dryness (cracked soil, dust, sparse vegetation), while the pine forest exhibits clear indicators of recent moisture (damp path, lush ferns, mist, glistening surfaces).
Figure 7. Example Dialogue 3. A complex interaction involving object insertion, cross-scene composition, and detailed visual comparison across distant turns.
and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024. [27] Hao Tang, Chenwei Xie, Xiaoyi Bao, Tingyu Weng, Pandeng Li, Yun Zheng, and Liwei Wang. Unilip: Adapting clip for unified multimodal understanding, generation and editing. arXiv preprint arXiv:2507.23278, 2025. [28] Gemini Team, Rohan Anil, Sebastian Borgeaud, JeanBaptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [29] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. [30] Kavana Venkatesh et al. Crea: A collaborative multi-agent framework for creative content generation with diffusion models. In Advances in Neural Information Processing Systems, 2025. [31] Guoyan Wang, Yanyan Huang, Chunlin Chen, Lifeng Wang, and Yuxiang Sun. Yanyun-3: Enabling cross-platform strategy game operation with vision-language models. arXiv preprint arXiv:2511.12937, 2025. [32] Hengyi Wang et al. Multimodal needle in a haystack. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2025. [33] Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long
15
Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. [34] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. [35] Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena YeungLevy. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision, pages 58–76. Springer, 2024. [36] Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal llm as an agent for unified image generation and editing. Advances in Neural Information Processing Systems, 37:128374–128395, 2024. [37] Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng, Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang, Yongqiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. Qwen-image technical report, 2025. Visual haystacks: A vision[38] Tsung-Han Wu et al. centric needle-in-a-haystack benchmark. arXiv preprint arXiv:2407.13766, 2024. [39] Jinheng Xie et al. Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506.15564, 2025. [40] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [41] Rui Ye, Zhongwang Zhang, Kuan Li, Huifeng Yin, Zhengwei Tao, Yida Zhao, Liangcai Su, Liwen Zhang, Zile Qiao, Xinyu Wang, et al. Agentfold: Long-horizon web agents with proactive context management. arXiv preprint arXiv:2510.24699, 2025. [42] Woongyeong Yeo et al. Worldmm: Dynamic multimodal memory agent for long video reasoning. arXiv preprint arXiv:2512.02425, 2025. [43] Qiying Yu et al. Dapo: An open-source llm reinforcement learning system. arXiv preprint arXiv:2503.14476, 2025. [44] Zijun Zhou et al. Multi-turn consistent image editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025.
[45] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
16