OpenDraco: A Platform for Engineering and Experimenting with Configurable Multi-Agent Architectures for Software Engineering

arXiv:2610.32686v1 [cs.SE] 23 Oct 2026

OpenDraco: A Platform for Engineering and Experimenting with Configurable Multi-Agent Architectures for Software Engineering Xiang Feng Ye Pan

Vincenzo De Martino

Matias Martinez

Universitat Politècnica de Catalunya Barcelona, Spain xiangfeng.yepan@estudiantat.upc.edu

Universitat Politècnica de Catalunya Barcelona, Spain vincenzo.de.martino@upc.edu

Universitat Politècnica de Catalunya Barcelona, Spain matias.martinez@upc.edu

Abstract—Experimenting with LLM-based multi-agent systems (MAS) involves an iterative cycle of modeling configurations, executing and evaluating them, comparing results, and refining system designs. Existing approaches provide limited end-to-end support for this experimentation lifecycle. We present OpenDraco, an open-source platform that integrates these activities into a single workflow. OpenDraco represents agents, prompts, tools, models, parameters, and communication topology in exportable, versioned JSON specifications, translated into executable LangGraph workflows supporting different interaction structures. Users can modify and re-evaluate configurations using local or cloud-based LLMs. We instantiate OpenDraco for automated program repair (APR) and evaluate a sequential baseline and seven reconstructed configurations on synthetic and real-world SWEbench Lite issues. The results reveal substantial cost differences among configurations with similar repair outcomes, while the baseline resolves 12 of 77 SWE-bench issues using open-source LLMs. Additional proof-of-concept tasks illustrate applicability beyond APR. OpenDraco supports iterative experimentation with heterogeneous MAS configurations, enabling researchers to examine their outcomes and costs within a common environment. Demonstration video: https://youtu.be/278sOR4lBEk. Index Terms—Multi-Agent Systems (MAS), Automated Program Repair (APR), Large Language Models (LLMs), SWEbench

I. I NTRODUCTION Large Language Models (LLMs) are increasingly used in agentic solutions for Software Engineering (SE) tasks, including automated program repair (APR) [1], [2]. Multiagent systems (MAS) address these tasks by coordinating specialized agents, such as bug localizers, patch generators, and validators [3], [4]. How these agents communicate is an important design choice: different topologies determine how information flows between agents and may affect both task outcomes and computational cost [5], [6]. Developing a MAS is only part of the challenge. Researchers investigating alternative architectures also need to adjust configurations, run them on comparable tasks, evaluate their outputs, and track the results. Existing frameworks and visual development environments simplify the construction and execution of agent workflows. However, comparing MAS configurations for SE tasks requires managing benchmark in-

stances and evaluation results. Without an integrated workflow, such experiments become harder to organize and reproduce[7]. We present OpenDraco, an open-source platform designed to support this experimentation process. OpenDraco represents agents as nodes and their communication topology as edges, while keeping prompts, tools, models, and parameters configurable. Each system is captured in an exportable, versioned JSON specification that is translated into an executable LangGraph workflow1 .Researchers can reuse agent definitions, modify their communication structure, and execute alternative configurations using local or cloud-based agents. By storing specifications and results, OpenDraco enables researchers and practitioners to reproduce experiments, compare configurations, and iteratively refine MASs. We demonstrate this workflow through APR, integrating OpenDraco with the SWE-bench evaluation harness [2]. We evaluate a sequential baseline and seven reconstructed configurations on synthetic and real-world bugs using the same locally hosted model. On a 23-instance development subset, the baseline and hyperagent_star each resolve two issues, while the baseline requires 39.78% less wall-clock time and 47.32% fewer tokens. The baseline also resolves 12 of 77 selected SWE-bench Lite test instances. These experiments illustrate how OpenDraco can be used to execute and compare heterogeneous MAS configurations under a common evaluation setup. Additional proof-of-concept workflows illustrate its applicability beyond APR. e Tool and demonstration video availability The source code and documentation of OpenDraco, together with the replication artifacts, are available on GitHub § [8] and Figshare [9]. A demonstration video of OpenDraco is available on YouTube Å [10]. II. R ELATED W ORK The organization of agents and their communication channels is an important design dimension of LLM-based MAS. 1 https://www.langchain.com/langgraph

Early approaches relied on predefined interaction patterns, including sequential pipelines, hierarchical structures, and ensembles. Li et al. [3], for example, investigate how aggregating multiple agents can improve task performance. More recent studies examine communication topology directly. Yang et al. [11] identify topology as an important research direction, while GPTSwarm and G-Designer investigate the optimization of agent interaction graphs [5], [6]. AFlow instead focuses on automatically generating agent workflows [12]. These approaches contribute methods for constructing or optimizing MAS designs. Our focus is complementary: providing an environment in which researchers can configure, execute, and evaluate alternative designs for SE tasks. Several development platforms already provide mechanisms for building and inspecting agent workflows. Flowise [13], Langflow [14], and Crew Studio [15] offer visual interfaces for composing agents and their interactions. AutoGen Studio [16] also supports declarative JSON specifications, reusable components, and interactive evaluation, while LangGraph Studio [17] provides graph visualization and execution debugging. Thus, visual modeling, reusable agents, and declarative workflow specifications are not unique to OpenDraco. OpenDraco differs primarily in its focus on comparative experimentation for SE. Rather than proposing a new topology optimization method or general-purpose workflow builder, it combines configurable agents and communication structures with versioned specifications, execution, task-specific evaluation, and result comparison. Its APR instantiation integrates reusable agents and topology templates with SWE-bench evaluation, allowing researchers to modify, reuse, and compare MAS configurations and their outcomes and costs without reimplementing orchestration logic.

Inference page

Evaluation page

Results page

Instance list

Eval script

Reports

Topology graph

Agent stream

Live logs

Eval runner

Backend · Python · FastAPI Config loader

Agent catalog

Graph builder

Shared state

Topology builder

Config history

Agent runtime

Workflows

Cloud API

Topology config

Env variables

Instances

Predictions

3 À Evaluate

Results page

Evaluation page

Metrics across designs

Task evaluator (SWE-bench)

B. Frontend Layer OpenDraco provides an Angular-based interface. The Topology page connects to the Agentic System Modeler (Section III-C1), enabling users to define MASs as agent nodes and communication edges and configure their models, prompts, parameters, and tools. Configurations are stored as importable/exportable JSON files and Git-versioned. The Inference page executes selected instances, streams agent activity, and stores outputs. The Evaluation page runs taskspecific evaluators; for APR, it uses the SWE-bench harness to apply patches and execute tests. The Results page reports outputs, patches, evaluation results, wall-clock time, token usage, passed tests, and resolution status.

Ollama · local

Storage · Local files · no database Execution Registry

Run tasks, stream agents

line and deployed locally through Docker. Users select or define a topology, configure each agent’s role, prompt, tools, model, and parameters, and save the configuration for reuse and versioning. They then execute it on selected task instances, evaluate the outputs, and inspect the results. Finally, they can modify or replace the configuration and rerun it on the same instances to compare MAS designs under the same setting.

Inference Agentic System Executor

Configure agents & topology

Fig. 2. OpenDraco user workflow.

Notebook

Agentic System Modeler

Inference page

4 ¡ Analyze & Compare

Frontend · Angular SPA Topology page

Topology page

Refine and Re-Evaluate

III. O PEN D RACO A RCHITECTURE

Agent editor

2 ’ Execute

1 ¨ Model

C. Backend Layer Logs

The backend is implemented in Python and exposes its services through a FastAPI server, which is consumed by the frontend and can also be accessed by third-party components. 1) Agentic System Modeler: This module supports the modeling of MAS and produces a structured specification (e.g., JSON) containing: a) each agent, defined by its model, prompts, parameters (e.g., temperature), and tools; and b) the topology, i.e., how the agents are connected. It further provides: i) a validation mechanism, which checks the specification against LangGraph’s constraints and detects structural anomalies (e.g., unreachable agents or the absence of a path

Fig. 1. Component architecture of OpenDraco.

Fig. 1 shows the architecture of OpenDraco. The framework follows an n-tier design. Each tier is described below. A. User Workflow Fig. 2 summarizes the workflow followed by a user when configuring and executing a MAS with OpenDraco. OpenDraco can be accessed through its web interface or command

2

from START to END). Cyclic flows are supported and bounded at runtime by a revisit limit; ii) an agent catalog, which organizes agents by attributes such as role to support reuse across different systems; iii) configuration history, which maintains a Git history for each configuration file, allowing changes to be tracked and previous versions to be restored; and iv) persistence, which stores specifications in the file system and loads them when needed for reuse or sharing. 2) Agentic System Executor: This module converts a modeled MAS, represented as a JSON specification (Section III-C1), into a LangGraph StateGraph and runs it on a given problem instance to produce a prediction (e.g., a patch for a SWE-bench instance). It comprises: i) a graph builder that creates the LangGraph representation; ii) a typed shared state for agent communication; iii) an agent runtime that calls the model with its configured parameters, prompt, tools, and current state; and iv) a workflow runner that schedules nodes and supports fan-out and conditional edges. Each agent follows a common reasoning-and-acting loop [18], alternating between LLM calls and tool invocations until it produces a final response. Outputs are normalized before being passed downstream. A shared registry exposes the tool catalog and enforces per-agent allow lists defined by the tools list. 3) Evaluation Runner: This module evaluates outputs produced by the Agentic System Executor, including correctness or quality against ground-truth solutions. For APR, OpenDraco provides three modes: i) the official SWE-bench harness in Docker, ii) remote execution via sb-cli, and iii) a local evaluator for repositories outside SWE-bench—including those containing our synthetic instances—that clones the repository, applies the generated patch, and runs its tests. 4) Additional Layers: The Storage Layer persists execution registries, predictions, reports, metrics, configurations, logs, and results as local files, enabling direct inspection and editing. The Inference Layer supports Ollama2 backends running on localhost (the default) or another host, and cloud-based services through OpenAI-compatible APIs (e.g., OpenRouter).

IV. E VALUATION AND P RELIMINARY R ESULTS A. Evaluation Goals Our preliminary evaluation addresses four questions: whether OpenDraco can represent and execute existing agentic APR topologies end-to-end; whether the modeled systems can repair real-world bugs from SWE-bench Lite; whether the evaluation pipeline scales to a larger set of real-world instances; and how temperature affects run-to-run variability. We use synthetic bugs for the initial end-to-end assessment. B. Evaluation Metrics and Setup We use three instance sets: five synthetic APR instances spanning five difficulty tiers, the 23-instance dev subset of SWE-bench Lite, and a 77-instance subset of the SWE-bench Lite test split, restricted to instances whose gold patch is shorter than 600 characters. The synthetic set allows us to evaluate all eight configurations end-to-end before running the more expensive real-world experiments. We evaluate eight MAS configurations that vary in their topology and/or agent definitions. Collectively, they cover four topology patterns: chain (sequential), star (via a central agent), tree (hierarchical), and hybrid (combined). First, we define a chain baseline following a conventional APR sequence by connecting a bug-localization agent, a patchgeneration agent, and a validation agent. The remaining seven configurations draw on agentic APR systems reported on the SWE-bench leaderboard. Two configurations, tree and hybrid, are inspired by the communication structures of Prometheus and AgentScope, respectively, while using our own agent definitions. The other five reconstruct systems by combining agents from the catalog according to their communication structures. To assess run-torun variability, we execute the chain baseline five times at each temperature (0/1) on the synthetic set, using a different random seed for each run and recording the resolved rate, wall-clock time, and token usage. TABLE I R ESULTS WITH Q W E N 3.5:9 B . AG . ( ED .): AGENTS ( EDGES ); R ESOLVED : SYNTHETIC DIFFICULTY FROM TRIVIAL TO EXPERT.

D. Customization of OpenDraco for APR

Configuration

We instantiate OpenDraco for APR by specializing three components: i) The agent catalog, comprising agents derived from 16 publicly available repositories identified in our previous study [19]. These repositories implement solutions reported on the SWE-bench leaderboard, and the agents are classified by role (e.g., Bug Localizer or Patch Synthesizer). The catalog includes only artifacts covered by permissive upstream licenses (MIT, BSD, or Apache-2.0) and records their provenance. ii) The evaluation pipeline, through integration with the SWE-bench harness. iii) The UI, which exposes the APR agent catalog and predefined APR topology templates, provides access to the SWE-bench evaluation, and reports APR-specific results and metrics.

Ag. (ed.)

Wall-clock

Configuration comparison: synthetic set (5 difficulty tiers) chain (baseline) 4 (3) 29 min 2 s tree 7 (6) 31 min 41 s hybrid 6 (7) 35 min 21 s experepair_star 5 (7) 41 min 18 s hyperagent_star 5 (7) 29 min 35 s joycode_star 5 (7) 39 min 15 s lingxi_star 5 (7) 42 min 22 s openhands_star 6 (9) 17 min 32 s Real-world repair: SWE-bench Lite (23-instance dev subset) chain (baseline) 4 (3) 4 h 37 min hyperagent_star 5 (7) 7 h 40 min openhands_star 6 (9) 3 h 15 min

2 https://ollama.com

3

Resolved −

Total tokens

+

226,605 172,084 254,316 197,781 229,188 256,908 300,563 82,840 2/23 2/23 0/23

2,178,309 4,134,877 1,348,773

Evaluation scalability: SWE-bench Lite (77-instance test subset) chain 4 (3) 16 h 4 min 12/77

6,344,990

Run-to-run variability: chain on the Synthetic set chain - temperature 0 4 (3) 30–32 min chain - temperature 1 4 (3) 25–34 min

206.0–206.5k 198.7–276.8k

3/5 (all) 2/5 to 4/5

OpenDraco supports end-to-end experimentation and comparison of configurations by effectiveness, time, and token usage. Future work will support runtime topology evolution, evaluate additional local and cloud-based models over repeated runs, and extend the agent catalog, topology templates, and evaluators to other SE tasks. We also plan a user study on modeling and reproducibility effort, and to incorporate sustainability metrics into OpenDraco.

C. Preliminary Results Table I reports the results. All configurations use the same model (qwen3.5:9b) to ensure a consistent comparison. On the synthetic set, openhands_star resolves all five instances and also has the shortest wall-clock time and lowest token usage among the eight configurations. On the 23-instance development subset, both chain and hyperagent_star resolve two instances. However, chain uses 39.78% less time and 47.32% fewer tokens. Given this effectiveness–cost tradeoff, we select chain for the larger 77-instance test subset. It resolves 12 of the 77 instances (15.58%). At temperature 0, chain resolves the same three synthetic instances in all five runs. At temperature 1, the number of resolved instances ranges from two to four, and both runtime and token usage vary more across runs. This variability suggests that configurations using nonzero temperature should be evaluated over repeated runs.

ACKNOWLEDGEMENTS ChatGPT was used only to improve language and readability. All authors reviewed and validated the final manuscript and take full responsibility. R EFERENCES [1] X. Hou, Y. Zhao et al., “Large language models for software engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024. [2] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve realworld github issues?” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=VTF8yNQM66 [3] J. Li, Q. Zhang, Y. Yu, Q. Fu, and D. Ye, “More agents is all you need,” 2024. [Online]. Available: https://arxiv.org/abs/2402.05120 [4] J. He, C. Treude, and D. Lo, “Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, pp. 1–30, 2025. [5] M. Zhuge et al., “Gptswarm: Language agents as optimizable graphs,” in Forty-first International Conference on Machine Learning, 2024. [6] G. Zhang, Y. Yue, X. Sun, G. Wan, M. Yu, J. Fang, K. Wang, T. Chen, and D. Cheng, “G-designer: Architecting multi-agent communication topologies via graph neural networks,” 2025. [Online]. Available: https://arxiv.org/abs/2410.11782 [7] V. De Martino et al., “From prompting to engineering: A research agenda for prompt engineering in software engineering,” arXiv preprint arXiv:2609.02248, 2026. [8] “Github OpenDraco,” https://github.com/orgs/OpenDraco/repositories, 2026. [9] X. F. Ye Pan, V. De Martino, and M. Martinez, “Opendraco,” 2026. [Online]. Available: https://figshare.com/articles/software/OpenDraco/ 33390001/2 [10] “Opendraco video demo,” https://youtu.be/278sOR4lBEk, 2026. [11] J. Yang, M. Zhang, Y. Jin, H. Chen, Q. Wen, L. Lin, Y. He, S. Kumar, W. Xu, J. Evans, and J. Wang, “Topological structure learning should be a research priority for llm-based multi-agent systems,” 2025. [Online]. Available: https://arxiv.org/abs/2505.22467 [12] J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang et al., “Aflow: Automating agentic workflow generation,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 34 040–34 077. [13] FlowiseAI, “Flowiseai/flowise: Build ai agents, visually.” [Online]. Available: https://github.com/flowiseai/flowise [14] Langflow, “Low-code ai builder for agentic and rag applications.” [Online]. Available: https://www.langflow.org/ [15] “Crewai studio.” [Online]. Available: https://docs-platform.crewai.com/ platform/en/features/crew-studio [16] “Autogen studio - autogen.” [Online]. Available: https://microsoft.github. io/autogen/stable/user-guide/autogenstudio-user-guide/index.html [17] L. Accounts, “Langgraph studio: The first agent ide,” Apr 2026. [Online]. Available: https://www.langchain.com/blog/ langgraph-studio-the-first-agent-ide [18] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” 2023. [Online]. Available: https://arxiv.org/abs/2210.03629 [19] M. Martinez and X. Franch, “What’s in a benchmark? the case of swebench in automated program repair,” in Proceedings of the IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice, 2026, pp. 647–658.

D. Discussion 1) Framework Extensibility: To assess OpenDraco beyond APR, we modeled and executed proof-of-concept workflows through its UI (Section III-B), including a translation workflow that reads a text file, translates it, and evaluates the output with BLEU against a reference. This required only task-specific agents, topologies, tools, and evaluators, without changes to the framework core. OpenDraco can support new tasks through configuration without framework modification. 2) Limitations of the Experiments: Reconstruction Fidelity: The reconstructed APR configurations approximate the original systems in terms of agents, prompts, tools, and communication structures, but not their exact runtime behavior. Thus, our results characterize their OpenDraco instantiations, and differences cannot be attributed to topology alone. Resources: Due to the 6 GB VRAM of the GPU, all agents were executed sequentially, including independent branches. Reported wall-clock times therefore do not capture potential parallelization gains. Model Choice: All experiments use the locally hosted qwen3.5:9b, limiting generalizability to other models. We chose an open local model to support reproducibility, keep source code local, and avoid proprietary paid APIs. Evaluation Scope: Except for the variability study, each configuration was executed once. Moreover, only three configurations were evaluated on real-world instances, with chain alone evaluated on the larger subset, limiting conclusions about variability and configuration differences. V. C ONCLUSION AND F UTURE W ORK This paper presented OpenDraco, an open-source platform for modeling, executing, and evaluating configurable LLMbased MAS for SE. OpenDraco captures agents, prompts, tools, models, parameters, and communication topology in a versioned, shareable configuration, allowing MAS experiments to be modified and reproduced without changing the framework core. A preliminary APR evaluation shows that

4

Related documents

• OpenDraco: A Platform for Engineering and Experimenting with Configurable Multi-Agent Architectures for Software Engineering#1448202
• Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report#448000
• CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology#1199619
• Evaluating LLM Agents on Automated Software Analysis Tasks#10421
• Understanding Issues, Causes and Solutions in Open-Source LLM-based Multi-Agent Systems#1169626