Unifying Temporal and Structural Credit Assignment in LLM-Based Multi-Agent Prompt Optimization
Wenwu Li1
arXiv:2605.30227v1 [cs.MA] 28 May 2026
1
Yuran Song1
Mingze Zhao2
Wenhao Li* 1
Tongji University, Shanghai, China, {wenwu,2250753,bjin,whli}@tongji.edu.cn 2 The University of Hong Kong, Hong Kong, China, [email protected]
Abstract While Multi-Agent Systems (MAS) empower Large Language Models to tackle complex reasoning tasks through collaborative interaction, optimizing their dynamics remains a formidable challenge due to the discrete, non-differentiable nature of the computation graph and the sparsity of global supervisory signals. Existing black-box optimizers struggle to attribute trajectory-level failure to specific local components, resulting in inefficient, high-variance exploration. We argue that tractable MAS optimization needs structural inductive biases to disentangle error signals. We propose temporal and structural credit assignment, which decomposes the objective along two axes: (i) temporal credit, using state-space bottlenecks to identify critical rounds, and (ii) structural credit, using stationary role policies to isolate agent contributions. Leveraging these decomposed signals, we introduce a discrete, verbalized block coordinate descent algorithm for iterative refinement. Rather than indiscriminate global updates, it alternates between optimizing role prompts and aggregation protocols, using LLM-generated “proxy gradients” to target only the identified weak links. Across diverse reasoning benchmarks, our approach substantially reduces query complexity while improving performance, providing a principled and interpretable path toward self-improving MAS.
1
Bo Jin1
INTRODUCTION
Large Language Models (LLMs) have evolved from static text generators into dynamic reasoning engines, achieving remarkable success in mathematical reasoning, code generation, and complex planning [Brown et al., 2020, Wei et al., * Corresponding author.
2022]. To transcend the limitations of single-model inference, recent research has shifted toward MAS, where specialized agents collaborate via iterative critique-and-revision cycles [Zhuge et al., 2024, Zhang et al., 2025a]. By distributing tasks across diverse roles (e.g., proposer, debater, synthesizer), MAS frameworks can theoretically solve problems that are intractable for a single monolithic model [Li et al., 2024a]. However, optimizing the interaction dynamics of MAS presents a formidable challenge. From an optimization perspective, an LLM-based MAS operates as a discrete, nondifferentiable computation graph. Unlike neural network training where backpropagation precisely attributes error to specific weights, MAS optimization suffers from the Credit Assignment Problem (CAP) in its most severe form: the supervisory signal is typically sparse (a single scalar score at the terminal state) and global (applying to the entire trajectory) [Sutton and Barto, 2018, Foerster et al., 2018]. Existing approaches often treat the entire system as a black box [Deng et al., 2022, Fernando et al., 2023, Guo et al., 2023], applying derivative-free optimization or heuristic aggregation [Nagpal et al., 2025, Ai et al., 2025]. While effective for simple chains, these methods struggle in complex multi-round collaborations because they ignore the internal causal structure of the reasoning process. Optimizing a system with N agents over R rounds essentially involves searching in a space of O(N ×R) coupled prompts. Without structural priors, this faces catastrophic variance and inefficient exploration [Zhou et al., 2024]. In this work, we argue that making MAS optimization tractable requires imposing structural constraints on the computation graph to disentangle the error signal. We propose two principled relaxations that transform the chaotic interaction graph into a structured optimization landscape. First, state-space bottleneck via aggregation. In a fully connected MAS, information flows diffusely, making it impossible to pinpoint when reasoning collapsed. We introduce an explicit Aggregation Module at each round to summarize
Preprint. May 29, 2026.
CONCRETE EXAMPLE ILLUSTRATION Section 3-Spatiotemporal Credit Assignment
Section 1-Multi-Agent Iterative Reasoning
Task Input
Agent A (Planner) Round 1
Section 2-Aggregation Module (Core Mechanism)
Agent B (Solver) Agent C (Critic) Round 2
Round R
Aggregation Module
Inputs: Agent messages from the current round
B. Spatial Credit Assignment (Across Agents)
Temporal Weights
Inputs: Aggregated system states S1, S2, ..., SR
Inputs: Agent outputs across rounds
Spatial Weights
Attribute outcome changes to agent-specific contributions. Aggregate evidence across rounds.
Prompt A
Measure marginal impact of each aggregation update on the final outcome. Rank and normalize round-level contributions.
Shared System State Sr
.. .
Temporal Credit Weights
Medium
Feedback for next round
Agent A:
Agent B: 0.21 Flight options...
Agent B
Identify limiting or effective agent roles
Identify critical reasoning rounds and suppress low-value states
Spatiotemporal Credit-Guided Prompt Optimization Framework
Low
Prompt C
EXAMPLE REASONING ROUNDS
0.89 Initial destinations...
Prompt Aggregation + Message A + Message B + Message C
Prompt B
Agent A Agent B Agent C
...
0.71
S2
Round 2 Agent C: 0.34 Hotel suggestions...
Agent B: 0.91
Agent A: 0.13
Refined plan based on flight...
Shared System State S1
Adjusted costs...
Shared System State S2
Shared System State Sr
Aggregation Module
Agent C
S1
Round 1 EXAMPLE TASK: TRAVEL ITINERARY PLANNING
Agent A
High
...
Prompts are reused across rounds, but contributions to the final outcome vary.
Section 4-Credit-Guided Prompt Update (Inference-Time)
A. Temporal Credit Assignment (Across Rounds)
0.27
...
Sr
0.34
TEMPORAL CREDIT
Based on Credit Assignment
Agent C: 0.24 Added local activities...
BEFORE / AFTER PROMPT UPDATE ...
BEFORE PROMPT General Goal: Create a 5-day travel itinerary.
AFTER PROMPT Specific Guidance (Based on R2 & Agent A): Focus on refining daily plans after initial flight selection (R2). Prioritize destination logic and flow over initial cost (Agent A).
Figure 1: Overview of the credit-guided prompt optimization pipeline. Top: a multi-agent, multi-round reasoning loop (planner/solver/critic) produces per-round messages that are aggregated into a shared system state Sr ; an aggregation module feeds back to the next round. From the completed trajectory, we compute temporal credit across rounds (identifying critical rounds) and structural/spatial credit across agents (identifying effective or limiting roles). These credits then drive inference-time prompt updates, selectively refining the lowest-credit rounds/roles while keeping strong components fixed. Bottom: an example travel-itinerary task illustrates per-round agent outputs, the evolving shared state (S1 , S2 , . . .), temporal credit weights, and a before/after prompt update that specializes guidance to the weak round/role. agent outputs into a unified state St . This effectively models the interaction as a Markov Decision Process (MDP), where the aggregator creates a “state bottleneck” [Puterman, 1994, Mnih et al., 2015]. This architectural choice is not merely functional; it is a prerequisite for Temporal Credit Assignment, allowing us to evaluate the quality of consensus at discrete time steps independent of individual agent noise.
descent, we utilize LLM-based critics to compute “proxy gradients”—decomposed credit scores that guide targeted textual updates only to the bottleneck components. Our contributions are summarized as follows: 1) We reformulate MAS prompt optimization by introducing statespace bottlenecks and stationary policy constraints. These inductive biases bridge the gap between sparse terminal rewards and dense, actionable feedback. 2) We propose a theoretically grounded optimization algorithm that alternates between structural (role) and temporal (aggregator) updates. This mimics Block Coordinate Descent in the discrete prompt space, ensuring stability and convergence compared to joint optimization baselines. 3) Numerical experiments demonstrate that our approach achieves higher task accuracy while reduces query complexity. By focusing updates on specific "weak link" roles or "phase transition" rounds, we avoid the inefficiency of indiscriminate global updates.
Second, stationary policy via parameter sharing [Gupta et al., 2017, Yu et al., 2022]. Optimizing unique prompts for every agent at every round creates an explosion of parameters. We impose a Stationary Policy Constraint, where each role (e.g., the Debater) shares the same system prompt across all interaction rounds. This reduces the search space from trajectoryspecific instructions to robust role definitions. Crucially, this constraint enables reliable Structural Credit Assignment: by observing an agent’s performance across multiple rounds, we can distinguish systematic role incompetence from transient stochastic errors.
2
Building on these structural insights, we propose a unified framework for Temporal and Structural Credit Assignment, as shown in Figure 1. We formalize the MAS optimization problem as a bi-level objective depending on two orthogonal variable blocks: (i) Role Prompts (defining agent behaviors) and (ii) Aggregation Prompts (defining state transition protocols). Instead of optimizing these jointly—which is prone to instability—we derive a novel Block Coordinate Descent (BCD) algorithm adapted for discrete prompt optimization [Tseng, 2001, Cai et al., 2023]. Our method alternates between refining role-specific instructions (fixing the aggregation logic) and refining the aggregation protocol (fixing the role behaviors). To drive this
RELATED WORKS
Prompt Optimization. Several methods have been developed for optimizing prompts, ranging from learning parameter-efficient soft prompts to automatically searching for optimal discrete prompts using reinforcement learning, meta-optimization, and evolutionary algorithms. PromptTuning [Lester et al., 2021] freezes the parameters of a pretrained language model and learns task-specific soft prompts via end-to-end backpropagation. RLPrompt [Deng et al., 2022] optimizes discrete textual prompts through reinforcement learning with a policy network. OPRO [Yang et al., 2023] frames prompt optimization as an iterative black2
box optimization problem. EvoPrompt [Guo et al., 2023] integrates evolutionary algorithms with large language models by employing the LLM to implement linguistically coherent crossover and mutation operations. PromptBreeder [Fernando et al., 2023] implements a self-referential selfimprovement mechanism for prompts using a genetic algorithm that co-evolves task-prompts and mutation-prompts.
teraction rounds. The base LLM parameters are fixed; the optimization variables are the prompts used by each agent at each round. Let ϕi,t denote the prompt fed to agent ai at round t. We collect all prompts as Φ ≜ {ϕi,t }N,R i=1,t=1 . Given an input instance x ∼ D, the system runs for R rounds. At round t, each agent ai produces an utterance ui,t conditioned on x and its prompt ϕi,t :
LLM-based MAS. Current research on LLM-based multiagent systems spans four key dimensions: system representation, optimization mechanisms, dynamic adaptation, and reasoning efficiency. GPTSwarm [Zhang et al., 2024] exemplifies offline structural learning via static post-training optimization with strong theoretical grounding in policygradient methods. DyLAN [Liu et al., 2024] represents unsupervised online selection, offering principled agent evaluation without labeled data and drawing theoretical connections to Shapley-value attribution. EvoMAC [Li et al., 2024b] reflects environment-driven online adaptation, enabling test-time improvement from objective feedback and linking classical control-theoretic feedback to LLM generation. MAS-GPT [Chen et al., 2024] illustrates generative offline learning, trading development-time cost for inference-time efficiency and cross-domain transfer.
ui,t ∼ p(· | x, ϕi,t ) , i ∈ {1, . . . , N }, t ∈ {1, . . . , R}. (1) Let Ut ≜ {u1,t , . . . , uN,t } be the set of role utterances at round t. An aggregation module summarizes Ut into a shared state St = ft (Ut ), t ∈ {1, . . . , R}, with an initial state S0 fixed. Prompt-as-Input and Output-as-Context. We define each round prompt as a composition of (i) a system instruction, (ii) the previous-round shared state, and (iii) a role-specific template. Concretely, for each agent ai and round t, ϕi,t = gi,t (x, St−1 ; θi,t ) , (2) where gi,t (·) is a (deterministic) prompt constructor and θi,t denotes its free text content (the part we optimize). This explicitly captures that each agent’s output at round t − 1 becomes part of the input for round t.
Credit Assignment. The credit assignment problem refers to determining the contribution of each agent to the collectively obtained reward in a multi-agent system. In reinforcement learning context, when an agent receives a reward, it is challenging to determine which actions should be credited or blamed for the outcome. In multi-agent systems context, under the centralized training with decentralized execution paradigm, it is required to allocate contributions from joint decisions to individual agents. Value decomposition methods address this challenge by employing mixing networks to decompose the joint state-action value function into individual local observation-action value functions. MATTRL [Hu et al., 2026] implements inference-time credit assignment via Difference Rewards for precise contribution attribution in multi-agent reasoning, while MAPRO [Zhang et al., 2025b] employs training-time credit assignment for joint optimization of discussion and answer generation.
3
Final-Round Scoring. After the final round, a terminal decision module produces the system output from (x, SR ): ŷ = J(x, SR ).
(3)
A task-specific scorer assigns a scalar score to the final output s(x, ŷ) ∈ R. Equivalently, the trajectory-level score can be written as J(τ ; x) ≜ s(x, j(x, SR )) , τ ≜ U1 , S1 , . . . , UR , SR . (4) Optimization Objective. The basic prompt optimization problem is to maximize the expected final-round score: h i max Ex∼D Eτ ∼p(·|x;Φ) J(τ ; x) . (5) Φ
PROBLEM FORMULATION
Since J(τ ; x) is only observed at the end of the R-round interaction, it depends on all upstream prompts {ϕi,t } through the coupled generation process Eq.(1)–Eq.(3).
We now formalize the multi-agent, multi-round prompting setting that our temporal and structural credit assignment targets. We specify the interaction protocol, notation, and the optimization variables, which will let us define per-round and per-role credit signals precisely. This section sets up the trajectory and objective used by the attribution method and the credit-guided prompt optimization procedure.
Textual-Gradient Prompt Update. Because prompts are discrete text and the base LLM parameters are fixed, we do not backpropagate numerical gradients through Eq.(1). Instead, we convert the terminal score into a textual gradient (natural-language feedback) [Xiao et al., 2024, Yüksekgönül et al., 2024] that specifies how to revise prompts:
Multi-Agent, Multi-Round Prompting. We consider a multi-agent, multi-round LLM reasoning system with a fixed set of agents (roles) A = {a1 , . . . , aN } and R in-
∆i,t = G(x, τ, J(τ ; x), i, t) , 3
(6)
where G(·) outputs a feedback string for prompt (i, t). Prompts are updated by applying the feedback to the current prompt text: (k+1) (k) (k) ϕi,t = U ϕi,t , ∆i,t , i = 1, . . . , N, t = 1, . . . , R, (7) where k indexes optimization iterations and U(·) is a textedit operator (e.g., rewrite / insert constraints). This completes the most basic definition of the optimization problem in a multi-agent, multi-round system, where the final-round score is propagated back to prompts via textual gradients.
4
instantiate a state-space bottleneck across rounds. This enables textual gradients to be grounded on intermediate representations ∆t = Gt (x, St , SR , J(τ ; x)), and further decomposed to individual agents: ∆i,t = Gi,t (x, ui,t , St , SR , J(τ ; x)) .
In this way, aggregation provides the structural interface that makes textual gradients representable, alignable, and decomposable across both temporal rounds and agent roles. New optimization variables. Crucially, the aggregation module is itself prompt-driven, so introducing shared states also introduces new decision variables: the aggregator prompts that control how Ut is summarized. Let ψt denote the aggregation prompt at round t, and write the state update as St = ft (Ut ; ψt ). The optimization target therefore changes from role-prompt-only tuning to a joint objective over role prompts and aggregation prompts: h i max Ex∼D Eτ ∼p(·|x;Φ,Ψ) J(τ ; x) , Ψ ≜ {ψt }R t=1 . (9)
METHODOLOGY
Building on the naive objective in Eq.(5), we make two relaxations to obtain a tractable optimization scheme. First, we introduce a state-space bottleneck by inserting a roundindexed aggregation module (one aggregator per round), so round-level decisions become explicit and scorable. Second, we impose parameter sharing by instantiating a fixed set of role-specialized agents whose prompts are shared across rounds, so credits can be accumulated by role rather than by individual turns. During optimization, we evaluate intermediate role outputs and per-round aggregation outputs with LLM-based critics to obtain proxy signals for credit estimation. We use the terminal scorer s(x, ŷ) as the optimization objective, while LLM critics provide auxiliary signals to estimate role- and round-level credits for targeted prompt updates. These critic signals are transformed into two forms of credit: (i) structural credit over roles, used to identify and optimize weak agents; and (ii) temporal credit over rounds, implemented as a per-round aggregator credit score used to trigger targeted aggregation-prompt optimization. 4.1
(8)
Φ,Ψ
This shift is important: optimizing aggregation can change the effective trajectory distribution and thus the credit signals used to update role prompts. 4.2
PARAMETER SHARING
We also analyze a constrained regime with parameter sharing, where each agent reuses a single prompt across all interaction rounds. Formally, for each agent ai , we impose ϕi,1 = ϕi,2 = · · · = ϕi,R ≜ ϕi .
(10)
This reduces the number of optimization variables, but increases the influence of each prompt because it is applied repeatedly under different aggregated contexts. Under this constraint, the optimization objective becomes max Ex∼D Eτ ∼p(·|x;{ϕi },Ψ) J(τ ; x) . (11)
STATE-SPACE BOTTLENECK
In MAS, the terminal score provides a single global optimization signal. To update prompts, this signal must be expressed as a textual gradient. Whether such feedback can be made actionable depends on the availability of a statespace bottleneck: an explicit shared state that concentrates, rather than disperses, the information flow.
{ϕi }N i=1 ,Ψ
This is also our final optimization target. See Fig. 2 for the overall evolution. Although simplified, the problem remains non-trivial: each shared prompt ϕi affects the final score through multiple uses across evolving states {St−1 }. Effective optimization therefore requires aggregating feedback across rounds, which naturally motivates temporal credit assignment under parameter sharing.
Without aggregation. If no aggregation is used, the final output is derived directly from the collection of all utterances {ui,t }i,t . In this case, the textual gradient can only be expressed as an unstructured global signal ∆ = G(x, {ui,t }i,t , J(τ ; x)). Because there is no explicit bottleneck to anchor intermediate credit, the feedback cannot be aligned with specific agents or interaction rounds, providing only coarse guidance for prompt updates.
Naive objective Eq. (5)
Add aggregation prompts Eq. (9)
Share role prompts Eq. (11)
Figure 2: Evolution of the optimization objective: from the naive terminal-score objective, to introducing round-wise aggregation prompts, to enforcing shared role prompts.
With aggregation. By introducing an aggregation module, the system maintains explicit shared states {St }, which 4
4.3
VERBALIZED BCD
Algorithm 1: BCD for Temporal and Structural CreditGuided Prompt Optimization Input: D; fixed base LLM; rounds R; roles A = {ai }N i=1 R Output: Optimized prompts {ϕi }N i=1 and {ψt }t=1
Having specified a new optimization objective, the next step is to design an algorithm that can effectively solve it. Our objective provides supervision only at the end of the R-round interaction (Eq. (11)), leaving the intermediate rounds and roles without direct learning signals. Crucially, terminal success does not imply that the final aggregation round (or speaker) is solely responsible; earlier rounds can be decisive or harmful. We therefore introduce LLM-based critics as auxiliary evaluators of intermediate utterances and aggregations, and translate their scores into temporal and structural credits. These credits then drive a targeted block coordinate descent procedure that alternates between updating low-credit role prompts and low-credit aggregation prompts, while keeping high-credit components fixed.
(0)
(0)
Initialize: {ϕi }, {ψt }; k ← 0; while not converged do // Step 1: Structural block update (fix {ψt }) Compute {Cirole }N i=1 ; Select low-credit roles I (k) ⊆ {1, . . . , N }; foreach i ∈ I (k) do (k+1) (k) ϕi ← U ϕi , ∆role ; i (k+1)
Set ϕi
Credit Computing. For the structural dimension, let qi,t ∈ [0, 1] denote a normalized critic score for role i’s utterance ui,t at round t. To account for interaction effects, we introduce a peer-view proxy pi,t ∈ [0, 1]: we do not require each agent to explicitly score others; instead, a single LLM-based judge conditions on the full set Ut and assesses each role’s contribution to group reasoning. We fuse these two signals as crole i,t = λ qi,t + (1 − λ) pi,t where λ ∈ [0, 1], and aggregate across PR rounds to obtain a role-level structural credit Cirole = R1 t=1 crole i,t . For the temporal dimension, let agg qt ∈ [0, 1] be the critic score for the round-t aggregation output St . We define the temporal credit as Cttime = qtagg .
(k)
← ϕi
for i ∈ / I (k) ;
// Step 2: Temporal block update (fix {ϕi }) Compute {Cttime }R t=1 ; Select low-credit rounds T (k) ⊆ {1, . . . , R}; foreach t ∈ T (k) do (k) (k+1) ← U ψt , ∆time ; ψt t (k+1)
Set ψt
(k)
← ψt
for t ∈ / T (k) ;
k ← k + 1; (k)
(k)
R return {ϕi }N i=1 , {ψt }t=1 ;
Verbalized BCD over Prompt Blocks. Our optimization is a special instance of BCD over two prompt blocks: (i) the structural prompts that instantiate role behaviors, denoted by Φrole (e.g., {ϕi }N i=1 under the shared-prompt constraint in Eq. (10)); and (ii) the temporal prompts that instantiate the round-indexed aggregation modules, denoted by Ψ ≜ role,(0) {ψt }R , Ψ(0) ), each outer iteration t=1 . Starting from (Φ k performs two alternating phases:
We repeat Phases A–B until a fixed budget is exhausted or the held-out performance saturates. This alternating design makes the procedure implementable and stable: at each step we optimize one block while treating the other as a fixed environment, reducing unnecessary drift in the coupled multi-round interaction.
5
Phase A: optimize roles while fixing aggregation prompts. We keep Ψ(k) fixed, roll out trajectories, compute {Cirole }, select a subset of low-credit roles (e.g., bottom-K or below a threshold), and update only their prompts using textual gradients (Eq. (7)): Φrole,(k+1) ← ROLE O PT Φrole,(k) ; Ψ(k) , {Cirole } . (12)
EXPERIMENTS
To validate the proposed temporal and structural credit assignment and the resulting prompt optimization routine, we now turn to empirical evaluation. The following section specifies datasets, protocols, and baselines, and then quantifies how the credit signals translate into measurable gains and interpretability across benchmarks.
Phase B: optimize aggregation prompts while fixing roles. We then keep Φrole,(k+1) fixed, roll out trajectories again (or reuse logged trajectories when applicable), compute {Cttime }, select low-credit rounds, and update only their aggregation prompts: Ψ(k+1) ← AGG O PT Ψ(k) ; Φrole,(k+1) , {Cttime } . (13)
5.1
SETTINGS
We evaluate our proposed credit-guided prompt optimization on multiple-choice reasoning benchmarks, where each question has four options and the correct answer is selected from {A, B, C, D}. Models generate free-form text, and we deterministically extract the predicted option using a fixed 5
parsing rule: the first occurrence of a standalone A/B/C/D token in the output. If no valid option is found, the prediction is considered invalid and counted as incorrect. These parsing rules are fixed prior to evaluation and remain constant across all runs. We report results on AQuA [Ling et al., 2017], MedMCQA [Pal et al., 2022], GPQA [Rein et al., 2024], and MMLU [Hendrycks et al., 2021b,a].
(ii) Temporal optimization. Temporal credit is assigned to per-round aggregators using the EMA update described in Section 3. Credit is updated only on informative failures (aggregator wrong but at least one agent correct). When the credit of round t drops below a threshold and sufficient failures are observed, only that round’s aggregator prompt is optimized. All hyperparameters are fixed across datasets.
For each dataset, we randomly select a fixed optimization set of 100 examples used exclusively for prompt search and optimization (i.e., updating role prompts and round-wise aggregator prompts). All reported metrics are evaluated on a larger, disjoint leave-out test set, which is never accessed during optimization. The split is fixed across runs. No test examples are used for prompt search, gradient updates, credit assignment, or hyperparameter tuning. Accordingly, we emphasize no test-set exposure, rather than zero-shot performance, on the reported test set.
5.2
MAIN RESULTS AND ANALYSIS
Based on the analyses in this section, the experiments aim to answer four research questions: RQ1: Does credit-guided prompt optimization consistently improve accuracy across datasets and models compared with unoptimized prompts and a black-box baseline? RQ2: How much do structural and temporal optimization each contribute, and does their combination yield the strongest and most consistent gains? RQ3: Is the optimization process efficient and stable (convergence speed and variance), and how do key hyperparameters affect performance and stability? RQ4: Do credit signals align with error types and outcome shifts, providing interpretable diagnostics and actionable guidance for follow-up optimization?
Random seeds are fixed for sampling the optimization set and for any stochastic components in decoding or optimization. Unless otherwise stated, results are averaged over multiple runs with different seeds, and we report the mean accuracy along with standard deviation. Prompt optimization is performed independently for each dataset. This constitutes dataset-specific prompt search over a fixed base model, not model fine-tuning. Cross-dataset generalization is not claimed unless explicitly evaluated.
5.3
Models and Runtime. All debate agents are instantiated with open-weight instruction-following LLMs, allowing inspection and modification of prompts. We use Qwen2.57B-Instruct, LLaMA3-8B-Instruct, and Gemma-7B-Instruct for the main experiments, and decoding settings are kept identical across baselines and our method.
RQ1: EFFECTIVENESS
Table 1 summarizes the performance of our credit-guided prompt optimization compared to two baselines: the unmodified prompts and a black-box optimization baseline. Across all datasets and models, our method consistently improves accuracy. For instance, on MedMCQA with LLaMA3-8B, our approach improves accuracy by 7.0% relative to the baseline, while on GPQA with Qwen2.5-7B the gain is 2.1%. Even on datasets with higher initial accuracy, such as AQuA and MMLU, we observe consistent improvements. Standard deviations across multiple runs remain low, indicating stable and reproducible performance. These results confirm that integrating structural and temporal credit signals effectively guides prompt optimization in multi-agent settings.
Frameworks and Baselines. We evaluate our approach on several multi-agent frameworks, with a focus on LLMDebate and DyLAN. In all settings, the interaction protocol (number of agents, number of rounds, and aggregator placement) is fixed. We compare against: (i) the unmodified baseline prompts, (ii) a black-box prompt optimization baseline using DSPy MIPRO (multi-stage instruction and demonstration optimization) [Opsahl-Ong et al., 2024] that performs agent-wise non-informed prompt search, and (iii) our credit-aware optimization.
RQ1 Answer. Credit-guided prompt optimization delivers broad, reliable accuracy gains across datasets and model families, outperforming both the unoptimized prompts and the black-box baseline, which supports its generalizability and robustness.
Training and Optimization Protocol. We follow a twostage prompt-only procedure; no model parameters are updated. First, DSPy (MIPROv2) is used solely for automated prompt initialization [Opsahl-Ong et al., 2024]. Second, fixed-budget prompt edits are performed using only the optimization split; the test split is never accessed. (i) Structural optimization. We evaluate each agent’s final-round answer with AgentTurnEval, aggregate scores into a risk statistic, and deterministically select the two worstperforming roles. Only these role prompts are optimized using RolePromptOpt with summarized error diagnosis.
5.4
RQ2: ABLATIONS
To understand the source of improvements, we perform ablation experiments. We isolate the contributions of structural and temporal components: Role-only optimization (structural): optimizing only the worst-performing role prompts selected by the risk statistic. Aggregator-only optimization 6
Table 1: Main results on multiple-choice reasoning benchmarks. The main number is the mean accuracy, and the right-side annotation reports the delta relative to the corresponding MAS baseline: “-” marks the baseline, ↑ denotes an increase, and ↓ denotes a decrease. Cell colors follow a red–white–green gradient from negative to zero to positive deltas, and the lower number in each cell is the standard deviation. DyLAN Domain
Debate
Model
baseline
optimized
ours
baseline
optimized
ours
LLaMA3-8B
54.13− ±2.3936
61.75↑7.63 ±3.3040
61.13↑7.00 ±1.0308
55.13− ±3.2243
61.63↑6.50 ±2.1360
64.63↑9.50 ±1.7970
Qwen2.5-7B
51.88− ±2.3229
53.88↑2.00 ±1.6520
55.75↑3.88 ±1.9365
54.50− ±1.0000
54.88↑0.38 ±3.3510
55.50↑1.00 ±4.3205
Gemma-7B
19.13− ±1.1087
19.38↑0.25 ±0.9465
21.33↑2.21 ±2.4664
26.75− ±3.1820
27.50↑0.75 ±2.4833
28.50↑1.75 ±2.5981
LLaMA3-8B
27.42− ±2.2728
29.61↑2.19 ±3.0873
28.11↑0.69 ±1.8779
39.75− ±1.7816
35.02↓4.73 ±1.5535
36.75↓2.30 ±4.4439
Qwen2.5-7B
32.83− ±2.3308
35.02↑2.19 ±2.0956
32.37↓0.46 ±1.6581
32.26− ±1.4546
33.07↑0.81 ±1.3212
35.83↑3.57 ±0.8752
Gemma-7B
11.52− ±0.4600
12.29↑0.77 ±0.5312
11.98↑0.46 ±0.6505
12.56− ±1.3778
17.75↑5.19 ±3.5850
18.67↑6.11 ±1.6334
LLaMA3-8B
51.15− ±3.0376
55.46↑4.31 ±2.5500
56.32↑5.17 ±1.6263
53.16− ±3.6784
52.01↓1.15 ±4.3354
52.30↓0.86 ±9.4087
Qwen2.5-7B
83.05− ±1.7250
85.35↑2.30 ±1.7250
83.91↑0.86 ±1.3279
83.05− ±2.8750
81.04↓2.01 ±1.4846
83.62↑0.58 ±3.3031
Gemma-7B
18.72− ±2.9480
22.42↑3.70 ±0.8132
23.57↑4.85 ±0.6640
18.62− ±1.4994
22.61↑3.99 ±1.3279
27.59↑8.97 ±2.3000
LLaMA3-8B
56.43− ±1.0820
56.20↓0.23 ±0.5952
56.30↓0.13 ±0.7407
68.78− ±1.0444
69.33↑0.55 ±2.6207
74.77↑6.00 ±2.8646
Qwen2.5-7B
65.35− ±1.1045
66.38↑1.03 ±1.1386
67.48↑2.13 ±0.6135
61.58− ±1.3371
61.95↑0.38 ±1.9265
63.38↑1.80 ±2.0126
Gemma-7B
29.40− ±0.3162
30.40↑1.00 ±0.4733
32.00↑2.60 ±0.2828
28.25− ±4.8946
33.73↑5.48 ±3.7754
33.60↑5.35 ±2.2627
MedMCQA
GPQA
AQuA
MMLU
(temporal): optimizing only the low-credit round aggregators while freezing role prompts.
RQ2 Answer. The ablations show complementary effects: role-only (structural) updates yield larger gains on role-sensitive datasets such as MedMCQA, while aggregator-only (temporal) updates are more beneficial on consolidation-heavy tasks such as GPQA. Combining both consistently gives the best accuracy across MedMCQA and MMLU, outperforming either component alone.
Figure 3 visualizes the LLaMA3-8B Debate results on MedMCQA and MMLU, highlighting the consistent gains from credit-guided prompt optimization. Both components independently improve accuracy relative to the baseline. Structural optimization contributes more on role-sensitive datasets (e.g., MedMCQA), whereas temporal optimization is most effective on datasets where aggregator consolidation is critical (e.g., GPQA). The combination consistently achieves the highest performance.
5.5
RQ3: SENSETIVITY
We evaluate performance across optimization iterations to assess efficiency and stability. Figure 4 shows the accuracy trajectories across iterations, highlighting faster gains for our method versus DSPy MIPRO and the baseline. Creditguided optimization converges rapidly within a few iterations and maintains low variance across runs. In contrast, the black-box baseline converges more slowly and exhibits higher variance, reflecting inefficient exploration.
We further analyze temporal optimization at the round level. Restricting updates to one round t at a time reveals which rounds are most error-prone. Early rounds often dominate failures, indicating that losing key evidence early can limit overall accuracy, while late rounds mainly affect final consolidation. These findings provide actionable guidance on where optimization should focus.
7
Table 2: Distribution of prediction shifts before and after optimization. Compared with DSPu(MIPRO), our method yields more incorrect→correct repairs and fewer correct→incorrect regressions, indicating that credit signals act as structured diagnostics rather than global perturbations.
Figure 3: LLaMA3-8B Debate results on MedMCQA and MMLU: the combined temporal+structural credit-guided optimization yields the strongest gains, with structural-only and temporal-only providing smaller improvements over the baseline.
Shift
DSPy(MIPRO) (%)
Ours (%)
X→X ✓→X X→✓ ✓→✓
38.00 7.00 7.00 48.00
34.50 5.00 11.00 49.50
specific optimizations mainly improve categories flagged as failing by the critic, and temporal credit αt correlates with aggregator reliability rather than mere signal presence. Together, these analyses confirm that structural and temporal credit signals provide both mechanistic interpretability and practical guidance for follow-up experiments. RQ4 Answer. Taken together with RQ1–RQ3, the credit signals function as actionable diagnostics: they pinpoint which rounds and roles actually drive outcome changes, explain when gains persist or regress under tuning, and indicate where the next optimization pass should focus.
Figure 4: Accuracy convergence over optimization iterations; ours converges faster and to higher accuracy than DSPy MIPRO and the baseline.
6 We also investigate hyperparameter sensitivity by varying the temporal credit threshold τ and buffer size m. Lower τ or larger m results in conservative updates, preserving stability, while higher τ or smaller m allows more aggressive updates, increasing the risk of performance drift. Monitoring credit trajectories αt and test accuracy variance confirms that our recommended settings achieve a favorable stability–plasticity balance.
In this paper, we introduce a test-time temporal and structural credit assignment framework that decomposes multiagent LLM trajectories along rounds and roles, enabling component-level attribution without altering inference dynamics or model parameters. Across the experiments, the resulting credit signals reveal pronounced contribution imbalance and identify the small set of stages and agents that consistently constrain outcomes. Leveraging these signals, credit-guided prompt optimization focuses updates on weak components and delivers reliable gains across benchmarks and model families, reducing wasteful global changes while improving effectiveness.
RQ3 Answer. Building on the consistent gains in RQ1 and the complementary effects in RQ2, the optimization is efficient because it targets the most influential components, achieving improvements with only a few focused iterations. Stability follows from this targeted selection, while τ and m provide explicit control over the trade-off between cautious refinement and faster but riskier updates.
5.6
CLOSING REMARKS
These findings position temporal and structural attribution as a practical diagnostic and optimization tool for multi-agent reasoning systems. At the same time, the framework remains limited to completed trajectories and relies on evaluative signals; temporal attribution can be computationally heavy for long horizons; and the current instantiation assumes fixed roles with centralized aggregation. Future directions include integrating credit signals into online or inference-time adaptation, extending attribution to weakly supervised settings via self-consistency or consensus critics, and applying the framework to more dynamic coordination regimes such as hierarchical, tool-augmented, or open-ended environments.
RQ4: INTERPRETABILITY
To validate that optimization aligns with failure diagnosis, we analyze outcome shifts and error types. Compared with black-box edits, credit-guided optimization reduces regressions (correct→incorrect) and increases repairs (incorrect→correct). Per-category accuracy shows that role8
References
Zhiyuan Hu, Yunhai Hu, et al. Collaborative multi-agent testtime reinforcement learning for reasoning, 2026. URL https://arxiv.org/abs/2601.09667.
Rui Ai, Yuqi Pan, et al. Beyond majority voting: Llm aggregation by leveraging higher-order information, 2025. URL https://arxiv.org/abs/2510.01499.
B. Lester, R. Al-Rfou, et al. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
Tom Brown, Benjamin Mann, et al. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877– 1901. Curran Associates, Inc., 2020.
Yunxuan Li, Yibing Du, et al. Improving multi-agent debate with sparse communication topology. In Yaser AlOnaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7281–7294, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp. 427. URL https://aclanthology.org/2024. findings-emnlp.427/.
Xufeng Cai, Chaobing Song, Stephen Wright, and Jelena Diakonikolas. Cyclic block coordinate descent with variance reduction for composite nonconvex optimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 3469–3494. PMLR, 2023. URL https://proceedings.mlr.press/ v202/cai23e.html.
Yuxuan Li, Zhenfang Chen, et al. Evomac: Evolutionary multi-agent collaboration for large language models. arXiv preprint arXiv:2403.01245, 2024b. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158– 167, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1015. URL https://aclanthology.org/P17-1015/.
Zhipeng Chen, Zihan Wang, et al. Mas-gpt: Training LLM s to build LLM -based multi-agent systems. arXiv preprint arXiv:2402.08960, 2024. M. Deng, J. Wang, et al. Rlprompt: Optimizing discrete text prompts with reinforcement learning. In EMNLP, 2022. C. Fernando, D. Banarse, et al. Promptbreeder: Selfreferential self-improvement via prompt evolution. In ICLR, 2023.
Rui Liu, Yifan Zhang, et al. Dylan: Dynamic large language model agent network. arXiv preprint arXiv:2401.12768, 2024.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Volodymyr Mnih et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
Q. Guo, R. Wang, et al. Connecting LLMs with evolutionary algorithms yields powerful prompt optimizers. In ICLR, 2023.
Kartik Nagpal, Dayi Dong, et al. Leveraging large language models for effective and explainable multi-agent credit assignment. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’25, page 1501–1510, Richland, SC, 2025. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9798400714269.
Jayesh K. Gupta, Maxim Egorov, and Mykel J. Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In AAMAS Workshops (Selected Papers), pages 66–83. Springer, 2017. doi: 10.1007/ 978-3-319-71682-4_5.
Krista Opsahl-Ong, Michael J. Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. CoRR, abs/2406.11695, 2024. doi: 10.48550/arXiv.2406.11695. URL https://arxiv.org/abs/2406.11695.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning (CHIL), volume 174 of Proceedings of Machine Learning Research, pages 248–260. PMLR,
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b. 9
2022. URL https://proceedings.mlr.press/ v174/pal22a.html.
Zheyuan Zhang, Lin Ge, et al. Mapro: Recasting multiagent prompt optimization as maximum a posteriori inference, 2025b. URL https://arxiv.org/abs/ 2510.07475.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, New York, 1994.
Han Zhou, Xingchen Wan, et al. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview.net/forum?id=L3FHMoKZcS.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduatelevel google-proof q&a benchmark. In First Conference on Language Modeling (COLM), 2024. URL https://arxiv.org/abs/2311.12022.
Mingchen Zhuge, Wenyi Wang, et al. Gptswarm: language agents as optimizable graphs. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 2 edition, 2018. Paul Tseng. Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of Optimization Theory and Applications, 109(3):475–494, 2001. doi: 10.1023/A:1017501703105. Jason Wei, Xuezhi Wang, et al. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088. Tim Z. Xiao, Robert Bamler, Bernhard Schölkopf, and Weiyang Liu. Verbalized machine learning: Revisiting machine learning with language models, 2024. URL https://arxiv.org/abs/2406.04344. Chengrun Yang, Xuezhi Wang, et al. Large language models as optimizers. In ICLR, 2023. Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. In Advances in Neural Information Processing Systems, volume 35, pages 24611–24624, 2022. Mert Yüksekgönül, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic "differentiation" via text, 2024. URL https://arxiv.org/abs/2406.07496. Dehua Zhang, Qingsong Yuan, Lei Meng, Ruixue Xia, Wei Liu, and Chunbin Qin. Reinforcement learning for singleagent to multi-agent systems: from basic theory to industrial application progress, a survey. Artificial Intelligence Review, 2025a. Geng Zhang, Qingyun Wu, Lunan Wang, and Zhiyuan Chen. Gptswarm: Language agents as optimizable graphs. arXiv preprint arXiv:2402.16823, 2024. 10
A
APPENDIX OVERVIEW
This appendix provides supplemental details to support reproducibility and analysis. We use standard appendix numbering (e.g., Table A.1, Figure A.1, Algorithm A.1) and keep all references consistent with the main text.
B
EVALUATION AND PROMPT OPTIMISATION PROMPTS
The prompts used for agent evaluation, diagnosis and role prompt optimisation are presented below to aid reproducibility. These prompts are treated as experimental protocols that can be referenced in the main text. B.1
AGENT TURN EVALUATION PROMPT Agent Turn Evaluation Prompt AGENT_TURN_EVAL_PROMPT = """ You are an experienced question evaluation specialist and failure-analysis prompt engineer. You are given: 1) A multiple-choice question with options A, B, C, and D. 2) The gold correct answer. 3) The final answer produced by ONE debating agent. Your task is NOT to solve the question again. Your task is to evaluate this agents final answer independently and assign a score that reflects the quality of its reasoning and decision-making. You must: - Determine whether the final answer is correct. - If incorrect, identify the primary reason for failure using a predefined category. - If correct, assess whether the reasoning is robust or fragile. - Assign a numerical score that can be accumulated across many questions. -------------------------------Failure pattern taxonomy (choose EXACTLY ONE): You MUST select exactly one of the following labels. Do NOT invent new labels. - DOMAIN_MISMATCH The agent reasons from an inappropriate domain or role (e.g., ethical, economic, or technical framing instead of task-relevant reasoning). - KNOWLEDGE_DEFICIT The agent lacks or misuses core domain knowledge, leading to factual or conceptual errors. - MISINTERPRET_QUESTION The agent misunderstands key constraints, conditions, or intent of the question. - INCOMPLETE_REASONING The agents reasoning is partially correct but missing critical logical steps or justification. - OVERGENERALIZATION
11
The agent applies generic patterns or heuristics without adequately considering case-specific details. - MISALIGNED_OBJECTIVE The agent answers a different question than what is being asked (e.g., treatment vs diagnosis). - INSUFFICIENT_JUSTIFICATION The conclusion may be correct, but the reasoning is weak, shallow, or insufficiently supported. - RANDOM_OR_UNGROUNDED The answer appears arbitrary, speculative, or not grounded in the provided information. - NONE The answer is correct and the reasoning is sound. -------------------------------Scoring rules: - Scores must be integers from 0 to 5. - 5 = Correct answer with strong, well-justified reasoning. - 34 = Correct answer but with weak, incomplete, or risky reasoning. - 12 = Incorrect answer due to reasoning or judgment errors. - 0 = Incorrect answer due to fundamental misunderstanding or systematic reasoning failure. Consistency rules: - If the answer is correct, failure_pattern MUST be "NONE". - If the answer is incorrect, failure_pattern MUST NOT be "NONE". Output format: - Final answer correctness: correct / incorrect - Primary failure or risk pattern: <one of the predefined labels> - Brief explanation: 12 sentences focused on reasoning quality - Score: <integer 05> Additional rules: 1. Do NOT restate the full question or options. 2. Do NOT compare with other agents. 3. Do NOT suggest prompt changes explicitly. 4. Focus on issues that could be mitigated by improving the agents prompt or debate behavior. 5. Be concise, consistent, and scoring-oriented. """
B.2
AGENT DIAGNOSIS PROMPT Agent Diagnosis Prompt AGENT_DIAGNOSIS_PROMPT = """ You are an attribution analyst for a multi-agent reasoning system. Your role is to analyze summarized error information produced by a single agent and generate a concise attribution summary of the agents systematic failure characteristics.
12
The input you receive is a structured summary grouped by failure types. For each failure type, the input provides: - how frequently this failure occurred, and - a small number of representative example explanations. You should treat the frequency information as an indicator of how systematic and dominant each failure pattern is. The example explanations are only illustrative signals and do NOT represent all errors. You are NOT responsible for fixing the errors or rewriting the agents role prompt. Your task is strictly to identify and summarize why the errors occurred. When producing the attribution summary: - Focus on dominant and recurring failure patterns, prioritizing those with higher frequency. - Identify shared reasoning weaknesses, perspective mismatches, or systematic misalignments. - Abstract away from individual questions, examples, or surface details. - Do NOT repeat or quote raw explanations. - Do NOT enumerate failure types or counts explicitly. - Do NOT include task-specific facts or domain knowledge. Your summary should capture, at an appropriate level of abstraction: - role or perspective mismatches, - reasoning or interpretation deficiencies, - knowledge usage or grounding issues, if they are reflected in the summarized failures. The output should be concise, structured in natural language, and suitable for downstream prompt refinement modules. Output only the attribution summary. Do not include analysis, bullet points, statistics, or recommendations. """
B.3
ROLE PROMPT OPTIMISATION PROMPT Role Prompt Optimisation Prompt ROLE_PROMPT_OPTIMIZE = """ You are a prompt refinement module for a multi-agent reasoning system. Your task is to correct and reconstruct an agent’s role prompt based on aggregated evaluation failures. The original role prompt may contain invalid assumptions, missing constraints, or misleading reasoning guidance. Do NOT assume the original prompt is correct. Use the failure summary to infer systematic issues in three aspects: (1) role and perspective alignment, (2) reasoning and interpretation discipline, (3) knowledge use and grounding. Follow these principles implicitly:
13
1. Role and objective correction: - If failures indicate domain mismatch or objective misalignment, remove or correct the role perspective, focus, or priorities that cause the agent to reason from an inappropriate viewpoint or answer the wrong question. 2. Reasoning discipline reconstruction: - If failures indicate misinterpretation, incomplete reasoning, overgeneralization, or weak justification, introduce clearer reasoning requirements such as careful question interpretation, constraint checking, step-by-step reasoning, and explicit justification. 3. Knowledge use and grounding control: - If failures indicate knowledge deficits or ungrounded responses, strengthen guidance on using only relevant, task-appropriate knowledge and avoiding speculative or unsupported conclusions. Constraints: - Do not preserve incorrect assumptions from the original prompt. - Do not add task-specific facts or external domain knowledge. - Do not overfit to individual examples; address systematic behavior only. - Keep the reconstructed prompt concise and suitable for debate-based interaction. Output only the reconstructed role prompt. Do not include analysis, explanations, or failure labels. """
C
EXTENDED ANALYSIS OF THE DYLAN BASELINE ON MMLU (EXAMPLE)
To aid reproducibility and provide deeper insight into system behavior, we analyzed the evaluation log file 8080_dylan_mmlu_2026-02-19_21-20-23.txt. This file contains 500 entries, each recording the final answer of a multi-agent debate, the evaluator’s judgment of correctness, a failure-pattern label, and a quality score. The analysis below summarizes the distributions of these values.
D
EXTENDED ANALYSIS OF OPTIMIZED DYLAN ON MMLU (EXAMPLE)
In addition to the baseline analysis, we examined the log file from the optimized DyLAN system (8080_dylan_mmlu_2026-02-19_21-20-44.txt). This experiment applied an optimization method to the role prompts before evaluation. The log comprises 500 multiple-choice questions from the MMLU dataset. D.1
OVERALL PREDICTION STATISTICS
Table 3 summarizes the aggregate performance of the optimized system. Out of 500 questions, the system answered 279 correctly (55.8% accuracy). The mean evaluation score was 3.38 on a 0–5 scale, comparable to the baseline run. Thus, despite optimization, there was no substantial improvement in overall accuracy or score. D.2
DISTRIBUTION ACROSS DEBATE ROUNDS
The optimization encouraged earlier convergence: 81.6% of questions were answered after one round, while only 3.4% of questions reached the third round (Table 4). The average score dropped sharply from round 1 to round 2 (3.59 to 2.27) and increased again for the few questions that reached round 3 (3.12). 14
Table 3: Overall statistics for the optimised DyLAN system on MMLU. Metric
Value
Description
Total predictions Correct predictions Incorrect predictions Average evaluation score
500 279 (55.8%) 221 (44.2%) 3.38
Number of evaluated questions Final answers matching the gold answer Final answers that were wrong Mean score on a 0–5 scale
Table 4: Number of questions and average score by debate round for the optimised system.
D.3
Round
Count
Proportion
Average score
1 2 3
408 75 17
81.6% 15.0% 3.4%
3.59 2.27 3.12
FAILURE PATTERN DISTRIBUTION
Table 5 reports the frequency and average score of each failure pattern. As with the baseline run, KNOWLEDGE_DEFICIT and MISINTERPRET_QUESTION remain the most common error modes, collectively accounting for nearly 38% of all predictions. Additional errors such as INCOMPLETE_REASONING and OVERGENERALIZATION appear more often than in the baseline, and a single RANDOM_OR_UNGROUNDED error is observed. Table 5: Failure patterns and average scores for the optimised system. “NONE” denotes correct answers. Failure pattern NONE (correct) KNOWLEDGE_DEFICIT MISINTERPRET_QUESTION INCOMPLETE_REASONING MISALIGNED_OBJECTIVE OVERGENERALIZATION INSUFFICIENT_JUSTIFICATION DOMAIN_MISMATCH RANDOM_OR_UNGROUNDED
D.4
Count
Share
Avg. score
279 123 65 11 8 6 4 3 1
55.8% 24.6% 13.0% 2.2% 1.6% 1.2% 0.8% 0.6% 0.2%
5.00 1.33 1.32 1.45 1.38 1.33 1.50 1.00 1.00
Interpretation Correct answer with robust reasoning Missing or misused domain knowledge Misunderstanding the question’s intent or constraints Partial reasoning missing critical steps Answering an off-target question Generic heuristics without context Plausible conclusion but weak justification Reasoning from an inappropriate domain Arbitrary or ungrounded answer
ROUND–ERROR CROSS ANALYSIS
Table 6 cross-tabulates failure patterns across debate rounds. Most correct responses occur in the first round. Knowledge deficits remain prevalent across rounds, whereas misinterpretation errors are more prominent in the third round. The distribution suggests that the optimization did not fully mitigate fundamental knowledge gaps. Table 6: Number of predictions by round and failure pattern for the optimised system. NONE KNOWL.DEFICIT MISINT.QUESTION MISALIGN.OBJ. INCOMP.REASON. DOMAINMISM. OVERGEN. INSUFF.JUST. RANDOMUNGR. Round 1 Round 2 Round 3
251 20 8
92 29 2
42 17 6
6 2 0
7 3 1
15
2 1 0
4 2 0
3 1 0
1 0 0