Compound Prompt Constraints in LLM Code Generation: A Factorial Study of Format, Persona, and Urgency

arXiv:2609.03156v1 [cs.SE] 2 Sep 2026

Compound Prompt Constraints in LLM Code Generation: A Factorial Study of Format, Persona, and Urgency Shrenik Jadhav

Nickalas La Placa

Caleb Stone

Doctoral Student Department of EECS Embry-Riddle Aeronautical University Daytona Beach, FL, USA jadhavs2@my.erau.edu

Undergraduate Research Assistant Department of EECS Embry-Riddle Aeronautical University Daytona Beach, FL, USA laplacan@my.erau.edu

Graduate Research Assistant Department of EECS Embry-Riddle Aeronautical University Daytona Beach, FL, USA stonec16@my.erau.edu

Ashok Raja

Omar Ochoa

Vidhyashree Nagaraju

Assistant Professor Associate Professor Assistant Professor Department of EECS Department of EECS Computer Information Technology and Graphics Embry-Riddle Aeronautical University Embry-Riddle Aeronautical University Purdue University Northwest Daytona Beach, FL, USA Daytona Beach, FL, USA Hammond, IN, USA Vidhyashree.Nagaraju@erau.edu ochoao@erau.edu raja22@pnw.edu

Abstract—Large language models (LLMs) are increasingly deployed in software engineering pipelines for code generation, where production prompts routinely combine multiple constraints in a single request. A typical prompt may require the response to follow a structured output format such as JSON or XML, may assign the model a specific developer persona through the system message, and may frame the request with urgency or quality language reflecting real-world deployment pressures. Prior work has studied each of these prompt-level interventions in isolation, but the interaction between them remains uncharacterized, leaving open whether single-factor evaluations can predict the behavior of the compound prompts that real systems actually issue. To address this gap, this paper presents a full-factorial empirical study of how output formatting, persona assignment, and urgency framing jointly affect the reliability of LLM code generation. Rather than evaluating each constraint separately, the framework manipulates all three factors simultaneously in a controlled 3 × 3 × 3 design and decomposes each compound condition into an additive prediction and a residual interaction term that quantifies super-additive degradation. The framework is evaluated on the 164 problems of HumanEval+ across five OpenAI models spanning the GPT-4o family, the GPT-4.1 family, and the o3-mini reasoning model, yielding 22,140 individual evaluations under greedy decoding. A format-aware code extraction pipeline with multi-strategy fallback separates formatting failures from reasoning failures, and statistical significance is assessed using McNemar’s test on perproblem outcomes with odds ratios and 95% confidence intervals. The results show that compound prompt constraints produce architecture-dependent degradation that cannot be predicted from single-factor experiments. The GPT-4o family exhibits a consistent super-additive pattern in which compound constraints reduce pass@1 by 3 to 12 percentage points beyond what the sum of individual effects would predict, with the largest interaction reaching −12.2 pp on GPT-4o-mini for a JSON plus expert persona plus moderate urgency combination, and JSON-based combinations producing substantially larger interaction effects

than XML-based combinations. In contrast, the GPT-4.1 family resists the effect at both parameter scales, and the o3-mini reasoning model exhibits a qualitatively different pattern in which structured output constraints improve rather than degrade performance. These findings indicate that the vulnerability is architecture-dependent rather than size-dependent, that individually neutral or beneficial constraints can combine to produce large degradation invisible to single-factor evaluation, and that compound-prompt testing should be a standard part of reliability assessment for LLM-assisted engineering pipelines. Index Terms—large language models, prompt engineering, code generation, reliability evaluation, factorial experiment

I. I NTRODUCTION Large Language Models (LLMs) have rapidly become a core component of modern software engineering practice. Beyond software engineering, LLMs are also being integrated into other engineering domains as decision-support and control components, including fairness-aware reward shaping in multi-agent reinforcement learning for peer-to-peer energy markets [1], [2] and explainable reinforcement learning for voltage control in power distribution networks [3]. As LLMs take on increasingly consequential roles across these domains, understanding how prompt construction affects output reliability becomes a cross-cutting concern. Tools built on models such as the GPT-4 family now assist developers with code completion, function synthesis, test generation, refactoring, and bug localization, and recent surveys document hundreds of studies applying LLMs across nearly every phase of the software lifecycle [4], [5]. Their appeal lies in a deceptively simple interface: a developer writes a natural language prompt and receives executable code in return. Behind this interface,

however, the mapping from prompt to output is highly sensitive to how the request is phrased, and small changes in wording, structure, or framing can produce measurable shifts in correctness. A growing body of work shows that prompt design is not a cosmetic concern but a first-order determinant of LLM behavior. Three categories of prompt-level interventions have received particular attention. First, structured output constraints such as JSON or XML schemas are routinely imposed so that downstream pipelines can parse model responses programmatically, yet recent work reports that requiring strict output formats can measurably degrade reasoning quality on the underlying task [6]. Second, persona or role assignment via the system message is widely used to specialize model behavior, with mixed evidence on whether expert personas reliably improve task performance [7], [8]. Third, urgency and emotional framing has been shown to shift LLM outputs in measurable ways, sometimes improving accuracy on benchmark tasks [9]. Each of these interventions corresponds to a common deployment pattern. Production prompts in code generation pipelines routinely combine a structured output schema, a role specification, and a quality or urgency framing in a single request, yet they are almost never studied in combination. Despite the practical relevance of compound prompts, the published literature has overwhelmingly evaluated these factors one at a time. Studies of structured output [6], persona assignment [7], [8], and urgency framing [9] each report effects in isolation, but do not characterize what happens when all three are imposed simultaneously. This leaves a concrete methodological gap. If the individual effects were additive, single-factor results would be sufficient to predict compound behavior. If they are not additive, single-factor evaluation is systematically misleading for the prompts that real systems actually issue, and any reliability claim derived from such evaluations may not transfer to deployment. The gap is not only quantitative but also architectural: it is unknown whether sensitivity to compound constraints is a property of model size, training procedure, or model family, which has direct implications for how practitioners should select models for constraint-heavy pipelines. In deployed software systems where LLM outputs feed automated test suites, code review tools, or production refactoring workflows, an unmodeled interaction between routine prompt elements becomes a latent reliability risk, since failures appear only under the combined conditions that real production prompts actually issue. To address this gap, this paper presents a full-factorial empirical study of how three categories of prompt constraints jointly affect the reliability of LLM code generation. Rather than evaluating output formatting, persona assignment, and urgency framing in isolation, the study manipulates all three factors simultaneously in a controlled 3 × 3 × 3 design and measures how their interaction shapes per-problem pass rates across multiple model families. This design enables direct quantification of compound effects that cannot be recovered from single-factor experiments, and isolates whether observed

degradation is driven by individual constraints, by their interaction, or by underlying model architecture. The framework is intended to support component-level reliability assessment of LLM-assisted code generation pipelines, where production prompts routinely combine structured output schemas, role specifications, and quality or urgency framing in a single request. The main contributions of this paper are as follows. We introduce a 3×3×3 full-factorial experimental framework that varies output format, persona, and urgency framing across the 164 problems of HumanEval+ [10], [11] and five OpenAI models spanning the GPT-4o family, the GPT-4.1 family, and the o3-mini reasoning model, yielding 22,140 individual evaluations under greedy decoding. • We propose an interaction analysis methodology that decomposes each triple-constraint condition into an additive prediction and a residual interaction term, supported by a format-aware code extraction pipeline that separates formatting failures from reasoning failures and by McNemar paired tests with odds ratios and 95% confidence intervals on per-problem outcomes. • We empirically identify a super-additive degradation pattern in the GPT-4o family, in which compound constraints reduce HumanEval+ pass@1 by 3 to 12 percentage points beyond what the sum of individual effects would predict, with the largest interaction reaching −12.2 pp on GPT-4o-mini and JSON-based combinations producing substantially larger interaction effects than XML-based combinations. • We show that this vulnerability is architecture-dependent rather than size-dependent. The GPT-4.1 family resists the effect at both parameter scales, and the o3-mini reasoning model exhibits a qualitatively different pattern in which structured output constraints improve rather than degrade performance. • We demonstrate that single-factor prompt evaluation is insufficient for predicting compound-prompt behavior, since individually neutral or beneficial constraints can combine to produce large degradation that is invisible without factorial testing, with implications for the reliability assessment of LLM-assisted engineering pipelines. •

The framework is evaluated across five models on 27 prompt conditions, producing a per-condition pass@1 surface that supports main-effect analysis across factor levels, additive interaction decomposition for all eight triple-constraint combinations, and pairwise McNemar significance testing on per-problem outcomes. Beyond the headline interaction findings, the analysis quantifies how individual constraints can be neutral or even beneficial in isolation while their combination produces large degradation, characterizes the JSON versus XML asymmetry as a practical mitigation lever, and tracks prompt token length as a confound inherent to compound-constraint research. Together, these results indicate that compound-prompt evaluation should be a standard component of reliability assessment for LLM-assisted engineering workflows, and that model selec-

tion, not just prompt design, is a critical factor in deployment reliability. The remainder of this paper is organized as follows. Section II introduces the background on large language models, prompt constraints, and the statistical methods used in the analysis. Section III describes the factorial design, models, benchmark, extraction pipeline, and evaluation procedures. Section IV presents results across all five models, decomposes the three-way interaction, and discusses implications for LLMassisted software engineering. Section V concludes. II. BACKGROUND This section briefly introduces the core techniques and concepts used in the proposed experimental framework, focusing on their role in eliciting code from large language models and quantifying the reliability of the resulting outputs. A. Large Language Models for Code Generation Large language models are neural networks based on the Transformer architecture [12] that are trained on large corpora of natural language and source code to learn statistical regularities of both. Code-capable LLMs such as the GPT4 family generate programs autoregressively, predicting one token at a time conditioned on a prompt that typically contains a task description, a function signature, and optional contextual instructions [4], [11]. In this framework, LLMs are treated as black-box code generators whose only controllable inputs are the system message, the user message, and the decoding parameters, mirroring the interface available to practitioners deploying these models in production pipelines. B. Prompt Constraints A prompt constraint is any instruction in the system or user message that restricts or shapes the model’s output beyond the core task description. Three categories of constraints are central to this work. Output format constraints require the response to conform to a specific structure such as JSON or XML, so that downstream systems can parse the result programmatically; recent evidence indicates that such constraints can degrade reasoning quality when enforced through prompt instructions rather than decoding-time mechanisms [6]. Persona constraints assign a role or identity to the model via the system message, such as a generic developer or a domain expert, with the goal of specializing behavior [7], [8]. Urgency framing prepends emotional or motivational language to the user message, such as statements describing high stakes or time pressure, which has been shown to measurably alter LLM outputs [9]. In this framework, these three categories define the three factors of the factorial design. C. HumanEval+ and the pass@k Metric HumanEval is a benchmark of 164 hand-written Python programming problems, each consisting of a function signature, a docstring, and a hidden test suite [11]. A model’s solution is considered correct only if the generated function passes all hidden tests when executed in a sandbox. The standard metric

is pass@k, the probability that at least one of k independently sampled solutions passes all tests: "
# pass@k = Eproblems 1 −

n−c k
n k

(1)

where n is the number of samples drawn per problem and c is the number of correct samples. HumanEval+ [10] augments each original problem with approximately 764 additional test cases generated through mutation and differential testing, substantially reducing the false-positive rate of base HumanEval and providing greater discriminative power for detecting subtle correctness failures. In this work, all conditions are evaluated using pass@1 on HumanEval+ under greedy decoding, which corresponds to a single deterministic sample per problem. D. Factorial Design and Interaction Effects A full-factorial design varies multiple independent factors simultaneously across all combinations of their levels, enabling estimation of both main effects, the average influence of each factor, and interaction effects, the extent to which the joint influence of multiple factors deviates from the sum of their individual effects [13]. For three factors with three levels each, the design contains 3×3×3 = 27 conditions. The threeway interaction term is defined as the residual between the actual outcome under the full combination and the prediction obtained by adding the individual main effects to the baseline: If,p,u = Yactual − (Ybase + ∆f + ∆p + ∆u )

(2)

where ∆f , ∆p , and ∆u are the main effects of the format, persona, and urgency factors measured in isolation. A negative value of If,p,u indicates super-additive degradation, meaning the compound constraint produces worse performance than the sum of its parts would predict. In this framework, the interaction term is the primary quantity used to detect compoundprompt failure modes. E. McNemar’s Test for Paired Binary Outcomes Because the same 164 HumanEval+ problems appear in every experimental condition, comparing two conditions yields paired binary outcomes for each problem, namely pass or fail under each condition, rather than independent samples. McNemar’s test is the appropriate significance test for this setting [14]. Given the discordant pair counts b, the number of problems passed under condition A but failed under B, and c, the number passed under B but failed under A, the test statistic and the corresponding odds ratio are: χ2 =

(b − c)2 , b+c

OR =

b c

(3)

An odds ratio greater than one indicates that problems solved under condition A are more likely to fail under condition B than the reverse. In this work, McNemar’s test is applied with 95% confidence intervals on the odds ratio to assess the statistical significance of degradation between single-constraint and compound-constraint conditions.

III. M ETHODOLOGY We employ a full-factorial experimental design to systematically evaluate how three categories of prompt constraints interact to affect LLM code generation reliability. Fig. 1 presents the five-stage experimental pipeline, from factorial design through evaluation and statistical analysis. The complete framework processes 22,140 individual evaluations (27 conditions × 164 problems × 5 models).

All standard models use greedy decoding (temperature = 0) with a maximum output length of 512 tokens. The o3mini model requires modified parameters: system messages are merged into the user message as o3-mini does not support the system role, the maximum completion token budget is set to 4096 to accommodate hidden reasoning tokens, and temperature is fixed at 1, the only value supported by reasoning models. These differences are noted when interpreting o3-mini results.

A. Factorial Design We manipulate three independent prompt constraint factors, each at three levels of increasing intensity, producing 3 × 3 × 3 = 27 experimental conditions: a) Format Constraint: The format factor controls the structured output requirement at three levels: no formatting requirement, in which the model returns free-form code; JSON, in which the response must be a JSON object containing the solution as a string value; and XML, in which the response must be wrapped in solution and code tags. Format compliance is enforced through prompt instructions only, not through API-level mechanisms such as the response format parameter, ensuring that the model must simultaneously manage both reasoning and formatting. b) Persona Assignment: The persona factor controls rolebased identity framing delivered through the system message at three levels: no system message, a generic developer persona, and a detailed senior Python developer persona specifying 15 years of experience with algorithmic specialization and edge-case expertise. c) Urgency Framing: The urgency factor controls emotional and motivational pressure prepended to the user message at three levels: no framing, moderate importance framing requesting thoroughness and accuracy, and extreme emergency framing describing a production outage with explicit job-loss pressure. B. Prompt Construction Each prompt is assembled by combining the three factor levels into a structured API call. The persona level determines the system message content, which is left empty when no persona is assigned. The user message is composed by concatenating the urgency prefix, format-specific instructions, and the target function signature from the benchmark. This design ensures that constraint complexity increases with factor intensity while maintaining consistent task semantics across all conditions. C. Models We evaluate five OpenAI models spanning two architectural families and one reasoning model. The GPT-4o family includes GPT-4o-mini and GPT-4o, representing a smaller and larger variant of the same architecture. The GPT-4.1 family includes GPT-4.1-mini and GPT-4.1, a newer family explicitly optimized for improved instruction following [15]. We also evaluate o3-mini, a reasoning model that generates hidden chain-of-thought tokens before producing visible output.

D. Benchmark We use HumanEval+ [10], an enhanced version of OpenAI’s HumanEval benchmark [11]. HumanEval+ contains 164 function-level Python programming problems, each augmented with approximately 764 test cases compared to roughly 10 in the original. This substantially more rigorous test suite reduces the false-positive rate inherent in base HumanEval and provides greater discriminative power for detecting constraintinduced degradation. E. Code Extraction Pipeline A critical challenge in evaluating structured output conditions is extracting executable code from format-wrapped responses. We implement a multi-strategy extraction pipeline that routes responses based on the expected format. For JSON conditions, extraction attempts direct JSON parsing first, then regex-based key matching, then automated JSON repair for malformed responses, and finally falls back to plain-text extraction. A key implementation detail is handling double-escaped newline characters, which appear when models embed code within JSON strings and cause near-zero pass rates if left uncorrected. For XML conditions, extraction searches for code content within the expected tag structure, applies HTML entity decoding, and falls back to plain-text extraction if tags are missing or malformed. For free-form conditions, extraction checks for markdown code fences, then attempts to identify function definitions by indentation structure. Each extraction records the method used, enabling post-hoc separation of formatting failures from reasoning failures as recommended for structured output evaluation [6]. F. Evaluation Metrics and Statistical Analysis Our primary metric is pass@1 on the HumanEval+ test suite, defined as the fraction of problems for which the extracted code passes all augmented test cases in a single attempt. To quantify compound constraint effects, we compute the three-way interaction term I for each triple combination as defined in Eq. 2 of Section II. A statistically significant negative interaction (I < 0) indicates super-additive degradation, in which compound constraints reduce performance beyond the sum of their individual effects. Because the same 164 problems appear in every condition, we use McNemar’s test for paired binary outcomes rather than

Fig. 1. Experimental framework. Stage 1 defines the 3 × 3 × 3 factorial design. Stage 2 assembles prompts by mapping factor levels to system and user messages. Stage 3 dispatches prompts across five models. Stage 4 applies format-aware code extraction with multi-strategy fallback. Stage 5 evaluates extracted code via EvalPlus and computes three-way interaction effects.

independent-sample tests. We report odds ratios with 95% confidence intervals as effect size measures alongside p-values from the McNemar tests. Prompt length measured in tokens necessarily increases with constraint intensity, as format instructions and urgency framing add tokens to the prompt. We track prompt token counts across conditions and discuss this as a limitation, noting that this confound is inherent to the phenomenon under study since real-world compound prompts are longer precisely because they contain more constraints.

TABLE I S UMMARY OF R ESULTS ACROSS F IVE M ODELS (H UMAN E VAL + PASS @1) Model 4o-mini 4o 4.1-mini 4.1 o3-mini

Base. 73.8 79.3 87.2 89.6 59.8

Best 84.8 87.8 90.2 90.9 91.5

Worst 70.1 75.0 83.5 84.8 59.8

Range 14.6 12.8 6.7 6.1 31.7

Avg.Int. −7.6 −4.2 +1.7 +1.3 —

SA 8/8 5/8 0/8 2/8 —

Base.=Baseline (%), Range=Best−Worst (pp), Avg.Int.=Avg. three-way interaction (pp), SA=Super-additive count. o3-mini excluded from interaction analysis (see Section IV-D).

IV. R ESULTS AND D ISCUSSION We evaluated five OpenAI models across all 27 prompt conditions using the HumanEval+ benchmark (164 problems, ∼764 test cases per problem). Each condition received a single greedy-decoded response (temperature=0) per problem, yielding 22,140 total evaluations. We report HumanEval+ pass@1 as the primary metric throughout, as the enhanced test suite provides substantially greater discriminative power than base HumanEval. Table I summarizes performance across all five models.

A. Performance Under Compound Constraints Fig. 2 presents the mean pass@1 as a function of the number of simultaneously active prompt constraints, ranging from baseline with no active constraint to the full triple combination. Two distinct behavioral clusters emerge. The GPT-4.1 family (4.1-mini baseline 87.2%, 4.1 baseline 89.6%) maintains near-flat performance across all constraint levels, with a total range of only 6.1 to 6.7 percentage points (pp) across the 27 individual conditions, as summarized in Table I.

95

TABLE II M AIN E FFECTS : AVERAGE PASS @1 (%) BY FACTOR L EVEL

HumanEval+ pass@1 (%)

90

Factor

85

Format

80 75

Persona

70 65 60

GPT-4o-mini

GPT-4.1

GPT-4o

o3-mini

Urgency

GPT-4.1-mini

0

1

2

3

Fig. 2. Mean HumanEval+ pass@1 as a function of the number of active prompt constraints across five models. Error bars indicate 95% confidence intervals. The GPT-4o family shows greater sensitivity to constraint accumulation than the GPT-4.1 family.

B. Super-Additive Degradation in Compound Constraints To test whether compound constraints produce degradation beyond the sum of their individual effects, we decompose each triple combination’s expected performance using the additive interaction model defined in Eq. 2 of Section II. For each combination of format f , persona p, and urgency u, the predicted pass@1 under additivity is Ybase + ∆f + ∆p + ∆u , and the interaction effect If,p,u is its residual against the actual measured pass@1. Negative values of If,p,u indicate super-additive degradation, in which the compound constraint

None 78.1 83.6 87.6 89.2 79.5 83.6 87.9 89.3 76.6 80.8 87.7 88.7

Mid 75.4 81.0 88.6 88.2 78.3 83.1 88.1 88.6 78.7 86.2 87.5 88.7

High 82.0 86.0 88.2 89.1 77.8 83.9 88.3 88.6 80.2 83.5 89.1 89.1

Format: None/JSON/XML. Persona: None/Generic/Expert. Urgency: None/Moderate/Extreme.

Number of Active Prompt Constraints

5.0

2.5

Interaction Effect (percentage points)

In contrast, the GPT-4o family exhibits substantially greater variability across the same set of conditions: GPT-4o-mini spans 14.6 pp from 70.1% to 84.8%, and GPT-4o spans 12.8 pp from 75.0% to 87.8%. The narrower ranges visible in the marginal main-effect averages of Table II reflect the smoothing effect of averaging across the other two factors and should not be interpreted as the per-condition extremes. Notably, individual constraints do not uniformly degrade performance. Table II shows that for GPT-4o-mini, urgency framing improves average pass@1 from 76.6% (none) to 80.2% (extreme), and XML formatting improves it from 78.1% (none) to 82.0%. These individual gains are consistent with prior findings that mild emotional stimuli and structured output instructions can sharpen model attention [6], [9]. The persona factor is the clear exception: across all four standard models, persona assignment produces negligible main effects, with marginal pass@1 differences of less than 1.7 pp between the no-persona, generic-developer, and expert-persona conditions in every row of Table II. This null result aligns with recent evidence that role-based system prompts do not reliably improve LLM task performance [8], and we therefore do not interpret persona as an active driver of the main effects in the remainder of the analysis. As we show in the following subsection, however, even individually neutral or beneficial constraints do not combine additively when imposed together.

Model 4o-mini 4o 4.1-mini 4.1 4o-mini 4o 4.1-mini 4.1 4o-mini 4o 4.1-mini 4.1

+1.7 pp (0/8)

+1.3 pp (2/8)

0.0

−2.5

-4.2 pp (5/8)

−5.0

−7.5

-7.6 pp (8/8) −10.0 GPT-4o-mini

GPT-4o

GPT-4.1-mini

GPT-4.1

Fig. 3. Average three-way interaction effect across eight triple-constraint combinations. Negative values indicate super-additive degradation. Numbers in parentheses indicate how many of the eight combinations exceeded a −2 pp threshold. The GPT-4o family shows consistent super-additive degradation; the GPT-4.1 family does not.

produces worse performance than the sum of individual effects would predict. Fig. 3 presents the average three-way interaction across all eight triple-constraint combinations for each model. GPT-4omini exhibits the strongest super-additive degradation (I¯ = −7.6 pp, 8/8 combinations super-additive), followed by GPT4o (I¯ = −4.2 pp, 5/8). In contrast, GPT-4.1-mini (I¯ = +1.7 pp, 0/8) and GPT-4.1 (I¯ = +1.3 pp, 2/8) show no systematic super-additive pattern, with interaction effects near zero or slightly positive. Fig. 4 provides a detailed breakdown for GPT-4o-mini, the most affected model. All eight triple combinations perform worse than the additive prediction, with interaction effects ranging from −3.0 pp (XML+generic+extreme) to −12.2 pp (JSON+expert+moderate). JSON-based combinations exhibit substantially larger interaction effects (I¯ = −10.7 pp) than XML-based combinations (I¯ = −4.6 pp), suggesting that the computational overhead of JSON string encoding (escape sequences, brace matching) compounds more heavily with concurrent persona and urgency processing than the simpler

HumanEval+ pass@1 (%)

C. Architecture-Dependent Resilience

Predicted (additive)

90

Actual

-3.0

85

-3.0 -4.3 -7.9

80 -9.1

-10.4

75

-12.2

-11.0

ex t

ex t X+

ex p

+

+ ge n X+

+ ge n X+

ex p

+

m

od

m od

ex t X+

ge n

+

ex t J+

+ ex p J+

+ ge n J+

J+

ex p

+

m

m

od

od

70

Fig. 4. Predicted (additive) vs. actual pass@1 for all eight triple-constraint combinations on GPT-4o-mini. Every combination underperforms its additive prediction, with JSON combinations showing the largest gaps (−9.1 to −12.2 pp).

TABLE III T HREE -WAY I NTERACTION E FFECTS ( PP ) FOR GPT-4 O FAMILY

Comb. J+exp+mod J+gen+mod J+exp+ext J+gen+ext X+exp+mod X+gen+mod X+gen+ext X+exp+ext

Pred 85.4 84.1 86.0 84.8 87.2 86.0 86.6 87.8

4o-mini Act Int 73.2 −12.2 73.2 −11.0 75.6 −10.4 75.6 −9.1 79.3 −7.9 81.7 −4.3 83.5 −3.0 84.8 −3.0

Pred 85.4 85.4 82.9 82.9 94.5 94.5 92.1 92.1

4o Act 84.8 84.8 83.5 79.3 87.8 86.0 84.8 85.4

Int −0.6 −0.6 +0.6 −3.7 −6.7 −8.5 −7.3 −6.7

J=JSON, X=XML, exp=expert, gen=generic, mod=moderate, ext=extreme. Pred=Additive prediction, Act=Actual, Int=Interaction.

tag-wrapping structure of XML. Table III presents the full interaction decomposition for both GPT-4o family models. To assess statistical significance of the compound degradation, we conducted McNemar’s test on paired per-problem outcomes, comparing conditions that isolate the effect of adding constraints. For GPT-4o-mini, comparing the best single-constraint condition (extreme urgency only, 82.9%) against the worst triple combination (JSON+expert+moderate, 73.2%) yielded an odds ratio of 3.67 (95% CI: [1.49, 9.04], p = 0.004), indicating that problems solved under a single constraint were 3.7 times more likely to fail under the triple constraint than vice versa. Similarly, comparing JSON-only (79.3%) to JSON+expert+moderate (73.2%) yielded OR = 2.25 (95% CI: [0.98, 5.17], p = 0.076), a marginally significant trend consistent with the interaction pattern. For GPT-4o, comparing moderate-urgency-only (87.8%) to JSON+generic+none (75.0%) was highly significant (OR = 6.25, 95% CI: [2.18, 17.96], p < 0.001). For GPT-4.1-mini, no pairwise comparison reached significance (p > 0.05 for all), consistent with the near-zero interaction effects.

The most striking finding is that super-additive degradation is architecture-dependent rather than size-dependent. Within the GPT-4o family, both the smaller model (4o-mini, I¯ = −7.6 pp) and the larger model (4o, I¯ = −4.2 pp) exhibit superadditive degradation, although the larger model shows partial resilience. Within the GPT-4.1 family, both models resist the effect entirely (I¯ ≈ +1.5 pp for both). This pattern suggests that the vulnerability to compound constraints is tied to architectural or training differences between model families rather than to parameter count. The GPT-4.1 family was explicitly optimized for instruction-following fidelity [15], and our results indicate that this optimization confers robustness to multi-constraint interference as a secondary benefit. An important practical implication follows. The effects of prompt constraints cannot be reliably predicted by testing them in isolation. For GPT-4o-mini, each individual constraint either improved performance or had negligible effect, yet their combination produced a 12.2 pp degradation beyond what additivity would predict. This nonlinear interaction represents a latent failure mode that standard single-factor prompt evaluation would not detect. D. The o3-mini Case The o3-mini reasoning model exhibited a pattern qualitatively distinct from all standard models. Its unconstrained baseline was anomalously low (59.8%), but performance increased dramatically with any structured output constraint (JSON: 88.6%, XML: 88.1%), an improvement of approximately 11 pp. Adding persona and urgency on top of format constraints did not produce further degradation; all 27 conditions with at least one active constraint clustered between 77% and 92%. The additive interaction framework is inapplicable to o3-mini because its individual factor effects (20–30 pp each) create ceiling effects that make the three-way interaction term uninterpretable. We attribute this pattern to o3-mini’s internal chain-of-thought architecture, which generates hidden reasoning tokens before producing visible output, potentially decoupling reasoning from output formatting. This architectural feature may render structured constraints beneficial rather than taxing, as they provide a concrete output template that complements the model’s internal deliberation process. E. Implications for LLM-Assisted Engineering Workflows These results have direct implications for the use of LLMs in software engineering pipelines. In production systems, prompts frequently combine structured output requirements (for downstream parsing), role-based instructions (for domain specialization), and urgency or quality framing (reflecting realworld deployment pressures). Our findings demonstrate that this common practice introduces an interaction-driven failure mode in GPT-4o-class models: compound constraints degrade code generation reliability by 3–12 pp beyond what individual constraint testing would predict.

From a component-level reliability perspective, a 12.2 pp degradation in pass@1 corresponds to approximately 20 additional failing solutions per 164 problems. In automated pipelines where LLM outputs feed directly into test suites or code review, this represents a meaningful increase in defect injection rate. The architecture-dependent nature of this vulnerability suggests that model selection, not just prompt design, is a critical factor in system reliability. Organizations deploying LLM-assisted code generation should evaluate candidate models under compound constraint conditions representative of their actual deployment prompts, rather than relying on single-factor benchmarks. The asymmetry between JSON and XML interaction effects further suggests a practical mitigation strategy: when structured output is required alongside persona and urgency framing, XML-style wrapping may impose lower compound overhead than JSON string encoding, reducing the magnitude of compound degradation. To illustrate the broader implications, consider a safetycritical software development workflow in which an LLM assists engineers in implementing functions such as geofence monitoring, battery management, or sensor validation for an unmanned aircraft system. A production prompt may simultaneously require structured JSON output for automated integration, assign an expert developer persona, and include urgency framing during operational troubleshooting. The results presented in this study indicate that evaluating these prompt elements independently may underestimate the probability of introducing logic defects into generated software. Consequently, reliability assessment should be performed using the complete production prompt configuration rather than its individual components. This framework is intended to complement existing system safety practices rather than replace them. The measured interaction effects can provide additional evidence during software tool assessment, support hazard analyses by identifying prompt configuration as a potential contributor to software defects, and guide verification activities toward prompt configurations representative of operational use. In this way, compound-prompt evaluation extends existing software assurance processes with empirical evidence about an emerging source of reliability degradation in LLM-assisted engineering workflows. V. C ONCLUSION This paper presented a full-factorial empirical study of how structured output formatting, persona assignment, and urgency framing jointly affect the reliability of LLM code generation. Using a 3 × 3 × 3 design across the 164 problems of HumanEval+ and five OpenAI models spanning the GPT-4o family, the GPT-4.1 family, and the o3-mini reasoning model, we produced 22,140 individual evaluations and decomposed each triple-constraint condition into an additive prediction and a residual interaction term, enabling direct measurement of compound effects that cannot be recovered from single-factor experiments.

The results show that the interaction between prompt constraints depends strongly on model architecture rather than parameter count. The GPT-4o family exhibits a consistent superadditive degradation pattern in which compound constraints reduce pass@1 by 3 to 12 percentage points beyond what the sum of individual effects would predict, with the largest interaction reaching −12.2 pp on GPT-4o-mini and JSONbased combinations producing substantially larger interaction effects than XML-based combinations. In contrast, the GPT4.1 family resists the effect at both parameter scales, and the o3-mini reasoning model exhibits a qualitatively different pattern in which structured output constraints improve rather than degrade performance, indicating that explicit instructionfollowing optimization and internal chain-of-thought generation both appear to confer resilience to multi-constraint interference. These findings carry two practical implications. The effects of prompt constraints on code generation cannot be reliably extrapolated from single-factor studies, since individually neutral or even beneficial constraints can combine to produce large degradation that standard prompt evaluation would not detect. Model selection, not just prompt design, is therefore a critical factor in deployment reliability, and where structured output is required alongside persona and urgency framing, XMLstyle wrapping may impose lower compound overhead than JSON string encoding. Important limitations remain: the study covers five OpenAI models on a single benchmark of functionlevel Python problems under greedy decoding, prompt token length is confounded with constraint intensity, and the additive interaction framework was inapplicable to o3-mini due to ceiling effects. Future work will extend this framework to additional model families such as Claude, Gemini, and open-weight code models, to additional benchmarks including MBPP+ and repository-level evaluations, and toward mechanistic analysis of why certain architectures resist compound constraints while others do not. A parallel direction is mitigation, exploring whether decoding-time format enforcement, prompt decomposition, or constraint-aware fine-tuning can reduce the magnitude of compound degradation in vulnerable models, moving from documenting compound-prompt failure modes toward predicting and preventing them in deployed LLMassisted engineering pipelines. DATA AND C ODE AVAILABILITY The benchmark used in this study is HumanEval+ [10], a publicly available EvalPlus project and all model responses were obtained through the OpenAI API using parameters reported in Section III. Any data and models used in this paper can be made available upon request. R EFERENCES [1] S. Jadhav, B. Sevak, S. Das, A. Hussain, W. Su, and V.H. Bui, “FairMarket-RL: LLM-guided fairness shaping for multiagent reinforcement learning in peer-to-peer markets,” arXiv preprint arXiv:2506.22708, 2025.

[2] ——, “Scalable fairness shaping with LLM-guided multi-agent reinforcement learning for peer-to-peer electricity markets,” Utilities Policy, 2026. [3] S. Jadhav, B. Sevak, and V.-H. Bui, “XRL-LLM: Explainable reinforcement learning framework for voltage control,” Energies, vol. 19, no. 7, p. 1789, 2026. [4] J. Jiang, F. Wang, J. Shen, S. Mo, and S. Wang, “A survey on large language models for code generation,” ACM Transactions on Software Engineering and Methodology, 2025. [5] Q. Zhang, C. Fang, Y. Ma, W. Sun, and Z. Chen, “A survey on large language models for software engineering,” arXiv preprint arXiv:2312.15223, 2023. [6] Z. R. Tam, C.-K. Wu, Y.-L. Tsai, C.-Y. Lin, H.-y. Lee, and Y.-N. Chen, “Let me speak freely? a study on the impact of format restrictions on large language model performance,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track (EMNLP Industry), 2024. [7] L. Salewski, S. Alaniz, I. Rio-Torto, E. Schulz, and Z. Akata, “In-context impersonation reveals large language models’ strengths and biases,” in Advances in Neural Information Processing Systems (NeurIPS), 2023. [8] M. Zheng, J. Pei, L. Logeswaran, M. Lee, and D. Jurgens, “When “a helpful assistant” is not really helpful: Personas in system prompts do not improve performances of large language models,” arXiv preprint arXiv:2311.10054, 2024. [9] C. Li, J. Wang, Y. Zhang, K. Zhu, W. Hou, J. Lian, F. Luo, Q. Yang, and X. Xie, “Large language models understand and can be enhanced by emotional stimuli,” arXiv preprint arXiv:2307.11760, 2023. [10] J. Liu, C. S. Xia, Y. Wang, and L. Zhang, “Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation,” in Advances in Neural Information Processing Systems (NeurIPS), 2023. [11] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017. [13] D. C. Montgomery, Design and Analysis of Experiments, 9th ed. Hoboken, NJ: Wiley, 2017. [14] Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947. [15] OpenAI, “Introducing GPT-4.1 in the API,” https://openai.com/index/gpt-4-1/, 2025, accessed: 2026-04-07.

Related documents

• Prompt Optimization for LLM Code Generation via Reinforcement Learning#204858
• Reducing Hallucinations in LLM-Generated Code via Semantic Triangulation#984927
• Pedagogical Prompt Engineering Patterns: A Vygotskian Approach to adaptive Scaffolding in Computing Education with LLMs#446855
• Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming#271916
• Case Studies and Reflections on Agentic Software Engineering for Rapid Development of Digital Music Instruments#217809