arXiv CSopen access

Knowledge-Conditioned, Single-Pass LLM Synthesis of Executable Unity Game Scenes: A Compiler Error Census across 26 Goal Playable Concepts

1

Knowledge-Conditioned, Single-Pass LLM Synthesis of Executable Unity Game Scenes: A Compiler Error Census across 26 Goal Playable Concepts

arXiv:2607.10187v1 [cs.LG] 11 Jul 2026

Hugh Xuechen Liu and Kıvanç Tatar

Abstract—Large language models (LLMs) write Unity C# for game scenes. Yet nearly all demonstrations rest on an iterative repair loop that regenerates code until it compiles, conflating what the model writes with what the loop fixes. We remove the loop and evaluate a single pass, where the first draft is final. This isolates the model’s parametric knowledge, the most stringent test of unaided generation. Models instantiate Goal Playable Concepts, playable counterparts of goal patterns, across 10,400 generations (four open-weight models, 7B–30B; two generation modes; four intermediate-representation (IR) conditioning levels; 26 goal patterns; 20 seeds). None compiled into a runnable scene, leaving no survivorship bias. To understand how the generated C# scripts fail, we categorize the 99 error codes behind 90,673 compiler-error occurrences as Grounding (invented or misused Unity types and APIs) or Hygiene (structural defects needing no Unity knowledge). The split differs sharply by goal pattern (e.g., Stealth fails mostly on invented engine references; Capture on plain C# structure). Larger models, stricter IRs, and different generation modes move the errors but never yield a compiling scene. The bottleneck is missing engine-specific knowledge. The census orders goal patterns by that demand, showing designers where single-pass generation breaks. Index Terms—error taxonomy, code generation, large language models, Unity, gameplay design patterns, goal playable concepts

I. I NTRODUCTION

L

ARGE language models (LLMs) have become a practical tool for game content generation. Demos and tutorials routinely show LLMs writing Unity C# scripts, placing game objects, and implementing mechanics on demand 1 . Yet almost every such demonstration rests on an iterative repair loop: the model generates a candidate, the compiler errors (if any) return to a human or to the model itself, and the loop repeats until the artifact compiles and runs. This workflow is productive in practice but conflates what the model writes for the scene with what the repair loop fixes. Also, the gameplay on show is likewise picked ad hoc rather than drawn from a gameplay design vocabulary [1]–[4]. We ask a simpler and harder question: what is the intrinsic capability ceiling of single-pass LLM generation of executable game artifacts with no human feedback and no iterative repair? Single-pass evaluation isolates the model’s parametric knowledge (what is stored in its weights) from the Hugh Xuechen Liu and Kıvanç Tatar are with Chalmers University of Technology and University of Gothenburg, SE-412 96 Göteborg, Sweden (email: [email protected]; [email protected]). 1 Some examples: https://www.youtube.com/watch?v=gSFHyso uuI; https: //github.com/keijiro/DungeonMatchHeroes

practitioner’s domain expertise. It is the most stringent and most diagnostic condition. In an iterative workflow, each error reflects the model plus the feedback it received. Removing the loop makes every error attributable to the model alone. Removing the loop also levels the comparison. Every model faces the same condition on every task. A difference in the error profile can be hence read as a difference in what the model knows or in what the task demands. Existing game generation studies typically condition on genre labels (e.g., First-person Shooter, platformer) or freeform natural language. However, genre is criticised as a culturally constructed, semantically unstable concept that cannot serve as a reproducible evaluation target for cumulative research [5]–[7]. We instead ground our evaluation in Goal Playable Concepts (GPC) [8], distinguished across three layers. The gameplay design pattern language of Björk and Holopainen [9] is a theoretical design vocabulary following the ontological and formal analysis approach of game research [10]. A pattern means a frequently recurring configuration in gameplay design. Within it, 26 goal patterns describe what a player is trying to achieve. A GPC is the playable counterpart of one goal pattern, an artifact built so the concept can be grasped by playing it. Each GPC in turn has a reference Unity instantiation (currently its only engine realization so the two are coextensive here). This layering makes GPC the right evaluation target, giving a theoretically grounded abstraction, an executable artifact, and an objective binary verdict from compilation across all 26 goal patterns. Our evaluation spans five [model, generation mode], where the generation mode is Editor-style or Runtime-builder (defined in Section III). Four open-weight models, each from a different family, generate in the Editor-style mode (7BQwen2.5, 16B-DeepSeek, 22B-Codestral, 30B-Qwen3-Ed). The largest additionally generates in the Runtime-builder mode (30B-Qwen3-Rt). Each [model, generation mode] runs the identical grid of 4 intermediate-representation (IR) conditioning levels, 26 goal patterns, and 20 random seeds. This yields 10,400 generation records, each a single generation pass producing one Unity C# script with no repair. Open-weight models are a deliberate choice. They deploy locally (which studios with confidential assets require), are reproducible (frozen checkpoints do not drift like commercial endpoints), and admit fine-tuning at realistic cost. The 7B–30B range covers the tier these constraints admit. We find that no generated C# script compiled into a runnable scene across any of the 10,400 records. We count a

2

record as successful only when it both compiles and exposes a valid Unity entry point (Section III details the outcome taxonomy). Rather than a verdict, this uniform outcome is what makes the dataset suited to exhaustive failure-mode enumeration. Every record contributes failure evidence, and no survivorship filter separates analysed from discarded attempts. Contributions. This paper makes four contributions. (1) A Grounding/Hygiene taxonomy for LLM-generated Unity C# scripts. Drawing on the domain-specific versus domainindependent error distinction from software-engineering and computing-education research, we categorize all 99 observed C# error codes as Grounding (missing or incorrect Unity API/type knowledge) or Hygiene (domain-independent structural defects), comprising 18 Grounding and 81 Hygiene error codes. (2) A 10,400-record error code census with per-pattern profiles. We count every error-code occurrence across the 10,400 records (90,673 total across 99 error codes). The per-pattern Grounding share ranges from 0 to 0.98 (a property of error composition, robust to the error-volume inflation of 22B-Codestral reported in Section IV), indicating that goal patterns surface qualitatively different regions of the model’s Unity API knowledge which we interpret rather than directly measure as a knowledge boundary, and visualise as a Grounding–Hygiene scatter across the four IR levels. (3) An error-derived lens on goal-pattern semantic complexity. Linking these profiles to each pattern’s game design semantics yields an empirical ordering of Björk and Holopainen’s goal patterns [9] by the engine knowledge their instantiation demands. Perception- and physics-coupled patterns (e.g., Stealth, Rescue) concentrate Grounding errors, whereas patterns reducible to general state manipulation (e.g., Capture) concentrate Hygiene errors. This connects a designpattern vocabulary to an observable error structure of playablescene synthesis and shows designers which gameplay concepts current LLMs struggle to realize. (4) Condition-axis observations: IR conditioning and generation mode. Across the four IR levels, per-pattern profiles shift in both magnitude and Grounding/Hygiene composition, and a 30B Editor-versusRuntime comparison shows that switching generation mode (editor versus runtime, Section III) redistributes rather than resolves Grounding errors, locating the bottleneck in the model’s missing engine knowledge rather than generation mode 2 . The remainder of this paper is organised as follows. Section II situates the work relative to prior LLM code generation, game content generation, and compiler error taxonomy literature. Section III describes the experimental design, IR conditioning protocol, and failure taxonomy construction, and walks through one pattern end to end as a concrete exemplar. Section IV reports results. Section V interprets the findings and discusses limitations. Section VI concludes. II. BACKGROUND This section situates our study against three literatures. We first review how LLM code generation is evaluated since our single-pass, compile-or-fail setup departs from the typical 2 This study is reproducible through https://anonymous.4open.science/r/ paper-repo-failure-taxonomy-LLM-Unity-Scene-2D47/README.md

function-level benchmarks. We then turn to LLM-assisted game content generation to position goal playable concepts against the genre-based and natural-language-based targets used in prior work. We finally draw on the compiler-error taxonomy tradition in computing education which supplies the domain-dependent versus domain-independent distinction that our “Grounding/Hygiene” split builds on. A. LLM Code Generation Evaluation Following prior work that contrasts function-, project-, and library-level code generation [11]–[14], we distinguish three settings: function-level tasks on self-contained snippets, project-level generation that must fit into a multi-file codebase, and library-level generation that must call a specific external API correctly. Difficulty rises as code leaves the self-contained setting. Function-level benchmarks with automated test suites are the current dominant evaluation setting for LLM code generation. HumanEval [11] and MBPP [12] each provide hundreds of self-contained Python programming problems with accompanying tests. Performance is reported as pass@k, the probability that at least one of k independently sampled completions passes all hidden unit tests. These benchmarks have driven rapid progress but assess a narrow profile, namely isolated functions with no external type dependencies. Difficulty rises as code leaves the self-contained setting in function-level tasks (as demonstrated before) which need no external types. For example, project-level generation must fit a multi-file codebase. Also, library-level generation must call a specific external API correctly with the right type names, signatures, and namespaces. Once the artifact depends on such an API, the model’s parametric knowledge of that API might become the limiting factor. Studies on domain-specific libraries and real-world repositories find accuracy degrades sharply relative to general benchmarks, as models conflate API versions, invent non-existent method names, or reference types absent from the target library [14], [15]. Engine-coupled generation, our setting, must bind to a live (game) engine scene rather than a static library. It remains largely unexamined. B. LLM-Assisted Game Content Generation Procedural content generation (PCG) has a long history in games, spanning level layout, terrain, narrative quest design, and item parametrisation [16]. PCG has established searchbased [17] and machine-learning [18] traditions. LLMs have more recently been applied to tasks once handled by explicit generators such as rule-based narrative systems, grammardriven level generators, and constraint solvers [19]–[21]. Much of this work focuses on genre labels (e.g., platformer, RolePlay Games) or free-form descriptions. However, genre is a coarse, surface-level label applied to a whole game. By contrast, game content and design form a deeper and structured space. Hendrikx et al. [22] organise that space into six layers ordered by complexity, running from low-level game bits through game space, game systems, and game scenarios up to the game-design layer of rules and goals that the lower layers serve. Vocabularies like game design pattern [9], [23] and game ontologies [24]–[26] sit at this game-design layer

3

rather than at the genre surface. That depth makes them a more principled evaluation target than a genre tag. Prior game research further criticises genre as culturally constructed and semantically unstable, too unstable to serve as a reproducible evaluation target for cumulative research [5]–[7]. Among the forementioned game design patterns and ontologies, Goal Playable Concepts (GPC) [8] provide a theoretically grounded alternative. Within Björk and Holopainen’s gameplay design pattern language [9], goal patterns describe player goals and their interactive affordances (listed in Table I), and a GPC realizes one goal pattern as a playable artifact (currently a verifiable Unity scene). This layering enables evaluation at the gameplay pattern level rather than the genre level, and makes failure analysis directly interpretable in terms of designconcept structure. Structured, executable game representations also underpin frameworks such as the Video Game Description Language [27] and GVGAI [28] which target agent play rather than generation from design concepts. To our knowledge, no prior game-content work has compared failure profiles systematically at the level of individual game design patterns. Closest to our setting, the contemporaneous Mage benchmark [29] evaluates LLM-generated executable game scenes along multiple quality axes on a related class of goal-pattern tasks. The two efforts are complementary and differ in their unit of analysis. Mage measures how well generation succeeds under benchmark metrics, whereas the present study measures how generation fails at the granularity of individual compiler error occurrences. Understanding how these errors distribute across pattern instantiations and model configurations matters for two reasons. First, it locates where an LLM’s scene-generation capability breaks down. Second, it shows how the semantics of each goal pattern constrains what its Unity instantiation must implement.

TABLE I T HE 26 GOAL PATTERNS AND THEIR ONE - LINE DESCRIPTIONS . Pattern Ownership Collection Eliminate Capture Overcome Evade Stealth Herd Conceal Rescue Delivery Guard Race Alignment Configuration Traverse Survive Connection Exploration Reconnaissance Contact Enclosure Gain Competence

C. Compiler Error Taxonomies Compilation is a strict and objective gate. A scene either builds and runs or it does not with no rubric or human judgement in between. This makes compile success or failure an unusually clean signal for an executable artifact. Unlike open-ended text, a failed compile is unambiguous and its cause is localised to a specific diagnostic. The type of compiler error is itself diagnostic of why the model failed. A missing semicolon and an invented engine type are both failures, but they implicate very different model capabilities. We treat the compiler not as a mere pass-or-fail oracle but as a structured probe into the model’s failure modes. The following taxonomy formalises that probe. Throughout the paper, failure refers to the outcome of a generation attempt that yields no runnable scene. Error refers to a compiler diagnostic that such an attempt emits. Classifying compiler errors by semantic type is well established in computer science education research. Altadmri and Brown’s analysis of the Blackbox dataset identified characteristic error distributions for novice programmers, showing that a small set of error types accounts for the majority of all compile-time failures and that certain errors (e.g., missing semicolons, unmatched braces) are structurally domain-

Gain Information Last Man Standing King of the Hill

One-line Description Gain ownership of a game element. Complete several goals that together form a coherent unit. Remove a game element from its location in the game space. Eliminate or take ownership of an actively resisting goal object. Defeat an opposing force in a test or series of tests. Avoid being captured or hit. Move through an area and act without being detected. Move a game element to a location without directly interacting with it. Hinder other players’ ability to gain information. Free someone or something that is guarded. Move a game element to a specified element or place in the game space. Hinder others from accessing a particular area or game element. Be the first to reach a goal, often a location via an approved route. Form a linear alignment of game elements. Form a spatial, temporal, or logical arrangement of game elements. Move a game element from one position in the game to another. Avoid being killed by other players’ actions and game events. Link or position game elements so they have a physical relation. Learn the layout of the game world, or locate parts or objects in it. Patrol a known area to detect changes. Bring two or more elements into physical contact. Surround game elements with a continuous line or wall. Gain the ability to perform a certain action in the game. Act in the game to receive information or make deductions. Be the last survivor. Reach and keep a sought-after game state that others also want.

independent while others reflect gaps in conceptual understanding [30]. Pettit et al. and subsequent studies confirm this concentration effect and connect error type to the underlying cognitive difficulty of the corresponding programming concept [31]. This domain-independent versus domain-dependent distinction is the conceptual basis we build on. For enginecoupled generation the domain-dependent side becomes engine-specific. Some errors cannot be resolved without knowing what types and methods the target engine actually provides, whereas others (e.g., a missing semicolon, an unmatched brace) are domain-independent and can be diagnosed with no engine knowledge. We do not assume such a taxonomy a priori. Section IV derives one from the error codes actually observed across the 10,400 records. Prior work has used error type profiles to characterize LLM code quality beyond pass@k, identifying that specific error categories concentrate in predictable model failure modes [32],

4

[33]. We extend this approach to game content generation, connecting per-code frequency profiles to the semantic complexity of the underlying goal pattern, a mapping that has not been made in prior game generation or LLM evaluation research.

Base pattern md

LLM

C#

Unity

err log

IR-cond pattern md

LLM1

IR

LLM2

C#

Unity

err log

schema (free/min/full)

III. R ESEARCH D ESIGN A. Task Definition We evaluate whether open-weight LLMs can instantiate the 26 GPC [8] (introduced in Section I) as executable Unity C# scripts in a single generation pass, with no human feedback and no iterative repair. We use “single-pass” in a specific sense: the absence of an error-driven repair loop that feeds compiler errors back into the code. The with_schema runs call the model twice, once for the IR and once for the code. Neither call sees compiler output. We never re-run a generation after it fails. The two calls are two steps of one pipeline, not two rounds of fixing. Single-pass is not one-shot or fewshot, terms that count the worked examples placed in the prompt [34]. Our prompts contain none and are zero-shot in that sense. This single-pass condition isolates the model’s own knowledge from a practitioner’s debugging skill, exposing failure modes that iterative workflows conceal. Every GPC has a reference Unity instantiation. These references are our proof of feasibility only. Generation runs inside a single pre-built Unity project holding all 26 scenes and their shared assets. The model sees none of this: not the reference instantiations, not the project contents. The task is to produce a complete and self-contained artifact. The script must create whatever it needs (e.g., game objects, components, and materials) through Unity’s programmatic APIs. The generated script is compiled inside the project. A reference to an asset that already exists under its correct name resolves against it. Anything else the script must create itself. A reference to a type or asset that neither exists nor is created becomes a compiler error. We use two generation pipelines (Figure 1). The no_schema pipeline gives the model only the goal pattern description, a markdown document defining the pattern’s game-design semantics, and asks for a Unity C# script directly. The with_schema pipeline adds an intermediate step. Step 1 generates an intermediate representation (IR) in JSON from the pattern description and a schema document. Step 2 has the same model generate the C# script from its own IR. The reference IRs that accompany the reference scenes are never used. The model builds its own IR, then implements it. The IR has seven top-level fields (scene, objects, scripts, params, runtime_params, links, rules). The with_schema pipeline runs at three levels (free, min, full). They differ in how much of the frozen schema the model sees. More schema means more field-level constraints to satisfy. free gives no schema text, leaving the model to structure the IR itself. min gives only the top-level field names. full gives the complete field definitions and hard constraints (Appendix A). This IR schema is fixed before the whole experiment. Building it was an offline design step by the authors. We started from three hand-authored reference implementations.

Fig. 1. Baseline and IR-conditioned pipelines. Data (white), LLM (light grey), compile target (dark grey), schema injection (dashed). Both share the same Unity batch-replay harness and error-log aggregation.

Over five versioned iterations we revised the IR schema into v0.2-runtime-evidence, then froze it before generation began. Freezing keeps the IR axis uniform across runs, not tuned to outcomes. This offline design does not weaken the single-pass condition. The model still writes every script in one pass with no compiler feedback. A field-frequency check across all 26 reference implementations confirms that every GPC populates all seven fields. The details of the IR schema’s design and iteration are also provided in Appendix A. B. Experimental Variables Table II lists all experimental factors. Because the target engine is Unity throughout, we first fix a distinction that Unity itself draws. A scene can be built along two different code paths. Our two Unity script generation modes correspond to them. The Editor-style mode requests a Unity Editor script that builds the scene at edit time through editor-side APIs (e.g., the UnityEditor namespace, AssetDatabase, menu actions).3 The Runtime-builder mode requests a single MonoBehaviour that constructs the scene programmatically in Awake() at play time, defines all gameplay classes in the same file, and may not use the UnityEditor namespace at all.4 The two paths exercise different parts of the Unity API. Contrasting them at a fixed model tests whether failures are tied to one API regime or intrinsic to the model’s engine knowledge. The five [model, generation mode] come from crossing four pre-trained and frozen-weight open models with the two generation modes. Three models generate in the Editor-style mode only: Qwen2.5-Coder-7B [35], DeepSeekCoder-V2-Lite-16B [36], and Codestral-22B 5 (7B-Qwen2.5, 16B-DeepSeek, 22B-Codestral). The fourth, Qwen3-Coder30B [37] (a newer generation than the Qwen2.5-Coder used at 7B), generates in both modes (30B-Qwen3-Ed and 30BQwen3-Rt). Switching models changes identity as a whole, not scale alone. Size, family, and pretraining corpus change together. The 30B Editor-versus-Runtime contrast holds identity fixed and isolates the generation mode. A still larger or newer open-weight model (e.g., a Gemma-class checkpoint) is the same scale remedy at greater magnitude. Checkpoints released after this study are a replication target, not an omission. 3 Unity Editor scripting API: https://docs.unity3d.com/ScriptReference/ AssetDatabase.html, https://docs.unity3d.com/ScriptReference/MenuItem. html. 4 Unity runtime scripting API: https://docs.unity3d.com/ScriptReference/ MonoBehaviour.html, https://docs.unity3d.com/ScriptReference/ MonoBehaviour.Awake.html. 5 https://huggingface.co/mistralai/Codestral-22B-v0.1

5

TABLE II E XPERIMENTAL FACTORS . E ACH OF THE FIVE [ MODEL , GENERATION MODE ] RUNS THE IDENTICAL IR- BY- PATTERN - BY- SEED GRID FOR 2,080 RECORDS (10,400 TOTAL ). Factor

Levels

Notes

[Model, generation mode] IR conditioning

7B-Qwen2.5, 16B-DeepSeek, 22BCodestral, 30B-Qwen3-Ed, 30BQwen3-Rt no_schema, with_schema_free, with_schema_min, with_schema_full 26 goal patterns

5 [model, generation mode]

20 per [[model, generation mode], condition, pattern]

Stochastic variation

Goal pattern Seed

Model axis (fixed mode) 7B-Qwen2.5 Qwen2.5-Coder Editor mode

16B-DeepSeek DeepSeek-V2-Lite Editor mode

Increasing schema detail

Unity inst. for all

Mode axis (fixed model) 22B-Codestral Codestral Editor mode

30B-Qwen3-Ed Qwen3-Coder Editor mode

30B-Qwen3-Rt Qwen3-Coder Runtime mode

shared factor grid per [model, generation mode]: 26 goal patterns × 4 IR levels × 20 seeds = 2,080 records

Fig. 2. The five [model, generation mode] and the shared factor grid. Four open-weight models generate in the Editor-style mode. The largest also generates in the Runtime-builder mode. Each runs the identical factor grid (bottom). Total: 5 × 2,080 = 10,400 records.

Four IR conditioning levels vary the structured context provided. At one end no_schema supplies only the pattern description. At the other with_schema_full adds a complete GPC schema with all field constraints. Twenty seeds per [[model, generation mode], condition, pattern] capture run-torun variation. The factor grid of 26 goal patterns × 4 IR conditioning levels × 20 seeds is identical for every [model, generation mode], giving 2,080 records each and 10,400 in total (Table II). Throughout the paper, we write the design axes as tuples. The five [model, generation mode] are fixed pairings, not a full model-by-mode crossing. We bracket the pair as one unit inside larger tuples such as [[model, generation mode], condition, pattern, seed]. A [model, generation mode] is one model generating in one mode, Editor-style or Runtime-builder (five: 7B-Qwen2.5, 16B-DeepSeek, 22B-Codestral, 30B-Qwen3-Ed, 30B-Qwen3-Rt). A condition is shorthand for an IR conditioning level (four: no_schema, free, min, full). A record is one generation attempt for a given [[model, generation mode], condition, pattern, seed]. Figure 2 summarises the five [model, generation mode] and the shared factor grid.

C. Generation and Compilation Pipeline Each record comes from one [[model, generation mode], condition, pattern, seed] (Table II). The model generates a Unity C# script. The script is compiled. . . . The result is stored (i.e., the generated C# script, its compilation outcome, and any emitted diagnostics). All [model, generation mode] generate with temperature 0.2 and top-p 0.95 on vLLM, under an output budget of 2,048 tokens for 7B-Qwen2.5 and 16BDeepSeek and 8,192 tokens for 22B-Codestral and both 30B modes. These settings follow common practice rather than task-specific tuning. The output budgets are rarely reached and

do not affect the results or their interpretation.6 Each record follows an automated pipeline: 1) The LLM generates a Unity C# script (and in the with_schema conditions, first generates an IR JSON in a preceding call). 2) A format sanitizer checks the raw output before it reaches the project. Records that fail this check (e.g., output that is empty after code-fence extraction, or that contains no C# type declaration) are recorded as sanitizer rejected and do not proceed to compilation. 3) A BatchRunner process writes the surviving script to the Unity project and triggers AssetDatabase.Refresh which imports and compiles the new .cs file and reloads the scripting domain. 4) Script compilation runs asynchronously relative to asset import. A watchdog enforces a 120-second timeout on the compile-and-domain-reload cycle, uniformly across every [model, generation mode] and IR conditioning level. Output is captured and parsed for C# diagnostic codes. A record with captured diagnostics is a compile error. A record where the timeout elapses before any diagnostics appear is timeout, no diagnostics. 5) If compilation succeeds, BatchRunner checks for a recognised entry point. A script with a valid entry point is a pass. One that compiles but exposes no valid entry point is no entry (discussed in Section V). 6) Two artifact codes are excluded from the census: CS2001 is a start-up artifact emitted before the AICommandRunStart log marker and is never counted; CS1029 lines containing the string BatchRunner_sanitize mark a sanitizer rejection and are tallied separately from compilation failures. Figure 3 summarizes the generation and compilation pipeline. Each record falls into exactly one of the five mutually exclusive outcomes of Table III. A timeout does not discard a record. All diagnostics emitted before the 120-second limit are captured. Timed-out records with captured output are a primary data source for the error census (Section IV).7 D. Worked Example: Stealth We walk through one record from input to failure to illustrate the research design pipeline. The record is Stealth in 30B-Qwen3-Ed (the Editor-style mode) under 6 Each budget is the cap on generated tokens, within the model’s context window (3,072 tokens for 7B-Qwen2.5 and 16B-DeepSeek, 32,768 for the others). Measured with each model’s tokenizer, outputs reach the budget in 4.0% of smaller-model records overall, 1.8% under schema conditioning, and 2.1% in 30B-Qwen3-Ed. The exceptions are the no_schema runs of 7BQwen2.5 and 30B-Qwen3-Rt (16.9% and 17.3%), where free-form generation runs long and error counts include a truncation contribution. 22B-Codestral is a separate case, with incomplete outputs that stop far below its budget for reasons given in Section V. 7 Because the 120-second compile limit is identical across every [model, generation mode] and IR conditioning level, cross-condition comparisons are not confounded by unequal timeouts. The limit is a heuristic, long enough for the compile-and-domain-reload cycle to finish in typical cases and short enough to keep the pipeline tractable. Its absolute value is revisited in the Limitations (Section V). Unity’s asset-import pipeline recompiles each script up to three times per record. Error lines are deduplicated at the raw-log level before code extraction to avoid triple-counting the same diagnostic.

6

The model returns the IR below (abridged). Its seven scripts entries name six component classes of its own invention (PlayerStealthController, GuardAI, . . . ), each bound to a scene object.

LLM output Unity C#

format sanitizer

fail

sanitizer rejected

pass no output

compile (120 s watchdog)

diagnostics

success

timeout, no diagnostics

compile error

no entry none

entry-point check valid

pass

Fig. 3. Outcome routing for a single record. Each record enters at the top and exits at exactly one of the five mutually exclusive outcomes of Table III (grey, right). Pass (compiled with a valid entry point) was never reached in any of the 10,400 records.

{ "scene": ..., "objects": [ {"id":"player_001","name":"Player Character", ...}, {"id":"guard_001","name":"Guard", ...}, ... ], "scripts": [ {"object_id":"player_001", "class_name":"PlayerStealthController"}, {"object_id":"guard_001", "class_name":"GuardAI"}, ... ], "params": ..., "runtime_params": ..., "links": ..., "rules": ... }

In Step 2, the coder prompt places the Step-1 IR after its own header and asks for raw C# only. With the placeholders filled it reads: [pattern: Stealth] [method: with_schema_full] Generate a Unity Editor script that instantiates a scene matching the following engine-specific Intermediate Representation (IR). Thereafter, you may refer to it as IR. Output only raw C# code. <full Step-1 IR>

Fig. 4. Reference Unity instantiation of the Stealth goal pattern. The player (blue square, lower left) must reach the goal region (white square, upper right) while moving within the walled area (grey rectangles as walls) and avoiding the vision cones (orange wedges) of patrolling guards (red circles). Reaching the goal undetected satisfies the pattern’s goal structure.

with_schema_full, seed 6. Its input is the Stealth pattern description, a multi-paragraph Markdown document that begins “Stealth is the goal to move through a certain area and perform an action without being detected” (full text in Appendix C). This document is the only description of the pattern. Figure 4 shows the reference Unity scene. The model never receives this Markdown on its own, and never the scene, its assets, or their names. The Markdown is filled into a short prompt template (Appendix D), and generation runs in two calls. Of the five outcomes in Figure 3, this record exits at compile error. It passes the format sanitizer, reaches compilation, and returns diagnostics rather than timing out. The rest of this section opens that outcome to show what the failure is. In Step 1 of Section III-C, the IR-maker prompt places a short header before the Markdown and asks for an enginespecific Intermediate Representation (IR) as JSON. With the placeholders filled for this record it reads: [pattern: Stealth] [method: with_schema_full] Generate an engine-specific Intermediate Representation (IR) JSON for the playable concept described below. Thereafter, you may refer to it as IR. Output ONLY valid JSON. No extra text. <IR schema and full Stealth pattern description>

The returned script attaches each component (e.g., guard1GO.AddComponent<GuardAI>()) but never defines any of the six classes, none of which exists in the project. Compiling this script is the attempt to build the scene. It fails with 31 occurrences of CS0246 (type or namespace not found), one per reference site. These diagnostics place the record in the compile error outcome of Figure 3. This is a domain-specific failure. Resolving it requires knowledge of what types the Unity engine and project provide. By contrast, a syntactic error such as a missing semicolon is domain-independent and requires no knowledge of Unity. Section IV-A2 formalises this distinction as the two error classes used throughout, Grounding (domain-specific) and Hygiene (domain-independent). This record’s invented types are a Grounding failure. IV. R ESULTS This section reports the results of the experiment designed in Section III. We begin with the aggregate failure landscape from record-level outcomes. Each record (defined in Section III) is one concrete run of the experiment, a single generation of C# script for one [[model, generation mode], condition, pattern, seed]. We categorize the observed error codes as Grounding or Hygiene. Using this Grounding/Hygiene taxonomy as a lens, we read the error-code distribution across the whole corpus, across conditioning levels and [model, generation mode]. We then turn to the per-pattern error structure and bound its range across the whole landscape. A zoom-in on a single [model, generation mode] (30B-Qwen3-Ed) closes the section at finer granularity. A. The Failure Landscape This subsection establishes the Grounding/Hygiene taxonomy that the rest of the paper applies. The record-level

7

TABLE III R ECORD - LEVEL OUTCOMES FOR EVERY [[ MODEL , GENERATION MODE ], CONDITION ], AS % OF n = 520 RECORDS PER ROW ( ROWS SUM TO 100%). S ANIT., SANITIZER REJECTED ; C OMP. ERR , COMPILE ERROR WITH CAPTURED DIAGNOSTICS ; TO/ NO CS, TIMEOUT, NO DIAGNOSTICS ; N O ENTRY, COMPILED BUT NO VALID ENTRY POINT; PASS , COMPILED WITH A VALID ENTRY POINT. PASS IS 0 IN EVERY ROW , THE PREMISE OF THE STUDY. [Model, gen. mode] Condition Sanit. Comp. err TO/noCS No entry Pass 7B-Qwen2.5 no schema 17.7 33.8 0.0 48.5 0.0 sch free 70.8 21.7 0.0 7.5 0.0 82.5 13.5 0.0 4.0 0.0 sch min 89.4 9.8 0.0 0.8 0.0 sch full 16B-DeepSeek no schema 0.0 37.5 0.0 62.5 0.0 77.5 8.5 0.0 14.0 0.0 sch free sch min 84.2 11.9 0.0 3.8 0.0 sch full 85.4 11.5 0.0 3.1 0.0 22B-Codestral no schema 11.5 36.0 0.4 52.1 0.0 sch free 64.0 13.3 0.4 22.3 0.0 sch min 31.2 57.1 0.4 11.3 0.0 14.4 74.4 0.4 10.8 0.0 sch full 30B-Qwen3-Ed no schema 0.0 23.5 0.0 76.5 0.0 sch free 1.0 56.9 0.6 41.5 0.0 15.2 74.6 0.4 9.8 0.0 sch min sch full 8.8 87.7 0.0 3.5 0.0 30B-Qwen3-Rt no schema 12.9 67.5 0.6 19.0 0.0 sch free 91.9 3.7 0.4 4.0 0.0 sch min 95.6 2.7 0.0 1.7 0.0 98.1 0.4 0.0 1.5 0.0 sch full

outcomes come first and show which records yield compiler evidence at all. The taxonomy is then derived from the 99 observed error codes. The closing distribution and composition views check how the two categories behave across [model, generation mode]. The differences between [model, generation mode] reported along the way are context for reading the perpattern analysis. 1) Record-Level Outcomes: Table III reports the recordlevel outcome distribution for all four IR conditioning levels in each [model, generation mode]. No generated C# script compiled into a runnable scene across any of the 10,400 records. The proportion producing compiler output (timeout_with_cs) varies sharply by [model, generation mode], from 18.5% (771 of 4,160) across 7B-Qwen2.5 and 16B-DeepSeek combined to 45.2% in 22B-Codestral, 60.7% in 30B-Qwen3-Ed, and 18.6% in 30B-Qwen3-Rt. Table III reveals conditioning responses. For 7B-Qwen2.5 and 16B-DeepSeek, sanitizer rejection climbs with every step of strictness (7B-Qwen2.5: 17.7%, 70.8%, 82.5%, 89.4%) and is already dominant at free. Any schema pushes most records out of the compiler’s reach. 22B-Codestral moves the opposite way, its sanitizer rate falling with stricter schemas (64.0% to 14.4%) while the share of records with compiler output rises to 74.4%. Stricter structure improves format compliance even as the generated C# scripts grow and break. 30B-Qwen3-Ed shows the same direction more mildly, the share with compiler output rising from 23.5% to 87.7% while sanitizer rejection stays under 16%. 30B-Qwen3-Rt collapses under any schema, with sanitizer rejection at 91.9% by free and 98.1% at full, leaving almost no records in the error-code pool. 2) Observed Error Codes and the Grounding/Hygiene Taxonomy: Across all records with compiler output, 99 distinct C# error codes appear (each listed with its message template and total count in Appendix B). Every code is a standard diagnostic defined by the C# compiler, with a doc-

umented entry in Microsoft’s compiler-error reference8 . Only the Grounding/Hygiene categorization is introduced by this study. They split cleanly into the two categories that define the taxonomy used throughout, Grounding and Hygiene. The 18 Grounding codes require Unity-specific knowledge to resolve. For example, CS0246 (type or namespace not found) names model-invented types such as EnemyAI, PlayerScript, and GuardAI that exist in neither the engine nor the project. CS1061 (no such member) names members such as health or movementSpeed that the model assumed a Unity component exposes. The full set spans undeclared types and namespaces (CS0246, CS0234), inaccessible or missing members (CS1061, CS0117, CS0122), incorrect overrides (CS0115, CS0534), and Unity-type mismatches (CS0311, CS0315, CS8121, among others). The remaining 81 Hygiene codes need no Unity knowledge to diagnose. They are structural and syntactic defects such as CS1003 (syntax error), CS1002 (missing semicolon), unmatched braces, undeclared variables, and invalid expressions. The boundary between the two categories is anchored in the compiler’s own diagnostic semantics, not in a post hoc judgement about each record. An error code is Grounding when resolving its diagnostic requires type, member, or namespace information external to the script, which only the Unity engine and project supply. An error code is Hygiene when its diagnostic is purely lexical or syntactic and consults no external type information. Because each code carries a fixed diagnostic meaning, the mapping from code to category is deterministic. The term hygiene is borrowed from programming-language theory [38] and software-engineering practice [39], where it denotes code that is locally well-formed independent of external domain conventions. Each error occurrence is one instance of an error code emitted when compiling the generated C# script. The Grounding/Hygiene category attaches to the error code (a distinct C# diagnostic such as CS0246), not to the record (one generation attempt for a given [[model, generation mode], condition, pattern, seed]). The two categories are mutually exclusive. Each error code is either Grounding or Hygiene. A single record may still contain both Grounding and Hygiene error codes because compiling one generated C# script typically produces several distinct errors. Our unit of analysis is the error occurrence, counted with its frequency. For instance, if CS0246 appears five times in one record and names a different missing type each time, all five occurrences are counted rather than just one. Per-pattern profiles sum these counts across all 20 seeds within each [[model, generation mode], condition, pattern]. Table IV traces one pattern from its individual records, through the error occurrences and their codes, to the aggregated per-pattern profile. 3) Error Code Distribution: Of the 90,673 error occurrences, 76% come from 22B-Codestral alone, whose verbose and structurally broken C# scripts accumulate far more error lines per compilation attempt than the other models. Table V lists the top-10 error codes by total occurrence. The most frequent overall is CS1003 (syntax error, Hygiene) with 28,571 8 E.g., CS0246: https://learn.microsoft.com/en-us/dotnet/csharp/ language-reference/compiler-messages/cs0246

8

TABLE IV I LLUSTRATIVE LINEAGE FROM RECORDS TO A PER - PATTERN PROFILE (Stealth, 30B-Q W E N 3-E D , W I T H _ S C H E M A _ F U L L ). E ACH OCCURRENCE ( COLUMN 2) INSTANTIATES ONE ERROR CODE , ABSTRACTING THE CONCRETE IDENTIFIER UP TO THE CODE AND ITS MESSAGE TEMPLATE ( COLUMN 3); THE G ROUNDING /H YGIENE CATEGORY ATTACHES TO THE CODE . T HE SAME CODE CS0246 RECURS WITH A DIFFERENT INVENTED TYPE ACROSS SEEDS . T WO OF THE 20 SEEDS ARE SHOWN TO ILLUSTRATE THE AGGREGATION ; THE REAL PER - PATTERN PROFILE SUMS ALL 20. H YGIENE ROWS (CS1002, CS1003) ARE REPRESENTATIVE . record

error occurrence

error code and message template

category

per-pattern profile

[[30B-Qwen3-Ed], full, Stealth, 6]

CS0246 → GuardAI

CS0246: The type or namespace name X could not be found CS1002: ; expected

Grounding

Stealth

Hygiene

this table (2 seeds): G 2, H 2 occ

CS0246: The type or namespace name X could not be found CS1003: Syntax error, X expected

Grounding

CS1002 → missing ; [[30B-Qwen3-Ed], full, Stealth, 7]

CS0246 → EnemyAI CS1003 → syntax error

Hygiene

all 20 seeds: G 334, H 44 occ (16.7 / 2.2 per seed)

TABLE V T OP -10 C# ERROR CODES BY TOTAL OCCURRENCE , WITH ABBREVIATED MESSAGE TEMPLATES AND A PER -[ MODEL , GENERATION MODE ] BREAKDOWN . X DENOTES A QUOTED IDENTIFIER , SHOWN WITH A REPRESENTATIVE EXAMPLE FROM THE LOGS IN PARENTHESES . G = G ROUNDING ; H = H YGIENE . T HE FIVE [ MODEL , GENERATION MODE ] ARE DEFINED IN S ECTION III. Rk Code Message template GH Total 7B-Qwen2.5 16B-DeepSeek 22B-Codestral 30B-Qwen3-Ed 30B-Qwen3-Rt 1 CS1003 Syntax error, X (e.g., ’,’) expected H 28,571 176 88 26,268 62 1,977 2 CS1002 ; expected H 15,943 7 312 14,674 246 704 3 CS0246 Type or namespace X (e.g., GuardGoal) not G 12,297 505 279 3,245 7,711 557 found 4 CS1525 Invalid expression term X (e.g., ’’) H 4,524 2 0 4,136 6 380 5 CS1001 Identifier expected H 4,343 239 110 3,776 64 154 6 CS1022 Type/namespace definition or EOF expected H 3,717 2 125 3,424 9 157 7 CS1513 } expected H 2,852 5 257 2,272 294 24 8 CS1044 More than one type in a declaration H 2,374 0 0 2,047 0 327 9 CS0103 Name X (e.g., whatToConceal) not in context H 2,070 131 22 51 1,864 2 10 CS1061 X has no member (e.g., startPosition) G 1,590 37 94 202 1,245 12

occurrences, or 31.5% of the 90,673 total. Of these, 26,268 (91.9%) come from 22B-Codestral. Next is CS1002 (missing semicolon, Hygiene) with 15,943 occurrences (17.6%). Among Grounding error codes, CS0246 (type or namespace not found) dominates with 12,297 occurrences (13.6%) spread across every [model, generation mode] (505, 279, 3,245, 7,711, and 557 from 7B-Qwen2.5 through 30B-Qwen3-Rt). Its presence in all five marks the Unity type system as a universal knowledge boundary. The second Grounding error code is CS1061 (member not found), with 1,590 occurrences (1.8%). Of these, 1,245 (78.3%) sit in 30B-Qwen3-Ed. Only CS0246 and CS1061 of the top 10 are Grounding error codes. The other eight are Hygiene. Hygiene errors accumulate through repetition. The same missing semicolons and unmatched braces recur line after line. Grounding errors are different. Each names a specific missing type and pinpoints a Unity API boundary the model failed to cross. 4) Composition Across Conditioning and [model, generation mode]: The previous sections ranked individual error codes by volume. From this composition analysis onward, we look only at their Grounding or Hygiene category, not the specific code. Figure 5 traces how the Grounding and Hygiene share (the fraction of error occurrences that are Grounding or Hygiene) of the failures shifts as the schema tightens. Because every error occurrence is either Grounding or Hygiene, the two shares sum to one. The Grounding share alone captures a [[model, generation mode], pattern]’s whole composition. A high Grounding share means failure mostly on missing Unity types and members, the errors that require engine knowledge to resolve. A high Hygiene share means failure on plain

syntax, diagnosable without any Unity knowledge. Each panel is one IR condition, holding up to 5 × 26 = 130 markers, one per [[model, generation mode], pattern] combination. A marker sums the error occurrences over that combination’s 20 seed-records, counted per seed. A marker appears only if its records yield at least one error code. A [[model, generation mode], pattern] whose records never produced one is absent, which is why the panel counts fall below 130. As the schema tightens, fewer markers remain. That means, under a stricter schema, more of a [[model, generation mode], pattern]’s records are rejected at the format sanitizer and never reach the compiler (Section III-C). With no records producing an error code, the marker disappears. Among those that remain, fewer reach one error code per seed. This thinning differs sharply by [model, generation mode]. Under no_schema 117 of the 130 markers are present. 30B-Qwen3Ed keeps all 26 goal patterns above one error code per seed at min and full. 22B-Codestral keeps 24 to 26. The smaller 7B-Qwen2.5 and 16B-DeepSeek still place 15 and 14 markers at full. Most of those fall below one error code per seed, leaving only 7 and 4 above. 30B-Qwen3-Rt nearly leaves the plot (two markers at full) because almost all its records are sanitizer-rejected. Only 30B-Qwen3-Ed and 22B-Codestral keep full coverage under a strict schema. Among the markers (i.e., [[model, generation mode], pattern] combination) that remain, composition moves toward Grounding as the schema tightens and converges across the surviving [model, generation mode]. Under no_schema the markers lean Hygiene. The Grounding share quantifies this. No [model, generation mode] reaches a median Grounding

9

Fig. 5. Error composition by IR conditioning. Each panel is one conditioning level. Each [[model, generation mode], pattern] appears as a marker, placed by total error occurrences per seed (x, log) and by Grounding share, the fraction of its error occurrences that are Grounding (y). The dashed line at Grounding share = 0.5 marks equal Grounding and Hygiene occurrences. Above it, Grounding occurrences are at least half of a marker’s total. Markers below one error per seed are shaded. A [[model, generation mode], pattern] with no compiler output is omitted (panel counts 117, 86, 82, and 83).

share of 0.5 (0.44 for 7B-Qwen2.5, 0.41 for 30B-Qwen3Ed, near 0 for the other three). These are per-marker medians, taken across a [model, generation mode]’s per-pattern markers, using markers at one error code per seed or more. The clearest trend is 30B-Qwen3-Ed. Its per-marker median Grounding share climbs level by level (0.41, 0.65, 0.79, 0.86). Its markers migrate rightward (median 1.2 to 10.4 error codes per seed) and their spread tightens, forming a small upper-right cluster by full. 22B-Codestral jumps later, from near 0 at no_schema and free to 0.70 at min and 0.99 at full. Its rise comes through verbose output filled with syntax errors (Section V). 7B-Qwen2.5 and 16B-DeepSeek never settle into a clean trend. Strict schema conditioning does not reduce failure. It concentrates the errors that remain onto the Grounding layer. 5) The Range of Per-Pattern Error Profiles: The failure landscape view in the previous sections says nothing about which gameplay goal concepts co-occur with these errors. This section reads the census pattern by pattern. It first bounds the range of per-pattern profiles across all five [model, generation mode], then zooms into 30B-Qwen3-Ed to trace them by pattern, conditioning level, and generation mode. As in Section IV-A4, we read each error occurrence by its Grounding or Hygiene category, not by its specific error code. The aggregated per-pattern error profile covers five [model, generation mode] × 26 goal patterns, or 130 [[model, generation mode], pattern] in total. Grounding occurrences exceed Hygiene in 50 of the 130 (38.5%). Hygiene occurrences exceed Grounding in 79 (60.8%). One yields no compiler output at all. Across the 130, the Grounding share spans nearly its entire possible range, from 0 to 0.98. At one end a pattern fails purely on plain syntax. At the other it fails almost only on missing Unity references. We examine the two endpoints in detail. The maximum is 0.98, for Alignment under 22B-Codestral. It has 206 Grounding occurrences (chiefly CS0246 with 181 and CS1061 with 21) against 5 Hygiene. Both error codes mean a referenced Unity type or member does not exist. With only 5 Hygiene occurrences, the generated C# scripts are almost free of structural defects. They fail at their Unity

references instead, the same invented-vocabulary failure as in the worked example (Section III-D). 22B-Codestral inflates its raw totals elsewhere with verbose and repetitive output (Section V). Repetition can inflate a Grounding count. It cannot produce the near-absence of Hygiene occurrences. The imbalance here is a property of composition, not volume. The minimum is 0, for Delivery under 16B-DeepSeek. It has 0 Grounding against 37 Hygiene, split between CS0101 (19, duplicate type in a namespace) and CS0111 (18, duplicate member in a type). The model re-declares the same types and members until the compiler rejects the file. The scripts collapse at the structural layer before any Unity reference is attempted. The two endpoints differ in model as well as in pattern. The maximum arises under 22B-Codestral and the minimum under 16B-DeepSeek. Between the two models, size, family, and pretraining corpus all change (Section III). The range mixes pattern effects with [model, generation mode] effects. B. A Zoom-in: 30B-Qwen3-Ed To read the pattern dimension cleanly, this section uses 30B-Qwen3-Ed alone, for three reasons. First, it carries the cleanest signal. Its sanitizer rejection stays below 16% across all four IR conditioning levels (Table III). At least 84% of its records clear the format gate at every level, so its per-pattern readings are the least affected by the compiler-reach selection discussed in the Limitations (Section V). The other four lose 85–98% of records to the format gate under the stricter schemas (with_schema_min and with_schema_full), or are swamped by verbose output that inflates their counts (Section V). Second, it has the broadest coverage, populating 101 of the 26 × 4 pattern-by-condition combinations, more than any other [model, generation mode]. Third, it is the largest model in the study. Its errors show how difficulty distributes across patterns once the model has its best chance, the most diagnostic setting for isolating the pattern dimension (Section III). The zoom-in reads this single [model, generation mode] from three views: by pattern, across the IR conditioning levels, and across the two generation modes. The first view is by pattern. In 30B-Qwen3-Ed, the highest per-seed counts of Grounding error occurrences fall on

10

Rescue (9.01), Exploration (8.11), Survive (7.59), and Stealth (6.69). Capture is the contrasting case, with an elevated perseed Hygiene count (7.88) against a comparatively low perseed Grounding count (2.38). For the four patterns with the highest per-seed Grounding counts, the most frequent error code is CS0246 (type or namespace not found). The identifiers it names extend the invented vocabulary of the worked example (Section III-D). Each is named after its pattern’s own mechanics: GuardAI and AlarmSystem for Stealth, RescueTarget for Rescue, ExplorationTracker for Exploration, and HealthSystem for Survive. The names encode the correct design semantics. None of these types exists in the engine or the project. The Grounding boundary is referential, not conceptual. It is a gap between what the pattern requires and what the model knows the engine to provide. The record-level outcomes vary less across these patterns than the code content does. Most reach the compiler at similar rates (50–58 of 80 records yield compiler diagnostics), while Capture and Ownership show higher sanitizer rejection (8 and 10 of 80) and correspondingly thinner code evidence. The second view follows the IR conditioning axis. Figure 6 reads 30B-Qwen3-Ed pattern by pattern. The shift toward Grounding appears again, now within a single [model, generation mode]. The column-mean Grounding share rises from 0.44 (no_schema) through 0.61 (free) and 0.77 (min) to 0.80 (full). Each column mean weights a pattern by its error occurrences, so its values differ from the equally weighted per-marker medians read off Figure 5. Most of the rise comes between free and min. After min the share levels off. Of the 23 patterns with data at both extremes, 18 end higher in Grounding share under full than under no_schema. The schema scaffolds the syntactic layer. Failures that would otherwise surface as Hygiene defects are displaced upward to the Grounding layer, where the model’s Unity knowledge becomes the limiting factor. The shift is a tendency, not a law. Five patterns (Stealth, Herd, Reconnaissance, Guard, and Capture) move the opposite way, ending lower in Grounding share under full. Capture is the sharpest case, falling from a Grounding share of 1.00 under no_schema to 0.14 under full. The error codes explain the reversal. Under no_schema, Capture produces only 7 error occurrences in total (5 CS0122, inaccessible member; 2 CS0246), all Grounding. The generated C# scripts are short and almost correct, failing only on a few Unity references. Under full, the same pattern produces 652 occurrences, 559 of them Hygiene, mostly unmatched braces (CS1513, 235) and missing semicolons (CS1002, 223). The schema drives the model to emit substantially longer and more structured C# scripts that collapse at the syntactic layer before any Unity reference is reached. The mechanism is the same one that moves most patterns upward (schema conditioning raises code complexity). Its direction depends on whether the model can keep the larger output syntactically intact for the pattern at hand. The third view switches the generation mode. It compares Qwen3-Coder-30B under its two modes, Editor-style (30BQwen3-Ed) and Runtime-builder (30B-Qwen3-Rt), holding condition, pattern, and seed constant. 30B-Qwen3-Ed produces

Fig. 6. Per-pattern error composition across IR conditioning levels, read from 30B-Qwen3-Ed. Each row is a goal pattern and each column an IR conditioning level. Each [model, generation mode] pools the error occurrences of all seeds for that pattern and level. The shade is the [model, generation mode]’s Grounding share (Section IV-A4): blue, Grounding share < 0.5; red, Grounding share > 0.5; white, the 0.5 balance. Rows are ordered by full Grounding share. Three no_schema entries with no compiler output are marked n/a. The bottom row gives the column mean.

12,160 error occurrences across 1,262 records with compiler output. 30B-Qwen3-Rt produces 5,989 occurrences across 386 such records. The lower Runtime figures reflect the sharp increase in sanitizer rejection under schema conditioning (Table III). At the per-pattern level (Table VI), the high per-seed Grounding counts noted above (Rescue, Exploration, Survive, Stealth) belong to the Editor mode. The Runtime-builder mode substantially reduces per-seed Grounding counts for most patterns but introduces severe Hygiene spikes for several patterns: Collection (per-seed Hygiene count 28.61 vs. 0.85 in Editor) and Connection (12.40 vs. 0.44). Configuration rises on both (Grounding 6.73 vs. 2.73, Hygiene 12.11 vs. 0.78). These Hygiene spikes mark structurally broken generated C# scripts, with no accompanying improvement in compilation outcomes. Overall, switching to the Runtime-builder mode does not eliminate Grounding errors. It redistributes the error landscape. Where the errors land depends on the pattern. For Collection

11

TABLE VI P ER - PATTERN COMPARISON OF THE TWO GENERATION MODES (30B, ALL IR LEVELS COMBINED ). E D -G / E D -H: PER - SEED COUNTS OF G ROUNDING AND H YGIENE ERROR OCCURRENCES IN THE E DITOR MODE (30B-Q WEN 3-E D ). RT-G / RT-H: THE SAME IN THE RUNTIME - BUILDER MODE (30B-Q WEN 3-RT ). P ER - SEED COUNTS ROUNDED TO TWO DECIMAL PLACES . Pattern Ownership Collection Eliminate Capture Overcome Evade Stealth Herd Conceal Rescue Delivery Guard Race Alignment Configuration Traverse Survive Connection Exploration Reconnaissance Contact Enclosure Gain Competence Gain Information Last Man Standing King of the Hill

Ed-G 2.45 3.78 4.13 2.38 5.88 3.88 6.69 3.20 3.81 9.01 4.19 3.83 5.41 1.96 2.73 5.38 7.59 2.20 8.11 4.48 1.86 2.74 3.55 3.05 5.71 5.56

Ed-H 1.54 0.85 0.93 7.88 1.46 1.64 1.51 0.69 1.16 1.63 1.51 2.09 1.21 0.35 0.78 0.81 1.61 0.44 1.79 1.14 0.98 0.75 0.99 0.91 2.16 1.69

Rt-G 0.01 0.09 0.00 0.00 0.01 0.00 0.00 0.05 0.11 0.10 0.00 0.08 0.01 0.06 6.73 0.00 0.03 0.00 0.00 0.11 0.03 0.01 0.06 0.01 0.00 0.03

Rt-H 0.64 28.61 0.76 0.10 0.78 0.65 0.48 0.01 0.16 0.49 0.58 1.04 0.58 1.74 12.11 0.40 0.79 12.40 0.68 0.29 0.39 0.51 0.28 0.89 0.45 1.56

and Connection, the Runtime-builder mode replaces Grounding errors with large Hygiene counts. For Configuration, both error categories are amplified. V. D ISCUSSION A. Interpretation of Per-Pattern Profiles The per-pattern Grounding share is a diagnostic lens on the Unity-specific knowledge each goal pattern demands. Its two ends fail at different layers. The four patterns with the highest per-seed Grounding error occurrence counts (Rescue, Exploration, Survive, Stealth) share one failure mode. The model posits a dedicated engine-specific component vocabulary and references it as though Unity provided it, producing CS0246. The reference implementations realize the same goals far more simply, with 2D trigger colliders and basic movement. The model understands the concept well enough to reference the right kind of operation but fails to name a class or method that exists, a knowledge-boundary failure, not a structural one. The invented names are not copied out of the pattern descriptions. The Rescue description (“free someone or something that is guarded”, Table I) nowhere mentions pathfinding or vision. The model still invents Pathfinding and ComputerVisionSystem for Rescue and attaches them as components. The Survive description contains neither collision nor health, yet the model invents AvatarCollisionHandler and AvatarHealth. None of these types exists in the engine or the project. The model infers from the pattern’s goal what gameplay machinery a scene would need, then invents engine types to supply it. That inference is itself a grounding failure, a claim about what the engine provides rather than an echo of the prompt.

Patterns with a low Grounding share, such as Capture, invert this. Their generated C# scripts collapse structurally before reaching the layer where Unity knowledge would be tested, as their mechanics reduce to general object-state tracking rather than engine-specific APIs. The gameplay pattern design language itself supports this reading at the two poles. Björk and Holopainen’s catalogue lists the patterns each goal pattern instantiates and is modulated by [9]. These relations spell out what a playable form of the pattern must simulate. Stealth (“move through an area and act without being detected”, Table I) instantiates Movement (“the action of moving game elements in the Game World”), Area Control (“being in control over who can move within an area in the game world”), and Tension (“the feeling of caring about the outcome of actions or events in a game without having full control over them”). The catalogue further describes it as a compound goal pattern built from Conceal (“hinder other players’ ability to gain information”) and Evade (“avoid being captured or hit”). Therefore, a scene that realizes Stealth must simulate concealment, evasion, and detection in some form. The relations do not dictate one embodiment. Read literally in a 2D scene, as our reference instantiation does, they become several mechanics at once: moving guards, fields of vision, and a rule for detection. In Unity that machinery lives in the perception and physics APIs, which is exactly where the model invents types such as GuardAI and AlarmSystem. Capture (“eliminate or take ownership of an actively resisting goal object”) instantiates Gain Ownership (“gain ownership of a game element”), Transfer of Control (“when the influence over a game element is passed from one player to another”), and Combat (“actions where the intent is to kill or otherwise overcome opponents”), relations that revolve around who owns or controls a game element. A scene that realizes Capture can keep that state in plain C# variables. Little of the engine is needed. Therefore, little of it can be named wrongly. Our per-pattern reading of the Grounding/Hygiene census turns the goal-pattern vocabulary of Björk and Holopainen [9], realized here as GPCs [8], into an empirical ordering of design concepts by the engine knowledge each demands. B. Model Scale and Identity From 7B to 30B, no model produces a runnable scene in a single pass. Within the deployable open-weight tier, a larger model is not a remedy. The trend is familiar from code-generation benchmarks. Pass rates climb with model size on self-contained routines [11], [12]. However, accuracy degrades sharply once the model must name real types and signatures from an external API [14], [15]. Engine-coupled generation, which constructs a live scene, sits at the far end of that axis. What varies by [model, generation mode] is the composition of failure. It does not track parameter count. 22BCodestral, larger than 7B-Qwen2.5 and 16B-DeepSeek, fails almost entirely at the syntactic layer. 30B-Qwen3-Ed reaches the Grounding layer. The two mixture-of-experts models differ again (Section IV). The sharpest case is at fixed scale. The same Qwen3-Coder-30B model reaches the compiler in the Editor-style mode but collapses to near-total sanitizer rejection

12

in the Runtime-builder mode. These differences are driven by model identity and generation mode, not scale. One model-specific behavior distorts count-based readings. 22B-Codestral contributes 68,813 of the 90,673 occurrences (76%) from only 2,080 records, with single records emitting up to 2,066 diagnostics. In its long records the model stops writing C# partway and continues in the style of the prompt’s documentation, emitting markdown that the C# compiler rejects line by line. This is an artifact of long output, not denser Unity API errors. We report per-seed counts of error occurrences and the Grounding share rather than raw totals, a caution for count-based profiling of LLM-generated code generally [32], [33]. For a team choosing a model, the takeaway is concrete. Within the studied tier, a larger model buys a different failure composition, not a runnable scene. Under a singlepass constraint, choosing a model is choosing which failure layer to face. A model that fails at the syntactic layer calls for output repair. A model that fails at the Grounding layer calls for engine knowledge that has to come from outside the model. The generation-mode contrast is measured only at 30B, where the Editor mode reaches the compiler far more often (1,262 against 386 records with compiler output). Smaller models ran in the Editor mode only, leaving the mode question below 30B open. Below the zero-compilation floor, pass-rate evaluation stops discriminating. The error composition is the signal that remains. This is what a failure census adds to passrate benchmarks of LLM code generation, and what it offers PCG evaluation at the game-design layer of goals [22]: a map of where generator capability ends before any content exists to evaluate. C. What IR Conditioning Buys, and What It Costs Like model scale and the generation mode in the previous section, IR conditioning never turns failure into success. In every [model, generation mode], adding a schema changes only where a record fails, not whether it fails. We use the IR to expose a different layer of failure for measurement. What the schema changes depends on the model. The two smallest, 7B-Qwen2.5 and 16B-DeepSeek, mostly stop producing usable output at all. Any schema sends most of their records to sanitizer rejection (70.8% and 77.5%). Under the strictest schema, most of those are not broken scripts but text that has stopped being code (a single token repeated, or a lone code-fence marker). Whatever Unity knowledge these models have, the schema makes it impossible to observe. 22B-Codestral and 30B-Qwen3-Ed do the opposite. A stricter schema helps them clear the format gate (22B-Codestral’s sanitizer rate drops from 64.0% to 14.4%). More records reach the compiler. They fail there too, with richer diagnostics but no runnable scene. The same Qwen3-Coder-30B in the Runtimebuilder mode collapses to 91.9–98.1% sanitizer rejection. Even for one fixed model, the schema’s effect depends on the generation mode. For the records that do reach the compiler, the failures move from syntax toward Unity knowledge as the schema tightens (mean Grounding share 0.44 to 0.80, Figure 6). The schema cleans up the syntax and leaves the

errors that need real engine knowledge, relocating the failure rather than removing it. This is the limit of structural conditioning. A schema constrains the shape of the output but supplies no Unity knowledge. It can move failure to the Grounding layer but cannot close the Grounding gap that blocks compilation. An IR that actually improved compilation would have to carry engine facts (real types, signatures, and members), not just structure. The direction of the schema’s effect appears to depend on model capacity. The schema is a structural contract: required fields, names, and shapes, with no engine content. Holding to that contract while still writing C# is itself a demand on the model. For 22B-Codestral and 30B-Qwen3-Ed, the contract keeps the output in code form, which lets more records survive the format gate. For 7B-Qwen2.5 and 16BDeepSeek, the same contract is more than the model can hold. Their output falls apart before any C# appears. The same dependence is reported elsewhere. Format restrictions degrade generation, with stricter restrictions degrading it more [40]. Structured-output reliability collapses as schemas grow more complex [41]. In this study the threshold sits between 16B and 22B. Having a contract matters more than how deep it goes. Most of the composition shift arrives by min (Figure 6). The Capture reversal (Section IV) marks where added depth turns against the model. D. Implications for LLM-Assisted Game Scene Development The Grounding/Hygiene taxonomy is a practical diagnostic for choosing interventions. Patterns with a high Grounding share, where the primary error is missing or incorrect Unity API references, are unlikely to benefit from generic prompt engineering or syntactic post-processing. They may require engine-knowledge augmentation such as retrieval over Unity documentation [42], fine-tuning on Unity corpora, or structured knowledge injection [43]. The 30B Editor-versusRuntime comparison illustrates this. Switching the generation mode redistributes rather than eliminates Grounding errors, and for several patterns (Collection, Connection) introduces severe Hygiene spikes without fixing the missing engine knowledge underneath. In gameplay terms, patterns with a high Grounding share are those whose core mechanic is coupled to the engine itself. Stealth’s detection, Exploration’s spatial search, and Survive’s collision and health all have to be expressed through specific Unity perception and physics components, exactly where the model invents nonexistent types. Meanwhile, Capture keeps a low Grounding share because its mechanic reduces to tracking who owns what in plain variables, with no such coupling. A designer can anticipate where single-pass generation will break by asking whether a concept’s defining mechanic lives in the engine’s perception and physics layer or in plain state logic. They can then budget engine-knowledge support (or hand-authoring) for the former. Patterns with a low Grounding share, by contrast, are better candidates for constrained or grammar-guided decoding [44], output sanitization, or syntactic post-processing. Their errors do not require Unity knowledge to diagnose or correct. A rule-

13

based sanitizer on general C# syntax can in principle intercept most Hygiene errors. The record-level outcomes add a deployment caution. On this evidence, pairing strict schema conditioning with sub-20B models is wasted computation. Of such records, 85–90% are lost at the format gate before any script compiles (Table III). Teams constrained to small open-weight models should either drop schema conditioning or invest first in format compliance (e.g., grammar-constrained decoding) before structured knowledge injection can pay off. The single-pass results also show what an iterative repair loop would have to add. A pattern with a high Grounding share forces each repair loop to supply Unity API knowledge the model cannot supply itself. A pattern with a low Grounding share is easier to repair because the model already has enough Unity knowledge for a locally targeted fix. The contemporaneous Mage benchmark, built on a related class of goal-pattern tasks with its own scenes and pipeline, similarly finds nearzero compilation rates [29]. The floor is not an artifact of our particular instantiations or harness. Whether it holds beyond goal-pattern tasks in Unity is untested. E. Limitations Several limitations bound these findings. (a) Pattern instantiation vs. pattern. Each goal pattern is an abstract design construct. Our 26 targets are its current Unity instantiations, one per pattern, so the error profiles characterize these particular instantiations, not goal patterns in general. All 26 are 2D scenes built from the same engine primitives, compiled through the same harness, and prompted through the same template. A cross-pattern difference in the census is unlikely to come from one pattern receiving a more elaborate implementation. This control is at the implementation level and does not make a single instantiation representative of its pattern. (b) Model axis is not a controlled scale sweep. Size, family, pretraining corpus, and architecture (the 16B and 30B are sparse mixture-of-experts, with 2.4B and 3.3B active parameters) change together along the axis. Differences by [model, generation mode] cannot be attributed to parameter count alone. The uniform compilation failure is unaffected, as every model fails regardless of family or size. Claims that separate scale from model identity are out of scope. (c) Census coverage and compiler-reach selection. The census covers only records that reach the compiler and emit diagnostics. The share of records that do varies sharply across the [model, generation mode] and conditioning grid. No-entry records compile but expose no entry point for the BatchRunner, typically a plausible gameplay component rather than a scene builder (e.g., a well-formed public class Contact : MonoBehaviour that never builds the scene). This is a prompt-specification failure, reported in Table III but contributing no error codes. Sanitizer rejection further removes 70–98% of records in several [[model, generation mode], condition] rows. A per-pattern Grounding share is read as conditional on producing compiler output, not as an unconditional property of the pattern. This selection is distinct from survivorship bias, which the all-fail corpus removes

(Section I). It acts earlier, on which failed records yield compiler evidence. The reference choice of the zoom-in, with sanitizer rejection below 16%, keeps it minimal (Section IV). (d) IR attribution and schema quality. Under schema conditioning the same LLM generates the IR (Step 1) and the C# script (Step 2), so compiler evidence alone cannot attribute a Grounding error to a hallucinated IR type or to the coding step. Settling that attribution would require running Step 2 with ground-truth IRs. The results also characterize one frozen schema (v0.2-runtime-evidence), built from the reference implementations and frozen before any run, which guards against tuning to outcomes but does not make it the best possible schema. The headline result does not rest on the schema. no_schema supplies no IR and still yields zero runnable scenes. (e) Timeout budget. The 120 s compilation budget is uniform across conditions, which protects comparisons. Its absolute value remains a choice that a replication could probe. (f) Single-pass ceiling. The Grounding/Hygiene profiles characterize the intrinsic capability ceiling before any intervention, not performance in real workflows with iterative repair. (g) Reading the Grounding share. A high Grounding share does not mean a model handles Grounding well, only that the syntax is clean enough for Grounding errors to be what is left. A low share usually means the generated C# scripts broke before they ever referenced Unity. The share marks where a [model, generation mode] fails, not how well it does anything. The same reading limits the schema’s help. For Capture, the structure the schema asks for is more than the model can write cleanly. Its Grounding share falls from 1.00 to 0.14 as syntax errors return (Section IV). VI. C ONCLUSION We conducted a controlled single-pass evaluation of Unity C# scene generation across 10,400 records (five [model, generation mode]; four IR conditioning levels; 26 goal patterns; 20 seeds). No generated C# script compiled into a runnable scene. From the failed compilations we extracted 90,673 error occurrences across 99 error codes (18 Grounding, 81 Hygiene). The per-pattern census orders Björk and Holopainen’s goal patterns by the engine knowledge their current specific Unity instantiations demand. Patterns coupled to Unity’s perception and physics APIs (e.g., Stealth, Rescue) concentrate Grounding errors. Patterns reducible to plain state manipulation (e.g., Capture) concentrate Hygiene errors. This ordering is an interpretation of the error evidence, not a direct measurement of pattern complexity. A larger model, a stricter schema, and a different generation mode each shift the error profiles without producing a compiling scene. IR conditioning supplies structure rather than engine knowledge. The bottleneck is the missing engine knowledge, not structure, generation mode, or scale. The resulting Grounding/Hygiene error taxonomy gives designers a concrete read on which gameplay concepts current LLMs can realize and which demand engine-knowledge support. Future work will extend the evaluation in three directions. First, multi-turn repair, feeding compiler errors back to the model, will test how many rounds it takes to reach compilation

14

and whether the Grounding/Hygiene profile predicts which patterns are easy to repair. Second, running the coding step (Step 2) on ground-truth IRs will separate IR-generation failures from code-generation failures. Third, expanding to more patterns and to larger or closed-source models will test whether the findings hold beyond the 7B–30B openweight tier. Beyond automated repair, the census also serves human-in-the-loop workflows. A person audits and approves the generated C# scripts, keeping accountability with people, not with the model. R EFERENCES [1] D. Church, “Formal abstract design tools,” Game Developer, vol. 6, no. 8, pp. 44–50, 1999. [2] G. Costikyan, “I have no words & i must design: Toward a critical vocabulary for games,” in Computer Games and Digital Cultures Conference Proceedings, 2002. [3] B. Kreimeier, “The case for game design patterns,” 2002. [4] P. Cairns, C. Power, M. Barlet, and G. Haynes, “Future design of accessibility in games: A design vocabulary,” International Journal of Human-Computer Studies, vol. 131, pp. 64–71, 2019. [5] R. I. Clarke, J. H. Lee, and N. Clark, “Why video game genres fail: A classificatory analysis,” Games and Culture, vol. 12, no. 5, pp. 445–465, 2017. [6] D. Arsenault, “Video game genre, evolution and innovation,” Eludamos: Journal for computer game culture, vol. 3, no. 2, pp. 149–176, 2009. [7] J. Juul, Half-real: Video games between real rules and fictional worlds. MIT press, 2011. [8] Z. Lyu, J. Holopainen, and S. Björk, “Goal playable concepts coupling gameplay design patterns with playable concepts,” in Proceedings of the 26th International Academic Mindtrek Conference, 2023, pp. 57–66. [9] S. Bjork and J. Holopainen, Patterns in game design. Charles River Media Hingham, 2005, vol. 11. [10] P. Lankoski and S. Björk, “Game research methods: An overview,” 2015. [11] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [12] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. [13] M. Rahman, S. Khatoonabadi, and E. Shihab, “Beyond synthetic benchmarks: Evaluating llm performance on real-world class-level code generation,” arXiv preprint arXiv:2510.26130, 2025. [14] J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” ACM Transactions on Software Engineering and Methodology, vol. 35, no. 2, pp. 1–72, 2026. [15] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” vol. 2024, pp. 54 107–54 157, 2024. [16] N. Shaker, J. Togelius, and M. J. Nelson, Procedural content generation in games. Springer, 2016. [17] J. Togelius, G. N. Yannakakis, K. O. Stanley, and C. Browne, “Searchbased procedural content generation: A taxonomy and survey,” IEEE Transactions on Computational Intelligence and AI in Games, vol. 3, no. 3, pp. 172–186, 2011. [18] A. Summerville, S. Snodgrass, M. Guzdial, C. Holmgård, A. K. Hoover, A. Isaksen, A. Nealen, and J. Togelius, “Procedural content generation via machine learning (pcgml),” IEEE Transactions on Games, vol. 10, no. 3, pp. 257–270, 2018. [19] R. Gallotta, G. Todd, M. Zammit, S. Earle, A. Liapis, J. Togelius, and G. N. Yannakakis, “Large language models and games: A survey and roadmap,” IEEE Transactions on Games, 2024. [20] G. Todd, S. Earle, M. U. Nasir, M. C. Green, and J. Togelius, “Level generation through large language models,” in Proceedings of the 18th International Conference on the Foundations of Digital Games, 2023, pp. 1–8. [21] S. Sudhakaran, M. González-Duque, M. Freiberger, C. Glanois, E. Najarro, and S. Risi, “Mariogpt: Open-ended text2level generation through large language models,” Advances in Neural Information Processing Systems, vol. 36, pp. 54 213–54 227, 2023.

[22] M. Hendrikx, S. Meijer, J. Van Der Velden, and A. Iosup, “Procedural content generation for games: A survey,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 9, no. 1, pp. 1–22, 2013. [23] C. Barney, Pattern language for game design. CRC Press, 2020. [24] J. P. Zagal, M. Mateas, C. Fernández-Vara, B. Hochhalter, and N. Lichti, “Towards an ontological language for game analysis,” Worlds in play: International perspectives on digital games research, vol. 21, p. 21, 2007. [25] J. Parkkila, F. Radulovic, D. Garijo, M. Poveda-Villalón, J. Ikonen, J. Porras, and A. Gómez-Pérez, “An ontology for videogame interoperability,” Multimedia tools and applications, vol. 76, no. 4, pp. 4981– 5000, 2017. [26] S. De Martino, M. N. Asmundo, S. A. Rizzo, and D. F. Santamaria, “Modeling the video game environment: the videowl ontology.” in WOA, 2023, pp. 191–205. [27] M. Ebner, J. Levine, S. M. Lucas, T. Schaul, T. Thompson, and J. Togelius, “Towards a video game description language,” 2013. [28] D. Perez-Liebana, J. Liu, A. Khalifa, R. D. Gaina, J. Togelius, and S. M. Lucas, “General video game ai: A multitrack framework for evaluating agents, games, and content generation algorithms,” IEEE Transactions on Games, vol. 11, no. 3, pp. 195–214, 2019. [29] H. X. Liu and K. Tatar, “Mage: Multi-axis evaluation of llm-generated executable game scenes beyond compile-pass rate,” 2026. [Online]. Available: https://arxiv.org/abs/2605.07342 [30] A. Altadmri and N. C. Brown, “37 million compilations: Investigating novice programming mistakes in large-scale student data,” in Proceedings of the 46th ACM technical symposium on computer science education, 2015, pp. 522–527. [31] R. S. Pettit, J. Homer, and R. Gee, “Do enhanced compiler error messages help students? results inconclusive.” in Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education, 2017, pp. 465–470. [32] F. Tambon, A. Moradi-Dakhel, A. Nikanjam, F. Khomh, M. C. Desmarais, and G. Antoniol, “Bugs in large language models generated code: An empirical study,” Empirical Software Engineering, vol. 30, no. 3, p. 65, 2025. [33] S. Dou, H. Jia, S. Wu, H. Zheng, M. Wu, Y. Tao, M. Zhang, M. Chai, J. Fan, Z. Xi et al., “What is wrong with your code generated by large language models? an extensive study,” Science China Information Sciences, vol. 69, no. 1, p. 112107, 2026. [34] G. Ramesh, M. Sahil, S. A. Palan, D. Bhandary, T. A. Ashok, J. Shreyas, and N. Sowjanya, “A review on nlp zero-shot and few-shot learning: methods and applications,” Discover Applied Sciences, vol. 7, no. 966, 2025. [35] B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu et al., “Qwen2. 5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024. [36] Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y. Wu, Y. Li, H. Gao, S. Ma et al., “Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,” arXiv preprint arXiv:2406.11931, 2024. [37] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al., “Qwen3 technical report,” 2025. [38] E. Kohlbecker, D. P. Friedman, M. Felleisen, and B. Duba, “Hygienic macro expansion,” in Proceedings of the 1986 ACM Conference on LISP and Functional Programming, 1986, pp. 151–161. [39] D. M. Tilbrook and J. McMullen, “Washing behind your ears: Principles of software hygiene.” [40] Z. R. Tam, C.-K. Wu, Y.-L. Tsai, C.-Y. Lin, H.-y. Lee, and Y.N. Chen, “Let me speak freely? a study on the impact of format restrictions on performance of large language models,” arXiv preprint arXiv:2408.02442, 2024. [41] S. Geng, H. Cooper, M. Moskal, S. Jenkins, J. Berman, N. Ranchin, R. West, E. Horvitz, and H. Nori, “Jsonschemabench: A rigorous benchmark of structured outputs for language models,” arXiv preprint arXiv:2501.10868, 2025. [42] S. Zhou, U. Alon, F. F. Xu, Z. Jiang, and G. Neubig, “Docprompting: Generating code by retrieving the docs,” in The Eleventh International Conference on Learning Representations, 2022. [43] X. Jiang, Y. Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao, “Self-planning code generation with large language models,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 7, pp. 1–30, 2024. [44] B. Wang, Z. Wang, X. Wang, Y. Cao, R. A Saurous, and Y. Kim, “Grammar prompting for domain-specific language generation with large language models,” vol. 36, 2023, pp. 65 030–65 055.

15

TABLE VII IR SCHEMA ITERATION HISTORY FROM INITIAL DRAFT TO FROZEN V 0.2- RUNTIME - EVIDENCE .

#

Iteration What Changed

Why

Impact

1

v0 static draft

Defined six top-level fields: scene, objects, scripts, params, links, rules.

Minimal structured representation between pattern description and Unity code generation.

Established baseline schema consumed by all pipeline stages.

2

MVP narrowing

Deferred params extraction; pipeline operates on objects, scripts, links, rules only. params emitted as {}.

params requires GUID resolution (scene → prefab → script → .cs), blocking the initial pipeline.

Unblocked end-to-end generation without serialized-field extraction.

3

Static .unity parsing misses prefabRuntime Added PrefabInstance/PrefabAsset object types, script-defined rules, and driven gameplay; core behavior emerges exfrom prefab instantiation and runtime runtime_params field. tenscript logic. sion

IR captures the actual gameplay loop; generation can reason about spawned entities and runtime configuration.

4

scripts[].object_id must refPerinstance erence a real objects[].id. No implicit aggregate placeholders (e.g. constraint circle_all).

Aggregate placeholders create ambiguous references unresolvable during code generation or evaluation.

Enforces 1:1 script-to-object binding; enables automatic referential integrity validation.

5

Evidence-Conditional relation labels (e.g. Unconditional labels over-assert deteraware can_trigger_game_win_if_aligned). minism for conditional code paths. EviseRequired evidence_type on dence attribution improves trust calibramanevery rules[] entry. Optional tion in generated output. tics confidence field.

Generation produces more accurate causal claims; evaluation can filter or weight rules by evidence type and confidence.

VII. ACKNOWLEDGEMENT The batch compilation pipeline adapts the write-to-asset approach introduced in AICommand by Keijiro Takahashi 9 . We thank Staffan Björk and Jussi Holopainen for their input on goal playable concepts and related background. This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program – Humanities and Society (WASP-HS) funded by the Marianne and Marcus Wallenberg Foundation and the Marcus and Amalia Wallenberg Foundation. The computations and data handling were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 202206725.

links: [{ source, target, relation, evidence_type? }, ...] rules: [{ id, type, description, pattern, evidence_type, confidence? }, ...] Hard constraints. 1) scripts[].object_id must reference objects[].id 2) scripts are per-instance (no sharing across objects) 3) no implicit aggregate placeholders 4) rules[].evidence_type is required, in { direct_code, scene_override, inferred }

A PPENDIX A IR S CHEMA : I TERATION H ISTORY AND D EFINITION Table VII records the five-iteration history from the initial static draft to the frozen v0.2-runtime-evidence schema. The three with_schema conditions expose progressively more of this frozen schema to the IR-generation step. free supplies no schema text, so the model structures the IR itself; min supplies only the top-level field names below; and full supplies the full field definitions and hard constraints below. Top-level fields (all required). scene: string objects: [{ id, name, type }, ...] scripts: [{ id, object_id, class_name }, ...] params: {} runtime_params: { "<script_id>": { ... }, ... } 9 https://github.com/keijiro/AICommand

A PPENDIX B F ULL E RROR C ODE C ENSUS

16

TABLE VIII: All 99 observed C# compiler error codes, categorized as Grounding (G) or Hygiene (H), with total occurrence count and message template. Quoted identifiers are shown as X; for Grounding codes, a representative identifier from the logs is given in parentheses, almost all of them model-invented, pattern-named types absent from the engine and project. Code

Total

Message template

Grounding (G), 18 codes CS0246 12,297 The type or namespace name X (e.g., EnemyAI, GuardAI) could not be found CS1061 1,590 X (e.g., PlayerController.health) does not contain a definition for X and no accessible extension m CS0115 213 X (e.g., StealthGoal.IsCompleted()): no suitable method found to override CS0234 138 The type or namespace name X (e.g., namespace Game) does not exist in the namespace ’UnityEngine.Animatio CS0117 71 X (e.g., RuntimeSceneBuilder) does not contain a definition for X CS0122 68 X (e.g., RescueGoal.Rescue()) is inaccessible due to its protection level CS0311 66 The type X cannot be used as type parameter X in the generic type or CS0030 20 Cannot convert type X (e.g., DetectableElement[]) to X CS0619 13 X (e.g., GameObject.light) is obsolete: ’GameObject.AddComponent with string argume CS0239 8 X: cannot override inherite CS0315 7 The type X cannot be used as type parameter X in the generic type or me CS8121 4 An expression of type X cannot be handled by a pattern of type ’ScriptableObject CS0506 3 X (e.g., PlayerController.Update()): cannot override inherited member ’PlayerController.Upd CS0534 3 X (e.g., ConfigurationRule.IsSatisfied()) does not implement inherited abstract member ’ConfigurationRule.IsSatisfied(Li CS1624 2 The body of X (e.g., TraverseInstantiator.MoveGameElement()) cannot be an itera CS0426 1 The type name X does not exist in the type X CS0509 1 X (e.g., CollectionClip): cannot derive from sealed type X CS7036 1 There is no argument given that corresponds to the required formal parameter X Hygiene (H), 81 codes CS1003 28,571 Syntax error, X expected CS1002 15,943 ; expected CS1525 4,524 Invalid expression term X CS1001 4,343 Identifier expected CS1022 3,717 Type or namespace definition, or end-of-file expected CS1513 2,852 } expected CS1044 2,374 Cannot use more than one type in a for, using, fixed, or declaration statement CS0103 2,070 The name X does not exist in the current context CS1013 1,342 Invalid number CS1529 1,138 A using clause must precede all other elements defined in the namespace except extern alia CS1026 951 ) expected CS0111 939 Type X already defines a member called X with the same parameter types CS0101 820 The namespace X already contains a definition for X CS1055 662 An add or remove accessor expected CS1056 526 Unexpected character X CS0270 523 Array size cannot be specified in a variable declaration (try initializing with a X ex CS1519 497 Invalid token X in class, record, struct, or interface member declaration CS1012 472 Too many characters in character literal CS1503 423 Argument 1: cannot convert from X to X CS0116 403 A namespace cannot directly contain members such as fields, methods or statements CS0650 369 Bad array declarator: To declare a managed array the rank specifier precedes the variable’ CS1514 339 { expected CS8124 332 Tuple must contain at least two elements. CS1010 328 Newline in constant CS8803 169 Top-level statements must precede namespace and type declarations. CS1014 167 A get or set accessor expected CS1031 165 Type expected CS1040 148 Preprocessor directives must appear as the first non-whitespace character on a line CS0165 147 Use of unassigned local variable X CS0106 115 The modifier X is not valid for this item CS1526 111 A new expression requires an argument list or (), [], or {} after type CS0595 107 Invalid real literal. CS1041 71 Identifier expected; X is a keyword CS0263 55 Partial declarations of X must not specify different base classes CS8635 49 Unexpected character sequence X CS1585 47 Member modifier X must precede the member type and name CS1024 42 Preprocessor directive expected CS0742 33 A query body must end with a select clause or a group clause CS0443 32 Syntax error; value expected CS8641 23 X cannot start a statement. CS0029 22 Cannot implicitly convert type X to X CS0145 22 A const field requires a value to be provided CS1018 17 Keyword X or X expected CS0708 14 X: cannot declare instance members in a static class CS0178 13 Invalid rank specifier: expected X or X CS0021 12 Cannot apply indexing with [] to an expression of type X CS0120 12 An object reference is required for the non-static field, method, or property ’HerdAttract CS1073 12 Unexpected token X continued on next page

17

Table VIII continued from previous page Code

Total

CS0687 CS7000 CS0176 CS0266 CS1524 CS1027 CS0136 CS1528 CS0200 CS0216 CS0721 CS1011 CS1597 CS1955 CS0034 CS0713 CS0722 CS1009 CS1501 CS1733 CS1950 CS0102 CS0230 CS0272 CS0664 CS0723 CS1020 CS1515 CS1553 CS1646 CS8180 CS8504 CS8652

10 10 9 9 7 6 5 4 3 3 3 3 3 3 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1

Message template The namespace alias qualifier X always resolves to a type or namespace so is illegal he Unexpected use of an aliased name Member X cannot be accessed with an instance reference; quali Cannot implicitly convert type X to X. An explicit conversion exists Expected catch or finally #endif directive expected A local or parameter named X cannot be declared in this scope because that na Expected ; or = (cannot specify constructor arguments in declaration) Property or indexer X cannot be assigned to – it is read only The operator X requires a matching operator X: static types cannot be used as parameters Empty character literal Semicolon after method or accessor block is not valid Non-invocable member X cannot be used like a me Operator X is ambiguous on operands of type X and X Static class X cannot derive from type X. Static classes X: static types cannot be used as return types Unrecognized escape sequence No overload for method X takes 2 arguments Expected expression The best overloaded Add method X for the collection initializer h The type X already contains a definition for X Type and identifier are both required in a foreach statement The property or indexer X cannot be used in this context because the Literal of type double cannot be implicitly converted to type X; use an X suffix t Cannot declare a variable of static type X Overloadable binary operator expected X expected Declaration is not valid; use X instead Keyword, identifier, or string expected after verbatim specifier: @ { or ; or => expected Pattern missing The feature X is currently in Preview and *unsupport

18

A PPENDIX C W ORKED -E XAMPLE I NPUT: S TEALTH PATTERN D ESCRIPTION The complete goal-pattern description fed to the model for the worked example (Section III-D). The model received this Markdown text verbatim and nothing else — no scene, assets, or asset names. The invented class names in the worked example (Section III-D), such as GuardAI, AlarmSystem, and SafeZoneTrigger, trace directly to terms in this text (Guards, Alarms, safe zone). # Stealth ## Description Stealth is the goal to move through a certain area and perform an action without being detected. ## Overview Sometimes favorable conditions in a game can be achieved by not having one’s actions noticed by other players. When this is the case, players have [Stealth](Stealth.md) goals that force them to plan actions that minimize the risks of being noticed while still completing the required actions. [Stealth](Stealth.md) is a compound goal pattern using [Conceal](Conceal.md) together with [Evade](Evade.md) with a secondary goal involving [Movement](Movement.md) or other actions from the player, normally [Rescue](Rescue.md), [Traverse](Traverse.md), [Delivery](Delivery.md), [Camping](Camping.md), or [Gain Ownership](GainOwnership.md) (including gaining [Area Control](AreaControl.md) simply by being undetected in a particular place). Designing the [Stealth](Stealth.md) goal consists not only of choosing between the different design options of these patterns, but also determining what player actions can reveal the players and what the [Tradeoffs](Tradeoffs.md) are between the various [Risk/Reward](RiskReward.md) relations for each action in a given context. Longer [Stealth](Stealth.md) goals can be divided into parts that require short [Stealth](Stealth.md) goals to be fulfilled in order to avoid [Guards](Guard.md) and [Alarms](Alarms.md), short bursts of action to [Overcome](Overcome.md) enemy [Units](Units.md) without them activating [Alarms](Alarms.md), and [Tension](Tension.md) -filled moments when the best option for the player is to perform [No-Ops](No-Ops.md). The complexity of [Stealth](Stealth.md) goals can be increased by letting [Guards](Guard.md) have [Reconnaissance](Reconnaissance.md) goals so that players have to take their [Movement](Movement.md) into consideration. [Stealth](Stealth.md) is the goal of trying to [Conceal](Conceal.md) one’s location while having to move. [Stealth](Stealth.md) goals may require players to pace themselves as quick [Movement](Movement.md) may have too high risks, and sometimes any action or [Movement](Movement.md) may cause the goal to fail. [Stealth](Stealth.md) can thus create [Tension](Tension.md) as players may have no [Freedom of Choice](FreedomofChoice.md) except to perform [No-Op](No-Ops.md) actions to continue to [Conceal](Conceal.md) themselves hoping not to be detected by opponents (which actually represents a form of [Area Control](AreaControl.md)). The slow tempo and possible pauses in completing [Stealth](Stealth.md) goalsgive players a chance to make use of [Strategic Knowledge](StrategicKnowledge.md), for example the locations of [Alarms](Alarms.md), making the game with the pattern have [Stimulated Planning](StimulatedPlanning.md). Most cases of [Stealth](Stealth.md) rely on opponents having [Guard](Guard.md) or [Reconnaissance](Reconnaissance.md) as [Preventing Goals](PreventingGoals.md), making [Stealth](Stealth.md) and these goals[Excluding Goals](ExcludingGoals.md). Giving players [Stealth](Stealth.md) goals combined with [Herd](Herd.md) goals increases the chances of failure and may limit the[Right Level of Difficulty](RightLevelofDifficulty.md) of the goals. ## Examples * [Thief: The Dark Project](../game/thief-the-dark-project.md) and the other games in the series exemplify a game using [Stealth](Stealth.md). The player is a master thief, Garrett, who lives in a medieval fantasy world and performs his duties by relieving the rich nobles of their riches. The main goal is to collect the valuable items, while the secondary goal is to avoid being detected by the [Guards](Guard.md) while moving around the [Level](Levels.md) s. * Many children’s’ games are based on one person trying to find the other players while at the same time trying to [Guard](Guard.md) an area that is a safe zone for the other players. If the other players, by a combination of stealth and running, make it to the safe zone they are home free and do not have to be the player guarding the safe zone in the next game. ## Relations ### Instantiates * [Evade](Evade.md) * [Stimulated Planning](StimulatedPlanning.md) * [Tension](Tension.md) * [Movement](Movement.md) * [Area Control](AreaControl.md)

19

### Modulates * [Delivery](Delivery.md) * [Rescue](Rescue.md) ### Instantiated by * [Reconnaissance](Reconnaissance.md) ### Modulated by * [Safe Havens](SafeHavens.md) * [No-Ops](No-Ops.md) * [Guard](Guard.md) * [Alarms](Alarms.md) * [Risk/Reward](RiskReward.md) * [Tradeoffs](Tradeoffs.md) * [Traverse](Traverse.md) * [Gain Ownership](GainOwnership.md) * [Camping](Camping.md) ### Potentially conflicting with * [Herd](Herd.md)

A PPENDIX D P ROMPT T EMPLATES All prompts are reproduced verbatim from src/prompts/ in the artifact repository. Four placeholders are substituted at runtime. <PATTERN_ID> becomes the goal-pattern name, <PATTERN_MD> the full Markdown description of that pattern (see Appendix C for the Stealth example), <IR_JSON> the Step-1 IR generated by the IR-maker prompt, and <METHOD> the IR conditioning level name (with_schema_free, with_schema_min, or with_schema_full). Coder prompts Editor-style, no_schema (7B-Qwen2.5, 16B-DeepSeek, 22B-Codestral, 30B-Qwen3-Ed; condition no_schema): [pattern: <PATTERN_ID>] [method: no_schema] Generate a Unity Editor script that implements the playable concept described below. Output only raw C# code. <PATTERN_MD> Editor-style, with_schema (7B-Qwen2.5, with_schema_free/min/full):

16B-DeepSeek,

22B-Codestral,

30B-Qwen3-Ed;

[pattern: <PATTERN_ID>] [method: <METHOD>] Generate a Unity Editor script that instantiates a scene matching the following engine-specific Intermediate Representation (IR). Thereafter, you may refer to it as IR. Output only raw C# code. <IR_JSON> Runtime-builder, no_schema (30B-Qwen3-Rt; condition no_schema): [pattern: <PATTERN_ID>] [method: no_schema_runtime] Generate a single C# file containing a Unity MonoBehaviour called RuntimeSceneBuilder that, in its Awake() method, programmatically implements the playable concept described below. Also define all gameplay MonoBehaviour classes in the same file.

conditions

20

Attach RuntimeSceneBuilder to an empty GameObject and press Play. The scene must build itself at runtime | no Editor API, no MenuItem, no UnityEditor namespace. Output only raw C# code. <PATTERN_MD> Runtime-builder, with_schema (30B-Qwen3-Rt; conditions with_schema_free/min/full): [pattern: <PATTERN_ID>] [method: <METHOD>] Generate a single C# file containing a Unity MonoBehaviour called RuntimeSceneBuilder that, in its Awake() method, programmatically creates a scene matching the following engine-specific Intermediate Representation (IR). Thereafter, you may refer to it as IR. Also define all gameplay MonoBehaviour classes in the same file. Attach RuntimeSceneBuilder to an empty GameObject and press Play. The scene must build itself at runtime | no Editor API, no MenuItem, no UnityEditor namespace. Output only raw C# code. <IR_JSON> IR-maker prompts (Step 1, with_schema conditions only) with_schema_free (all five [model, generation mode]): [pattern: <PATTERN_ID>] [method: with_schema_free] Generate an engine-specific Intermediate Representation (IR) JSON for the playable concept described below. Thereafter, you may refer to it as IR. Output ONLY valid JSON. No extra text. <PATTERN_MD> with_schema_min (all five [model, generation mode]): [pattern: <PATTERN_ID>] [method: with_schema_min] Generate an engine-specific Intermediate Representation (IR) JSON for the playable concept described below. Thereafter, you may refer to it as IR. Output ONLY valid JSON. No extra text. Required top-level fields: "scene" | string "objects" | [ { "id", "name", "type" }, ... ] "scripts" | [ { "id", "object_id", "class_name" }, ... ] "params" | {} "runtime_params" | { "<script_id>": { ... }, ... } "links" | [ { "source", "target", "relation" }, ... ] "rules" | [ { "id", "type", "description", "pattern", "evidence_type" }, ... ] <PATTERN_MD> with_schema_full (all five [model, generation mode]):

21

[pattern: <PATTERN_ID>] [method: with_schema_full] Generate an engine-specific Intermediate Representation (IR) JSON for the playable concept described below. Thereafter, you may refer to it as IR. Output ONLY valid JSON. No extra text. Follow the IR v0.2-runtime-evidence schema precisely. Top-level fields (all required): "scene" | string, scene identifier "objects" | array of { "id", "name", "type" } type in { "GameObject", "PrefabInstance", "PrefabAsset" } "scripts" | array of { "id", "object_id", "class_name" } one entry per component instance on one object "params" | always {} "runtime_params" | object keyed by scripts[].id; values are flat { field: value } maps "links" | array of { "source", "target", "relation", "evidence_type"? } "rules" | array of { "id", "type", "description", "pattern", "evidence_type", "confidence"? } Hard constraints: 1. Every scripts[].object_id MUST reference a real objects[].id (no dangling refs). 2. Scripts are per-instance; no shared script entries across objects. 3. Every entity must be listed explicitly in objects (no aggregate placeholders). 4. Every rules[] entry MUST include evidence_type in { "direct_code", "scene_override", "inferred" }. <PATTERN_MD>