The Patchwork Problem in LLM-Generated Code Viraaji Mothukuri, Reza M. Parizi Decentralized Science Lab, College of Computing and Software Engineering Kennesaw State University, GA, USA [email protected] [email protected]
arXiv:2607.08981v1 [cs.SE] 9 Jul 2026
Abstract LLM-generated code often compiles, passes tests, and appears correct, yet breaks once deployed. The root cause is frequently structural rather than logical. A generated endpoint references configuration keys never declared in the project, an import targets a package that does not exist in any registry, or a new route omits the authentication guard applied to every sibling endpoint. Each patch is locally valid but globally incoherent, and standard CI toolchains rarely surface these failures. As LLM-powered coding tools see widespread adoption, this blind spot poses a growing risk to software quality. We call this the patchwork problem. This paper formalizes structural coherence as consistency invariants over graph representations of repository artifacts, including import, call, dependency, configuration, schema, resource, control-flow, and routing graphs, and introduces an eight-category failure taxonomy distinguishing defects specific to LLM generation from those merely amplified by it. We present a hybrid verification framework that delegates to mature static analysis tools where they already excel and deploys purpose-built detectors for cross-cutting invariants underserved by existing toolchains, targeting provable constraint violations rather than heuristic pattern matching. Empirical evaluation across two frontier models under four prompting strategies reveals that the vast majority of structural failures evade type checking, testing, and SAST entirely, and that failure patterns diverge qualitatively between models in ways that challenge model-agnostic mitigation strategies. External validation on real-world AI-generated repositories confirms that these failures are not artifacts of controlled experimentation but are prevalent wherever LLMs write code with minimal human oversight. Keywords: LLM code generation, structural coherence, static analysis, graph invariants, code quality, neural code synthesis.
1
Introduction
tools typically focus on taint flows rather than structural coherence. The result is a blind spot in which generated code enters codebases carrying latent defects that remain invisible to standard toolchains. We term this the patchwork problem. LLM-generated patches may be individually well-formed yet fail to cohere into a consistent whole, particularly at repository scale, where consistency constraints span imports, dependencies, configurations, schemas, and security contracts [7]. Our approach formalizes structural coherence as consistency invariants over graph representations of repository artifacts [8]. A key design insight is that reliable detection requires matching each invariant class to an appropriate verification strategy. Categories where mature static analysis tools already capture the relevant language semantics can be delegated to those tools, while categories that require cross-graph reasoning, absent from existing toolchains, call for purpose-built detectors that target constraint violations under explicit assumptions and produce actionable evidence. This work makes three contributions: (1) We introduce a taxonomy of eight structural failure categories, each defined by graph-based consistency invariants, distinguishing failures characteristic of LLM-generated patches from
Code generation from Large Language Models has achieved remarkable results on isolated programming tasks [1–4], driving rapid adoption, with millions of engineers using LLM-powered assistants daily. Yet a gap persists between benchmark performance and production utility [5]. Code that appears correct in isolation frequently fails when integrated into real software systems [6], and the dominant failure mode is structural rather than functional. A generated patch may compile, pass type checking, and satisfy local tests while violating invariants that span the repository. Consider a FastAPI endpoint referencing a Pydantic model with hallucinated field names, or a Django view assuming environment variables that are never declared in the project configuration. Such patches exhibit local correctness but global incoherence: they pass the checks developers rely on and fail only when exercised in the context of the full system. Current evaluation methodologies do not surface these failures systematically. Type checking and linting often miss semantic inconsistencies that cross file boundaries. Test suites cannot cover every integration point. SAST 1
issues merely amplified by them. (2) We present a multi-graph verification framework that integrates mature static analysis tools (mypy, tsc, pylint, ESLint) with purpose-built cross-graph detectors over eight repository graphs, producing localized evidence traces for each violation. (3) We provide an empirical study across 336 generations from two frontier models under four prompting conditions, along with external validation on 43 real-world AI-generated repositories.
2
Table 1: Summary of related work Focus
Gap
[6]
Taxonomy of hallucinations in repo-level generation
Descriptive; invariants or detection
[9]
Quantitative measurement of fabricated package references Cross-file code completion with retrieval subtasks
Dependency-only; cross-artifact verification Evaluates completion, structural coherence
[5]
GitHub issue resolution via test passage
Test-based; misses failures that evade tests
[10]
Repo-aligned generation with dependency annotations
Pass@k metric; no structural invariant checking
[11]
Backend generation with exploit-based security assessment
Exploit-focused; no config or schema verification
[12]
Security-focused evaluation
Vulnerability labels, structural constraints
[13]
Multi-file vulnerability scenarios Autonomous multi-step repo editing Unified code property graph for vulnerability discovery
Security-scoped; no general structural coverage Evaluates resolution rate, not edit coherence Single-file; no cross-file or config constraints
[15]
Multi-view pre-training
contrastive
Graphs as training signal, not verification layer
[16]
Graph-aligned fine-tuning for structural semantics Constrained decoding for secure generation
Alignment target, not constraint checking Decoding-time; no post-hoc structural verification
Dual-judge detection
Vulnerability-scoped; structural coherence
[7]
Related Work
Table 1 situates our contribution relative to prior work across four research threads. Hallucination characterization work [6, 9] establishes the empirical prevalence of structural defects in generated code but characterizes them descriptively rather than as verifiable constraint violations. Repository-level benchmarks, including RepoBench [7], SWE-bench [5], EvoCodeBench [10], BaxBench [11], SecRepoBench [12], SecureVibeBench [13], and SWE-agent [14] demonstrate that snippet-level performance does not transfer reliably to repository-scale tasks, yet their evaluation criteria remain outcome-oriented (test passage, exploit success) rather than diagnosing which structural invariants are violated. Graph-based representations such as Code Property Graphs [8], CODE-MVP [15], and GALLa [16] leverage graphs as representational substrates but do not operationalize them as a constraint verification layer. Secure generation approaches, including CodeGuard+ [17] and SafeGenBench [18], target vulnerability prevention without formalizing structural coherence. Our work addresses this gap by defining structural incoherence as violated consistency constraints across graph representations and producing localized evidence traces that attribute failures to specific constraint violations.
3
Work
[14] [8]
[17]
[18]
patch
vulnerability
no formal automated
no not
not
no
requires the import graph and symbol table, optionally augmented with type information for generic resolution. Evidence traces record file, line, symbol, expected module, and resolution outcome. This failure class is amplified in LLM outputs because models operating with incomplete context frequently invent plausible but non-existent module names or reference deprecated APIs. Phantom Internal API (PIA): Phantom API failures occur when generated code invokes internal functions with incorrect signatures or semantics inconsistent with declared interfaces. The invariant requires signature compatibility such that for call site 𝑐 invoking symbol 𝑠 with signature registry Σ, compatible(sig(𝑐), Σ(𝑠)) = true. Detection leverages the call graph and signature registry extracted from type annotations and protocol declarations. This category is strongly amplified as LLMs hallucinate method signatures based on naming conventions rather than actual declarations, particularly for internal APIs underrepresented in training data. Dependency Hallucination (DHI): Algorithm 2 validates that all external imports reference packages declared in the project’s dependency manifests. External imports not found in the dependency graph trigger registry queries to PyPI or npm, distinguishing fully hallucinated packages (nonexistent in registries) from undeclared-but-existing packages. For npm packages, import names match registry names directly. For PyPI, the detector assumes direct correspondence between import names and package names, which holds for the majority of packages but not for cases where import and distribution names diverge (e.g., yaml vs. PyYAML, cv2 vs. opencv-python). The resulting phantom module set
Structural Failure Taxonomy
The patchwork problem manifests through structural failures, defined as violations of consistency invariants at the repository scale, verifiable via static graph analysis without execution. These failures differ from functional bugs in that individual patches appear correct yet collectively violate project contracts. Our taxonomy comprises eight categories, each defined by formal invariants, required graph artifacts, and failure characteristics specific to LLM outputs. Table 2 summarizes the classification. Symbol Resolution Failures (SRF): A symbol resolution failure occurs when a referenced name cannot be resolved within the repository’s module graph. Formally, for symbol reference 𝑟 in file 𝑓 with module graph M, the invariant requires ∀𝑟 ∈ refs( 𝑓 ) : ∃𝑑 s.t. resolve(𝑟, 𝑓 , M) → 𝑑. Detection 2
feeds downstream into SRF and PIA detection. schema(𝑃.output) ⊇ schema(𝐶.input) with type Build/Configuration Incoherence (BCI): Build consistency. Detection requires the call graph with configuration failures arise when generated code assumes edges spanning files and the schema graph extracted configurations inconsistent with the repository’s declared from OpenAPI specs, Pydantic models, or Zod schemas. state. The invariant requires that for each configuration LLMs amplify this failure by hallucinating response assumption 𝑎 implied by generated code, a satisfying fields based on naming conventions. declaration exists in the configuration space C. Detection Security Structural Regressions (SSR): Security operates over the build graph and configuration graph structural regressions occur when application wiring encompassing entrypoints, environment variables, and violates security contracts absent classic taint flow framework settings. Four invariant classes apply across vulnerabilities. The invariant requires guard coverage languages, namely entrypoint existence, environment such that ∀𝑟 ∈ 𝑅 : guarded by(𝑟, 𝑀) for security-critical variable declaration, module system consistency, and routes 𝑅 and required guards 𝑀. Detection requires framework configuration alignment. This category is routing and middleware attachment graphs specific amplified under weak retrieval context where models to each framework. We scope detection to FastAPI default to standard configurations divergent from project (dependency injection guards), Django (permission settings. decorators), Express (middleware chains), and Next.js Resource Coherence Failures (RCF): Resource (middleware matchers). LLMs amplify this failure by coherence failures occur when code fails to provide wiring middleware incorrectly, even when producing declared resources, encompassing both filesystem syntactically correct code. resources and computational contracts. Filesystem resource failures arise when code references files, assets, 4 Verification Framework templates, or migrations that do not exist, with the existence invariant requiring exists(resolve path(𝑟, root)) = Architecture Overview: The verification framework true for each resource reference 𝑟 and additional ordering employs a hybrid architecture guided by a precision-first constraints for sequential resources such as database design philosophy in which each taxonomy category is migrations. Return contract failures arise when functions matched to the verification strategy that maximizes with declared return types contain execution paths that detection precision for its invariant class. Categories do not produce a value matching the declared type, where mature static analysis tools already handle the violating the contract that the function’s signature relevant language semantics (symbol resolution, signature promises to callers. Schema completeness failures arise compatibility) delegate to those tools, inheriting their when model definitions omit fields required by consuming years of edge-case handling. Control flow coherence code. Detection operates over the resource graph employs a hybrid approach combining custom graph mapping code references to filesystem paths, the CFG reachability and pattern matching with SAST tool for return-path reachability analysis, and the schema delegation, reflecting the observation that no single graph for field completeness validation. LLMs exhibit layer achieves adequate coverage alone. Categories amplified failure rates across all three sub-categories requiring cross-graph reasoning absent from existing by generating plausible paths based on conventions toolchains (configuration incoherence, dependency rather than actual repository structure, omitting return hallucination, security regressions, resource coherence, statements on error-handling branches, and producing cross-file contracts) employ purpose-built detectors incomplete schema definitions. that target provable constraint violations rather than Control Flow Coherence (CFC): Control heuristic pattern matching. This division reflects the flow failures manifest as CFG anomalies including empirical observation that reimplementing established unreachable blocks, contradictory conditions, exception analyses from scratch produces low-precision detectors flow misuse, and dead error handling. Invariants due to the long tail of language-specific edge cases require full reachability from entry (∀𝑣 : reachable(𝑣 0 , 𝑣)) (exception flows, generators, context managers, async and exception handler type consistency. Detection patterns), while the novel cross-cutting invariants central operates on intraprocedural CFGs with exception edge to the patchwork problem have no existing tool coverage. annotations. While unreachable code is a general bug The framework operates on each repository state after class, LLMs exhibit amplified characteristic patterns generation, constructing graph representations from the such as overbroad exception handling, redundant null combined original and generated code and routing each checks, and copy-paste control flow inconsistencies. to the appropriate verification backend. The framework’s Cross-File Contract Violations (CCV): Contract output for each finding is a localized evidence trace violations occur when producer and consumer modules recording the violated invariant, the implicated files and exhibit interface mismatches across file boundaries, line numbers, and the constraint that would need to hold including wrong field names, incompatible serialization, for the code to be structurally sound. All source code, incorrect error codes, and misaligned assumptions. graph construction scripts, detection pipelines, evaluation The invariant requires schema compatibility whereby
3
LLM-Generated Patch
task only blind generation
partial view
P2: Local Files Only
P3: Retrieved Files fragmented view Oracle Files
Schema Graph (type analysis)
Resource and Call Graph (pycg / ts-morph)
CCV Cross File Contract Violations
schema mismatches
RCF Resource Coherence Failures
BCI Build/Configuration Incoherence
invalid settings
undefined calls signature mismatch
Config Graph (config parsing)
Control Flow Graph (branch analysis)
unreachable/dead code dead branches CFC Control Flow Coherence Detection Output Errors & Warnings
SSR Security Structural Regressions
missing auth guards
DHI Dependency Hallucination
PIA Phantom Internal API
SRF Symbol Resolution Failures
Routing Graph (route + middleware)
Generated Code
P1: Minimal
P4: Full Context
Dependency Graph (manifest parsing) unresolved imports phantom dependencies
undeclared packages Hallucinated packages
Import Graph (AST parsing) stale references
Figure 1: Overview of the Proposed Framework Table 2: Structural failure taxonomy ID
Category
Primary Graph(s)
LLM Profile
SRF PIA DHI BCI RCF CFC CCV SSR
Symbol Resolution Phantom Internal API Dependency Hallucination Build/Config Incoherence Resource Coherence Control Flow Coherence Cross-File Contracts Security Structural
Import + Symbol Table Call + Signature Dependency + Registry Build + Config Resource + CFG + Schema CFG Call + Schema Routing + Middleware
Amplified Strongly Amplified Specific to LLMs Amplified Amplified General/Amplified Amplified Amplified
configurations, and external validation datasets required to reproduce our results are publicly available [19]. Figure 1 illustrates the end-to-end verification pipeline, showing how graph construction connects prompting conditions to failure detection. Graph Construction: For each repository state after generation, the framework constructs eight graph representations spanning structural, behavioral, and configurational dimensions. Table 3 summarizes the construction method for each graph type. Figure 2 visualizes the many-to-many mapping between graph representations and failure categories, illustrating why the framework requires multiple coordinated analyses rather than a single monolithic pass. Chords connect each graph representation (left) to the failure categories it enables detecting (right). Some categories require multiple graphs, and some graphs serve
Type check evasion
Test Evasion
Partial Yes Yes Yes Yes Partial Yes Yes
Partial Yes Yes Yes Yes Partial Yes Yes
multiple categories, motivating the hybrid architecture. Detection Algorithms: The following paragraphs formalize detection for each taxonomy category. Purpose-built detectors (Algorithms 1–3, 4, 5, and 7) target provable constraint violations, while control flow coherence (Algorithm 6) combines custom analysis with SAST tool delegation. Detection order reflects data dependencies, with DHI running first to produce the phantom module set consumed by SRF and PIA. Configuration Incoherence Detection (BCI): Algorithm 1 detects provable runtime failures from unguarded environment variable accesses. It extracts strict access patterns that throw on missing values (os.environ["KEY"] in Python, unguarded process.env.KEY in TypeScript), eliminates accesses protected by guards (try/except, membership tests, fallback operators), and validates remaining accesses against the repository’s 4
Algorithm 1 Configuration Incoherence Detection (BCI)
Table 3: Graph construction methods by language Graph
Python
TypeScript
Import
ast module; relative import resolution pycg flow-insensitive points-to analysis pyproject.toml, requirements.txt, poetry.lock + PyPI validation Pydantic BaseModel, SQLAlchemy Column .env, .env.example, Docker Compose, framework settings open(), pathlib.Path, template loaders, migration deps ast branch analysis; pylint delegation FastAPI route decorators, Django URL conf
ts.createProgram with tsconfig.json resolution Compiler API type-directed resolution package.json, package-lock.json, yarn.lock + npm validation Zod z.object(), Prisma schema.prisma Same sources
Call Dependency
Schema Config
Resource
CFG Routing
Config
CFG
Require: Repository 𝑅 with generated code 𝐺, config files 𝐶 Ensure: Set of validated configuration incoherence findings 𝐹 1: Extract unsafe accesses 2: 𝐴 ← ∅ 3: for each file 𝑓 in 𝐺 do 4: if 𝑓 is Python then 5: 𝐴 ← 𝐴 ∪ {os.environ["K"] patterns in 𝑓 } 6: else if 𝑓 is TypeScript then 7: 𝐴 ← 𝐴 ∪ {unguarded process.env.K in 𝑓 } 8: end if 9: end for 10: Guard elimination 11: for each access 𝑎 ∈ 𝐴 do 12: if 𝑎 inside try/except KeyError or preceded by membership test or has ||/?? fallback then 13: 𝐴 ← 𝐴 \ {𝑎} 14: end if 15: end for 16: Config-space validation keys from .env, .env.example, 17: C ← docker-compose.yml, settings 18: 𝐹 ← ∅ 19: for each access 𝑎 ∈ 𝐴 with key 𝑘 do 20: if 𝑘 ∉ C then 21: 𝐹 ← 𝐹 ∪ {(𝑎, 𝑘, file, line)} 22: end if 23: end for 24: return 𝐹
Equivalent TS patterns
ts-morph branch analysis; ESLint delegation Express router chains, Next.js middleware matchers
Schema Dependency
Resource
Call
Routing
SRF
Import
PIA
SSR
DHI
CCV BCI
RCF
CFC
Figure 2: Graph-to-category mapping guarded by exception handlers that re-raise or call sys.exit. Filesystem resource violations check that configuration space. Arithmetic expressions and safe referenced paths (templates, migrations, assets) resolve access patterns with explicit defaults are excluded at to existing files. Schema completeness violations flag extraction time. Every reported finding represents a consuming code that accesses fields absent from the configuration access that will produce a runtime crash if producing model’s definition. Every finding represents a the variable is absent. provable violation: a reachable path missing a declared Dependency Hallucination Detection (DHI): return, a path literal that does not resolve, or a field Algorithm 2 validates that all external imports reference access targeting an undefined name. packages declared in the project’s dependency manifests. Cross-File Contract Violation Detection External imports not found in the dependency graph (CCV): Algorithm 5 detects interface mismatches trigger registry queries to PyPI or npm, distinguishing across module boundaries via call graph and schema fully hallucinated packages (nonexistent in registries) graph analysis. It targets four patterns: disconnected from undeclared-but-existing packages. The resulting middleware (registered but never imported in route phantom module set feeds downstream into SRF and modules), unused decorators referencing nonexistent PIA detection. permission classes, duplicate middleware registrations Symbol Resolution and Phantom API causing double execution, and field naming mismatches Detection (SRF, PIA): Algorithm 3 leverages between producer response schemas and consumer access the phantom module set from Algorithm 2. A patterns (e.g., user name vs. username). Each finding LocalModulePattern filter excludes intra-generation identifies a statically verifiable disconnect between two cross-references and intentional placeholders to prevent code locations that must agree for correct execution. false positives from multi-file generation tasks. Control Flow Coherence Detection (CFC): Resource Coherence Detection (RCF): Algorithm 4 Algorithm 6 employs a hybrid three-layer approach with targets three sub-categories. Return contract violations findings deduplicated by line number. Layer 1 performs construct intraprocedural CFGs for functions with BFS reachability from function entry nodes, flagging declared return types and identify execution paths that only entirely dead functions. Layer 2 applies pattern terminate without producing a value, excluding branches
5
Algorithm 2 Dependency Hallucination Detection (DHI)
Algorithm 4 Resource Coherence Detection (RCF) Require: CFGs F , resource graph R, schema graph S, generated code 𝐺 Ensure: Set of resource coherence findings 𝐹 1: 𝐹 ← ∅ 2: for each function 𝑓 in 𝐺 with declared return type 𝑇 do 3: Build intraprocedural CFG; for each path 𝜋 to exit, if 𝜋 has no return of type 𝑇 and is not exception-terminated, add ( 𝑓 , 𝜋, 𝑇) to 𝐹 4: end for 5: for each resource ref 𝑟 with path 𝑝 in 𝐺 do 6: if ¬exists(resolve( 𝑝, root)) then 7: 𝐹 ← 𝐹 ∪ {(𝑟, 𝑝)} 8: end if 9: end for 10: for each consumer access 𝑐. 𝑓 𝑖𝑒𝑙𝑑 produced by model 𝑀 do 11: if 𝑓 𝑖𝑒𝑙𝑑 ∉ fields(𝑀) then 12: 𝐹 ← 𝐹 ∪ {(𝑐, 𝑀, 𝑓 𝑖𝑒𝑙𝑑)} 13: end if 14: end for 15: return 𝐹
Require: Repository 𝑅 with manifests 𝑀, generated code 𝐺 Ensure: Set of dependency hallucination findings 𝐹 1: 𝐷 ← parse packages from 𝑀 (pyproject.toml, requirements.txt, package.json) 2: Extract and validate external imports 3: 𝐹 ← ∅ 4: for each import 𝑖 in 𝐺 referencing package 𝑝 do 5: if 𝑝 ∉ stdlib and 𝑝 ∉ local modules and 𝑝 ∉ 𝐷 then 6: Query registry R (PyPI/npm) for 𝑝 7: if 𝑝 ∉ R then 8: 𝐹 ← 𝐹 ∪ {(𝑖, 𝑝, “hallucinated”)} 9: else 10: 𝐹 ← 𝐹 ∪ {(𝑖, 𝑝, “undeclared”)} 11: end if 12: end if 13: end for 14: return 𝐹
Algorithm 3 Symbol Resolution (SRF) and Phantom API (PIA) Detection Require: Import graph I, call graph K, phantom set 𝑃 from Algorithm 2 Ensure: Sets of SRF findings 𝐹𝑆 and PIA findings 𝐹𝑃 1: 𝐹𝑆 ← ∅, 𝐹𝑃 ← ∅ 2: for each import edge ( 𝑓 , 𝑚, 𝑠) in I do 3: if (𝑚 ∈ 𝑃 or ¬resolve(𝑠, 𝑚)) and 𝑚 ∉ LocalModulePattern then 4: 𝐹𝑆 ← 𝐹𝑆 ∪ {( 𝑓 , 𝑚, 𝑠)} 5: end if 6: end for 7: for each call edge ( 𝑓 , 𝑚.𝑠, args) in K do 8: if 𝑚 ∈ 𝑃 and 𝑚 ∉ LocalModulePattern then 9: 𝐹𝑃 ← 𝐹𝑃 ∪ {( 𝑓 , 𝑚, 𝑠, args)} 10: end if 11: end for 12: return 𝐹𝑆 , 𝐹𝑃
Illustrative Example: hypertropher-app [20] A Next.js/React web application built with AI coding tools and published on GitHub. 72 files analyzed
tsc-strict ✓
SAST ✓
11 structural failures ×
BCI — Configuration Incoherence 7 findings Four environment variables (NEXT PUBLIC SUPABASE URL, NEXT PUBLIC SUPABASE ANON KEY, SUPABASE SECRET API KEY, NEXT PUBLIC GOOGLE MAPS API KEY) accessed without defaults across three Supabase client files and the application layout. None declared in any .env or config file. Each resolves to undefined at runtime. Invisible to tsc because process.env access is structurally valid regardless of key existence.
DHI — Dependency Hallucination 1 finding Import references @vercel/analytics/next, a package absent from package.json. The path alias filter correctly excludes 98 @/components/ui/* local aliases, isolating the single genuinely unresolvable external dependency.
matching for dead code after terminators, tautological conditions, duplicate handlers, and infinite loops. Layer 3 delegates to pylint/ESLint with post-processing filters suppressing known false positives from context managers, generators, and heavy exception scaffolding. Security Structural Regression Detection (SSR): Algorithm 7 identifies endpoints lacking authentication guards present on sibling routes. Routes are clustered by resource segment, public endpoints are filtered, and majority-rule analysis flags routes where a dominant guard (≥90% coverage) is absent on destructive HTTP methods (POST, PUT, DELETE, PATCH).
RCF — Resource Coherence
2 findings
Functions loadingCities and previewUrl declare return types but contain conditional branches that never return a value. CFG reachability analysis identifies the gaps. The type checker misses these because exception flow masks the incomplete returns.
CFC — Control Flow Coherence
1 finding
Dead code after a return statement at line 461. This category was absent from all 336 controlled generations yet appears in real-world AI-generated code, consistent with the hypothesis that less supervised generation surfaces failure modes that controlled experiments do not elicit.
Illustrative Example: We trace a real-world AI-generated repository through the verification pipeline to illustrate how structural failures manifest and evade standard toolchains.
A companion repository, VoiceTradeWithSchwab [21] (voice-controlled stock trading, 100% AI-generated Python), exhibits 92 findings across five categories including phantom imports, an infinite loop, and unguarded trading configuration variables.
6
Algorithm 5 Cross-File Contract Violation Detection (CCV)
Algorithm 6 Control Flow Coherence Detection (CFC) Require: Generated code files 𝐺, constructed CFGs F Ensure: Set of control flow findings 𝐹 1: 𝐹 ← ∅ 2: for each file 𝑓 in 𝐺 do 3: Graph-based reachability (Layer 1) 4: for each function CFG 𝑔 ∈ F ( 𝑓 ) do 5: 𝑅 ← BFS from entry node of 𝑔 6: if all body nodes ∉ 𝑅 then 7: 𝐹 ← 𝐹 ∪ {( 𝑓 , 𝑔, “dead function”)} 8: end if 9: end for 10: Pattern-based detection (Layer 2) 11: 𝐹 ← 𝐹∪ detect dead-code-after-terminator in 𝑓 12: 𝐹 ← 𝐹∪ detect tautological conditions in 𝑓 13: 𝐹 ← 𝐹∪ detect duplicate except/switch-case in 𝑓 14: 𝐹 ← 𝐹∪ detect infinite loops without exit in 𝑓 15: SAST delegation (Layer 3) 16: 𝑟 ← invoke pylint/ESLint on 𝑓 (skip if syntax error) 17: Remove findings in context manager, generator, or heavy try/finally contexts 18: 𝐹 ← 𝐹 ∪𝑟 19: Deduplicate 𝐹 by line number (priority: graph > pattern > SAST) 20: end for 21: return 𝐹
Require: Call graph K, schema graph S, middleware config M, generated code 𝐺 Ensure: Set of contract violation findings 𝐹 1: 𝐹 ← ∅ 2: for each middleware 𝑚 registered in M do 3: if 𝑚 ∉ import edges of any route module then 4: 𝐹 ← 𝐹 ∪ {(𝑚, disconnected)} 5: end if 6: if 𝑚 appears >1 time in registration then 7: 𝐹 ← 𝐹 ∪ {(𝑚, duplicate)} 8: end if 9: end for 10: for each cross-file call edge ( 𝑝𝑟𝑜𝑑𝑢𝑐𝑒𝑟, 𝑐𝑜𝑛𝑠𝑢𝑚𝑒𝑟) in K do 11: 𝑆 𝑃 ← output fields of 𝑝𝑟𝑜𝑑𝑢𝑐𝑒𝑟; 𝑆𝐶 ← accessed fields in 𝑐𝑜𝑛𝑠𝑢𝑚𝑒𝑟 12: if ∃ 𝑓 ∈ 𝑆𝐶 : 𝑓 ∉ 𝑆 𝑃 then 13: 𝐹 ← 𝐹 ∪ {( 𝑝𝑟𝑜𝑑𝑢𝑐𝑒𝑟, 𝑐𝑜𝑛𝑠𝑢𝑚𝑒𝑟, 𝑆𝐶 \ 𝑆 𝑃 )} 14: end if 15: end for 16: return 𝐹
5
Evaluation and Results
5.1
Experimental Setup
representing static CI checks: type checking (mypy [26] and tsc, test execution, SAST via bandit [27] and semgrep [28], and regex heuristics. Dependency installation is excluded as the evaluation operates on generated code before environment builds; it would catch at most the 3 DHI findings but none of the remaining 64. Detection metrics include per-category precision against ground truth and evasion rates quantifying findings that pass each baseline undetected. Ground truth labels were established through two approaches. For categories with small finding counts (BCI, DHI, PIA, SRF), every finding was manually reviewed and verified as a provable constraint violation. For categories with larger counts (RCF, CCV), precision was established through iterative pipeline refinement that systematically eliminated false positive patterns, with boundary cases resolved by consulting the formal invariants for each category.
We evaluate structural failure detection across 336 code generations from two frontier models, GPT-4o (2024-08-06, 128K context) and Claude 3.5 Sonnet (2024-10-22, 200K context), both at temperature zero. The evaluation corpus consists of 10 curated open-source production repositories spanning Python (Django [22], FastAPI [23]) and TypeScript (Express [24], Next.js [25]), selected for active maintenance, with a minimum 50 files and 10K LOC, type annotation coverage exceeding 50%, test coverage above 60%, and explicit schema definitions.1 From these repositories, we extract 60 tasks derived from merged pull requests and closed issues at three complexity levels, namely L1 single-file (30 tasks), L2 multi-file (20 tasks), and L3 cross-cutting (10 tasks). Four prompting strategies control context richness ranging from P1 (minimal, task description only) through P2 (local, 2–5 same-directory files) and P3 (retrieved, 10 similarity-ranked files) to P4 (oracle, 5–15 ground-truth files). The evaluation has a partially unbalanced design in which 24 early tasks were evaluated with GPT-4o under P1 and P2 only, while the remaining 36 tasks received both models across all four strategies, yielding 192 GPT-4o generations (60 each for P1/P2, 36 each for P3/P4) and 144 Claude generations (36 per strategy). P1 and P2 therefore contain 96 generations each and P3 and P4 contain 72 each; all analyses use appropriate denominators to account for this asymmetry. We compare our framework against four baselines,
5.2
Results
Detection Performance: Table 4 reports detection results across 336 generations (192 GPT-4o, 144 Claude 3.5 Sonnet) under four prompting conditions. Our framework identifies 67 structural failures across eight active categories, with 65 (97.0%) invisible to all baseline methods. Compilation and type checking detect only 2 findings independently (both RCF return-type violations), while test execution, SAST, and regex heuristics detect none. By category, RCF accounts for the most findings (29), followed by CCV (18), BCI (12),
1 Please refer to [19] for the complete list of curated repositories, external validation datasets, and all evaluation metadata.
7
Algorithm 7 Security Structural Regression Detection (SSR)
across all evaluations. This is consistent with two properties of the evaluation corpus. The detector requires route clusters with at least 4 endpoints exhibiting a dominant per-route guard pattern, and most vibe-coded projects apply authentication at the framework level (e.g., global middleware, app-level decorators) rather than per-route, leaving no inconsistency to detect. The two highest-finding repositories, hypertropher-app and VoiceTradeWithSchwab (detailed in the Illustrative Example above), exemplify how structural failures cluster and compound in real-world AI-generated code.
Require: Routing graph G with routes {( 𝑝𝑎𝑡ℎ, 𝑚𝑒𝑡ℎ𝑜𝑑, 𝑔𝑢𝑎𝑟𝑑𝑠)} Ensure: Set of security regression findings 𝐹 1: Filter and cluster 2: for each route 𝑟 in G do 3: Assign 𝑟 to cluster 𝐶resource by path segment 4: end for 5: Remove clusters matching public whitelist (health, auth, docs, webhook, metrics) 6: Majority-rule guard analysis 7: 𝐹 ← ∅ 8: for each remaining cluster 𝐶 with |𝐶 | ≥ 4 do 9: 𝑔 ∗ ← most common guard in 𝐶 10: ratio ← |{𝑟 ∈ 𝐶 : 𝑔 ∗ ∈ guards(𝑟)}|/|𝐶 | 11: if ratio ≥ 0.9 then 12: for each route 𝑟 ∈ 𝐶 where 𝑔 ∗ ∉ guards(𝑟) do 13: if method(𝑟) ∈ {POST, PUT, DELETE, PATCH} then 14: 𝐹 ← 𝐹 ∪ {(𝑟, 𝑔 ∗ , 𝐶)} 15: end if 16: end for 17: end if 18: end for 19: return 𝐹
Table 4: Detection results across 336 generations Method
Findings
TP
Precision
Unique
Type Check/Lint Test Execution SAST Regex Heuristics
2 0 0 0
2 0 0 0
100% N/A N/A N/A
0 0 0 0
Patch Work Framework (Ours)
67
67
see text
65
Table 5: Per-category detection results
DHI (3), PIA (3), and SRF (2). Manual validation confirms 100% precision for BCI, DHI, PIA, and SRF (20 of 20 verified as provable constraint violations), while RCF and CCV precision was established through iterative pipeline refinement. No baseline method detects any CCV, BCI, DHI, PIA, or SRF finding; type checkers catch only 2 of 29 RCF findings. Evasion rates reinforce this gap: 97.0% of findings evade compilation (mypy --strict/tsc --strict), and 100% evade test suites and SAST tools. Two categories, CFC and SSR, produced zero findings in the controlled evaluation despite having active detectors. To determine whether these detectors function correctly or whether controlled generation simply does not elicit these failure modes, we applied the full pipeline to 43 real-world AI-generated repositories spanning vibe-coded projects (Cursor AI, Google Gemini, GitHub Copilot), GPT-Engineer/Lovable applications, and self-declared fully AI-generated projects.2 Across 1,581 analyzed files the pipeline detected 1,152 findings in 35 of 43 repositories (81.4% repo-level incidence), with 474 DHI findings, 270 RCF findings, 177 PIA findings, 148 BCI findings, 62 SRF findings, 16 CFC findings (6 duplicate switch cases, 2 dead-code-after-return, 2 infinite loops), and 5 CCV findings. The CFC findings confirm that the hybrid three-layer detector functions correctly; controlled frontier generation with explicit task specifications simply does not produce the unstructured code patterns that trigger control flow failures. SSR remained at zero
Category
N
Precision
Baseline
RCF CCV BCI DHI PIA SRF CFC SSR
29 18 12 3 3 2 0 0
Refined Refined 100% 100% 100% 100% N/A N/A
2/29 0/18 0/12 0/3 0/3 0/2 N/A N/A
Detection Strategy CFG return-path + schema Cross-graph disconnect Unsafe-access + config-space Registry validation Cross-graph phantom check Import + symbol resolution Hybrid 3-layer Resource-clustered auth
Model Comparison: Failure distributions diverge qualitatively between GPT-4o and Claude 3.5 Sonnet as visualized in Figure 3. Overall failure rates are comparable (GPT-4o: 39 findings in 25/192 generations, 13.0%; Claude: 28 findings in 17/144 generations, 11.8%), but the models exhibit distinct failure profiles rather than simply differing in magnitude. GPT-4o produces all 18 CCV findings and all import-related failures (DHI, PIA, SRF) exclusively, while Claude generates 22 of 29 RCF findings. Both contribute equally to BCI (6 each). Chi-squared testing confirms distributional independence (𝜒2 = 25.1, 𝑝 = 2.73 × 10−7 ). With 67 total findings, these patterns are descriptive observations warranting replication rather than definitive model characterizations. Prompt Sensitivity: Table 6 reports failure counts by prompting strategy. P3 (retrieved context) exhibits the highest count (24) and P1 (minimal) the lowest (8), indicating that richer context reshapes rather than uniformly reduces failure distributions. Notably, BCI appears in P1 through P3 but not P4 (oracle), consistent with ground-truth files helping models identify correct configuration variables. L3 (cross-cutting) tasks exhibit 44.6% finding incidence compared to 16.1% for L1 and 13.4% for L2, confirming that tasks spanning configuration, middleware, and schema boundaries are substantially more failure-prone. Figure 4 shows how failure category composition shifts across complexity levels, with BCI and CCV concentrated in L3 tasks that require cross-layer reasoning.
2 Please refer to [19] for the complete list of all repositories with per-repo metadata and finding counts.
8
SRF
2
PIA
3
DHI
RCF
CCV 8
L2 Multi-file (N=112)
3
BCI
RCF 21
L1 Single-file (N=168)
RCF 7
L3 Cross-cutting (N=56)
6
6
BCI 4
0
CCV 10 20
40
BCI 8 60
80
100
Failure composition (%)
22
RCF
7
CCV
BCI
DHI
PIA
SRF
CFC CCV SSR
Figure 4: Failure category composition by task complexity level.
18
GPT-4o Claude
−20
−15
−10
−5
0
5
10
15
20
Finding Count
CI integration and developer review workflows, providing actionable diagnostics rather than opaque pass/fail signals.
Figure 3: Model divergence in structural failure categories. Bars extend left for GPT-4o and right for Claude 3.5 Sonnet.
6
Table 6: Findings by prompt strategy and category Category
P1
P2
P3
P4
RCF CCV BCI DHI PIA SRF
0 4 4 0 0 0
8 5 2 1 1 0
13 5 6 0 0 0
8 4 0 2 2 2
Total
8
17
24
18
This work formalized the patchwork problem in LLM-generated code through a graph-based failure taxonomy and a hybrid verification framework combining mature static analysis tools with purpose-built detectors. Across controlled generations and real-world repositories, the overwhelming majority of detected structural failures evade type checking, testing, and SAST entirely, and failure patterns diverge qualitatively between models. Our future work will focus on extending the framework to additional models and programming languages and on subjecting the categories currently validated through iterative refinement to independent precision audits. Further directions include repair mechanisms that leverage detected constraint violations to prompt for missing declarations, and integration into agentic coding workflows and continuous integration pipelines where incremental patch review replaces complete-file generation.
Runtime: The pipeline analyzes each file in a median of 47 ms end-to-end (graph construction through all seven detectors), with a 120-second timeout per external analysis script. Registry validation queries (PyPI/npm) are cached across runs. The largest repository in Track D (435 files, 70K LOC) completes in 233 seconds. Runtime is dominated by graph construction (99.8% of per-file time); all detectors combined complete in under 1ms per file. These times indicate the framework is practical as a CI integration for repositories of moderate size.
5.3
Conclusion
Practical Implications
References
These findings have direct consequences for teams adopting LLM-generated code. First, standard CI pipelines (type checking, testing, SAST) are insufficient as quality gates for generated code; 97% of detected failures pass all four baselines, meaning structurally broken code can merge undetected. Teams relying solely on existing toolchains face a growing blind spot as LLM-generated code volume increases. Second, the qualitative divergence between models suggests that switching or combining models does not uniformly reduce risk; different models produce different failure profiles, and mitigation strategies should be model-aware. Third, the concentration of failures in L3 cross-cutting tasks (44.6% incidence versus 13–16% for simpler tasks) indicates that structural verification is most critical for tasks spanning configuration, routing, and schema boundaries, precisely the tasks where LLMs are increasingly deployed. The framework’s localized evidence traces are designed to support both automated
[1] J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” ACM Transactions on Software Engineering and Methodology, vol. 35, p. 1–72, Jan. 2026. [2] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and 9
[12] C. Shen, C. Dilgren, P. Chiniya, L. Griffith, Y. Ding, and Y. Chen, “Secrepobench: Benchmarking code agents for secure code completion in real-world repositories,” 2026.
W. Zaremba, “Evaluating large language models trained on code,” 2021. [3] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. Sutherland Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals, “Competition-level code generation with alphacode,” Science, vol. 378, p. 1092–1097, Dec. 2022.
[13] J. Chen, H. Huang, Y. Lyu, J. An, J. Shi, C. Yang, T. Zhang, H. Tian, Y. Li, Z. Li, X. Zhou, X. Hu, and D. Lo, “SecureVibeBench: Benchmarking secure vibe coding of AI agents via reconstructing vulnerability-introducing scenarios,” in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens, eds.), (San Diego, California, United States), pp. 24144–24168, Association for Computational Linguistics, July 2026.
[4] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021.
[14] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” 2024.
[5] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?,” in The Twelfth International Conference on Learning Representations, 2024.
[15] X. Wang, Y. Wang, Y. Wan, J. Wang, P. Zhou, L. Li, H. Wu, and J. Liu, “Code-mvp: Learning to represent source code from multiple views with contrastive pre-training,” in Findings of the Association for Computational Linguistics: NAACL 2022, 2022.
[6] Z. Zhang, C. Wang, Y. Wang, E. Shi, Y. Ma, W. Zhong, J. Chen, M. Mao, and Z. Zheng, “Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation,” Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. 481–503, 2025.
[16] Z. Zhang, H. Yu, S. Li, P. Di, J. Li, and R. Wang, “Galla: Graph aligned large language models for improved source code understanding,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), 2025.
[7] T. Liu, C. Xu, and J. McAuley, “Repobench: Benchmarking repository-level code auto-completion systems,” in International Conference on Learning Representations, vol. 2024, pp. 47832–47850, 2024.
[17] Y. Fu, E. Baker, Y. Ding, and Y. Chen, “Constrained decoding for secure code generation,” 2024.
[8] F. Yamaguchi, N. Golde, D. Arp, and K. Rieck, “Modeling and discovering vulnerabilities with code property graphs,” in 2014 IEEE symposium on security and privacy, pp. 590–604, IEEE, 2014.
[18] X. Li, J. Ding, C. Peng, B. Zhao, X. Gao, H. Gao, and X. Gu, “Safegenbench: A benchmark framework for security vulnerability detection in llm-generated code,” arXiv preprint arXiv:2506.05692, 2025.
[9] J. Spracklen, R. Wijewickrama, A. N. Sakib, A. Maiti, and B. Viswanath, “We have a package for you! a comprehensive analysis of package hallucinations by code generating {LLMs},” in 34th USENIX Security Symposium (USENIX Security 25), pp. 3687–3706, 2025.
[19] V. Mothukuri, “Source code of the paper: The patchwork problem in llm-generated code.” https: //github.com/decentralizedsciencelab/Patch Work.git, 2026. Source code, evaluation pipelines, detection configurations, and datasets.
[10] J. Li, G. Li, X. Zhang, Y. Dong, and Z. Jin, “Evocodebench: An evolving code generation benchmark aligned with real-world code repositories,” 2024.
[20] wowashuwow, “hypertropher-app.” https://gith ub.com/wowashuwow/hypertropher-app, 2025. [21] BSalita, “VoiceTradeWithSchwab.” https://gith ub.com/BSalita/VoiceTradeWithSchwab, 2025.
[11] M. Vero, N. Mündler, V. Chibotaru, V. Raychev, M. Baader, N. Jovanović, J. He, and M. Vechev, “Baxbench: Can LLMs generate correct and secure backends?,” in Forty-second International Conference on Machine Learning, 2025.
[22] Django Software Foundation, “Django: The web framework for perfectionists with deadlines.” https: //www.djangoproject.com, 2024. Version 5.0. [23] S. Ramı́rez, “FastAPI: Modern, fast (high-performance) web framework for building APIs 10
with Python.” https://fastapi.tiangolo.com, 2024. Version 0.110. [24] OpenJS Foundation, “Express: Fast, unopinionated, minimalist web framework for Node.js.” https:// expressjs.com, 2024. Version 4.x. [25] Vercel, “Next.js: The React framework for the web.” https://nextjs.org, 2024. Version 14. [26] mypy, “mypy: Optional static typing for Python.” https://mypy-lang.org, 2024. Version 1.8. [27] PyCQA, “Bandit: A tool designed to find common security issues in Python code.” https://bandit.r eadthedocs.io, 2024. Version 1.7. [28] Semgrep, Inc., “Semgrep: Lightweight static analysis for many languages.” https://semgrep.dev, 2024. Open-source edition.
11