Natural-Language Workflows Are Not Software Yet: Artifact-Driven Compilation for Reliable Agent Execution

Natural-Language Workflows Are Not Software Yet: Artifact-Driven Compilation for Reliable Agent Execution Xiangzhe Xu∗ , Hanxi Guo∗ , Guangyu Shen, Siyuan Cheng, Xiangyu Zhang

arXiv:2608.21341v1 [cs.SE] 21 Aug 2026

Purdue University West Lafayette, USA {xzx, guo778, shen447, cheng535, xyzhang}@purdue.edu ∗ Equal contribution

Abstract—Natural-language workflows offer a software-like interface for agents: domain experts can write reusable procedures, and agents can execute them as instructions. This promise is not yet reliable. Workflow descriptions often leave data dependencies implicit, so the executor must infer which prior results a step should use; agents can also fail to follow long or branching instructions under context pressure. We propose A RTIC, an artifact-driven workflow compiler that transforms a naturallanguage workflow into an artifact-driven workflow in which each step declares the artifacts it reads and writes, constraints gate produced artifacts, and explicit control transfers route execution. This representation exposes the enforcement burden placed on agent execution, allowing the compiler to identify steps that depend on too much state or contain difficult control logic and refine them through constrained optimization. To validate the LLM-assisted transformation, A RTIC decomposes faithfulness checking into local obligations and uses scenario-based dry runs to test whether compiled workflow regions conform to the source workflow. We evaluate A RTIC on 488 problem instances from 11 real-world domain workflows; it improves task resolve rate by 28 percentage points over the original text workflow. We also show that workflows compiled by A RTIC are 32 and 56 percentage points more consistent in cross-model and repeated-execution setups, respectively. Index Terms—AI agents, software engineering, skill compilation

I. I NTRODUCTION Natural-language workflows are emerging as a software-like layer for agents. They encode intended behavior in natural language, while the agent runtime interprets and executes that behavior [1]–[4]. This software-like layer is attractive to domain experts. Clinicians, analysts, and operations specialists can specify how agents should perform domain work through skills, workflow prompts, and procedural instruction templates [5]–[15]. In this view, a workflow becomes a portable unit of domain expertise that can be shared, reused, and executed by agents. This software-like view carries a reliability expectation grounded in two properties of software. First, software can be shared and reused across similar environments because its behavior is predictable. Second, software execution is reliable because the machine is expected to follow the program instructions exactly.

However, these properties are not yet realized for naturallanguage workflows due to the variability of how agents execute them. Unlike the underlying environment that runs conventional software, an agent may not faithfully execute the natural-language workflow it receives. The gap has two sources. First, natural-language workflows often leave data dependencies implicit. Each step inherits all intermediate results in the past context, resembling a program in which intermediate results are stored as global variables: a later operation may read any of them even though the dependency is not visible at the operation boundary. This implicit data flow undermines executor reliability. At each step, an executor agent must distinguish which prior results are relevant, and it may make mistakes when the context contains noisy information. Second, the effort of faithfully following workflow instructions grows with the workflow’s control flow and context load. Different workflows impose different burdens on an agent. A short linear workflow may be easy to follow when it fits within the agent’s capability and context window; in contrast, a workflow with many branches or heavy intermediate results in context is more likely to cause skipped steps, wrong transfers, or lost information during context compaction [16]–[22]. The key challenge is to define an execution model for agentic workflows that exposes their enforcement burden, so difficult workflow regions can be identified and refined before runtime. Inspired by state-space views of problem solving [23], we formulate a natural-language workflow as an explicit artifact-driven workflow. Each step declares the artifacts it reads and writes, and explicit control transfer conditions determine which step executes next. The artifactdriven workflow is semi-structured: the local action of a step can still be a natural-language instruction executed by an agent, but data dependencies and control transfers are explicit and inspectable. This representation preserves agent flexibility inside each step while giving the workflow concrete state and execution gates. Since artifacts and control transfers are explicit, the artifactdriven workflow thus exposes enforcement burden as concrete, analyzable signals. For example, steps that depend on many

artifacts require the agent to select relevant state from a larger context and preserve it through context compaction. Likewise, nested or compound routes increase the burden of following the intended control flow. These signals identify which workflow regions are likely to be difficult for an agent to execute faithfully. We implement this idea as A RTIC1 , an artifact-driven workflow compiler, so domain experts do not need to manually rewrite their workflows in this domain-specific language. Unlike existing natural-language-to-workflow approaches that rely on unconstrained LLM decisions or repeated manual review [24]–[26], A RTIC compiles a natural-language workflow into an artifact-driven form through constrained optimization: an LLM proposes and refines the artifact-driven workflow, while programmatic analyses over its explicit artifacts and control transfers provide feedback on context pressure and decision complexity. The constraints used in the optimization above focus on reducing the execution burden of the compiled workflow; they do not check whether the compiler LLM has correctly translated the source workflow. In our ablation study, a compiler variant without this correctness validation performs 16 percentage points worse than the full system. It is therefore important to check whether the artifact-driven workflow is source-consistent. This check is challenging because, unlike a traditional compiler whose correctness can be stated against formal source and target semantics [27], A RTIC transforms natural-language instructions whose semantics depend on domain context and common-sense interpretation [28]–[30]. We address this challenge by exploiting an asymmetry in LLM systems: generating a source-consistent workflow is difficult, but checking a smaller, concrete source-consistency claim can be easier [31]–[33]. Concretely, the validator checks source consistency in two ways. First, checking a whole workflow at once is unreliable, but the artifact-driven workflow language lets the compiler decompose the validation goal into smaller obligations that follow the language structure. Each local obligation is simpler and easier to validate reliably. Second, following recent agenttesting work [34], [35], the compiler grounds validation in concrete scenarios: it enumerates diverse cases and asks the LLM to dry-run those cases to check whether the compiled workflow remains source-consistent with the source workflow. Contributions. This paper makes four contributions. • We formulate workflow enforcement as artifact-driven execution, exposing data dependencies, control transfers, and analyzable burden signals that indicate where faithful agent execution is at risk. • We implement A RTIC , a compiler prototype that transforms natural-language workflows into executable artifact-driven workflows through constrained optimization. • We propose a validation paradigm for natural-languageto-workflow transformation, decomposing faithfulness

checking into local obligations and scenario-based dry runs. • We evaluate A RTIC on 488 problem instances from 11 real-world domain workflows; it improves task resolve rate by 28 percentage points over the original text workflow. Workflows compiled by A RTIC are also 32 and 56 percentage points more consistent in cross-model and repeated-execution setups, respectively. II. M OTIVATING E XAMPLE We use a patient-referral workflow derived from χ-Bench [36], a realistic healthcare benchmark for testing whether LLM agents can route patients to the correct program. This example lets us examine why directly executing a textual workflow is unreliable, why a naive executable workflow is still insufficient, and how artifact-driven compilation motivates our design. Unstructured workflow text is not a reliable enforcement mechanism. Figure 1 shows the textual workflow setting. The agent receives a referral id and retrieves the necessary medical data to decide between complex care and chronic disease care. Error mode A illustrates how a dense naturallanguage context can lead the agent to retrieve and use the wrong information. The context contains the doctor’s referral request, medical-record evidence of chronic conditions, utilization notes about a recent heart-failure exacerbation, and an intermediate risk assessment. The agent is expected to use the utilization evidence to make the complex-care decision, but because this dependency is not explicit, the executor agent may not consistently identify the correct information. In the example, it instead attends to the doctor’s chronic-care request and chronic-condition evidence and makes the wrong decision. Error mode B illustrates unreliable control-flow enforcement: the complex-care route requires either a high risk score or recent acute utilization. The agent may misread the subtle control-flow requirement, mistakenly treating the condition as high risk and recent acute utilization, which leads to wrong instruction following. Naively converting textual workflows to executable form may not improve enforcement. It is natural to translate a long textual workflow into several subagent calls with separate contexts. The subagents then share information by sending messages to the next agent, while deterministic Python code handles tool calls, parsing, and routing. However, converting a natural-language workflow into this step-by-step subagentdriven workflow does not necessarily improve enforcement. Figure 2 contrasts two executable workflows for the same textual procedure. In the step-by-step workflow in Figure 2(a), Step 4 still combines referral information, utilization, medical records, and complex route conditions. The subagent at that step still needs to select the correct data from complex context, and the decision at that step still contains complex control-flow structure. We enable more reliable execution by compiling workflows to an artifact-driven language and adaptively decomposing challenging steps. To formulate the challenge introduced by

1 A RTIC : ARTIfact-driven workflow Compiler.

2

chronic-care eligibility.

Agent input: Text workflow about patient referral You are given a referral id RF-123, … (instructions on how to get referral details, recent utilization, and medical record). …(how to calculate the risk score for a patient.) High risk OR acute util. If the current risk score is high or the utilization shows recent acute utilization, …, enroll the member in complex care.

III. M ETHOD Figure 3 gives the overall pipeline. The compiler has two key designs. First, it formulates workflow compilation as constrained optimization: the generated workflow should be easy for agents to enforce while still preserving the source procedure. Second, it validates faithfulness before accepting the generated workflow, because the transformation itself relies on LLM reasoning and cannot be trusted from syntax alone. Given an input natural-language workflow (step 1), the compiler asks an LLM to compose a candidate artifact-driven workflow (step 2). The candidate isolates the context needed by each step and decomposes steps that place too much reasoning or context burden on one agent. This draft is guided by a constraint checker that runs lightweight program analysis over the structured workflow and reports suboptimal regions, such as excessive control-flow complexity or live artifacts that would be hard to preserve through context compaction. After a candidate workflow is drafted, a faithfulnessvalidation component checks whether it preserves the intended behavior of the source workflow. The validator has three passes. Static validation catches syntactic and well-formedness errors. Inductive validation decomposes the global faithfulness claim into smaller local validation goals over workflow regions. Dry-run simulation helps discharge these local goals by simulating the execution of the source and compiled regions on diverse hypothetical cases (step 3). If validation finds an inconsistency, the compiler feeds the diagnostic back to the constrained-optimization stage (step 4); otherwise, it accepts the workflow (step 5).

Otherwise, enroll the member in chronic disease care if the doctor referral requests it and the medical record has a chronic condition. Text color conventions: data entity, condition, output option Dense context w/ competing information Let me get the record first, get_record(“RF-123”) Agent

Content: chronic kidney disease, type 2 diabetes, heart failure… (long records w/ 2k words) Let’s see the utilization, get_util(“RF-123”) Content: … the patient has a recent heart failure exacerbation (another 2k words)

Env

Mis-reads Control-flow

… (many steps calculating the risk score) Risk score: medium Now let’s see the referral, get_referral(“RF-123”) Member_id: M-778, content: The doctor requests chronic disease care for … (1k words) Refers to wrong info

Misses key info

Error A: Use wrong contexts for decision making I see that the doctor requests to chronic care and the patient indeed has two chronic condition, let open a case for chronic care. Error B: Fail to follow complex control-flow instructions I see that the patient has an acute utilization about heart failure, but his risk score is medium, so it is not eligible for complex care.

High risk AND acute util.

Reference correct reasoning I see that the patient has a recent heart failure. Although the risk score is medium, this is an acute utilization, so he is eligible for complex care.

Fig. 1. Common failure modes when an agent follows a complex textual workflow. The figure shows two potential failure modes together with the reference correct reasoning. In Error A, the agent fails to select the correct information from context for decision making: the key information is the recent heart-failure exacerbation, which should support complex care, but the agent mistakenly relies on the doctor’s chronic-care referral message. In Error B, the agent misreads a subtle control-flow instruction. The workflow specifies high risk or recent acute utilization, but the agent interprets it as high risk and recent acute utilization.

A. Artifact-Driven Workflow Formulation The core design of our language is to formulate a workflow as a sequence of transformations over artifacts. We refer to this representation as an artifact-driven workflow. This formulation lets the compiler explicitly measure and analyze the enforcement challenge of a workflow step. To support this discussion, we introduce a small language that formalizes artifact-driven workflows. We illustrate the design by defining the semantics of this language; the optimization and validation procedures in the following subsections are also defined over the same language. Language definition. Figure 4 defines the core language. The four statements denote the basic workflow structures. An agent step agent(ω) asks a subagent to follow a natural-language instruction ω and complete a local artifact transformation. A sequential step connects two consecutive workflow steps. An explicit branch chooses between two subworkflows according to predicate p, and a loop repeats a subworkflow while p holds.

complex context, we explicitly introduce artifacts into the workflow: named intermediate results that a step consumes or produces. In this representation, if a step accumulates or inputs too many artifacts, it becomes an indicator of potential enforcement challenge. Figure 2(b) shows an example. The complex-care decision receives only the utilization and risk-score artifacts; it does not receive the doctor’s referral message, which is relevant to the chronic-care decision. This separation reduces the chance that an agent selects the wrong evidence from a long context. Furthermore, the artifact-driven language lets the compiler measure control-flow complexity. In the example, the compiler identifies that the original decisionmaking step combines risk-score calculation, complex-care eligibility, and chronic-care eligibility. It therefore decomposes the step into three simpler transformations: computing the risk score, checking complex-care eligibility, and checking

S ::= agent(ω) | S1 ; S2 | if p then S1 else S2 | while p do S

agent step sequential step explicit branch loop

Fig. 4. Core artifact-driven workflow language.

3

Get utilization by calling …

Get referral info. Get referral info by calling … Extract member id and doctor’s message.

Input query

Get medical record by calling …

Complex Care

Make decision. Calculate risk score, if…, otherwise …

Unconstrained message passing

Chronic Care ❌ Not Eligible

(a) Existing step-by-step workflow (b) Our artifact-centric workflow

Input query

Get referral info

C Check chronic decision

DoctorMsg: str

Check

Not eligible for chronic care

MemId: str … Data dependences

Eligible for Chronic: bool Reason: str

…

Active components

Not eligible for complex care

Utilization: str

Inactive components Sub-agent

Output

Chronic Care

Medical Record: str

Control dependences

Artifact

Eligible for Chronic Care

Not Eligible

Constraint

Cal risk score from … and …

Check complex care decision C

Check

Eligible for Complex: bool Reason: str

Risk score: Int

Eligible for complex care

Complex Care ✅

Fig. 2. Workflow structure affects enforcement reliability. (a) shows a naive step-by-step workflow. The subagent in Step 4 still needs to select the correct information from a dense context that includes the referral message, medical record, utilization history, and other intermediate results. Moreover, Step 4 still contains complex control-flow instructions that the agent may misapply. (b) shows our artifact-driven workflow. The compiler identifies the step that needs complex artifacts and decomposes it into Step 3 and Step 4, so Step 4 only receives focused utilization information and the risk score. The compiler also decomposes the complex control flow into Step 4 and Step 5, so the workflow structure enforces the or relation specified in the original textual workflow.

the implementation, artifact data may be backed by a typed JSON object, a document, or a file, while constraints record the semantic properties that later checks and control transfers may inspect. We further define two standard operations on artifact states. Reading a set of artifact identifiers A restricts the store to those artifacts:

Syntax & Type Check Faithfulness Validation

Induct. Validation Dry-Run Simulation

Draft Workflow

Compiled Workflow

Constrained Optimization

Faithfulness Feedback

NL Workflow

Σ[A] = { i 7→ Σ(i) | i ∈ A }.

Updating an artifact state with produced artifact entries U overrides the current data–constraint pair for the identifiers in U : ( U (i), i ∈ dom(U ), Σ ⊕ U = λi. (3) Σ(i), otherwise.

Check Constraint

Fig. 3. Compiler workflow. The compiler drafts an artifact-driven workflow under program-analysis feedback, validates faithfulness through static, inductive, and dry-run checks, and either accepts the workflow or feeds diagnostics back to the drafting stage.

We denote the semantics of a statement with the following big-step notation: ⟨S, Σ⟩ ⇓ Σ′ (4) This means that statement S, when executed from artifact state Σ, terminates in artifact state Σ′ . Figure 5 shows the core rules. In the agent rule, Runω (Σ) denotes the possible artifact updates produced by executing instruction ω from state Σ, and Checkω (Σ, U ) denotes the constraint check over the produced update U . The interpretation of S EQ and I F -T RUE is direct: sequential composition executes the second statement from the state produced by the first, and the true branch executes S1 when predicate p evaluates to true. The remaining branch and loop rules are standard and are shown in Figure 14 in Appendix A.

Semantics definition. The semantics of the language is defined as transformations over artifact states. We formally define an artifact state as a record that maps each artifact identifier to its current data and semantic constraints: Σ : Aid ⇀ (D × 2C ).

(2)

(1)

Here, Aid is the set of artifact identifiers, D is the domain of artifact data, and C is the domain of semantic constraints over artifacts. Thus Σ(i) = (di , Ci ) says that artifact i currently stores data di and is governed by the constraint set Ci . In

4

Easier to compact

U ∈ Runω (Σ) Checkω (Σ, U ) = ok ⟨agent(ω), Σ⟩ ⇓ Σ ⊕ U

(AGENT)

⟨S1 , Σ⟩ ⇓ Σ1 ⟨S2 , Σ1 ⟩ ⇓ Σ2 ⟨S1 ; S2 , Σ⟩ ⇓ Σ2

(S EQ)

JpK(Σ) = true ⟨S1 , Σ⟩ ⇓ Σ′ ⟨if p then S1 else S2 , Σ⟩ ⇓ Σ′

Harder to compact Data deps

Workflow steps

Context compaction

Artifact that needs to preserve

(I F -T RUE)

Artifact that can be compacted

Fig. 6. Context-compaction pressure. A step is harder to compact when many live artifacts must be preserved across the compaction point.

Fig. 5. Core big-step semantics for artifact-driven workflows.

B. Optimization Objectives

take the larger route because only one route executes. Loops scale by the estimated iteration count b k. We denote the context burden of a prompt ω as bctx (ω):

We formulate workflow compilation as an iterative constrained optimization process. The compiler uses a set of optimization objectives to guide the LLM toward a workflow that is easier to enforce while preserving the source procedure. The intuition follows the challenges illustrated in Section II: if a natural-language step contains too much context or too much control structure, an agent may struggle to follow it faithfully. The compiler should therefore keep only enforceable instructions in one step and decompose complex instructions into multiple steps. Estimating context and control complexity is nontrivial because the structure is latent in a prompt. The artifact-driven language makes this analysis possible by normalizing a prompt into a workflow and then applying program analysis to that workflow. Given a prompt ω, we write fg(ω) for its finestgrained artifact-driven workflow representation. In this view, each agent step has one local goal and produces one focused artifact. This fine-grained view exposes the latent context and control-flow structure that the original prompt would otherwise ask one agent to manage implicitly. We use four optimization objectives to formulate this constrained optimization problem. Control-flow complexity. The intuition of control-flow complexity is that a prompt with many logical branches is harder for an agent to enforce faithfully. We denote the control-flow burden of a prompt ω as bcf (ω) and estimate it by applying cyclomatic complexity to its finest-grained workflow:

bctx (ω) = C(fg(ω)).

(7)

Compaction pressure. The intuition of compaction pressure is that a prompt is harder to execute robustly when important information must survive a context-compaction point. For a fine-grained workflow G, assume LiveG (ℓ) is provided by standard live-variable analysis at program point ℓ. The pointwise compaction pressure is: X pcmp (G, ℓ) = s(a). (8) a∈LiveG (ℓ)

We denote the compaction burden of a prompt ω as bcmp (ω) and use the worst internal compaction point: bcmp (ω) =

max ℓ∈pts(fg(ω))

pcmp (fg(ω), ℓ).

(9)

Figure 6 illustrates this case: when many live artifacts cross a compaction point, the compactor must preserve more state for future execution to remain well-defined. Artifact cost. Artifact cost tracks the overhead introduced by materializing intermediate results. We denote the artifact cost of a workflow W as A(W ): X A(W ) = s(a), (10) a∈Art(W )

[ bcf (ω) = cyclo(fg(ω)).

(5)

where Art(W ) is the set of materialized artifacts in workflow W , and s(a) is the estimated size of artifact a. The compiler therefore searches for workflows that reduce local controlflow, context, and compaction pressure without introducing unnecessary artifact materialization.

[ is an estimate of the cyclomatic complexity of the Here cyclo prompt’s latent workflow. Context pressure. The intuition of context pressure is that a prompt is harder to enforce when one agent invocation must carry a long execution context before reaching a checkpoint boundary. We first define a context estimator C over a finec grained workflow. Let ctx(X) be an atomic estimate of the context length for a primitive body or guard X:

C. Faithfulness Validation The optimization step relies on LLM reasoning, so its edits can involve nontrivial natural-language judgments. The compiler must therefore check whether the produced workflow is a faithful representation of the source workflow, but this check also requires LLM reasoning. The key challenge is to make validation easier than generation, so that an LLM validator with comparable capability can still improve the compiler. This generation-validation asymmetry has been exploited in verifier, process-supervision, and self-feedback methods for LLM reasoning [31]–[33].

c C(agent(ω)) = ctx(ω) C(S1 ; S2 ) = C(S1 ) + C(S2 )
(6) c C(if p then S1 else S2 ) = max ctx(p), C(S1 ), C(S2 ) C(while p do S) = b k(p, S) · C(S). The sequential case adds pressure because the same prompt asks one agent to carry both pieces of latent work. Branches

5

We reduce the challenge of validation by two key ideas. First, validating a whole workflow at once is difficult, but the artifact-driven workflow language lets the compiler decompose the validation goal into smaller obligations that follow the language structure. Each local obligation is simpler and easier to validate reliably. Second, the compiler facilitates LLM reasoning by concretizing the context: it enumerates diverse concrete test cases and asks the LLM to check whether the compiled workflow remains faithful when simulating those cases. Static well-formedness. Before invoking LLM validators, the compiler applies deterministic static checks to the artifactdriven workflow program. These checks validate the surface syntax, ensure that every artifact read is declared and available, and type-check artifact availability at control-flow joins. The key type check is path stability at joins. If a program point can be reached from multiple predecessors, then every incoming path must provide compatible types for the artifacts read at that point: ∀ℓ. ∀p1 , p2 ∈ Pred(ℓ). typep1 (Reads(ℓ)) ≃ typep2 (Reads(ℓ)).

enumerate concrete scenarios, including cases where changing one predicate changes the expected artifact or route. Third, it asks LLM validators to independently simulate the source and compiled workflows on each scenario, and compares the simulated trajectories and outcomes to detect inconsistencies. Figure 8 illustrates the process. IV. I MPLEMENTATION We implement the compiler prototype core in 5.9K lines of Python code. The code and data will be available upon publication. The resulting executable workflow package has three parts. The orchestrator is a deterministic Python script that invokes individual statements and transfers control according to each step’s output. A step delegates local natural-language instructions to subagents and parses their produced artifacts. The runtime implements low-level interfaces, including resolving concrete artifact paths from artifact identifiers, managing artifact versions, and invoking subagents. Appendix B gives additional implementation details. V. E VALUATION

(11)

A. Setup Benchmarks. Our workflow compiler is intended to help non-programming experts build more reliable agentic workflows for domain-specific procedures. We therefore evaluate on 11 domains drawn from SOP-Bench [37] and χBench [36]. SOP-Bench contains standard operating procedures (SOPs) designed to reflect real-world industrial workflows, with executable interfaces and human-validated test cases. χ-Bench is a realistic healthcare workflow simulator with MCP tools, medical-operation handbooks, and multi-role interaction. Rather than running the full end-to-end healthcare process, we use its care-management intake and programeligibility stage, which aligns with our goal of evaluating complex domain-workflow enforcement. We report this domain as Medical. For domains with more than 50 instances, we sample at most 50 for budget reasons. In total, our test set contains 488 instances from 11 domains; detailed domain statistics appear in Table III. Execution setup. We evaluate three compiler models: GPT5.4, Sonnet-4.6, and GLM-5. These are frontier large models used to translate textual workflows into artifact-driven workflows. We also evaluate six executor models, ranging from 3B active parameters to 700B+ total parameters among models with public size metadata. These executors are costefficient enough for realistic workflows in real-world domains. The full model list appears in Appendix Table IV. All executions use the same Codex-based agent harness, because Codex provides a general tool-executing interface and has been used beyond software engineering for knowledge-work tasks such as document drafting, data analysis, and communication coordination [38], [39]. The evaluated agent interacts with each benchmark environment through its MCP tools. The primary metric is task resolve rate: the fraction of benchmark instances whose final output satisfies the benchmark oracle. These benchmarks are designed to test workflow following: an

Here, Pred(ℓ) is the set of predecessor paths that can reach program point ℓ, Reads(ℓ) is the set of artifacts read at ℓ, and typep (·) gives the artifact types available along path p. This check helps the compiler make latent execution obligations explicit. A natural-language workflow may describe only the condition under which an artifact is normally produced, while a later step implicitly assumes that the artifact is available. For example, suppose a workflow says: “if the patient has a high risk score, retrieve the patient’s medical record and report any chronic disease,” followed by “cross-check the patient’s utilization.” A verbatim branch interpretation leaves the medical record unavailable on the low-risk path before the second step, even though the intended cross-check may still require that record. The join type check exposes this mismatch: either the workflow must retrieve a compatible medical-record artifact on all paths, or the later step must be rewritten so that its declared inputs match the artifacts actually available. Inductive validation. The compiler validates by induction over the workflow program structure. It decomposes the global validation goal into smaller obligations that match the structure of the compiled program, and asks the validator to discharge each obligation separately. Figure 7 illustrates this decomposition for a sequence, where a single validation goal is reduced to an artifact-boundary check and two local validation obligations. Appendix A-A gives the formal proof sketch for this validation structure. Dry-run simulation. The compiler further facilitates faithfulness validation by enumerating concrete test scenarios and asking LLM validators to reason over those scenarios. Scenariobased testing has been shown effective for exposing agent misinterpretation and execution failures [34], [35]. Our dry-run simulation has three stages. First, it extracts a logical formula from the validation target and identifies the atomic predicates that should be tested. Second, it uses predicate coverage to

6

Input: user_id

1. Retrieve user’s deliver address of the last unfulfilled order 2. Obtain recent regional delays. 3. Check whether the user’s address is in by one of the regions.

Output: is_order_delayed

The process output whether the user’s last unfulfilled order is affected by any recent regional delays.

Source: One step workflow Target: Two steps with an artifact Input: user_id

Retrieve address of …

Artifact: addr

The process outputs the user’s address on the last unfulfilled order

Obtain and check …

Output: is_order_delayed

The process checks whether the input addr is in a recent delay region.

Artifact Validation target Workflow step

Fig. 7. How we validate workflow by induction. Suppose the source workflow describes one natural-language step, but the compiled workflow refines it into two subworkflows S1 ; S2 connected by an intermediate artifact. The validator first checks that the artifact is a valid conceptual boundary in the source workflow: after S1 , the artifact should contain exactly the intermediate state that the remaining source workflow may rely on. It then validates the prefix up to that artifact and validates the continuation from that artifact. If both local obligations hold, the sequence validation composes them into a validation of the full compiled fragment.

B. RQ1: Does Artifact-Driven Compilation Improve Workflow Enforcement?

Condition to validate: Both workflows output true iff. member meets the risk score threshold (0.8) AND has at lease one chronic disease or is using drugA.

Table I compares artifact-driven workflows with representative baselines from two families. The first family relies on the general capability of LLM agents to improve the workflow. SkillCreator takes the original workflow as input and rewrites it into a clearer skill document, but the result is still textual instructions rather than an executable workflow. Code instructs a coding agent to write Python code with LLM calls that implements the workflow, using the same harness and executor model as our compiler. We provide this baseline with the necessary Python wrappers for MCP tools and utility functions for invoking subagents. Besides, Text denotes the setup that directly gives the original workflow to the executor, and A RTIC denotes the compiled artifact-driven workflow. Overall, our compiled workflows achieve the best average task resolve rate for every executor model: 85% on GLM4.7-Flash, 82% on GPT-OSS-120B, and 85% on Qwen3235B. This indicates that artifact-driven compilation improves workflow enforcement beyond simply rewriting the instruction or asking an agent to implement it as code. On domains such as patient intake and referral abuse, the code baseline can slightly outperform our workflow while our workflow remains nearperfect. Inspection suggests that these code implementations often use stable heuristics such as keyword matching, which can work well for simple domain rules but may not generalize to complex workflows, as reflected by the suboptimal performance of Code on other domains. For domains such as Know Your Business and warehouse inspection, all representations have relatively low performance. These workflows contain properties that require model judgment, such as distinguishing an ordinary typo from fake business information. This challenge is orthogonal to workflow enforcement; Figure 13 gives a concrete example. The second baseline family consists of specially designed pipelines that convert natural language into executable workflows [25], [26]. Table II compares our compiler with two such baselines on GLM-4.7-Flash. MASFactory often fails

Predicate coverage on extracted logic formula: Out = Risk /\ (Chronic \/ DrugA) Out

Risk

Chronic

DrugA

T

T

T

F

F

F

T

F

F

T

F

F

T

T

F

T

Dry run Enumerate test cases NL Workflow Dry run

Risk Chronic DrugA

Consistency Check

Compiled Workflow

Fig. 8. Dry-run simulation validation. The compiler extracts predicates, enumerates coverage-guided scenarios, and compares independent dry runs of the source and compiled workflows.

instance is resolved only when the agent follows the prescribed procedure and makes the correct domain decision. We compare against textual execution, text-space skill rewriting, direct code generation, and existing natural-language-to-workflow baselines. Research questions. The evaluation is organized around four research questions. RQ1 evaluates whether artifact-driven compilation improves workflow enforcement. RQ2 evaluates the reliability of the compiled workflows under model variation, repeated execution, and execution stressors. RQ3 analyzes compilation time and cost. RQ4 ablates how each system component contributes to system performance. We then present two case studies to illustrate how dry-run simulation improves compilation and where the current pipeline remains limited.

7

TABLE I C OMPARISON OF EFFECTIVENESS ACROSS DIFFERENT WORKFLOW REPRESENTATIONS . N UMBERS DENOTE TASK RESOLVE RATE IN PERCENT. GLM-4.7-Flash

Domain

GPT-OSS-120B

Qwen3-235B

Text

SkillCreator

Code

A RTIC

Text

SkillCreator

Code

A RTIC

Text

SkillCreator

Code

A RTIC

Medical Customer service Dangerous goods Know your business Order fulfillment Patient intake Referral abuse v1 Referral abuse v2 Traffic spoofing Video annotation Warehouse inspection

48 52 38 48 86 85 84 50 68 72 52

72 72 84 52 83 91 80 36 66 34 58

12 48 94 66 90 100 100 82 66 90 50

88 88 98 64 90 100 98 96 74 90 50

36 24 68 44 0 9 92 76 46 50 58

72 34 90 32 0 12 74 66 36 40 48

12 48 94 66 90 100 100 82 66 90 50

76 66 98 62 90 97 100 100 74 90 50

40 64 54 48 90 94 54 36 72 32 80

56 90 90 52 90 100 56 64 74 86 82

8 48 94 66 90 100 100 82 66 90 50

76 92 98 62 90 100 100 100 74 90 50

Average

62

66

73

85

46

46

73

82

60

76

72

85

TABLE II C OMPARISON BETWEEN OUR COMPILER AND EXISTING NL- TO - WORKFLOW BASELINES . ERR DENOTES THAT THE BASELINE FAILED TO GENERATE AN EXECUTABLE WORKFLOW.

Medical Customer service Dangerous goods Know your business Order fulfillment Patient intake Referral abuse v1 Referral abuse v2 Traffic spoofing Video annotation Warehouse inspection Average

MASFactory

Chat2Workflow

A RTIC

0 0

ERR

52 0 0 0 0 100 100 100 0 0 0

88 88 98 64 90 100 98 96 74 90 50

0

32

85

ERR ERR

0 0 ERR

0 ERR ERR

Domain

Cases

Length

Nodes

Edges

Artifacts

25 50 50 50 29 34 50 50 50 50 50

22,133 9,166 5,638 8,772 2,631 4,848 8,277 12,558 4,954 31,431 4,292

8 10 9 15 6 7 6 12 8 14 8

13 18 11 19 6 7 6 14 8 20 9

8 12 8 11 6 7 6 10 8 14 11

44.4

10,427

9.4

11.9

9.2

Medical Customer service Dangerous goods Know your business Order fulfillment Patient intake Referral abuse v1 Referral abuse v2 Traffic spoofing Video annotation Warehouse inspection Average

Performance (%)

Domain

TABLE III D OMAIN - LEVEL WORKFLOW STATISTICS .

before producing an executable workflow on these benchmark domains, and Chat2Workflow can produce runnable workflows for some domains but expects more concrete human guidance about the intended workflow structure. It is less accurate when workflows become more complex. These results suggest that existing techniques lack an autonomous validation and feedback mechanism that can ensure workflow quality in complex domains with limited human involvement.

80 60 40 20

Text ARTIC

Models

Fig. 9. Performance across executor models on the Medical domain.

C. RQ2: Are Compiled Workflows More Reliable? We call an instance consistent when all six executor models either pass it or fail it under the same workflow representation. Suppose a user develops and debugs a workflow on one executor model, observes which cases succeed or fail, and later deploys the same workflow with a different executor. This metric measures how often that development-time observation transfers across executor environments. Figure 10 shows pass/fail outcomes for the same Medical-domain instances across executor models. The compiled workflow is consistent on 80% of instances, compared with 48% for the text baseline. Consistency across repeated executions. We use pass^k,

Consistency across executor models. The results show that compiled workflows behave more stably when executed by different executor models. Figure 9 compares the Medical domain from χ-Bench across six executor models. The text workflow varies from 28% on its worst model to 60% on its best model, twice the spread of the compiled workflow, which ranges from 72% to 88%. The smaller degradation suggests that explicit artifacts and control transfers make execution less sensitive to the executor model. Per-instance consistency. Besides overall performance variance, it is also important to measure per-instance consistency.

8

piled workflows reduce context burden by decomposing challenging steps into smaller subagents. Across the Table I samples, the compiled workflow reduces average per-agent input tokens by 63% and output tokens by 50% compared with an agent working on the text workflow. Figure 17 in the appendix gives the full box plots by executor model.

Text - 48% Consistency Nemotron GLM-5 Nova Micro GLM-4.7-F Qwen3-235B GPT-OSS

ARTIC - 80% Consistency Nemotron GLM-5

D. RQ3: What Is the Compilation Cost? Compilation is an offline cost paid once per workflow. For the studied SOP-Bench domains, the median first-package compilation time with GLM-5 is under 5 minutes, and the average compilation cost is under $3. Figure 18 in the appendix gives the full time-and-cost distribution.

Nova Micro GLM-4.7-F Qwen3-235B GPT-OSS

Problem Instances Text pass

ARTIC pass

Fail

Fig. 10. Per-instance consistency across executor models on the Medical domain. Each row is a model and each block is one problem instance; colored blocks pass and gray blocks fail.

100

Text

Estimate

ARTIC

Observed

E. RQ4: How Does Each System Component Contribute? We ablate the dry-run feedback, validation-by-induction, and optimization components of our system. Each component contributes meaningfully: the observed performance changes range from 8 to 32 percentage points. Optimization contributes the most, which is expected because it identifies when a prompt places too much context or decision burden on the executor and decomposes that step. We also test different compiler models and find that the generated workflow performance is stable across compiler choices. Detailed ablation and compiler-model results appear in Appendix C.

pass^k

75 50 25 0

1

2

3

4

5

6

7

8

9

10

F. Case Studies How dry-run simulation helps validate compiled workflows. Figure 12 shows a concrete case in which dry-run simulation identifies an inconsistency in a compiled workflow. The compiled workflow used an overly vague prompt for checking the source of a referral: as shown in Figure 12(b), the first decision node checked whether a referral message exists, but did not verify whether the referral came from a qualifying source. The dry-run process begins with test case enumeration guided by predicate coverage, as shown in Figure 12(c). The exposing case is a referral from a non-qualifying person with a reason that otherwise supports complex-care review. Figure 12(d) then runs two independent simulations: one over the source natural-language workflow and one over the compiled workflow. Neither dry run is given the oracle label. Instead, each simulation follows its workflow step by step. The resulting traces disagree, which localizes the error to the referral-source predicate and provides the compiler with targeted feedback. Limitation of workflow compiler. The workflow compiler does not achieve perfect accuracy. Cases that depend more on model judgment than workflow enforcement may remain unresolved. Figure 13 illustrates a concrete example in the “Know your business” domain, where the SOP asks the executor to decide whether mismatches among a business name, website, address, and email are ordinary typos or evidence of fabricated information. Artifact-driven compilation can make the relevant inputs explicit and route the decision through a focused step, but the final judgment still depends on the executor model’s knowledge and calibration. This challenge is orthogonal to workflow enforcement.

K repeated runs

Fig. 11. Consistency of workflows in repeated runs. The metric pass^k denotes the probability that all k repeated runs are correct. Solid and dashed lines show the unbiased estimator and observed performance, respectively. Shaded bands show ± one standard deviation.

following Tau-Bench [40], to measure how likely all k repeated runs are correct when the same workflow is executed by the same model multiple times. Figure 11 shows the result on the Medical domain in χ-Bench. We report both the unbiased estimator and the observed performance; Appendix C gives the calculation details. At k = 10, the compiled workflow has about 72% passing cases, while the text workflow has only 16%. From k = 1 to k = 10, the compiled workflow degrades by 16 percentage points, compared with 35 points for the text workflow. This means the compiled workflow is significantly more stable than the text workflow under repeated execution. Robustness under environment perturbations. We test the robustness of each workflow under environment perturbations. In real-world execution, an agent may encounter transient model request errors or context compaction. This test measures how well a workflow tolerates such perturbations. We use all data samples where both the text and compiled workflow give the correct answer in the original setting, then observe how their performance changes under different conditions. The compiled workflow preserves 100% performance, while the text workflow drops to 80% under the strongest perturbations. See Figure 16 in the appendix for details. Per-agent context pressure. We further evaluate how com-

9

NL workflow: If referral source indicates treating physician or hospital discharge planner and referral reason supports complex care enrollment review, then complex care eligibility is met. C1: Is there a referral message?

NL1: Referral from treating physician? Y

NL2: Reason supports complex care? Y

N

Y

Not eligible

…

N

N

Not eligible

N

C2: Reason supports complex care? Y

Eligible

NL3: Referral from hospital discharge planner? Y

N

Misses checks for referral source Not eligible

Eligible

Not eligible (b) Decision tree derived from compiled workflow

(a) Decision tree derived from NL workflow

Dry run on NL Extracted condition: (Treating Physician OR Hospital Discharge Planner) AND Refer to Complex Care

Enumerated test case: condition: (False, False, True) Referral from a medical specialist for complex care.

(c) Test case enumeration guided by predicate coverage

NL1

N

NL3

N

N. E.

Dry run on compiled workflow C1

Y

C2

Y

(d) Dry run simulation

E. Inconsistent

Fig. 12. Dry-run feedback case study. The dry run compares the intended natural-language condition with the compiled decision structure on a representative case, identifying missed checks for referral source before runtime evaluation.

Domain workflows have a different status: they encode expert requirements about the procedure itself, so the compiler cannot arbitrarily change the workflow to improve outcome metrics. Our problem is therefore not only to obtain a better-performing prompt, but to preserve and enforce the workflow constraints that the expert intended. Skill compilers and skill adaptation. Concurrent work on skill compilation studies how agent systems can retrieve relevant skills and adapt skills across different harnesses [55]– [59]. That layer is complementary to ours. Skill retrieval decides which skill should be used, and harness adaptation rewrites a skill into a semantically equivalent counterpart for a different execution environment. This paper focuses on a different question: once a domain workflow is selected, how can the runtime measure and enforce whether the agent faithfully follows it? Our artifact-driven representation makes enforcement itself an object of analysis through explicit artifacts, constraints, and control transfers. Natural language to workflow generation. Existing work around natural-language-to-workflow generation falls into three related streams. First, some systems synthesize or evolve new workflows and rules to improve task performance [60], [61]. This direction is orthogonal to ours: expert workflows are requirements to be preserved, not search spaces that the system can freely rewrite. Second, systems such as Chat2Workflow and MASFactory help users construct executable workflows, but typically rely on human steering or correction to reach a valid workflow [25], [26]. Our setting instead asks the compiler to validate and refine the workflow with limited manual intervention. Third, platform-oriented systems and benchmarks operate in settings where component semantics are more fixed, such as workflow orchestration platforms or enterprise workflow services [24], [62]–[64]. These systems provide useful front-end techniques, but the fixed platform semantics may not generalize to other domains. A RTIC focuses

Original text from the SOP of “Know your business”. The SOP first instructs the agent to obtain the website, address, and email address of the business; this paragraph explains how to decide whether further information is needed. “They should use their experience to check if the website and/or address and/or email matches with the pre-validated business name. Due to errors by associates, there may be typos. . . . Use your experience to identify if the errors are genuine typos by associates entering the data or made-up names submitted by the businesses. If you believe that there is some irregularities here, please enter in the system awaiting information.”

Fig. 13. KYB example in which the source workflow explicitly requires judgment about whether a mismatch is an ordinary typo or a data irregularity.

VI. R ELATED W ORK Specification languages and workflow enforcement. Software-engineering research has long studied specification languages for making expected behavior precise [41]–[45]. Our goal is different from asking users to write such a specification language directly. Domain experts often start from natural-language workflows, and the hard step is to translate those workflows into an enforceable representation. Existing specification languages typically still rely on humans to decompose a procedure, choose the right abstraction boundaries, and encode the intended control and data dependencies [46]– [48]. This paper instead studies how a compiler can recover an artifact-driven representation from natural language and then make the result checkable and enforceable by an agent runtime. Automatic skill generation and prompt optimization. Another line of work generates skills, prompts, or agent policies from task feedback [33], [49]–[54]. This direction is largely orthogonal to our goal. It is especially useful when the task is outcome driven and the system can freely search for a better prompt or strategy, as in test-driven software development.

10

on that enforcement problem: the generated workflow must make intermediate artifacts, control transfers, and validation obligations explicit enough for the runtime to check and enforce.

[17] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 2024. [Online]. Available: https://aclanthology.org/2024.tacl-1.9 [18] Y. Jiang, Y. Yi, Y. Wang, Z. Xia, W. Shi, P. Zhang, X. Li, L. Li, Z. Gu, Q. Liu, Z. Ge, W. Chen, Y. R. Fung, C. Z. Yang, Y. Li, S. He, and X. Xie, “FollowBench: A multi-level fine-grained constraints following benchmark for large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 4667–4688. [Online]. Available: https://aclanthology.org/2024.acl-long.257 [19] X. Wu, M. Wang, Y. Liu, X. Shi, H. Yan, X. Lu, J. Zhu, and W. Zhang, “Lifbench: Evaluating the instruction following performance and stability of large language models in long-context scenarios,” 2025. [Online]. Available: https://arxiv.org/abs/2411.07037 [20] Y. Shi, S. Cai, Z. Xu, Y. Qin, G. Li, H. Shao, J. Chen, D. Yang, K. Li, and X. Sun, “Flowagent: Achieving compliance and flexibility for workflow agents,” arXiv preprint arXiv:2502.14345, 2025. [21] K. Chen, Q. Wu, T. Hou, T. Tang, X. Hu, Y. Hou, B. Li, C. Qian, G. Wang, H. Chen et al., “Agentif-oneday: A task-level instructionfollowing benchmark for general ai agents in daily scenarios,” arXiv preprint arXiv:2601.20613, 2026. [22] A. Purpura, L. Wang, S. Badyal, E. Beaufrand, and A. Faulkner, “Enhancing llm instruction following: An evaluation-driven multiagentic workflow for prompt instructions optimization,” arXiv preprint arXiv:2601.03359, 2026. [23] A. Newell and H. A. Simon, Human Problem Solving. Englewood Cliffs, NJ: Prentice-Hall, 1972. [24] J. Xu, W. Du, X. Liu, and X. Li, “Llm4workflow: An llm-based automated workflow model generation tool,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 2394–2398. [Online]. Available: https://doi.org/10.1145/3691620.3695360 [25] Y. Liu, J. Cai, Y. Li, Q. Meng, Z. Liu, X. Li, C. Qian, C. Shi, and C. Yang, “Masfactory: A graph-centric framework for orchestrating llm-based multi-agent systems with vibe graphing,” arXiv preprint arXiv:2603.06007, 2026. [Online]. Available: https: //arxiv.org/abs/2603.06007 [26] Y. Zhong, B. Xu, Y. Wang, Z. Shan, S. Qiao, G. Zheng, and N. Zhang, “Chat2workflow: A benchmark for generating executable visual workflows with natural language,” 2026. [Online]. Available: https://arxiv.org/abs/2604.19667 [27] X. Leroy, “Formal verification of a realistic compiler,” Communications of the ACM, vol. 52, no. 7, pp. 107–115, 2009. [Online]. Available: https://doi.org/10.1145/1538788.1538814 [28] S. Ghosh, D. Elenius, W. Li, P. Lincoln, N. Shankar, and W. Steiner, “Arsenal: automatic requirements specification extraction from natural language,” in NASA Formal Methods Symposium. Springer, 2016, pp. 41–46. [29] M. Cosler, C. Hahn, D. Mendoza, F. Schmitt, and C. Trippel, “nl2spec: Interactively translating unstructured natural language to temporal logics with large language models,” in International Conference on Computer Aided Verification. Springer, 2023, pp. 383–396. [30] F. Fuggitti and T. Chakraborti, “Nl2ltl–a python package for converting natural language (nl) instructions to linear temporal logic (ltl) formulas,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 13, 2023, pp. 16 428–16 430. [31] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021. [Online]. Available: https: //arxiv.org/abs/2110.14168 [32] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” arXiv preprint arXiv:2305.20050, 2023. [Online]. Available: https://arxiv.org/abs/2305.20050 [33] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, “Self-refine: Iterative refinement with self-feedback,” in Advances in Neural Information Processing Systems, 2023. [Online]. Available: https://arxiv.org/abs/2303.17651

VII. C ONCLUSION Natural-language workflows are easy to write but hard to enforce. This paper formulates them as artifact-driven workflows, where artifacts expose state, constraints gate progress, and control transfers are explicit. A RTIC compiles natural-language workflows into this form and improves enforcement across realistic domains. Its validation combines well-formedness checks, compositional reasoning, and scenario-based dry runs. R EFERENCES [1] Z. M. Jiang, A. E. Hassan, T. Zimmermann, and M. Harman, “AIware in the foundation model era,” IEEE Software, vol. 43, no. 1, pp. 26–30, 2026. [Online]. Available: https://doi.org/10.1109/MS.2025.3624446 [2] A. E. Hassan, D. Lin, G. K. Rajbahadur, K. Gallaba, F. R. Cogo, B. Chen, H. Zhang, K. Thangarajah, G. A. Oliva, J. Lin, W. M. Abdullah, and Z. M. Jiang, “Rethinking software engineering in the foundation model era: A curated catalogue of challenges in the development of trustworthy fmware,” 2024. [Online]. Available: https://arxiv.org/abs/2402.15943 [3] Z. Chen, C. Wang, W. Sun, X. Liu, J. M. Zhang, and Y. Liu, “Promptware engineering: Software engineering for prompt-enabled systems,” ACM Transactions on Software Engineering and Methodology, 2026. [Online]. Available: https://doi.org/10.1145/3796535 [4] J. T. Liang, M. Lin, N. Rao, and B. A. Myers, “Prompts are programs too! understanding how developers build software containing prompts,” Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 1591–1614, 2025. [Online]. Available: https://doi.org/10.1145/3729342 [5] A. Korinek, “AI Agents for Economic Research,” National Bureau of Economic Research, NBER Working Paper 34202, September 2025. [Online]. Available: https://www.nber.org/papers/w34202 [6] L. Wang, Y. Li, J. Wu, H. Gao, X. Bo, X. Chen, and J.-R. Wen, “Llm agents as social scientists: A human-ai collaborative platform for social science automation,” arXiv preprint arXiv:2604.01520, 2026. [7] L. Zhao, S. Liu, T. Xin, J. Tan, X. Wang, Y. Li, Z. Bian, Y. Chen, F. Kong, J. Bian et al., “Ai agent in healthcare: applications, evaluations, and future directions,” npj Artificial Intelligence, vol. 2, no. 1, p. 31, 2026. [8] S. Liu, R. Zhang, R. Ma, Y. Deng, L. Zhu, J. Li, Z. Li, Z. Shen, and M. Du, “Llm agents in law: Taxonomy, applications, and challenges,” arXiv preprint arXiv:2601.06216, 2026. [9] W. Hua, X. Yang, M. Jin, Z. Li, W. Cheng, R. Tang, and Y. Zhang, “Trustagent: Towards safe and trustworthy llm-based agents,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 10 000–10 016. [10] H. Wang, C. M. Poskitt, and J. Sun, “Agentspec: Customizable runtime enforcement for safe and reliable llm agents,” 2025. [Online]. Available: https://arxiv.org/abs/2503.18666 [11] A. Kamath, S. Zhang, C. Xu, S. Ugare, G. Singh, and S. Misailovic, “Enforcing temporal constraints for llm agents,” arXiv preprint arXiv:2512.23738, 2025. [12] OpenClaw, “Skills - openclaw,” 2026. [Online]. Available: https: //docs.openclaw.ai/tools/skills [13] Hermes Agent, “Skills System – Hermes Agent,” Nous Research documentation, 2026. [Online]. Available: https://hermes-agent. nousresearch.com/docs/user-guide/features/skills [14] Anthropic, “Agent skills - claude api docs,” 2026. [Online]. Available: https://platform.claude.com/docs/en/agents-and-tools/agent-skills/ overview [15] Oracle, “Open agent spec,” 2026. [Online]. Available: https://github. com/oracle/agent-spec [16] S. Liu, S. Dehghan, J. Ganhotra, M. Hirzel, and R. Jabbarvand, “From plan to action: How well do agents follow the plan?” arXiv preprint arXiv:2604.12147, 2026.

11

[56] Y. Ouyang, Y. Xiao, Y. Gu, and X. Zhang, “Skcc: Portable and secure skill compilation for cross-framework llm agents,” arXiv preprint arXiv:2605.03353, 2026. [57] S. Chen, J. Gai, R. Zhou, J. Zhang, T. Zhu, J. Li, K. Wang, Z. Wang, Z. Chen, K. Kaleb et al., “Skillcraft: Can llm agents learn to use tools skillfully?” arXiv preprint arXiv:2603.00718, 2026. [58] X. Li, “When single-agent with skills replace multi-agent systems and when they fail,” arXiv preprint arXiv:2601.04748, 2026. [59] Y. Ma, Y. Huang, H. Bao, H. Zhuang, S. Shukla, M. Galley, X. Zhang, and S. Feuerriegel, “Skillgen: Verified inference-time agent skill synthesis,” arXiv preprint arXiv:2605.10999, 2026. [60] Z. Li, S. Xu, K. Mei, W. Hua, B. Rama, O. Raheja, H. Wang, H. Zhu, and Y. Zhang, “Autoflow: Automated workflow generation for large language model agents,” arXiv preprint arXiv:2407.12821, 2024. [61] C. Gurkan, N. K. Jwalapuram, K. Wang, R. Danda, L. Rasmussen, J. Chen, and U. Wilensky, “Lear: Llm-driven evolution of agent-based rules,” in Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2025, pp. 2309–2326. [62] S. Fan, X. Cong, Y. Fu, Z. Zhang, S. Zhang, Y. Liu, Y. Wu, Y. Lin, Z. Liu, and M. Sun, “Workflowllm: Enhancing workflow orchestration capability of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2411.05451 [63] H. Liu, R. Li, W. Xiong, Z. Zhou, and W. Peng, “Workteam: Constructing workflows from natural language with multi-agents,” in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track), 2025, pp. 20–35. [64] R. Xiao, W. Ma, K. Wang, Y. Wu, J. Zhao, H. Wang, F. Huang, and Y. Li, “Flowbench: Revisiting and benchmarking workflow-guided planning for llm-based agents,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 10 883–10 900.

[34] S. Feng, X. Xu, X. Chen, K. Zhang, S. Y. Ahmed, Z. Su, M. Zheng, and X. Zhang, “Tai3: Testing agent integrity in interpreting user intent,” 2025. [Online]. Available: https://arxiv.org/abs/2506.07524 [35] S. Y. Ahmed, S. Feng, C. Bae, C. Barrus, and X. Zhang, “SpecOps: A fully automated AI agent testing framework in real-world GUI environments,” Proceedings of the IEEE/ACM International Conference on Software Engineering, 2026. [Online]. Available: https://arxiv.org/abs/2603.10268 [36] H. Chen, D. Metelski, L. Qi, T. Xia, J. Lee, S. Brown, K. Riley, F. Wang, T. Y. A. Liu, H. Capps et al., “Chi-bench: Can ai agents automate endto-end, long-horizon, policy-rich healthcare workflows?” arXiv preprint arXiv:2605.16679, 2026. [37] S. Nandi, A. Datta, R. Nama, U. Patel, N. Vichare, I. Bhattacharya, P. Grover, S. Asija, G. Carenini, W. Zhang, A. Gupta, S. Bhaduri, J. Xu, H. Raja, S. Ray, A. Chan, E. X. Fei, G. Du, Z. Akhtar, H. Asnani, W. Chan, M. Xiong, F. Carbone, and J. Mirchandani, “Sop-bench: Complex industrial sops for evaluating llm agents,” 2026. [Online]. Available: https://arxiv.org/abs/2506.08119 [38] OpenAI, “Codex ide extension,” 2026. [Online]. Available: https: //developers.openai.com/codex/ide [39] D. Johnston, D. Holtz, A. M. Richmond, C. Ong, P. Tambe, and A. Chatterji, “The shift to agentic ai: Evidence from codex,” arXiv preprint arXiv:2606.26959, 2026. [40] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ -bench: A benchmark for tool-agent-user interaction in real-world domains,” arXiv preprint arXiv:2406.12045, 2024. [Online]. Available: https: //arxiv.org/abs/2406.12045 [41] D. Jackson, “Alloy: a lightweight object modelling notation,” ACM Transactions on software engineering and methodology (TOSEM), vol. 11, no. 2, pp. 256–290, 2002. [42] J.-R. Abrial, Modeling in Event-B: system and software engineering. Cambridge University Press, 2010. [43] W. M. van Der Aalst, A. H. Ter Hofstede, B. Kiepuszewski, and A. P. Barros, “Workflow patterns,” Distributed and parallel databases, vol. 14, no. 1, pp. 5–51, 2003. [44] W. M. Van Der Aalst and A. H. Ter Hofstede, “Yawl: yet another workflow language,” Information systems, vol. 30, no. 4, pp. 245–275, 2005. [45] M. Pesic, H. Schonenberg, and W. M. Van der Aalst, “Declare: Full support for loosely-structured processes,” in 11th IEEE international enterprise distributed object computing conference (EDOC 2007). IEEE, 2007, pp. 287–298. [46] D. L. Parnas, “On the criteria to be used in decomposing systems into modules,” Communications of the ACM, vol. 15, no. 12, pp. 1053–1058, 1972. [47] M. Jackson, “The world and the machine,” in Proceedings of the 17th international conference on Software engineering, 1995, pp. 283–292. [48] P. Zave and M. Jackson, “Four dark corners of requirements engineering,” ACM transactions on Software Engineering and Methodology (TOSEM), vol. 6, no. 1, pp. 1–30, 1997. [49] Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” 2023. [Online]. Available: https://arxiv.org/abs/2211.01910 [50] R. Pryzant, D. Iter, J. Li, Y. Lee, C. Zhu, and M. Zeng, “Automatic prompt optimization with “gradient descent” and beam search,” in Proceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 7957–7968. [51] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” Advances in neural information processing systems, vol. 36, pp. 8634–8652, 2023. [52] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, “Voyager: An open-ended embodied agent with large language models,” arXiv preprint arXiv:2305.16291, 2023. [53] O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam et al., “Dspy: Compiling declarative language model calls into self-improving pipelines,” arXiv preprint arXiv:2310.03714, 2023. [54] E. Zelikman, Y. Wu, J. Mu, and N. Goodman, “Star: Bootstrapping reasoning with reasoning,” Advances in Neural Information Processing Systems, vol. 35, pp. 15 476–15 488, 2022. [55] D. Xu, Z. Chen, Z. Pan, J. Guan, D. Dong, J. Li, and B. Pu, “Skillsmith: Compiling agent skills into boundary-guided runtime interfaces,” arXiv preprint arXiv:2605.15215, 2026.

12

A PPENDIX A VALIDATION D ETAILS

JpK(Σ) = false ⟨S2 , Σ⟩ ⇓ Σ′ ⟨if p then S1 else S2 , Σ⟩ ⇓ Σ′ JpK(Σ) = false ⟨while p do S, Σ⟩ ⇓ Σ

A. Faithfulness Validation Formal Sketch This subsection gives the formal sketch behind the faithfulness-validation discussion in Section III. Let Wn denote a source natural-language workflow fragment, and let Wc denote the compiled artifact-driven workflow fragment. The validation goal is not to prove syntactic equivalence between the two representations. Instead, it asks whether the compiled workflow preserves the observable artifact behavior of the source workflow under dry-run simulation. Figure 14 gives the branch and loop rules omitted from the main text. They use the same artifact-state judgment as Equation 4. Observational correspondence. We first define when two artifact stores agree on the artifacts that matter to the outside workflow:

JpK(Σ) = true

⟨S, Σ⟩ ⇓ Σ1 ⟨while p do S, Σ1 ⟩ ⇓ Σ2 ⟨while p do S, Σ⟩ ⇓ Σ2 (W HILE -T RUE)

Goal. Dry-run observational preservation The validation goal is observational refinement: every observable behavior produced by the compiled workflow should be allowed by the source natural-language workflow under the dry-run policy. Formally, we write: Wc ⪯obs Wn := ∀Σn , Σc , Σ′c . Σn ∼obs Σc ∧ ⟨Wc , Σc ⟩ ⇓π Σ′c

(14)

⇒ ∃Σ′n .

(12)

⟨Wn , Σn ⟩ ⇓π Σ′n ∧ Σ′n ∼obs Σ′c .

∧ ∀b ∈ Obs(Σc ). ∃a ∈ Obs(Σn ). a ≃ b. Here, Obs(Σ) is the set of observable artifacts in store Σ, including final outputs, exposed route decisions, and intermediate artifacts whose values affect later observable behavior. The relation a ≃ b means that artifacts a and b are semantically equivalent for the workflow: they may differ in surface wording, but they agree on the decisions, required fields, and constraint-relevant content. The correspondence is intentionally one-directional in the validation goal below: compiled workflow behavior should be allowed by the source workflow, but the compiled workflow need not reproduce every possible interpretation a reader might infer from underspecified natural-language text. The comparison also tracks only observable artifacts. Runtime scratch artifacts, provenance records, and repair feedback may differ as long as they affect observable behavior only through declared artifact interfaces. Dry-run big-step semantics. Because the source workflow is written in natural language, validation uses an LLM policy π as a dry-run simulator. Given workflow W and artifact store Σ, the policy samples a rollout r ∼ π(· | W, Σ). The rollout records a simulated step-by-step execution plan, including the artifacts produced, decisions made, and rationale used by the simulator. A deterministic interpretation function Apply updates the artifact store according to the rollout. We write this as the dry-run big-step judgment: r ∼ π(· | W, Σ) Σ′ = Apply(r, Σ) . ⟨W, Σ⟩ ⇓π Σ′

(W HILE -FALSE)

Fig. 14. Supplemental big-step semantic rules.

Σn ∼obs Σc := ∀a ∈ Obs(Σn ). ∃b ∈ Obs(Σc ). a ≃ b

(I F -FALSE)

Here, Wc ⪯obs Wn means that every observable behavior of the compiled workflow is allowed by the source naturallanguage workflow under the dry-run policy. This is the property approximated by the dry-run validation procedure in the main text. The test-case generator selects representative input stores (Σn , Σc ), including corner cases guided by predicate coverage. The paired dry runs instantiate the two ⇓π judgments. The consistency checker then tests whether the resulting stores satisfy ∼obs . Initial. The base case is the trivial compiled form: an atomic agent step whose instruction is exactly the source naturallanguage fragment. This rule has no premise because it is definitional. agent(Wn ) ⪯obs Wn

(VAL -I NITIAL)

Sequential. The sequential induction step simultaneously checks a source decomposition and composes the two compiled refinements. The source-decomposition premise says that if the original region Wn dry runs from Σ to Σ2 , and the proposed source pieces Wn,1 and Wn,2 dry run through an intermediate state Σ1 , then their final state Σ′2 must match Σ2 : Decomp(Wn , Wn,1 , Wn,2 ) := ∀Σ, Σ1 , Σ2 , Σ′2 .

(13)

⟨Wn , Σ⟩ ⇓π Σ2 ∧ ⟨Wn,1 , Σ⟩ ⇓π Σ1 ∧ ⟨Wn,2 , Σ1 ⟩ ⇓π Σ′2

The judgment ⟨W, Σ⟩ ⇓π Σ′ therefore means that policy π, when asked to dry run workflow W from input artifact store Σ, produces a rollout whose artifact effects yield Σ′ .

⇒ Σ′2 ∼obs Σ2 .

13

(15)

Under this decomposition condition, refinements of the two pieces imply refinement of the original region: Wc,1 ⪯obs Wn,1

Under the case-split and guard-consistency conditions, refinements of the two source cases imply that the compiled branch refines the original source region:

Wc,2 ⪯obs Wn,2

Wc,t ⪯obs Wn,t

Decomp(Wn , Wn,1 , Wn,2 ) Wc,1 ; Wc,2 ⪯obs Wn

(VAL -S EQ)

Splitχ (Wn , Wn,t , Wn,f ) Consistentχ if χ then Wc,t else Wc,f ⪯obs Wn

Proof. Suppose the compiled sequence runs from a compiled input Σc that matches the source input Σn . Formally, assume: (16)

⟨Wc,2 , Σc,1 ⟩ ⇓π Σc,2 . We must construct a source output that matches Σc,2 . By Wc,1 ⪯obs Wn,1 and the first two lines of Equation 16, there exists a source intermediate state Σ1 such that: ⟨Wn,1 , Σn ⟩ ⇓π Σ1

∧

Σ1 ∼obs Σc,1 .

JχK(Σc ) = true,

∧

The guard-consistency premise gives the same route on the source state: JχK(Σn ) = true. (24)

(17)

Σ′2 ∼obs Σc,2 .

By Wc,t ⪯obs Wn,t and Equation 23, there exists a true-case source output Σt such that:

(18)

Finally, let Σ2 be the dry-run result of the original source region: ⟨Wn , Σn ⟩ ⇓π Σ2 . (19)

⟨Wn,t , Σn ⟩ ⇓π Σt

(20)

Σt ∼obs Σ′c .

(25)

By transitivity of ∼obs , Equations 25 and 26 imply Σ′n ∼obs Σ′c . Thus Σ′n is the witness required by the refinement definition. If the compiled guard evaluates to false, the same argument uses Wc,f ⪯obs Wn,f and the false half of Splitχ . □

By transitivity of ∼obs , Equations 18 and 20 imply Σ2 ∼obs Σc,2 . Thus Σ2 is the witness required by the refinement definition in Equation 14. □

Loop. The loop induction step validates that a source region can be represented as repeated execution of a body Wn,b under guard χ. Instead of assuming that the source side already contains a structured while, we define an operational iteration relation. The base case performs zero iterations when the guard is false: JχK(Σ) = false (I TER -Z ERO) Iterχ (Wb , Σ, Σ)

Branch. The branch induction step checks that guard χ partitions the source region into the intended true and false cases. The source case-split premise says that, on states where the guard evaluates to true, every behavior of the true source fragment is allowed by the original source region; on states where the guard evaluates to false, every behavior of the false source fragment is allowed by the original source region: Splitχ (Wn , Wn,t , Wn,f ) := ∀Σ, Σ′ .

The step case performs one body dry run and then continues iterating:

JχK(Σ) = true ∧ ⟨Wn,t , Σ⟩ ⇓π Σ′ b ′ .⟨Wn , Σ⟩ ⇓π Σ b′ ∧ Σ b ′ ∼obs Σ′ ⇒ ∃Σ ∧

∧

Applying the source case-split condition from Equation 21 to Equations 24 and 25 gives a source output Σ′n for the original region: ⟨Wn , Σn ⟩ ⇓π Σ′n ∧ Σ′n ∼obs Σt . (26)

Applying the source-decomposition condition from Equation 15 to Equations 17, 18, and 19 gives: Σ′2 ∼obs Σ2 .

(23)

⟨Wc,t , Σc ⟩ ⇓π Σ′c .

Then, by Wc,2 ⪯obs Wn,2 , Equation 17, and the last line of Equation 16, there exists a state Σ′2 such that: ⟨Wn,2 , Σ1 ⟩ ⇓π Σ′2

(VAL -B RANCH)

Proof. Suppose the compiled branch runs from a compiled input Σc that matches the source input Σn . Consider the truebranch case; the false-branch case is symmetric. Formally, assume: Σn ∼obs Σc ,

Σn ∼obs Σc , ⟨Wc,1 , Σc ⟩ ⇓π Σc,1 ,

Wc,f ⪯obs Wn,f

!

JχK(Σ) = false ∧ ⟨Wn,f , Σ⟩ ⇓π Σ′ b ′ .⟨Wn , Σ⟩ ⇓π Σ b′ ∧ Σ b ′ ∼obs Σ′ ⇒ ∃Σ

JχK(Σ) = true

(21)

Iterχ (Wb , Σ1 , Σ ) Iterχ (Wb , Σ, Σ′ )

! .

JχK(Σn ) = JχK(Σc ).

(I TER -S TEP)

The source loop-decomposition premise then says that every behavior produced by iterating the proposed source body is allowed by the original source region:

The guard-consistency premise says that the guard has the same value on observationally corresponding source and compiled states: Consistentχ := ∀Σn , Σc . Σn ∼obs Σc ⇒

⟨Wb , Σ⟩ ⇓π Σ1 ′

LoopDecompχ (Wn , Wn,b ) := ∀Σ, Σ′ . Iterχ (Wn,b , Σ, Σ′ ) ⇒ b ′ .⟨Wn , Σ⟩ ⇓π Σ b′ ∧ Σ b ′ ∼obs Σ′ . ∃Σ

(22)

14

(27)

Under this loop-decomposition condition, a refinement of the loop body implies that the compiled loop refines the original source region:

Together, VAL -S EQ, VAL -B RANCH, and VAL -L OOP are the induction steps: each checks that a natural-language region can be decomposed by one workflow construct, and each composes recursively validated subworkflows into a refinement of the original region.

Wc,b ⪯obs Wn,b LoopDecompχ (Wn , Wn,b ) Consistentχ while χ do Wc,b ⪯obs Wn

(VAL -L OOP)

A PPENDIX B I MPLEMENTATION D ETAILS

Proof. Suppose the compiled loop runs from a compiled input Σc that matches the source input Σn : Σn ∼obs Σc , ⟨while χ do Wc,b , Σc ⟩ ⇓π Σ′c .

This appendix gives additional details on how the abstract artifact workflow is lowered into the prototype runtime. These details are not part of the core workflow language; they describe the concrete package that executes a compiled workflow. Figure 15 shows the package structure. The orchestrator is a deterministic driver. It invokes one step module at a time, checks the artifacts produced by that step, and transfers control according to the returned route. Each step module implements the shared step interface: exec_step transforms declared input artifacts into declared output artifacts, and check_artifact validates the produced artifact before the orchestrator continues. The runtime provides the shared services used by step modules, including artifact I/O, version metadata, and calls to external agents or benchmark tools. Implementation lowering. Implementation lowering turns each workflow statement into a Python step module. The compiler synthesizes a exec_step function whose signature matches the artifact-driven workflow interface. The function receives only the artifacts declared as inputs and returns the declared output artifacts, a next-step decision, and optional feedback for later steps. The body may combine deterministic code with calls to benchmark tools or subagents, but these calls must pass through the runtime and must write results to the declared artifact paths. The lowering validator checks the generated module against this interface. It checks Python syntax, function signatures, declared output types, and routing targets. It also uses program analysis and model-assisted prompt/entity analysis to detect undeclared reads, such as a subagent prompt that mentions an input field absent from the step interface. Runtime connection. Runtime connection keeps environmentspecific code out of the generated step modules. The runtime writes artifacts, retrieves the latest or earlier versions, records metadata, and exposes concrete artifact paths to step code. It also provides actor-call primitives for subagents, LLM checks, and benchmark tools, so generated modules do not reimplement environment setup or artifact passing. Artifact versioning is the main runtime contract. Each artifact has a stable name, while each committed value has a version and metadata. A step normally reads the latest committed version of its input artifacts and writes a new version for each output artifact. If a retry or loop produces a replacement, the runtime preserves the previous version instead of overwriting it. This lets diagnostics point to the exact artifact that failed a quality gate and lets execution resume from the latest valid artifacts. The runtime also implements the retry protocol. It runs a step, applies the quality gate, and commits the artifacts

(28)

We first prove a stronger intermediate claim by induction on the compiled loop execution: there exists a source-iteration result Σi such that Iterχ (Wn,b , Σn , Σi )

∧

Σi ∼obs Σ′c .

(29)

For the zero-iteration case, the compiled guard is false and the compiled loop returns the input state: JχK(Σc ) = false

∧

Σ′c = Σc .

(30)

By Consistentχ , the source guard is also false: JχK(Σn ) = false.

(31)

Therefore I TER -Z ERO gives Iterχ (Wn,b , Σn , Σn ). Together with Σn ∼obs Σc = Σ′c , this establishes Equation 29 with Σi = Σn . For the step case, the compiled guard is true, the compiled body runs once, and the compiled loop continues from the body output: JχK(Σc ) = true,

⟨Wc,b , Σc ⟩ ⇓π Σc,1 ,

(32)

⟨while χ do Wc,b , Σc,1 ⟩ ⇓π Σ′c . By Consistentχ , the source guard is also true: JχK(Σn ) = true.

(33)

By Wc,b ⪯obs Wn,b , there exists a source body output Σ1 such that: ⟨Wn,b , Σn ⟩ ⇓π Σ1 ∧ Σ1 ∼obs Σc,1 . (34) Applying the induction hypothesis to the remaining compiled loop execution from Σc,1 , using Σ1 ∼obs Σc,1 , gives a state Σi such that: Iterχ (Wn,b , Σ1 , Σi )

∧

Σi ∼obs Σ′c .

(35)

Combining Equations 33, 34, and 35 with I TER -S TEP yields Iterχ (Wn,b , Σn , Σi ), which establishes the strengthened claim. Finally, apply the source loop-decomposition condition in Equation 27 to Equation 29. There exists a source output Σ′n such that: ⟨Wn , Σn ⟩ ⇓π Σ′n

∧

Σ′n ∼obs Σi .

(36)

By transitivity of ∼obs , Equations 29 and 36 imply Σ′n ∼obs Σ′c . Thus Σ′n is the witness required by the refinement definition. □

15

Deterministic Orchestrator Chains steps in a workflow and validates artifacts after each step. def run_workflow(inp: Artifact): # Step S_i a_i = S_i.exec_step(inp) S_i.check_artifact(a_i)

Per-Step Module — shared step interface

Agent Calls & Artifact I/O

# Step S_j a_j = S_j.exec_step(a_i) S_j.check_artifact(a_j) # Step S_k a_k = S_k.exec_step(a_j) S_k.check_artifact(a_k) return a_k

def check_artifact(inp: Artifact) -> None: ... """Validate artifact schema, routes, and invariants."""

def exec_step(inp: Artifact) -> Artifact: ... """Execute the step and produce an output artifact."""

Runtime def call_agent(prompt: str, tools: list[dict], timeout: int): """Invoke an external agent and return its response. """ return subprocess.run( ["codex",...], timeout=timeout).stdout

class Artifact(BaseModel): path: Path metadata: dict def write(self) -> None: ... def read(cls, id: str) -> Artifact: ...

Fig. 15. Implementation structure of a compiled workflow package. The left panel is the deterministic orchestrator, which invokes step modules and checks their artifacts before continuing. The top-right panel shows the generated per-step interface: exec_step produces artifacts, while check_artifact enforces schema and quality gates. The bottom-right panel is the runtime, which implements shared artifact I/O and external agent calls. Arrows denote the orchestrator’s invocation/return path and the step module’s runtime calls.

Original

TABLE IV C OMPILER AND EXECUTOR MODELS USED IN EVALUATION . M ODEL SIZE

100 75

IS REPORTED ONLY WHEN IT IS PUBLIC IN THE MODEL NAME OR PROVIDER - FACING IDENTIFIER .

50 25

Model

Role

Public size

GPT-5.4 Sonnet-4.6 GLM-5 GLM-4.7-Flash GPT-OSS-120B Qwen3-235B-A22B Amazon Nova Micro NVIDIA Nemotron Nano 3

Compiler Compiler Compiler/executor Executor Executor Executor Executor Executor

undisclosed undisclosed 744B total / 40B active 30B total / 3B active 120B 235B total / 22B active undisclosed 30B total / 3.5B active

Model Fail

Compact/20K

Compact/10K Baseline/Text

only when the gate succeeds. If the gate fails, the runtime records the feedback and invokes the step again with the previous artifact path and the constraint feedback. After a step commits, the orchestrator follows the generated controltransfer logic to choose the next step. The runtime-level checks therefore enforce execution invariants: artifacts are versioned consistently, failed attempts are not silently committed, retry feedback is preserved, and every selected route exists in the workflow package.

ARTIC

Fig. 16. Robustness comparison on the Medical domain in χ-Bench. The compiled workflow is evaluated against textual execution under controlled perturbations: transient model request failure, context compaction at 10K tokens, and context compaction at 20K tokens.

repeat from 1 through k. This observed curve is orderingdependent, while the unbiased estimator uses all ten repeats symmetrically. The shaded range is ± one standard deviation under an independent-case plug-in variance estimate. Environment perturbations. Figure 16 summarizes the robustness study discussed in Section V. We select cases where both the text workflow and compiled workflow pass in the original setting, then introduce environment perturbations such as transient model request failure and forced context compaction. This setup measures whether a workflow that appears correct under normal execution remains reliable when the executor faces common operational frictions. Context pressure plots. Figure 17 reports the per-agent token pressure behind the summary in Section V. Each point is one executor turn: for textual execution, this is a turn in the single text-workflow agent; for compiled execution, this is a turn in

A PPENDIX C A DDITIONAL E VALUATION R ESULTS Evaluation models. Table IV lists the compiler and executor models used in the evaluation. Repeated-execution metric. pass^k measures the probability that all k repeated executions of the same instance succeed. In our repeated-run study, each instance is executed ten times. If an instance succeeds in x of these ten executions,

the unbiased estimator for its pass^k value is xk / 10 k ; averaging this quantity over instances gives the solid curves in Figure 11. We also report an observed diagnostic: for each k, we count the fraction of instances that passed every observed

16

Output tokens (log)

Input tokens (log)

Text ARTIC

106

105

10

4

Text ARTIC

103

104 GLM-4.7 Flash

GPT-OSS 120B

Qwen3 235B

GLM-4.7 Flash

(a) Input-token pressure.

GPT-OSS 120B

Qwen3 235B

(b) Output-token pressure.

Fig. 17. Per-agent token pressure for text execution and compiled workflow execution.

TABLE V A BLATION OF OPTIMIZATION , VALIDATION , AND DRY- RUN FEEDBACK ON THE M EDICAL DOMAIN IN χ-B ENCH USING GLM-4.7-F LASH . Dry-run

Performance

✓ ✓ ✓ × ×

✓ ✓ × ✓ ×

✓ × × × ×

88% 80% 72% 48% 0%

6

10.0 7.5

4

5.0 2

2.5 0.0

TABLE VI C OMPILER MODEL COMPARISON ON THE M EDICAL DOMAIN IN χ-B ENCH UNDER GLM-4.7-F LASH EXECUTION . Metric

GLM-5

GPT-5.4

Sonnet-4.6

Performance Workflow steps Artifacts Cost

80.0% 8 8 $2.02

76.0% 9 9 $38.19

84.0% 8 8 $13.27

Cost (USD)

Validation

8

12.5

Time (min)

Optimization

15.0

Time

Cost

0

Fig. 18. First SOP-Bench compilation time and estimated model cost. The left axis reports reconstructed compilation time in minutes, and the right axis reports estimated GLM-5 compilation cost in USD.

the same executor setting. Compilation overhead. Figure 18 summarizes the time and estimated model cost of the first package-producing compilation run for each retained SOP-Bench domain. The estimates use the reconstructed compilation-time and prompt/responsecharacter cost records described in the data manifest; they exclude later targeted repair runs.

one of the subagents invoked by the artifact workflow. The plot therefore measures local context burden rather than total workflow cost. System-component ablation. Table V reports the component ablation study on the Medical domain in χ-Bench using GLM4.7-Flash execution. Removing dry-run feedback reduces pass rate from 88% to 80%; removing validation in addition reduces it to 72%; keeping validation but removing optimization reduces it to 48%; and removing all three leaves no passing cases. The largest drop comes from removing optimization, because optimization formulates where a prompt places too much context or decision burden on the executor. Validation and dry-run feedback catch transformation errors that remain after optimization. Table VI shows that the compiler is also stable across compiler models: GLM-5, GPT-5.4, and Sonnet4.6 produce workflows with 80%, 76%, and 84% pass rates, respectively, compared with 48% for the text workflow under

17

Related documents

• APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows#622623
• Avatar: Toward Autonomous End-to-End Orchestration of Scientific Workflows using LLMs#997381
• The Hallucination Regime: When Coherence Replaces Reality#96642
• ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developer#259549
• GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMs#1000225