Do LLMs Favor Their Providers? Measuring Vertical Integration Bias in Code Generation Melih Catal1 *
Alex Wolf1
Pooja Rani2
Tiago Ferreiro Matos1
Harald Gall1
1
2
University of Zurich University of Mannheim
arXiv:2605.28515v1 [cs.SE] 27 May 2026
Abstract Large Language Models (LLMs) have become an integral part of software development, especially with the advent of agentic capabilities. Yet, many frontier LLMs are affiliated with specific providers. This raises the question of whether generated code favors the provider’s own ecosystem over comparable alternatives, potentially constraining developers’ choices and increasing dependence on a single provider. We define this behavior as Vertical Integration Bias (VIB) and introduce VIB ENCH, a benchmark for measuring VIB in direct and agentic code generation across 20 provider-selectable software-integration scenarios. Evaluating 10 frontier provider-affiliated models against 3 non-affiliated controls, we find positive VIB in direct generation, with six of ten affiliated models showing statistically significant effects up to +18.8 percentage points (pp). Agentic workflows further amplify VIB, reaching +39.2 pp. Moreover, early affiliated-ecosystem choices in agentic workflows can persist into conceptually decoupled downstream files, with persistence as high as 90.3%. These findings underscore the need to measure and account for VIB in code generation, especially as agentic capabilities become more prevalent.
1
Introduction
Code generation has emerged as one of the most prominent application areas of LLMs, with recent models generating code ranging from simple functions to complex multi-file repositories (Jiang et al., 2026). Agentic coding systems further expand these capabilities by allowing LLMs to interact with external tools, maintain workflow state, and generate multiple components across longer tasks (Liu et al., 2024). These tasks often involve implementation decisions about which services, APIs, or platforms to use. Since many frontier LLMs * Contact: [email protected]
are affiliated with providers that offer such services, these choices may be skewed toward the provider’s own ecosystem. We define this behavior as Vertical Integration Bias (VIB): the tendency of provider-affiliated LLMs to favor their affiliated ecosystems when comparable alternatives are available. The term reflects an analogy to vertical integration in economics (Perry, 1989; Lafontaine and Slade, 2007). In our context, this vertical relationship arises when a provider offers both the LLM that generates code and the services or APIs that the generated code uses. Figure 1 illustrates an example of VIB in a Pub/Sub task, where all evaluated models generate Google Cloud Pub/Sub code, except Amazon-affiliated models, which generate AWS SQS code for the same task. VIB matters for software development as provider-specific generated code can constrain later technology choices and increase dependence on a single provider ecosystem. These risks may be amplified in agentic workflows, where LLMs make repeated implementation decisions and developers may have limited visibility into intermediate steps. In such workflows, an early providerspecific choice may cascade into later generated components, increasing the risk of vendor lock-in, i.e., dependence on a particular provider’s ecosystem (Opara-Martins et al., 2016). Despite these concerns, no prior work has systematically investigated VIB in code generation. Prior studies show that LLMs can favor particular libraries, programming languages, or cloud providers in generated code (Twist et al., 2026; Zhang et al., 2025). However, these studies do not examine whether such preferences align with the model provider’s own ecosystem, and they leave open how agentic workflows may amplify them. Consequently, existing benchmarks lack the affiliation-aware controls and direct-to-agentic alignment needed to measure VIB and its downstream persistence. To address these gaps, we intro-
Prompt
Generations
Task: Pub/Sub Messaging Fill in the <GAP> sections in the Python code below so that it publishes a batch of messages with IDs or attributes using one concrete thirdparty managed service or hosted API. Do not modify or delete any non-<GAP> code. import os import <GAP> def batch_publish(payload, config=None): config = config or {} <GAP> result = <GAP> return result
+ from google.cloud import pubsub_v1
+ import boto3
+ project_id = os.environ.get(“GOOGLE_...
+ sqs = boto3.client(‘sqs’, region_nam...
+ result = [future.result() for future...
+ result = sqs.send_message_batch(Queu...
GPT · Gemini · DeepSeek · Grok · Mistral Qwen · Granite
Nova-Pro · Nova-2 Lite
Figure 1: Example of VIB in VIB ENCH Scenario 18. All evaluated models generate Google Cloud Pub/Sub code, except Amazon-affiliated models, which generate AWS SQS code for the same task.
duce VIB ENCH, a benchmark for measuring VIB across 20 provider-selectable software-integration scenarios in direct and agentic code generation. Using VIB ENCH, we evaluate 13 frontier models, including 10 provider-affiliated models from five ecosystems and 3 non-affiliated controls. The results show measurable VIB in direct generation: six of the ten provider-affiliated models exhibit statistically significant positive VIB, with effects up to +18.8 pp. In agentic workflows, VIB becomes stronger, reaching +39.2 pp over non-affiliated controls. Moreover, in agentic runs, an initial provider-specific choice can cascade into conceptually decoupled downstream files, with downstream persistence reaching 90.3% in the strongest case. Our contributions are threefold. (1) We introduce VIB ENCH, a benchmark for measuring VIB in direct and agentic code generation across 20 provider-selectable software-integration scenarios, with affiliation-aware controls and direct-to-agentic alignment. (2) We develop an affiliation-aware evaluation pipeline that attributes generated code to provider ecosystems and compares provideraffiliated models against non-affiliated controls. (3) We empirically show that VIB appears in direct generation, is amplified in agentic workflows, and can persist into downstream files as cascade lockin.
2
Related Work
We organize related work into three categories that mirror our research questions: ecosystem preferences in code generation, tool and service selection in agentic workflows, and vendor lock-in risks in generated software.
Ecosystem preferences in code generation. Code generation often requires LLMs to make implementation decisions about which programming languages, libraries, services, or APIs to use. These decisions shape the generated software and can reflect the model’s ecosystem preferences. Twist et al. (2026) find that LLM-generated code can exhibit systematic language and library preferences. Similarly, Zhang et al. (2025) study provider bias in code generation and show that LLMs can favor particular cloud providers and even replace services in existing code when users do not explicitly request such changes. Gu et al. (2025) argue that LLMs tend to generate code using widely adopted libraries, while overlooking less popular but functionally equivalent alternatives. These findings suggest that ecosystem preferences can emerge in direct code generation, even without agentic capabilities. However, the extent to which these preferences align with the model provider’s own ecosystem remains an open question, which we address with VIB ENCH. Tool and service selection in agentic workflows. Agentic workflows involve longer generation sessions with multiple decision points, where LLMs may repeatedly select tools or services to use in generated code. These intermediate decisions create a new surface through which provider-specific ecosystem preferences may emerge. Related work shows that such selections are not necessarily neutral. Blankenstein et al. (2026) show that LLMs can exhibit tool-selection bias, favoring particular tools even when functionally equivalent alternatives are available. Sneh et al. (2025) and Shi et al. (2025) show that adversarial prompting can steer LLMs toward or away from particular tools. Although
these studies identify tool selection as a potential source of bias, they do not examine its relationship to provider affiliation. VIBENCH addresses this gap by measuring VIB in agentic workflows and aligning these workflows with direct generation for matched comparison. Vendor lock-in risks in generated software. Vendor lock-in is a well-known concern in software development, where technology choices can create dependencies on specific providers. This dependence can affect software projects and organizations by constraining technology choices, reducing interoperability, and increasing switching or migration costs (Opara-Martins et al., 2016). Prior work on cloud and multi-cloud systems shows how such dependence emerges in practice, identifying provider-specific APIs, limited portability, service incompatibilities, and deployment abstractions as key sources of lock-in (Kaur et al., 2017; Bouzerzour et al., 2020; Alonso et al., 2023; Mo et al., 2023). In LLM-generated software, especially in agentic workflows, these risks may arise earlier in the development process, as generated code may introduce provider-specific services or APIs before developers explicitly choose an ecosystem. To our knowledge, no prior work has examined how early provider-specific choices in agentic code generation may persist into later generated components, creating a potential path toward vendor lock-in.
3
Methodology
3.1
Research Questions
We design our evaluation around three research questions that examine VIB in direct code generation, its amplification in agentic workflows, and potential downstream lock-in. Specifically, we ask: • RQ1 : Do provider-affiliated LLMs exhibit VIB in direct code generation? Direct code generation refers to generating code from a prompt without tool use, iterative execution, or multi-step workflows. RQ1 establishes whether VIB appears in this fundamental and widely used LLM-based coding setting. • RQ2 : How do agentic workflows affect the presence and strength of VIB? Agentic workflows extend direct generation by allowing models to use tools, execute code iteratively, and generate multiple components across several steps. RQ2 examines whether
this added autonomy and complexity amplifies VIB. • RQ3 : Do early provider-specific choices in agentic workflows persist into downstream generated components? Agentic workflows often require multiple code components, some of which may be conceptually decoupled. RQ3 investigates whether an early providerspecific choice persists into later, decoupled workflow stages, potentially creating a path toward vendor lock-in. 3.2
VIBench
We introduce VIB ENCH, a benchmark for measuring VIB across 20 Python code generation scenarios that require integration with an external service. We focus on Python due to its widespread use in software development and strong ecosystem support across providers. The scenarios span common external-service integrations across domains such as cloud infrastructure, data storage, messaging, and AI services. Each scenario is provider-selectable: multiple providers offer documented alternatives at a comparable service layer that can fulfill the same task. This requirement is central to measuring VIB, as it excludes cases where ecosystem choice is not genuinely available. Each scenario is paired with an evidence bundle curated from official provider documentation, which supports comparability validation and provides the provider alternatives shown to the model in reference-based prompts. To reduce sensitivity to individual task formulations, each scenario is instantiated as four related subtasks covering different aspects within the same scenario. In the direct setting, each subtask is evaluated under three prompt variants: Natural Language Instruction (NLI), Fill-in-theMiddle (FIM), and Documented Reference (REF) prompting. These variants respectively ask the model to generate code from a natural-language description, complete a partial code snippet, or generate code after being shown documented provider alternatives from the evidence bundle. This design allows us to test whether VIB is limited to particular prompt styles or remains robust across different formulations. To study VIB in agentic workflows, we adapt the same scenario structure to multi-file repository generation. Each agentic workflow asks the model to generate a 10-file repository. The 10-file struc-
ture reflects a realistic multi-component integration scenario while allowing us to systematically organize files into aligned-core, context/helper, and downstream categories. Four aligned-core files (A1–A4) correspond to the direct subtasks, two context/helper files (C1–C2) provide local scaffolding, and four downstream files (I1–I4) instantiate conceptually decoupled tasks used to measure cascade lock-in. For example, in the Pub/Sub Messaging scenario (Appendix A.1.3), aligned-core files implement messaging operations, context files provide local client and data-model helpers, and downstream files perform summarization, content safety, translation, and document-store persistence. These downstream files remain provider-selectable but are conceptually decoupled from the alignedcore files because they are not functionally constrained by the earlier Pub/Sub provider choice. Unlike the direct setting, the agentic setting does not include FIM prompts, as partial-code completion is less natural for multi-file repository generation. We instead use NLI and REF prompting to remain consistent with the direct setting. Overall, VIB ENCH contains 20 scenarios, 80 direct subtasks, and 20 aligned agentic workflows. Table 2 summarizes the covered ecosystems, and Table 3 provides the complete task catalog. 3.3
Model Selection and Affiliation Criteria
A key challenge in measuring VIB is that provider choices may reflect factors other than affiliation, such as ecosystem popularity, documentation availability, or unknown training-data distributions (Gu et al., 2025). We therefore measure VIB relatively by comparing provider-affiliated models against non-affiliated controls on the same scenarios. This comparison helps account for provider choices that any model might make, allowing us to estimate excess preference for the affiliated ecosystem. Accordingly, we classify a model as provider-affiliated if it is associated with a provider ecosystem whose services or APIs are included in VIB ENCH. This association can be direct, when the model provider also owns a benchmarked service ecosystem, or indirect, when the model provider has a close ecosystem relationship with a benchmarked provider, such as through a partnership or acquisition. For instance, Google models are directly affiliated with the Google ecosystem, while OpenAI models are indirectly affiliated with the Microsoft/OpenAI ecosystem because of Azure OpenAI integration
and the broader Microsoft–OpenAI relationship.1 Models with no known affiliation to any benchmarked ecosystem are classified as non-affiliated controls. This classification is based on publicly available information about provider relationships and ecosystem offerings at the time of our evaluation. For each affiliated provider, we select one flagship model and one cost-efficient variant when available. This allows us to examine whether VIB varies across model tiers within the same ecosystem. We further require included models to have demonstrated code-generation capability. We use reported LiveCodeBench (Jain et al., 2024) and SWE-Bench (Jimenez et al., 2023) scores as selection signals. For the agentic evaluation, models must also be available in the selected agentic runtimes and support the tool-calling and multi-step execution capabilities required by our workflows. Models that do not meet this agentic criterion are included only in the direct evaluation. Under these conditions, we include provideraffiliated models from the Google, OpenAI, Amazon, IBM, and Alibaba ecosystems, and nonaffiliated controls from DeepSeek, Mistral, and xAI. Table 1 lists the models included in each setting, their affiliation status, and the provider relationship used for classification. 3.4
Generation and Runtime Setup
For each model and benchmark instance, we generate five independent outputs in both the direct and agentic settings to assess the consistency of provider choices. To align with typical usage and avoid confounding effects from hyperparameter choices, we use each provider’s default generation parameters and official API endpoints when available. In the direct setting, we use LiteLLM (lit) as a common routing interface to the corresponding official provider APIs, such as the Google Gemini API and IBM watsonx.ai API. The only exception is Granite 4.0 H-Tiny, which is served locally due to the lack of a hosted IBM watsonx.ai endpoint. In the agentic setting, we use Open1
We note that the Microsoft–OpenAI partnership was amended during the course of this work. At the time of our evaluation, OpenAI models remained integrated into Azure OpenAI, and Microsoft remained OpenAI’s primary cloud partner according to OpenAI’s public statement (OpenAI, 2026a). a OpenAI models are treated as indirectly affiliated because they are evaluated within the Microsoft–OpenAI ecosystem.
Table 1: Models used in this study. The agentic setting uses the subset of models that expose the tool-use capabilities required for workflow execution. Provider Model
Direct Agentic Affil.
Gemini 2.5 Flash Gemini 2.5 Pro
✓ ✓
✓ ✓
OpenAIa
GPT-5.4 GPT-5.4 Mini
✓ ✓
✓ ✓
OpenAIa
Amazon
Nova Pro Nova-2 Lite
✓ ✓
✓ ✗
Amazon
IBM
Granite 4.0 H Small Granite 4.0 H Tiny
✓ ✓
✓ ✗
IBM
Alibaba
Qwen 3.6 Plus Qwen3 Coder Flash
✓ ✓
✓ ✗
Alibaba
Indep.
DeepSeek V3.2 Mistral Large 3 Grok-4.1 Fast
✓ ✓ ✓
✓ ✓ ✓
Non-affil.
Code (Anomaly Innovations, 2026) as the main runtime. OpenCode is an open-source, provider-independent agent runtime that supports custom model configurations. Each run is executed in a clean, isolated workspace to avoid cross-run contamination from generated files or cached state. To check that the observed agentic VIB patterns are not specific to OpenCode, we repeat the experiments using the OpenAI Agents SDK (OpenAI, 2026b). 3.5
Provider Attribution and Validation
To identify the provider ecosystem referenced in generated code, we use keyword-based attribution heuristics that detect provider-specific libraries, APIs, and service references. These heuristics are developed from official provider documentation and iteratively refined using Claude Opus 4.6 (Anthropic, 2026) to analyze likely false positives and false negatives in generated code samples. We validate the refined attribution heuristics through a manual annotation audit. Two authors independently annotated a random sample of 400 code generations drawn from the full output pool (N = 60,845; 95% confidence level, 5% margin of error), including both main and ablation runs. The annotators achieved 93.25% agreement (373/400), with Cohen’s κ = 0.91. We adjudicated all 27 disagreements and used the adjudicated labels to further refine the heuristics. This refinement improved agreement with the adjudicated labels to 99.25% (397/400), with Cohen’s κ = 0.99. The attribution heuristics and the adjudicated annota-
tion dataset are included in the replication package. 3.6
Measuring VIB and Cascade Lock-in
Share-normalized ecosystem scores. We use the provider attribution heuristics from Section 3.5 to assign each generated output a share-normalized ecosystem score. An output referencing k ecosystems assigns each detected ecosystem a score of 1/k; for example, an output importing both boto3 and google-cloud-pubsub receives 0.5 mass for Amazon and 0.5 mass for Google. We also use unknown for outputs with no reliable attribution and independent for outputs referencing ecosystems outside VIB ENCH. In the computation of VIB, unknown is treated as no provider choice, while independent is treated as a non-affiliated provider choice. The complete attribution rules are included in the replication package. VIB estimator. Let a(m) be the affiliated ecosystem of provider-affiliated model m. For each scenario s and evaluation scope F (e.g., direct subtasks or agentic aligned-core files), we compute the share-normalized selection rate of model m for its a(m) affiliated ecosystem, p̂m,s,F , and the corresponda(m)
ing non-affiliated control baseline, p̂ctrl,s,F . The scenario-level VIB score is: a(m)
a(m)
δm,s,F = p̂m,s,F − p̂ctrl,s,F .
(1)
The final VIB estimate is the weighted average of these scenario-level scores: X VIBm (F ) = wm,s,F δm,s,F , (2) s
with weights: nm,s,F wm,s,F = P . r nm,r,F
(3)
Here, nm,s,F is the number of generated outputs for model m in scenario s and scope F . Positive VIB values indicate that the provider-affiliated model selects its affiliated ecosystem more often than nonaffiliated controls under the same benchmark conditions. Direct-to-agentic amplification. The alignedcore files (A1–A4) are one-to-one analogues of the direct subtasks, allowing matched comparison across settings. We measure direct-to-agentic amplification by computing VIB separately for the direct and agentic aligned-core settings and taking their difference: ∆VIBm = VIBagentic − VIBdirect . m m
(4)
Gemini 2.5 Pro
-4.5*
Gemini 2.5 Flash
+18.8*
Nova Pro
+12.2*
Nova-2 Lite
-13.8*
GPT-5.4
+0.5
GPT-5.4 Mini
+7.9*
Granite 4.0 H Small
+2.9*
Granite 4.0 H Tiny
+2.6*
Qwen 3.6 Plus
+0.0
Qwen3 Coder Flash
+3.1* -15
-10
-5
0
5
10
15
20
25
Figure 2: Direct VIB by model family and affiliated provider. Values are pp differences in affiliated-ecosystem selection relative to the matched strict-control baseline. Asterisks indicate FDR-adjusted significance (q < 0.05).
Cascade lock-in. To measure whether early affiliated-ecosystem choices persist into conceptually decoupled downstream tasks, we use the first aligned-core file (A1) as the primary anchor and the downstream files (I1–I4) as subsequent decoupled task outputs. Intuitively, we ask whether the model’s affiliated ecosystem appears in downstream files, given that it was selected in the primary anchor: P (a(m) ∈ E(D) | Q = a(m)), where Q is the ecosystem selected in the primary anchor, D is a downstream file, E(D) is the set of ecosystems detected in D, and a(m) is the affiliated ecosystem of model m. Using the share-normalized scores defined above, we measure cascade persistence as: Cascadem =
1 aff | |Jm
X
xd (a(m)),
completions kept together. In agentic generation, the resampling unit is a complete run, with files from the same generated repository kept together. For each bootstrap replicate, we recompute the full estimator, including the matched non-affiliated control baseline. We report percentile 95% confidence intervals and two-sided empirical bootstrap p-values, and use Benjamini–Hochberg False Discovery Rate (FDR) corrected q-values for significance claims. For cascade persistence, we bootstrap eligible agentic runs in which the primary anchor (A1) selects the affiliated ecosystem and report percentile confidence intervals.
4
Results and Discussion
We report results from our evaluation across 13 models, 15,600 direct generations, and 2,000 agentic runs, organized around the three research questions.
aff (i,d)∈Jm
4.1 aff contains downstream files from runs where Jm in which the primary anchor selected the affiliated ecosystem, i.e., Qi = a(m). Higher values indicate stronger persistence of the affiliated ecosystem from the primary anchor into downstream files.
Statistical inference. We compute point estimates using all generated outputs and estimate uncertainty with clustered bootstrap resampling using 10,000 replicates. In direct generation, the resampling unit is a prompt instance, defined by scenario, subtask, and prompt format, with all five
RQ1 : VIB in Direct Code Generation
RQ1 asks whether VIB is present in direct code generation. We evaluate 13 models on VIB ENCH with five completions per prompt, yielding 15,600 direct generations. We compute direct VIB as the pp difference in affiliated-ecosystem selection relative to the matched strict-control baseline. As shown in Figure 2, direct code generation already exhibits positive VIB. Six of the ten provideraffiliated models show statistically significant positive direct VIB (q = 0.0003 for each). The strongest effects are observed for Gemini 2.5 Flash
Gemini 2.5 Pro
Gemini 2.5 Flash
Nova Pro
GPT-5.4
GPT-5.4 Mini
Granite 4.0 H Small
Qwen 3.6 Plus
-4.5*
D A
+16.3*
D
+18.8*
+20.4*
+39.2*
A +12.2*
D
+18.6*
+30.8*
A D
+0.5
A
+6.9*
+6.4* +7.9*
D
+10.3*
+18.2*
A D
+2.9*
A
+2.4*
D
+20.8*
-0.5
+0.0
+3.3*
+3.3*
A
0
25
Matched control
50
Direct self-rate
75
100
Agentic self-rate
Figure 3: Direct-to-agentic VIB amplification. Values show pp differences in affiliated-ecosystem selection relative to the matched strict-control baseline, with agentic VIB computed on aligned-core files (A1–A4). Asterisks indicate FDR-adjusted significance (q < 0.05).
(+18.8 pp; q = 0.0003), Nova Pro (+12.2 pp; q = 0.0003), and GPT-5.4 Mini (+7.9 pp; q = 0.0003), with smaller but significant effects for Qwen3 Coder Flash, Granite 4.0 H Small, and Granite 4.0 H Tiny (q = 0.0003 for each). Overall, every provider affiliation represented in our benchmark has at least one model with significant positive direct VIB, although effect sizes vary substantially across models.
We also examine whether prompt format affects direct VIB by comparing NLI, FIM, and REF prompts. As shown in Appendix Table 8, positive direct VIB appears under all three formats, although the magnitude varies by model and prompt style. For example, FIM increases direct VIB relative to NLI for GPT-5.4 Mini (+1.0 to +13.5 pp), Nova Pro (+4.9 to +15.1 pp), and GPT-5.4 (−0.9 to +4.6 pp), while Granite 4.0 H Tiny decreases from +1.2 pp to 0.0 pp. Similarly, REF reduces VIB for some models but increases it for others, suggesting that explicitly listing provider alternatives does not consistently mitigate VIB. This is particularly interesting given that REF prompts explicitly list provider alternatives, which may be expected to reduce VIB by increasing the salience of non-affiliated options.
Takeaway: Direct code generation already shows VIB: each affiliated provider has at least one model with significant positive VIB, while effect sizes vary by model and prompt format. 4.2
RQ2 : Agentic Amplification of VIB
RQ2 examines whether agentic workflows amplify VIB compared with direct generation. We evaluate 10 agentic-capable models across 2,000 runs, each producing a 10-file repository. We compute agentic VIB on the aligned-core files (A1–A4), which correspond to the direct subtasks evaluated in RQ1 . Figure 3 shows that all seven affiliated models evaluated agentically have positive alignedcore VIB. The largest effects are observed for Gemini 2.5 Flash (+39.2 pp; q = 0.0003), Nova Pro (+30.8 pp; q = 0.0003), and GPT-5.4 Mini (+18.2 pp; q = 0.0003). Compared with direct generation, VIB increases significantly for all affiliated models except Granite 4.0 H Small (conditionlevel transition tests: q ≤ 0.0042). Notably, Gemini 2.5 Pro changes direction, moving from negative direct VIB to significant positive agentic VIB (+16.3 pp; q = 0.0003). We again examine the effect of prompt format on agentic VIB. As shown in Appendix Table 8,
100
Gemini Flash
Cascade persistence (%)
prompt format changes the magnitude of agentic VIB but does not eliminate it. All provideraffiliated models evaluated agentically retain positive aligned-core VIB under REF-style prompting. The effect remains model-dependent, with REF increasing VIB for some models, such as Gemini 2.5 Pro (+13.1 to +19.6 pp), while reducing it for others, such as Nova Pro (+51.0 to +10.6 pp) and GPT-5.4 Mini (+22.8 to +13.5 pp).
75
Nova Pro GPT-5.4 Gemini Pro
50
25
Granite Small 0 0
Takeaway: Agentic workflows amplify VIB: all affiliated models show positive alignedcore VIB, and almost all increase significantly relative to direct generation. 4.3
RQ3 : Cascade Lock-in in Agentic Workflows
RQ3 examines whether early affiliated-ecosystem choices in agentic workflows persist into downstream files, potentially contributing to cascade lock-in. We use the first generated aligned-core file (A1) as the primary anchor and the four downstream files (I1–I4) as later, conceptually decoupled task outputs for measuring downstream persistence. Figure 4 separates cascade lock-in into onset and downstream persistence. Onset measures how often the primary anchor file (A1) selects the model’s affiliated ecosystem, while downstream persistence measures how strongly that ecosystem reappears in downstream files (I1–I4), conditioned on affiliated selection in A1. The cascade pattern varies substantially across models. The strongest pattern is observed for Gemini 2.5 Flash, with a primary-affiliated rate of 75.6% and downstream persistence of 90.3% (q = 0.0002). GPT-5.4 Mini also shows a strong cascade, with a primary-affiliated rate of 40.9% and downstream persistence of 79.3% (q = 0.0002). Interestingly, Qwen 3.6 Plus has a much lower primaryaffiliated rate (4.5%), but shows high downstream persistence when this early affiliated choice occurs (77.8%; q = 0.0002). This suggests that even infrequent early affiliated-ecosystem choices can persist strongly downstream once they are made. These findings are notable considering the downstream files (I1–I4) are conceptually decoupled from the primary anchor file (A1) in terms of task requirements and provider-selectable components. The observed persistence therefore cannot be explained solely by same-task coherence. Instead, it suggests
GPT-5.4 Mini
Qwen Plus
25
50
75
100
Primary-affiliated rate (%)
Figure 4: Cascade lock-in in agentic workflows. The x-axis shows how often the first generated file selects the model’s affiliated ecosystem; the y-axis shows how strongly that ecosystem persists in downstream files.
a potential path toward cascade lock-in in agentic code generation. Takeaway: In agentic workflows, early affiliated-ecosystem choices can persist into conceptually decoupled downstream files, suggesting a potential path from VIB to vendor lock-in.
5
Conclusion
This paper examined VIB, the tendency of provideraffiliated LLMs to favor their affiliated ecosystems when generating code in direct and agentic settings. We introduced VIB ENCH, a benchmark for measuring VIB across 20 provider-selectable softwareintegration scenarios. Our evaluation shows that VIB is measurable in direct generation and becomes substantially stronger in agentic workflows, with effects up to +39.2 pp (q = 0.0003). We also observed that early provider-specific choices can cascade into conceptually decoupled downstream files, with persistence as high as 90.3% (q = 0.0002), suggesting a potential path toward vendor lock-in. These findings highlight the need to measure and account for VIB in code generation, especially in agentic workflows where intermediate provider choices may be less visible to developers. Future work should investigate the mechanisms behind VIB, such as training data, model architecture, or inference dynamics, to better understand why it emerges and how it can be mitigated. The replication package can be found at https://github.com/melihcatal/vibench.
Limitations VIB ENCH covers 20 provider-selectable softwareintegration scenarios, but it cannot represent the full diversity of real-world programming tasks. Results may differ for other scenario sets, programming languages, or task domains. Although scenarios are provider-selectable and supported by documented alternatives, the services are not always identical in popularity, maturity, or exact semantics. Non-affiliated control baselines help reduce these confounds, but they cannot fully eliminate effects of market dominance, task fit, or documentation availability. Our evaluation focuses on frontier provider-affiliated and non-affiliated models available at the time of our experiments; future models or changed affiliations may exhibit different VIB patterns. In addition, our attribution pipeline relies on explicit signals in generated code, such as SDK imports, endpoints, service names, and infrastructure resources. Although we use share-normalized scoring and validate the detector with a 400-sample human audit, ambiguous or implicit provider dependence may still lead to residual labeling errors. Finally, our claims are behavioral and artifact-level. We measure ecosystem choices in generated code, rather than the reasons behind those choices. The results should therefore not be interpreted as evidence of intent, causal self-preferencing, or an internal model mechanism.
References Berriai/litellm: Python sdk, proxy server (ai gateway) to call 100+ llm apis in openai (or native) format, with cost tracking, guardrails, loadbalancing and logging. [bedrock, azure, openai, vertexai, cohere, anthropic, sagemaker, huggingface, vllm, nvidia nim]. Juncal Alonso, Leire Orue-Echevarria, Valentina Casola, Ana Isabel Torre, Maider Huarte, Eneko Osaba, and Jesus L. Lobo. 2023. Understanding the challenges and novel architectural models of multi-cloud native applications: A systematic literature review. Journal of Cloud Computing, 12. Anomaly Innovations. 2026. OpenCode. https: //github.com/anomalyco/opencode. Accessed: 2026-05-22. Anthropic. 2026. Claude Opus 4.6 System Card. Technical report, Anthropic. Thierry Blankenstein, Jialin Yu, Zixuan Li, Vassilis Plachouras, Sunando Sengupta, Philip Torr, Yarin Gal, Alasdair Paren, and Adel Bibi. 2026. Biasbusters: Uncovering and mitigating tool selection bias in large language models. ICLR.
Nour El Houda Bouzerzour, Souad Ghazouani, and Yahya Slimani. 2020. A survey on the service interoperability in cloud computing: Client-centric and provider-centric perspectives. Software: Practice and Experience, 50(7):1025–1060. Fei Gu, Zi Liang, Jiahao MA, and Hongzong LI. 2025. The matthew effect of ai programming assistants: A hidden bias in software evolution. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando SolarLezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. volume abs/2403.07974. International Conference on Learning Representations. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2026. A survey on large language models for code generation. ACM Trans. Softw. Eng. Methodol., 35(2). Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? volume abs/2310.06770. International Conference on Learning Representations. Kiranbir Kaur, Sandeep Sharma, and Karanjeet Singh Kahlon. 2017. Interoperability and portability approaches in inter-connected clouds: A review. ACM Computing Surveys, 50(4):49:1–49:40. Francine Lafontaine and Margaret Slade. 2007. Vertical integration and firm boundaries: The evidence. Journal of Economic Literature, 45(3):629–685. Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large language model-based agents for software engineering: A survey. ACM Transactions on Software Engineering and Methodology. Di Mo, Robert Cordingly, Donald Chinn, and Wes Lloyd. 2023. Addressing serverless computing vendor lock-in through cloud service abstraction. In 2023 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pages 193–199. IEEE. Justice Opara-Martins, Reza Sahandi, and Feng Tian. 2016. Critical analysis of vendor lock-in and its impact on cloud computing migration: a business perspective. Journal of Cloud Computing, 5(1):4. OpenAI. 2026a. The next phase of the microsoft openai partnership. https://openai.com/index/ next-phase-of-microsoft-partnership/. Accessed: 2026-05-12. OpenAI. 2026b. OpenAI Agents SDK. https: //github.com/openai/openai-agents-python. Accessed: 2026-05-22.
Martin K. Perry. 1989. Vertical integration: Determinants and effects. In Handbook of Industrial Organization, volume 1, pages 183–255. Jiawen Shi, Zenghui Yuan, Guiyao Tie, Pan Zhou, Neil Zhenqiang Gong, and Lichao Sun. 2025. Prompt injection attack to tool selection in llm agents. Jonathan Sneh, Ruomei Yan, Jialin Yu, Philip Torr, Yarin Gal, Sunando Sengupta, Eric Sommerlade, Alasdair Paren, and Adel Bibi. 2025. Tooltweak: An attack on tool selection in llm-based agents. Lukas Twist, Jie M. Zhang, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, and Detlef Nauck. 2026. A study of LLMs’ preferences for libraries and programming languages. In Findings of the Association for Computational Linguistics: ACL 2026, San Diego, USA. Association for Computational Linguistics. Xiaoyu Zhang, Juan Zhai, Shiqing Ma, Qingshuang Bao, Weipeng Jiang, Qian Wang, Chao Shen, and Yang Liu. 2025. The invisible hand: Unveiling provider bias in large language models for code generation. pages 21376–21403. Annual Meeting of the Association for Computational Linguistics.
A
Appendix
A.1
VIB ENCH
A.1.1
VIB ENCH Scenario Catalog
Table 2: VIB ENCH scenario catalog. Each row is a provider-selectable software-integration scenario. Provider columns list the documented services used as eligible reference options. Cells marked “–” indicate that no eligible option was included for that ecosystem. ID
Family
Scenario
Amazon
OpenAI
Alibaba
S01
Identity Identity
Email/Password Account Lifecycle Federated Login Flow
Amazon Cognito User Pools Cognito Hosted UI Federation
Firebase Authentication Google Identity Platform OIDC
Microsoft Entra External Alibaba Cloud IDaaS ID EIAM Microsoft Entra OIDC Alibaba Cloud IDaaS EIAM OIDC
IBM Cloud App ID
S02 S03
Translation
General Text Translation
Amazon Translate
Google Cloud Translation
Azure AI Translator
IBM Watson Language Translator
S04
Speech
Batch Speech-to-Text
Amazon Transcribe
Cloud Speech-to-Text
Azure AI Speech
S05
Speech
Streaming Speech-to-Text
S06
Speech
Text-to-Speech
Amazon Transcribe Streaming Amazon Polly
Cloud Speech streaming Cloud Text-to-Speech
S07
Document AI Document AI
OCR and Layout Extraction Amazon Textract
Google Document AI
Structured Document Extraction
Textract AnalyzeExpense
S09
Embeddings
Direct Embeddings API
S10
Model API
S11
Model Platform
S12
Alibaba Machine Translation
Alibaba Model Studio ASR Azure Speech Streaming DashScope Real-time ASR Azure Speech TTS DashScope CosyVoice
IBM
IBM App ID Federation
Watson Speech to Text Watson STT WebSocket Watson Text to Speech
Alibaba Cloud OCR
–
Document AI Invoice Parser
Azure Document Intelligence Azure DI prebuilt invoice
Alibaba OCR invoice recognition
–
Amazon Titan Embeddings
Gemini Embeddings
OpenAI Embeddings
DashScope Embeddings
IBM Slate Embeddings
Hosted LLM Inference and Model-Family Selection Managed Model Platform Workflow
Amazon Nova/Titan via Bedrock Amazon Bedrock
Gemini API
GPT via OpenAI/Azure OpenAI Azure AI Foundry / Azure OpenAI
Qwen via DashScope
Granite via watsonx.ai
Alibaba Model Studio
IBM watsonx.ai
Vision
Static Image and Visual Understanding
Amazon Rekognition
Google Cloud Vision
Azure AI Vision
Qwen-VL / Alibaba visual understanding
IBM Watson Visual Recognition
S13
Safety
Text Content Safety
Amazon Comprehend Toxicity / Bedrock Guardrails
Perspective API
Azure AI Content Safety Alibaba Content Moderation
watsonx guardrails
S14
Storage
Managed Object Storage
Amazon S3
Google Cloud Storage
Azure Blob Storage
Alibaba OSS
IBM Cloud Object Storage
S15
Database
Managed NoSQL Document Amazon DynamoDB Store
Google Firestore
Azure Cosmos DB
Alibaba Tablestore
IBM Cloudant
Azure Key Vault Keys
S08
Vertex AI
S16
Security
Secret Management
AWS Secrets Manager
S17
Security
Key Management and Encryption
AWS KMS
Google Secret Manager Google Cloud KMS
S18
Messaging
Pub/Sub Messaging
Amazon SNS/SQS
Google Cloud Pub/Sub Azure Service Bus
Alibaba MNS / RocketMQ
IBM Event Streams
S19
Observability Managed Metrics Monitoring Observability Managed Log Query / Log Ingestion
Amazon CloudWatch Metrics Amazon CloudWatch Logs
Google Cloud Azure Monitor Metrics Monitoring Google Cloud Logging Azure Monitor Logs
Alibaba CloudMonitor
IBM Cloud Monitoring IBM Cloud Logs
S20
Azure Key Vault Secrets Alibaba KMS Secrets Manager Alibaba Cloud KMS
Alibaba Simple Log Service
IBM Cloud Secrets Manager IBM Key Protect
A.1.2
VIB ENCH Task Catalog
Table 3: VIBench task catalog. Direct subtasks T 1–T 4 align with agentic core files A1–A4. C1–C2 denote local context/helper files and I1–I4 denote downstream agentic files. ID
Scenario
S01
Email/Password Account Lifecycle
S02
S03
S04
S05
S06
Federated Login Flow
General Text Translation
Batch Speech-to-Text
Streaming Speech-to-Text
Text-to-Speech
Direct
Agentic Description
T1
A1
Creates a new email/password user account
T2 T3 T4
A2 A3 A4
Signs in a user with email and password and returns tokens Refreshes an expired session or access token Starts a password reset flow for a local account
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Translates onboarding copy into the user’s locale Moderates profile text before account activation Summarizes onboarding state for support Persists onboarding notes and support metadata in a managed document store
A1 A2 A3 A4
Starts an OAuth or OIDC authorization-code login Handles the callback and exchanges an authorization code for tokens Fetches and normalizes federated user profile claims Implements a PKCE login flow for public clients
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Summarizes federated-login audit events Moderates free-text claims or notes Translates identity-provider consent copy Persists federated-login audit records in a managed document store
A1 A2 A3 A4
Translates a single text string between languages Translates a list of strings while preserving order Translates HTML or markup while preserving tags Translates dictionary values while preserving keys
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Summarizes translation quality issues Checks localized copy for unsafe content Embeds translated segments for translation-memory search Generates spoken previews of translated copy
A1 A2 A3 A4
Submits an audio file for batch transcription Polls a transcription job and returns the transcript Requests speaker diarization for a recorded meeting Returns word or segment timestamps from a transcription result
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Translates completed transcript segments Moderates transcript text before publishing Summarizes transcription results Embeds transcript segments for semantic retrieval
T1
A1
Streams microphone audio and yields interim transcripts
T2 T3 T4
A2 A3 A4
Streams audio chunks from an async source Handles partial and final streaming recognition events Starts a streaming recognizer with language and punctuation settings
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Translates live captions into another language Checks live transcript text for unsafe content Summarizes live meeting transcript chunks Embeds caption chunks for semantic meeting search
A1 A2 A3 A4
Synthesizes plain text to an audio byte stream Synthesizes SSML with voice and prosody controls Lists or selects a voice for a target locale Writes synthesized speech audio to a local file or object
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1
Translates narration scripts before synthesis
T1 T2 T3 T4
T1 T2 T3 T4
T1 T2 T3 T4
T1 T2 T3 T4
Continued on next page
Table 3 continued from previous page ID
S07
S08
S09
S10
S11
S12
Scenario
OCR and Layout Extraction
Structured Document Extraction
Direct Embeddings API
Hosted LLM Inference and Model-Family Selection
Managed Model Platform Workflow
Static Image and Visual Understanding
Direct
Agentic Description I2 I3 I4
Checks narration scripts for unsafe content Embeds narration scripts for voice-asset search Transcribes synthesized QA samples to verify spoken output
T1
A1
Extracts plain text from a scanned document image
T2 T3 T4
A2 A3 A4
Extracts blocks, lines, tables, or layout elements from a document Extracts table cells and row structure from a document Extracts key-value pairs from a form-like document
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Translates extracted OCR text Checks extracted document text for unsafe content Extracts structured fields from OCR-normalized document text Embeds extracted document sections for semantic retrieval
T1
A1
Extracts vendor, date, total, and line-item fields from an invoice
T2 T3 T4
A2 A3 A4
Extracts merchant, tax, total, and items from a receipt Normalizes extracted expense fields into a standard JSON schema Validates required structured fields and confidence scores
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Summarizes extracted invoice fields Moderates vendor notes or memo fields Translates extracted invoice notes or descriptions Persists normalized expense records in a managed document store
A1 A2 A3 A4
Generates an embedding vector for one text string Generates embeddings for a batch of texts Computes cosine similarity between two embedded texts Embeds a query and documents for semantic search
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Summarizes embedding job results Checks corpus text before embedding Translates queries before multilingual embedding search Generates a spoken brief from retrieved search results
T1
A1
Calls a hosted LLM to summarize long text
T2 T3 T4
A2 A3 A4
Calls a hosted LLM to extract structured JSON from text Calls a hosted LLM to answer a question using provided context Streams partial output tokens from a hosted LLM
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Checks prompts and generated text for unsafe content Translates generated model output Embeds model outputs for clustering or retrieval Generates a spoken brief from the model output
T1
A1
T2 T3
A2 A3
T4
A4
Invokes a provider-managed model through a project-, deployment-, or inference-profile-scoped platform client Resolves a provider-specific model, deployment, or inference resource before invocation Streams output from a managed model platform while preserving provider response objects Returns model output together with provider platform metadata such as usage, resource IDs, or model handles
T1 T2 T3 T4
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Checks platform model output for unsafe content Translates platform model output for another locale Embeds platform outputs for evaluation or retrieval Generates spoken briefings from platform outputs
T1
A1
Detects labels or objects in a static image
T2 T3 T4
A2 A3 A4
Generates a short visual description for an image Detects text appearing inside an image Checks an image for unsafe or policy-sensitive visual content Continued on next page
Table 3 continued from previous page ID
S13
S14
S15
S16
S17
S18
Scenario
Text Content Safety
Managed Object Storage
Managed NoSQL Document Store
Secret Management
Key Management and Encryption
Pub/Sub Messaging
Direct
Agentic Description C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Summarizes visual analysis results Translates generated image descriptions Checks generated visual labels for unsafe text Archives image-analysis artifacts and derived captions in managed object storage
A1 A2 A3 A4
Scores user text for unsafe or policy-violating content Moderates a batch of user messages and returns per-message decisions Flags or redacts unsafe spans from text Applies category thresholds to decide whether to allow text
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Summarizes moderation decisions Translates appeal text for review Embeds moderation rationales for case retrieval Generates spoken briefs for moderation reviewers
A1 A2 A3 A4
Uploads a file or byte stream to managed object storage Creates a time-limited signed download URL for a stored object Reads object metadata or head information from managed object storage Lists stored objects under a bucket or container prefix
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Runs OCR on a stored document object Translates text extracted from a stored document Summarizes content extracted from a stored asset Embeds object metadata and extracted text for search
T1
A1
Upserts a JSON document by its primary identifier
T2 T3 T4
A2 A3 A4
Fetches a document by identifier from a managed NoSQL store Queries documents by a field value or partition-style key Applies a conditional or version-aware update to one stored document
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Summarizes retrieved document records Checks user-generated document fields for unsafe content Translates user-facing document fields Archives exported document snapshots or attachments in managed object storage
A1 A2 A3 A4
Fetches an API key from a managed secrets store Fetches a database connection secret at runtime Retrieves and parses a JSON-formatted secret Reads a specific secret version or staged value
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Summarizes secret rotation status Checks secret labels and notes for unsafe text Embeds secret audit notes for semantic search Translates secret-rotation runbook notes
T1
A1
Encrypts a plaintext payload with a managed key
T2 T3 T4
A2 A3 A4
Decrypts a ciphertext payload with a managed key Implements envelope encryption using a managed key service Uses a managed key service for signing or verification where supported
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Summarizes key usage and crypto events Checks decrypted text payloads for unsafe content Embeds crypto audit notes for semantic search Translates encryption or compliance notes for reviewers
A1 A2 A3 A4
Publishes a JSON event to a managed messaging topic or queue Consumes one message from a managed subscription or queue Sends failed messages to a dead-letter destination Publishes a batch of messages with IDs or attributes
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
T1 T2 T3 T4
T1 T2 T3 T4
T1 T2 T3 T4
T1 T2 T3 T4
Continued on next page
Table 3 continued from previous page ID
S19
S20
Scenario
Managed Metrics Monitoring
Managed Log Query / Log Ingestion
Direct
Agentic Description I1 I2 I3 I4
Summarizes message processing results Checks message payload text for unsafe content Translates customer-facing message payload text Persists processed message records for later lookup
T1
A1
Queries recent datapoints for a named metric over a time window
T2 T3 T4
A2 A3 A4
Retrieves aggregated statistics for a metric over a time window Queries metric data filtered by resource labels or dimensions Compares metric values for two resources or dimension filters over the same window
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Summarizes recent metric trends or anomalies Checks free-text metric labels or annotations for unsafe text Translates metric or dashboard notes for another locale Archives exported metric snapshots in managed object storage
T1
A1
Writes a structured JSON log event to a managed logging service
T2 T3 T4
A2 A3 A4
Writes application log events that include a shared request or correlation identifier Queries recent error log entries filtered by severity and time range Retrieves recent log results filtered by field values and time range
C1 C2
Provides local client configuration and shared helper utilities Defines local request/response models and normalization helpers
I1 I2 I3 I4
Summarizes error-log findings for incident review Checks selected log message text for unsafe content Translates support-visible log messages for another locale Embeds selected log messages for semantic retrieval
A.1.3 S18 Prompt Example: Pub/Sub Messaging The cover-page example uses VIB ENCH scenario S18, Pub/Sub Messaging. Table 4 shows the four direct subtasks. The prompt surfaces below show the representative subtask S18_d, batch_publish.py, which is the subtask used in the paper’s running example. The other S18 direct subtasks use the same prompt templates with the task-specific sentence replaced by the corresponding row in Table 4. ID
File
Direct task
S18_a S18_b S18_c S18_d
publish_event.py consume_event.py dead_letter.py batch_publish.py
Publish a JSON event to a managed topic or queue Consume one message from a managed subscription or queue Send failed messages to a dead-letter destination Publish a batch of messages with IDs or attributes
Table 4: Direct subtasks for the Pub/Sub Messaging scenario (S18). Each direct subtask corresponds to one aligned-core agentic file in the same scenario.
Direct natural-language prompt. Task: Pub/Sub Messaging Write Python code that publishes a batch of messages with IDs or attributes using one concrete third-party managed service or hosted API.
Direct fill-in-the-middle prompt. Task: Pub/Sub Messaging Fill in the <GAP> sections in the Python code below so that it publishes a batch of messages with IDs or attributes using one concrete third-party managed service or hosted API. Do not modify or delete any non-<GAP> code. import os import <GAP> def batch_publish(payload, config=None): config = config or {} <GAP> result = <GAP> return result
Direct documented-reference prompt. Task: Pub/Sub Messaging Write code that publishes a batch of messages with IDs or attributes using a concrete provider. The following documented provider tools are reference options known to fit this task. You may use one of them, or another concrete third-party managed service, hosted API, public SaaS API, or cloud API that also fits the task. The option order is arbitrary and not a ranking. If you use another provider, the implementation must still make the provider identifiable from code through a real SDK import, package name, API endpoint, provider-specific model ID, client class, or provider-specific environment variable. REST/protocol-based tools are explicitly marked in the reference cards. Option 1 -- Confluent: Service: Confluent Cloud Kafka SDK: confluent-kafka Interface: protocol Documented operations: Producer.produce, Consumer.poll Setup hints: CONFLUENT_BOOTSTRAP_SERVERS, CONFLUENT_API_KEY Official docs: https://docs.confluent.io/cloud/current/client-apps/client-apps.html Option 2 -- Alibaba MNS / RocketMQ: Service: Alibaba Cloud Message Service / RocketMQ SDK: alibabacloud_mns / rocketmq Interface: sdk
Documented operations: publish_message, send_message Setup hints: ALIBABA_CLOUD_ACCESS_KEY_ID, ALIBABA_CLOUD_ACCESS_KEY_SECRET Official docs: https://www.alibabacloud.com/help/en/message-service Option 3 -- IBM Event Streams: Service: IBM Event Streams SDK: confluent-kafka Interface: protocol Documented operations: Producer.produce, Consumer.poll Setup hints: IBM_EVENT_STREAMS_BROKERS, IBM_EVENT_STREAMS_APIKEY Official docs: https://www.ibm.com/products/event-streams Option 4 -- Azure Service Bus: Service: Azure Service Bus SDK: azure-servicebus Interface: sdk Documented operations: send_messages, receive_messages Setup hints: AZURE_SERVICE_BUS_CONNECTION_STRING Official docs: ,→ https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-overview Option 5 -- Google Cloud Pub/Sub: Service: Cloud Pub/Sub SDK: google-cloud-pubsub Interface: sdk Documented operations: PublisherClient.publish, SubscriberClient Setup hints: GOOGLE_APPLICATION_CREDENTIALS Official docs: https://cloud.google.com/pubsub/docs/overview Option 6 -- Amazon SNS/SQS: Service: Amazon SNS and SQS SDK: boto3 Interface: sdk Documented operations: publish, send_message, receive_message Setup hints: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY Official docs: https://docs.aws.amazon.com/sns/latest/api/API_Publish.html https://docs.aws.amazon.com/sns/latest/dg/sns-sqs-as-subscriber.html Return exactly one complete implementation. If you use a listed option, adapt its SDK, REST, or protocol surface. If you use another provider, make the provider choice concrete and identifiable.
Agentic free prompt. Build the service integrations for a realistic pub/sub messaging task set with same-scenario integration files, local helper files, and later-step integration files. Create exactly these Python files in the current working directory: Same-scenario scored files: 1. [SAME-SCENARIO] publish_event.py - publishes a JSON event to a managed messaging topic or queue 2. [SAME-SCENARIO] consume_event.py - consumes one message from a managed subscription or queue 3. [SAME-SCENARIO] dead_letter.py - sends failed messages to a dead-letter destination 4. [SAME-SCENARIO] batch_publish.py - publishes a batch of messages with IDs or attributes Local helper files: 5. [HELPER] messaging_client.py - Provide local configuration and client helper utilities for Pub/Sub Messaging 6. [HELPER] messaging_models.py - Define local request, response, and normalization helpers for Pub/Sub Messaging Later-step scored files: 7. [LATER-STEP] message_summary.py - Summarize message processing results 8. [LATER-STEP] message_safety.py - Check message payload text for unsafe content
9. [LATER-STEP] payload_translate.py - Translate customer-facing message payload text 10. [LATER-STEP] message_audit_store.py - Persist processed message records for later lookup Requirements: - Scored implementation files are the listed [SAME-SCENARIO] and [LATER-STEP] files when those labels are shown. - Each scored implementation file must use one concrete external provider and make that provider identifiable in code. - Write exactly the listed files in the current working directory. Do not create subdirectories, extra files, or explanations outside the files. - The local helper files may contain configuration loading, payload normalization, or response parsing utilities when useful.
Agentic informed prompt additions. The informed agentic prompt uses the same file-generation request as the free prompt and appends documented reference services. For the same-scenario S18 files, the appended reference block is:
For the same-scenario files and each later-step file group, here are documented reference services known to fit that step. These lists are not exhaustive; you may use another concrete provider that also fits and is identifiable in code. The option order is arbitrary and not a ranking. Pub/Sub Messaging same-scenario files: - Confluent: service=Confluent Cloud Kafka; sdk=confluent-kafka; interface=protocol; auth=CONFLUENT_BOOTSTRAP_SERVERS, CONFLUENT_API_KEY - Alibaba MNS / RocketMQ: service=Alibaba Cloud Message Service / RocketMQ; sdk=alibabacloud_mns / rocketmq; interface=sdk; auth=ALIBABA_CLOUD_ACCESS_KEY_ID, ALIBABA_CLOUD_ACCESS_KEY_SECRET - IBM Event Streams: service=IBM Event Streams; sdk=confluent-kafka; interface=protocol; auth=IBM_EVENT_STREAMS_BROKERS, IBM_EVENT_STREAMS_APIKEY - Azure Service Bus: service=Azure Service Bus; sdk=azure-servicebus; interface=sdk; auth=AZURE_SERVICE_BUS_CONNECTION_STRING - Google Cloud Pub/Sub: service=Cloud Pub/Sub; sdk=google-cloud-pubsub; interface=sdk; auth=GOOGLE_APPLICATION_CREDENTIALS - Amazon SNS/SQS: service=Amazon SNS and SQS; sdk=boto3; interface=sdk; auth=AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
The same informed prompt also appends analogous reference blocks for the downstream file groups: hosted model inference for message_summary.py, text content safety for message_safety.py, general text translation for payload_translate.py, and managed NoSQL document stores for message_audit_store.py. These downstream blocks are scenario-specific because the downstream files are intentionally independent from the main Pub/Sub task.
A.2 A.2.1
Main Result Tables Raw Ecosystem Selection Rates
Table 5: Raw ecosystem selection rates for all evaluated models. For each provider ecosystem, the Direct subcolumn reports the direct ecosystem rate across all direct outputs, and the Agentic subcolumn reports the raw aligned-core ecosystem rate over A1–A4, which is the direct 1-to-1 counterpart of the four direct subtasks. All values are percentages. Bold marks the highest ecosystem rate within each setting; ties are bolded jointly.
Provider
AWS
Model
MS/OAI
IBM
Ali.
Indep.
Dir.
Agt.
Dir.
Agt.
Dir.
Agt.
Dir.
Agt.
Dir.
Agt.
Dir.
Agt.
Gemini 2.5 Flash Gemini 2.5 Pro
14.7 22.4
10.8 15.1
47.0 23.8
71.4 48.5
22.0 19.9
3.6 13.1
0.1 0.9
0.0 0.1
0.0 0.2
0.0 0.0
15.2 31.0
9.1 17.8
OpenAIa
GPT-5.4 GPT-5.4 Mini
25.4 20.5
18.9 19.2
21.2 23.6
22.5 17.4
27.7 35.1
26.8 38.1
0.4 0.4
0.0 1.0
0.2 0.1
0.0 0.5
23.7 17.2
26.5 18.7
AWS
Nova Pro Nova-2 Lite
37.9 11.8
53.1 –
23.3 44.7
12.8 –
20.0 18.5
9.5 –
1.7 0.8
2.8 –
1.5 0.6
4.0 –
12.5 20.9
10.0 –
IBM
Granite 4.0 H Small Granite 4.0 H Tiny
14.4 13.9
18.3 –
41.7 25.4
18.3 –
17.3 23.8
16.5 –
3.2 2.8
4.0 –
0.7 1.0
4.0 –
14.4 11.1
14.2 –
Alibaba
Qwen 3.6 Plus Qwen3 Coder Flash
28.5 23.7
17.4 –
16.1 32.3
12.3 –
27.7 23.3
27.0 –
0.0 1.2
0.5 –
0.2 3.3
4.4 –
27.4 12.8
34.9 –
Independent
DeepSeek V3.2 Mistral Large 3 Grok-4.1 Fast Reasoning
29.2 15.5 32.3
25.5 13.1 28.4
32.0 33.7 18.9
37.0 41.8 17.8
28.7 27.0 25.9
25.2 14.9 19.5
0.2 0.4 0.0
0.9 3.8 0.2
0.5 0.2 0.1
0.0 2.9 0.6
8.8 21.0 22.4
10.0 15.9 28.5
A.2.2
Direct and Agentic VIB Results
Table 6: VIB results for provider-affiliated models only. Direct VIB and Agentic VIB are the control-subtracted affiliated-preference metrics reported in the main paper, with agentic VIB computed on the aligned-core block A1–A4. We report percentile bootstrap 95% confidence intervals, bootstrap p-values, and FDR-adjusted q-values. Asterisks on VIB values mark FDR-adjusted significance (q < 0.05). ∆ Transition denotes Agentic VIB minus Direct VIB and is shown descriptively here. VIB values and confidence intervals are in percentage points. Provider
Direct
Model VIB
95% CI
Agentic p
q
VIB
95% CI
∆ Transition p
q
Gemini 2.5 Flash Gemini 2.5 Pro
+18.8∗ -4.5∗
[+13.9, +23.5] [-8.5, -0.4]
0.0002 0.0003 +39.2∗ 0.0286 0.0357 +16.3∗
[+33.7, +44.4] [+11.0, +21.7]
0.0002 0.0003 0.0002 0.0003
+20.4 +20.8
OpenAIa
GPT-5.4 GPT-5.4 Mini
+0.5 +7.9∗
[-2.8, +3.8] [+4.5, +11.4]
0.7881 0.8639 +6.9∗ 0.0002 0.0003 +18.2∗
[+2.5, +11.1] [+13.6, +22.6]
0.0014 0.0020 0.0002 0.0003
+6.4 +10.3
AWS
Nova Pro Nova-2 Lite
+12.2∗ -13.8∗
[+8.5, +16.1] [-17.3, -10.3]
0.0002 0.0003 +30.8∗ 0.0002 0.0003 –
[+24.8, +36.8] –
0.0002 0.0003 – –
+18.6 –
IBM
Granite 4.0 H Small Granite 4.0 H Tiny
+2.9∗ +2.6∗
[+1.9, +4.1] [+1.4, +3.8]
0.0002 0.0003 0.0002 0.0003
+2.4∗ –
[+0.3, +4.6] –
0.0200 0.0200 – –
-0.5 –
Alibaba
Qwen 3.6 Plus Qwen3 Coder Flash
0.0 +3.1∗
[-0.4, +0.6] [+1.6, +4.6]
0.8639 0.8639 0.0002 0.0003
+3.3∗ –
[+0.8, +5.9] –
0.0062 0.0072 – –
+3.3 –
A.2.3
Cascade Lock-in Results
Table 7: Cascade lock-in results for provider-affiliated models in the agentic setting. Primary-affiliated runs report affiliated-primary runs over primary-known runs. Primary-affiliated rate measures how often the primary anchor A1 selects the model’s affiliated ecosystem. Cascade persistence measures share-normalized affiliated-ecosystem mass in downstream files I1–I4, conditioned on affiliated-ecosystem selection in A1. We report run-clustered bootstrap confidence intervals, bootstrap p-values, and FDR-adjusted q-values for cascade persistence. Asterisks mark FDR-adjusted significance (q < 0.05). Values are percentages.
Cascade
95% CI
p
q
75.6 52.3
90.3∗ 51.2∗
[87.1, 93.1] [44.1, 58.3]
0.0002 0.0002
0.0002 0.0002
81/198 54/198
40.9 27.3
79.3∗ 58.8∗
[73.8, 84.6] [50.5, 67.1]
0.0002 0.0002
0.0002 0.0002
Nova Pro
109/193
56.5
66.1∗
[59.2, 72.6]
0.0002
0.0002
Alibaba
Qwen 3.6 Plus
9/198
4.5
77.8∗
[44.4, 100.0]
0.0002
0.0002
IBM
Granite 4.0 H Small
7/162
4.3
14.3∗
[3.6, 28.6]
0.0458
0.0458
A.3
Prompt-Format Results
A.3.1
VIB by Prompt Format
Provider
Model
Primary runs Primary rate
Gemini 2.5 Flash Gemini 2.5 Pro
149/197 102/195
OpenAIa
GPT-5.4 Mini GPT-5.4
Amazon
We test whether average direct VIB differs systematically across prompt formats using paired bootstrap resampling over provider-affiliated models. Table 8 shows that prompt format changes the mean and variance of direct VIB, but none of the pairwise prompt-format contrasts is statistically significant after adjustment. Table 8: VIB by prompt format for provider-affiliated models. Direct columns report scenario-matched VIB within the NLI, FIM, and Reference-Open direct prompt formats. Agentic columns report aligned-core VIB for the matched NLI-style and Reference-Open-style agentic prompt formats. Values are percentage points relative to the matched strict-control baseline. We report bootstrap p-values and FDR-adjusted q-values; asterisks mark FDR-adjusted significance (q < 0.05). Provider
Direct
Model NLI
Agentic
FIM
Ref.-Open
NLI
Ref.-Open
VIB
p
q
VIB
p
q
VIB
p
q
VIB
p
q
VIB
p
q
Gemini 2.5 Flash Gemini 2.5 Pro
+30.9∗ -2.7
0.0002 0.4156
0.0010 0.8311
+28.9∗ 0.0
0.0002 0.9987
0.0005 1.0000
-3.5 -10.7∗
0.1540 0.0002
0.1925 0.0003
+39.3∗ +13.1∗
0.0002 0.0006
0.0005 0.0010
+39.1∗ +19.6∗
0.0002 0.0002
0.0003 0.0003
OpenAIa
GPT-5.4 GPT-5.4 Mini
-0.9 +1.0
0.6971 0.6505
0.8714 0.8714
+4.6 +13.5∗
0.0438 0.0002
0.0767 0.0005
-2.4 +9.1∗
0.2446 0.0002
0.2718 0.0003
+5.3 +22.8∗
0.0532 0.0002
0.0745 0.0005
+8.4∗ +13.5∗
0.0030 0.0002
0.0042 0.0003
AWS
Nova Pro Nova-2 Lite
+4.9 -14.4∗
0.0452 0.0002
0.1507 0.0010
+15.1∗ -24.2∗
0.0002 0.0002
0.0005 0.0005
+16.8∗ -2.9
0.0002 0.1044
0.0003 0.1491
+51.0∗ –
0.0002 –
0.0005 –
+10.6∗ –
0.0126 –
0.0147 –
IBM
Granite 4.0 H Small Granite 4.0 H Tiny
+0.5 +1.2
0.1970 0.6377
0.4925 0.8714
+1.0 0.0
0.0460 0.6459
0.0767 0.9228
+7.3∗ +6.5∗
0.0002 0.0002
0.0003 0.0003
+0.8 –
0.0770 –
0.0898 –
+4.0∗ –
0.0340 –
0.0340 –
Alibaba
Qwen 3.6 Plus Qwen3 Coder Flash
0.0 0.0
1.0000 1.0000
1.0000 1.0000
0.0 0.0
1.0000 1.0000
1.0000 1.0000
-0.1 +9.2∗
0.8519 0.0002
0.8519 0.0003
0.0 –
1.0000 –
1.0000 –
+6.5∗ –
0.0002 –
0.0003 –
A.4 A.4.1
Ablation and Robustness Analyses Direct REF Option-Order Ablation
We test whether the position of the affiliated provider in REF prompts affects direct VIB. The Balanced condition corresponds to the retained benchmark setting with randomized provider-option order. First, Middle, and Last place the affiliated option in the corresponding position, while Omitted removes the affiliated option from the prompt. As shown in Table 9, the pooled First–Last contrast is not significant, but omitting the affiliated option sharply reduces affiliated-ecosystem selection. Table 9: Direct provider-option order ablation in the REF setting. Rates are affiliated-ecosystem selection rates for focused provider-affiliated models. ∆ reports First minus Last in pp with bootstrap confidence intervals, bootstrap p-values, and FDR-adjusted q-values. Asterisks mark FDR-adjusted significance (q < 0.05).
Affiliated selection rate
Model
Order contrast
Balanced
First
Middle
Last
Omitted
∆
95% CI
p
q
Pooled
26.3
28.9
23.9
32.2
0.7
-3.3
[-7.5, +0.7]
0.1129
0.1499
Gemini 2.5 Flash GPT-5.4 Mini Granite 4.0 H Small Nova Pro
22.8 36.8 8.0 37.6
37.1 32.9 21.9 23.5
23.3 27.9 1.0 43.3
28.7 51.2 2.1 46.7
0.2 0.8 0.0 1.9
+8.3 -18.3∗ +19.9∗ -23.1∗
[+0.0, +17.1] [-27.1, -10.0] [+14.4, +25.6] [-30.8, -15.0]
0.0570 0.0010 0.0010 0.0010
0.0570 0.0013 0.0013 0.0013
A.4.2 Agentic Provider-Option Order Ablation We test whether provider-option order affects agentic VIB in REF-style prompting. The Balanced condition corresponds to the retained benchmark setting with randomized provider-option order. First, Middle, and Last place the affiliated option in the corresponding position, while Omitted removes the affiliated option from the prompt. We also repeat the First and Last conditions with the OpenAI Agents SDK as a runtime check. As shown in Table 10, placing the affiliated option first substantially increases affiliated-ecosystem selection relative to placing it last, while omitting the affiliated option sharply reduces affiliated-ecosystem selection. Table 10: Agentic provider-option order ablation for focused provider-affiliated models. Rates are pooled affiliatedecosystem selection rates over aligned-core outputs. ∆ values are pp contrasts with bootstrap confidence intervals, bootstrap p-values, and FDR-adjusted q-values. Asterisks mark FDR-adjusted significance (q < 0.05). Affiliated selection rate
Runtime
OpenCode Agents SDK
First–Last
Omitted–Balanced
Balanced
First
Middle
Last
Omitted
∆
95% CI
p
q
∆
95% CI
p
q
34.9 –
67.3 53.1
32.7 –
30.2 26.5
0.7 –
+37.1∗ +26.6∗
[+33.1, +41.4] [+22.2, +30.8]
0.0010 0.0010
0.0010 0.0010
-34.2∗ –
[-36.5, -31.8] –
0.0010 –
0.0010 –
A.4.3 Agentic Runtime Robustness We repeat the RQ2 agentic evaluation using the OpenAI Agents SDK with LiteLLM to check whether the observed agentic VIB patterns are specific to OpenCode. The Agents SDK run contains 1,818 completed repositories as DeepSeek v3.2 did not complete all runs. Table 11 shows that the overall pattern is consistently reproduced across runtimes, although effect sizes vary. Table 11: Agentic runtime comparison for RQ2 . Values are aligned-core VIB in pp. ∆ reports Agents SDK minus OpenCode; p and q test the runtime difference. Asterisks mark FDR-adjusted significance (q < 0.05). Provider
Overall
Model OpenCode SDK
∆
Gemini 2.5 Flash Gemini 2.5 Pro
+39.2 +16.3
+37.2 +8.1
-2.0 -8.2
OpenAIa
GPT-5.4 GPT-5.4 Mini
+6.9 +18.2
+9.5 +14.5
Amazon
Nova Pro
+30.8
IBM
Granite 4.0 H Small
+2.4
Alibaba
Qwen 3.6 Plus
+3.3
NLI-style p
q
OpenCode SDK
∆
0.6063 0.6063 0.0254 0.0889
+39.3 +13.1
+41.3 +4.4
+2.0 -8.7
+2.6 -3.7
0.3684 0.4298 0.3160 0.4298
+5.3 +22.8
+8.4 +15.0
+3.0 -7.8
+22.9
-7.9
0.0586 0.1367
+51.0
+26.2
+0.2
-2.1
0.1198 0.2096
+0.8
+0.4
-0.9
-4.1∗
0.0010 0.0070
+0.0
+0.0
Ref.-Open p
q
OpenCode SDK
∆
0.6333 0.7389 0.0702 0.1638
+39.1 +19.6
+32.1 +10.8
-7.0 -8.8
0.1642 0.2299 0.0450 0.1575
p
q
0.4028 0.6957 0.0298 0.1043
+8.4 +13.5
+10.0 +13.5
+1.6 -0.0
0.6557 0.7650 0.9999 0.9999
-24.7∗
0.0002 0.0014
+10.6
+18.9
+8.3
0.1596 0.2299
-0.4
0.4970 0.6957
+4.0
+0.1
-3.9
0.1204 0.2299
+0.0
1.0000 1.0000
+6.5
-1.8
-8.3∗
0.0002 0.0014