Alipay-PIBench: A Realistic Payment Integration Benchmark for Coding Agents Shiyu Ying1 , Xuejie Cao1 , Yingfan Ma1 , Yuanhao Dong1 , Wenyu Chen1 , Bowen Song1 , Lin Zhu1 1
Ant Group
arXiv:2607.14573v1 [cs.AI] 16 Jul 2026
Coding agents are increasingly being evaluated on realistic software development tasks that require repository-level editing, cross-component integration, domain knowledge, and taskspecific verification. Payment integration is a representative domain-specific setting because a correct implementation must select an appropriate payment product, keep credentials and signing logic within server-side trust boundaries, coordinate frontend and backend flows, verify payment outcomes, and maintain consistency between payment and local business states. We introduce Alipay-PIBench, a repository-level benchmark proposed by Alipay for evaluating AI coding agents on realistic payment integration tasks. Built around Alipay Open Platform products and business-oriented repositories, Alipay-PIBench evaluates functional correctness, reliability, security, and business-state consistency. The benchmark comprises nine product-specific projects and 18 task instances organized into two progressive scenarios: Basic (Functional Payment Completion), which evaluates the core end-to-end payment loop, and Advanced (Risk-Aware Payment Hardening), which evaluates agents’ ability to harden payment integrations against payment-specific risks through notification idempotency, abnormal-transaction handling, refund safeguards, and fund-safety controls. Generated solutions are evaluated using deterministic checks derived from scenario-specific rubrics, supplemented by rubric-aligned LLM-assisted assessment. Across the six evaluated models, mean RPR under the with-skill condition ranges from 68.58% to 91.37%. Compared with the corresponding without-skill condition, the with-skill condition improves mean RPR by 10.31 percentage points on average, with gains varying across models, products, and scenarios. Method-level results further distinguish source-level completion, executable payment behavior, and payment-domain requirements. Correspondence: [email protected] Code: https://github.com/inclusionAI/PIBench
1
Introduction
Recent evaluations of coding agents are moving beyond isolated code generation toward more realistic software delivery tasks. Benchmarks such as HumanEval, SWE-bench, and LiveCodeBench have been effective for measuring function-level programming, general bug fixing, and repositorylevel code modification (Chen et al., 2021; Jimenez et al., 2024; Jain et al., 2024). In real development workflows, however, many tasks require more than locally correct code: an agent must understand the business scenario, choose an appropriate domain solution, coordinate changes across components,
1
and preserve safety and state consistency after implementation. Payment integration is a typical example of this shift. For many applications, payment capability is a key step from product workflow to commercial operation. Yet integrating a payment product is not simply adding an API call. A correct implementation must select a suitable payment product, keep credentials and signing logic on the server side, coordinate frontend and backend payment flows, verify asynchronous notifications, and update local order or service state only after reliable payment confirmation. If an agent only makes the code appear runnable while trusting client-side payment status, skipping signature verification, mishandling repeated notifications, or causing inconsistency between payment state and business fulfillment, the result may introduce real fund-safety and business-consistency risks. These properties make payment integration a challenging domain-specific setting for evaluating coding agents. It jointly tests code generation, cross-component integration, payment product understanding, risk handling, and business-state maintenance. Existing general-purpose benchmarks are not designed to systematically measure these capabilities, because the correctness of payment integration depends not only on local code behavior, but also on payment-domain rules, security boundaries, and consistency across the business workflow. To address this setting, we propose Alipay-PIBench, an Alipay-led repository-level benchmark for evaluating AI coding agents on realistic Alipay payment integration tasks. Built on Alipay Open Platform products and business-style repositories, Alipay-PIBench evaluates not only whether an agent can complete a payment integration, but also whether the implementation is secure, reliable, convenient to integrate, and consistent with local business state. Alipay-PIBench follows a systematic evaluation pipeline. First, we use Product-Grounded Task Construction to connect real payment products, business scenarios, initial repositories, and naturallanguage integration requests, so that each task evaluates both product understanding and repositorylevel implementation. Second, we divide payment integration into two progressive scenarios: Basic (Functional Payment Completion), which focuses on the core payment loop including payment creation, payment interaction, backend confirmation, and local state update; and Advanced (RiskAware Payment Hardening), which focuses on risk hardening such as signature verification, idempotency, abnormal transaction states, refunds, and fund-safety boundaries. Third, we adopt Rubric-Derived Evaluation, deriving deterministic checks and LLM-assisted assessment criteria from scenario-specific rubrics so that evaluation items remain traceable to payment-domain requirements rather than ad hoc tests or generic judge prompts. Finally, through a paired withoutskill/with-skill setting, Alipay-PIBench evaluates whether the alipay-payment-integration skill helps agents complete payment integration more efficiently, conveniently, and safely. This paper makes the following contributions: • We propose the first Alipay-led repository-level benchmark for evaluating coding agents on realistic Alipay payment integration tasks. • We propose Product-Grounded Task Construction, which organizes payment products, business scenarios, initial repositories, and natural-language integration requests into a unified task format. • We introduce a progressive capability decomposition of payment integration into Basic (Functional Payment Completion) and Advanced (Risk-Aware Payment Hardening), distinguishing end-to-end payment completion from the ability to harden payment integrations against
2
payment-specific risks and boundary failures. • We develop Rubric-Derived Evaluation, which derives deterministic checks and supplementary large-language-model (LLM)-assisted rubric assessment criteria from scenario-specific payment rubrics, making evaluation evidence traceable to domain requirements. • We establish a paired without-skill/with-skill intervention protocol to measure how access to structured Alipay payment guidance changes adherence to product-specific APIs and integration patterns, payment safety, payment–business state consistency, and output-token efficiency. Across the 18 task instances and six evaluated models, mean RPR under the with-skill condition ranged from 68.58% to 91.37%. Compared with the corresponding without-skill condition, the with-skill condition improved mean RPR by 10.31 percentage points on average, with gains in 101 of 108 model–product–scenario comparisons. The average improvement was larger for Basic tasks than for Advanced tasks (+11.27 versus +9.35 percentage points). Method-level results further distinguish source-level completion, executable payment behavior, and payment-domain requirements, motivating the use of progressive scenarios and complementary rubric-grounded evaluation signals.
2
Related Work
Repository-level benchmarks for coding agents. Coding benchmarks have expanded from functionlevel generation toward realistic software engineering. HumanEval, APPS, and LiveCodeBench evaluate isolated programming and algorithmic problem solving (Chen et al., 2021; Hendrycks et al., 2021; Jain et al., 2024), while SWE-bench evaluates issue resolution in existing repositories through executable tests (Jimenez et al., 2024). Recent benchmarks further cover repositorylevel feature implementation, project construction, and longer-horizon development. FEA-Bench studies feature addition (Li et al., 2025); ProjDevBench and NL2Repo-Bench evaluate project or repository construction from natural-language requirements (Lu et al., 2026; Ding et al., 2026); and BeyondSWE broadens evaluation to cross-repository reasoning, domain-specific repair, dependency migration, and repository generation (Chen et al., 2026). DeepSWE further introduces original repository-scale tasks and behavior-oriented verifiers to reduce contamination and implementationspecific grading (Huang et al., 2026). Domain-specific benchmark design is also exemplified by ABench-Physics, which evaluates model performance on high-difficulty, dynamic physics problems (Zhang et al., 2025). These benchmarks reflect a transition toward realistic software delivery, but mostly remain organized around general software-engineering task forms. Production and payment-integration evaluation. Production-oriented evaluations increasingly emphasize realistic repositories, deployment constraints, and reliable verification. ProdCodeBench derives executable tasks from production developer-agent sessions and applies test-relevance and stability checks during benchmark construction (Jha et al., 2026). Security-oriented evaluations further examine whether functionally successful implementations satisfy deployment-critical security requirements (Endor Labs, 2026). Most closely related to the payment setting, Stripe reports an industry evaluation suite for coding agents on realistic Stripe integrations, using full coding environments and deterministic graders over APIs, user interfaces, and Stripe-side artifacts (Stripe, 2026). Together, these efforts demonstrate a broader shift from generic code correctness toward production-grounded evaluation under domain and platform constraints. Alipay-PIBench follows this
3
Payment Benchmark Construction
A
B
How are realistic payment-integration tasks constructed?
Rubric-Grounded Evaluation How are payment-domain requirements converted into evaluators?
Paired Skill Study
C
How is the practical value of the Alipay payment skill measured?
A1. Product–Project Coverage B1. Deterministic Evaluation
C1. Paired Skill Comparison (Within Task)
Static checks and executable validation
Same agent and task; project, instruction, and environment fixed
Each payment product is paired with a representative project 1
2
In-App Payment ez_tickets_app
4
Mobile Web Payment
3
JSAPI Payment
edoc-doctor-appointmentsystem
PC Web Payment
5
Order QR Code Payment
6
Merchant Initiated Transaction
9
litemall
Authorization Hold
8
bookcars
Static Checks
QR Code Payment
bill-express
7
Same Agent With alipay-paymentintegration
laravel-gymie
SaaS Starter
2
Business Workflow
3
E2E Tests
↓
Usage Based Payment
B2. LLM-Assisted Domain Assessment
A2M Recipes
Initial Repository
4
C2. Paired Comparison Metrics
Rubric-dependent examples requiring semantic judgment
Product Selection
Four elements defined in the task format
Payment Product
Integration Tests
bill-express
A2. Product-Grounded Task Construction 1
Unit Tests
Same Agent Without Skill
VS
Integration Request
State Coherence
Security Placement
Integration Outcome
Output-Normalized Efficiency
Rubric-Aligned Scores
Output Tokens per Unit RPR
CrossComponent Linkage
↓ Target Alipay product for the business flow
Existing workflow that requires payment integration
Business codebase and surrounding application state
Scenario-specific naturallanguage instruction
B3. Rubric-Aligned Evidence Deterministic Checks
A3. Benchmark Composition 9
Product Projects
×
2
=
Scenarios
18
Task Instances
C3. Comparison Axes
Both channels are derived from scenario-specific rubrics LLMAssisted Criteria
+
→
RubricItem Evidence
By Product
By Scenario
By Agent
By Skill Condition
Paired Conditions Without / With Skill
↓
↓ RUBRIC CONTRACT (Shared Evaluation Reference)
Tasks define rubrics; rubrics instantiate evaluators; shared rubrics support paired comparison
1. Task–Rubric Alignment Basic Advanced
2. Evaluator Derivation
3. Rubric-Aligned Evidence
→
→ Deterministic Check
LLM-Assisted Criteria
4. Paired Comparison
→ Criterion-level evidence and diagnosis
Both conditions use the same task rubric and evaluator artifacts With alipaypaymentintegration
Without Skill
↓
↓
↓
Task Construction → Defines Rubric Items
Rubric Items → Instantiate Evaluators
Skill Study → Reuses the Same Rubric
Figure 1 Overview of the Alipay-PIBench framework. Product-specific projects are paired with business workflows, initial repositories, and integration requests to form Basic and Advanced task scenarios. Deterministic checks and LLM-assisted criteria are derived from shared rubrics, while paired without-skill and with-skill conditions measure the practical effect of alipay-paymentintegration.
direction by studying repository-level payment integration in the Alipay ecosystem, with additional attention to payment-specific risks and business-state consistency. Rubric-grounded evaluation and domain guidance. Executable tests provide reproducible evidence of software behavior, but may not fully capture product selection, security placement, or cross-component business consistency. Some recent benchmarks therefore combine execution-based evaluation with LLM-assisted review, although RuVerBench shows that LLM rubric verification in agentic coding remains noisy (Lu et al., 2026; Peng et al., 2026). In Alipay-PIBench, scenariospecific payment rubrics provide a common specification from which deterministic checks and supplementary semantic criteria are derived, while executable evaluation remains the primary correctness signal. Separately, Stripe’s agent-steering experiments indicate that explicitly loaded, structured skills can guide integration behavior more effectively than passive warnings or documentation (Beswick and Epsteen, 2026). Motivated by this practical question, Alipay-PIBench performs paired without-skill/with-skill trials under the same task, environment, and evaluation criteria to measure the effect of the Alipay payment skill.
3
Alipay-PIBench Overview
3.1
Benchmark Composition
Alipay-PIBench is a benchmark suite for repository-level payment integration. It is organized around product-specific projects, each corresponding to one Alipay payment product and an associated 4
business setting. The selected products span multiple common Alipay integration patterns, including mobile-app payments, web-based payments, in-person code payments, pre-authorization, and recurring or subscription-style payments. These products differ in user entry points, paymentinitiation flows, backend confirmation paths, and risk profiles, allowing the benchmark to evaluate both product understanding and repository-level implementation. Each product-specific project contains two progressive scenarios. Basic (Functional Payment Completion) starts from a business workflow in which the target Alipay product has not yet been integrated, although another payment capability may already exist. Depending on the project, the agent may connect an existing payment entry point or add a product-specific entry point. Advanced (Risk-Aware Payment Hardening) starts after the core Alipay integration is present and evaluates payment-specific safety and boundary handling. The two scenarios are therefore progressive rather than independent: Basic evaluates whether the agent can establish the end-to-end payment loop, whereas Advanced evaluates whether it can harden that loop against risks such as invalid or duplicated notifications, non-idempotent state updates, abnormal transaction states, and unsafe refund behavior. We define a task instance as a project–scenario pair. Each task instance contains agent-facing inputs and evaluator-facing materials. The agent-facing inputs consist of an initial repository and a scenario-specific natural-language instruction, and the expected output is a modified repository. The evaluator-facing materials consist of scenario-specific rubrics, deterministic checks derived from those rubrics, and supplementary LLM-assisted criteria for rubric items requiring semantic or payment-domain judgment. Pairing each of the nine product-specific projects with Basic and Advanced scenarios yields 18 task instances. This shared format supports consistent evaluation across products, scenarios, agents, and without-skill/with-skill conditions. Table 1 Product and project coverage of Alipay-PIBench. A checkmark indicates an existing payment integration in the initial repository, while a cross indicates no initial payment integration. Basic and Advanced columns report the number of rubric items in each scenario. Product
Project
Backend stack
Initial Payment
Basic
Advanced
In-App Payment Usage Based Payment Mobile Web Payment JSAPI Payment Authorization Hold PC Web Payment Order QR Code Payment QR Code Payment Merchant Initiated Transaction
ez_tickets_app A2M Recipes eDoc-doctor-appointment-system laravel-gymie bookcars litemall bill-express bill-express saas-starter
Node.js + Express + MySQL Next.js + TypeScript + Node.js PHP + Apache + MySQL Laravel + PHP + SQLite Node.js + TypeScript + MongoDB Java + Spring Boot + MySQL Node.js + Express + SQLite Node.js + Express + SQLite Next.js + TypeScript + Postgres
× × × × ✓ ✓ × × ✓
21 20 13 26 18 18 17 17 18
41 22 21 24 33 36 40 41 33
Table 1 summarizes the product-level coverage of the current release. The project column identifies the corresponding repositories or project sites (Saleem, 2026; A2M Recipes, 2026; HiEventsDev, 2026; lubusIN, 2026; Elassas, 2026; linlinjava, 2026; opensourcepos, 2026; UniBee-Billing, 2026). For each product, the table records the project, backend technology stack, initial payment status, and the number of rubric items in the Basic and Advanced scenarios. Although product details differ, Basic tasks typically require coordinated changes to payment configuration, backend payment creation and confirmation, local business state, and application-facing payment entry points where applicable. Advanced tasks build on this foundation and require the agent to address paymentspecific risks such as duplicate notifications, repeated confirmation calls, inconsistent local states, unsafe refund behavior, and insufficient verification of asynchronous notifications. 5
3.2
Product-Grounded Task Construction
Project and scenario initialization We construct task instances from business-style applications and payment-specific integration requirements. For each Alipay payment product, we select a repository in which the surrounding business workflow is already implemented. The repository’s existing payment state is product-dependent: some projects contain no payment capability, whereas others contain an existing payment integration. These differences are retained as part of the task context and are summarized in Table 1. For each project, we prepare two independently initialized scenario states. The Basic state is a repository snapshot taken before the target Alipay product is integrated. The Advanced state is a separately prepared snapshot in which the core target-product payment path is already present, so that the task can focus on product-specific safety, fund-risk, and boundary requirements. Basic and Advanced are evaluated as separate controlled task instances with their own scenario-specific instructions, rubrics, and starting repositories. Instruction and rubrics For each product–scenario pair, we prepare a payment-specific naturallanguage instruction and a corresponding set of rubrics. The instruction defines the task presented to the agent. It specifies the business objective, repository context, payment behavior, and constraints that the solution must satisfy. Written as a developer-facing integration request, it generally leaves internal code organization open while specifying externally observable interfaces, payment semantics, security constraints, and, where necessary, required API, SDK, or configuration behavior. The target payment product is explicitly specified through the task metadata and corresponding instruction, which may additionally provide product-specific API and business-context details. Where relevant, the instruction also identifies the interfaces and observable behaviors used to evaluate the completed integration. The instruction reflects the starting state of each scenario. A Basic instruction asks the agent to add or complete the target payment capability within an existing business workflow. An Advanced instruction addresses safety-sensitive or boundary requirements in a separately prepared paymentintegration context. The task description and interface requirements remain specific to each product and project, with product-specific API and integration requirements grounded in the Alipay Open Platform documentation (Alipay Open Platform, 2026). The rubrics decompose each instruction into checkable requirements. Their composition varies across products and projects. Basic rubrics may assess payment creation, signing and credential handling, application-facing invocation, payment-result processing, and consistency between payment and business states. Advanced rubrics may assess safety-sensitive properties such as idempotent state transitions, duplicate-notification handling, abnormal transaction states, notification authenticity, or refund boundaries, where relevant to the task. Running example Figure 2 illustrates the construction and evaluation of the LaravelGymieJSAPIPayment-basic task. The initial Laravel repository provides a gym-membership workflow, including membership plans, order creation and query APIs, and an Alipay Mini Program page, but stops after creating a pending order. The instruction asks the agent to complete the existing workflow with Alipay Mini Program payment rather than introduce a replacement API. A valid solution must create the payment trade on the server, return the case-sensitive tradeNO field for native Mini Program invocation, and advance the order and membership fulfillment only after a trusted server-side result. The task rubrics further require invalid notifications to be rejected, 6
Example: LaravelGymie-JSAPIPayment-basic A task instance is constructed by turning a real membership checkout gap into a verifiable Alipay mini-app payment loop.
1 · Repository context
2 · Task-instance construction
Gymie membership system
Business request
Laravel 12 + SQLite + mini app pages
Buy a gym membership inside
Product inference Alipay JSAPI / mini-app payment
Alipay mini app
Payment capability gap
Complete the payment loop rather than harden refund or
rubric-derived artifacts
Runtime evidence given to agent
Constructed payment loop
→ sandbox proxy
key paths from env
Fixed surface
Static SDK/OpenAPI, env config, no hardcoded keys
dispute logic
agent-facing task package
sandbox buyer identifiers
Create order preserve existing checkout API
OpenAPI trade real sandbox proxy; no fake trade
Integration Miniapp pay my.tradePay(tradeNO)
query / refresh only
→ tradeNO
GET orders/{checkout_no} POST notify
Fulfillment order member billing; idempotent
→
trade creation, product mapping, op_app_id, notify checks
LLM-assisted review
GET plans POST orders
payment criteria become
Basic scenario
already exist; actual Alipay payment and membership activation are missing.
APP_ID / MINIAPP_APP_ID
Rubrics as the bridge executable and semantic checks
plans, order creation, order query
ALIPAY_GATEWAY_URL
3 · Verification lens
Trusted notify verify signature, amount, app id
→
product fit, secret handling, persistent state design, stable endpoints
Figure 2 Running example of task construction and rubric-derived evaluation for the LaravelGymie-JSAPIPayment-basic task. The task covers server-side trade creation, Mini Program payment invocation, trusted notification processing, and idempotent membership fulfillment, and is evaluated under the paired without-skill/with-skill condition.
repeated success notifications to be handled idempotently, and client-side callbacks not to determine the final payment state. The same scenario-specific rubrics are instantiated as task-specific deterministic checks, including static, unit, integration, and E2E tests where applicable, with product-specific client checks where applicable, together with supplementary LLM-assisted criteria for properties that require semantic code inspection. The resulting evaluators are then reused for the paired without-skill/with-skill comparison. Rubric-Derived Evaluation The scenario-specific rubrics serve as the source specification for evaluation. For rubric items that can be operationalized as executable checks, we derive deterministic evaluators, including static checks, unit tests where applicable, integration tests, and E2E or environment-level validation. Static checks examine structural properties such as the presence and placement of payment-related code, configuration, and credential handling. Unit and integration tests examine local payment logic, backend API behavior, state transitions, and interactions between payment and business modules. When evaluation requires Alipay SDK calls, we use the Alipay sandbox environment where reliable support is available. When a product is not supported by the sandbox, or when rare abnormal states and risk boundaries cannot be triggered reproducibly, we use deterministic mock responses or fixtures. These cases are derived from the same scenario-specific rubrics and manually reviewed to ensure that they evaluate payment behavior rather than incidental implementation details. For rubric items that cannot be fully captured by executable checks, we derive supplementary LLM-assisted assessment criteria from the same rubrics. These criteria examine properties such as payment-product fit, end-to-end flow consistency, security-sensitive logic placement, and payment– business state consistency. We manually review both deterministic checks and LLM-assisted criteria 7
for consistency with the task instruction and the intended rubric item. This procedure makes the resulting evidence traceable to the same task specification rather than defining executable and semantic evaluators independently. Paired skill intervention Finally, we define the paired intervention used to study the practical value of structured payment guidance. In the without-skill condition, the agent receives the project and task instruction without access to alipay-payment-integration. In the with-skill condition, the same agent also receives the official alipay-payment-integration skill, which provides reusable integration guidance rather than task-specific solutions. Its scope includes payment-product guidance, integration procedures, interface explanations, SDK and sandbox guidance, code examples, and common payment-integration pitfalls. For each pair, we keep the agent, project, task instruction, and execution environment fixed, and evaluate both outputs using the same scenario-specific rubrics and evaluators. The paired protocol supports analysis of how access to structured Alipay payment guidance affects payment-product selection, integration efficiency, payment safety, and payment–business state consistency. 3.3
Evaluation and Scoring
3.3.1
Evaluation Signals
We evaluate each generated solution using test items aligned with its scenario-specific rubrics. The evaluator executes these checks on the modified repository. Each item is assigned to static analysis, unit testing where applicable, integration testing, E2E validation, or LLM-assisted review. Static checks. Static checks examine source-level and structural signals, including SDK or OpenAPI usage, credential handling, payment entry points, signature-verification capability, field binding, and state or refund models where applicable. For safety tasks, they also inspect notification-verification hooks, fake-success bypasses, idempotency or terminal-state guards, refund or query capability, and secret-management safeguards. These checks provide lightweight evidence of the expected implementation structure, but do not establish end-to-end runtime correctness. Unit tests. Unit tests are used where a task provides applicable local test coverage, such as projectspecific application tests. Integration tests. Integration tests verify application startup, product-specific payment flows, backend state transitions, and interactions with payment gateways or deterministic fixtures. Safety tasks additionally cover duplicated notifications, repeated requests, invalid signatures, wrong amounts, exceptional transaction states, refund boundaries, and related risks where applicable. Some tasks use local mock Alipay gateways or deterministic fixtures for reproducibility rather than relying uniformly on live sandbox behavior. E2E tests. End-to-end (E2E) tests exercise the user-facing or integration-facing entry points required by each task. They verify endpoint reachability, backend invocation, and observable states such as success, failure, cancellation, and pending where applicable. Some combined business-andpayment logic is also exercised by integration tests.
8
LLM-assisted domain assessment. LLM-assisted domain assessment examines rubric-defined semantic properties, including product mapping where applicable, server-side signing and confirmation, state-machine coherence, security-sensitive logic placement, and cross-component linkage. For safety tasks, it further assesses whether the implementation addresses the requested risk class rather than merely preserving the happy path. This signal helps identify plausible but unsafe implementations that deterministic checks may miss. 3.3.2
Rubric-Level Metric and Aggregation
We use Rubric Pass Rate (RPR) as the primary metric. RPR is the fraction of rubric requirements satisfied by a generated solution. A configuration c denotes a model evaluated under one skill condition, either without-skill or with-skill. For task instance t and trial r, let Mt denote the evaluation methods used for t. Let It,m denote the rubric items evaluated by method m. We encode each outcome as yc,t,r,i ∈ {0, 1}: pass is 1, and fail or evaluator error is 0. The weighted RPR is P P 1 y α i∈It,m c,t,r,i m∈Mt m |It,m | P , RPRc,t,r = m∈Mt αm where αm is the relative weight of method m. We use conservative relative weights: αstatic = αunit = αllm = 1,
αintegration = αe2e = 2.
The denominator normalizes over the methods available for t. Thus, RPR remains in [0, 1] when task instances use different evaluator sets. We assign higher weight to integration and end-to-end checks because they more directly test whether the generated repository preserves a coherent payment flow across backend services, application state and user-facing entry points. Static, unit and LLM-assisted checks provide complementary but more localized evidence. Reported scores are macro-averages over project-scenario task instances. For repeated trials of the same task instance, we first compute its mean RPR, so each task instance contributes equally regardless of its rubric count. LLM-assisted domain assessment enters the same aggregation as a rubric-aligned method. Evaluator errors are also retained separately for diagnostic analysis.
4
Experiments and Results
To assess the performance of coding agents on realistic Alipay payment integration tasks, we evaluated six models on all 18 task instances in Alipay-PIBench. Each instance paired a project with a scenario and was evaluated under paired conditions with and without access to alipay-paymentintegration. The experiments were organized around three research questions: • RQ1: Model capability. How capable are current models at completing Alipay payment integration across products and scenarios? • RQ2: Skill intervention. How does access to alipay-payment-integration change performance? • RQ3: Evaluation-method diagnostics. What aspects of Alipay payment integration capability are revealed by different evaluation methods?
9
4.1
Experimental Setup
We conducted all experiments on Alipay-PIBench using Claude Code version 2.1.200. The evaluation covered six high-performing model variants available in our environment at the time of the study: Claude Opus 4.8, GLM-5.2, Kimi K2.7 Code, DeepSeek-V4-Pro, MiniMax M3, and Qwen3.7-Max. Official documentation is available for the Claude Opus 4.8, GLM-5.2, Kimi K2.7 Code, DeepSeek V4, and MiniMax M3 variants used in this study (Anthropic, 2026; Z.AI, 2026; Moonshot AI, 2026; DeepSeek, 2026; MiniMax, 2026). Each trial started from a fresh Docker container initialized with the repository and evaluation environment for the corresponding task. Consequently, files, dependency caches, service states, and evaluator outputs generated during one trial did not affect any other trial. For each combination of model, project, scenario, and skill condition, we kept the task configuration and evaluation procedure fixed. 4.2
RQ1: Model Capability
RQ1 characterizes model capability across Alipay payment integration products and scenarios. The results reported here come from configurations in which alipay-payment-integration was installed in the coding agent. Under this setting, the comparison focuses on how models use repository context, payment-domain instructions, and project-specific code structure. Figure 3 reports mean RPR for each model, product, and scenario. The top and bottom panels correspond to Basic and Advanced tasks, respectively, with an independent color scale in each panel. First, model capability varies substantially across Alipay payment integration tasks. Claude Opus 4.8 achieves the highest overall mean RPR (91.37%), followed by GLM-5.2 (87.12%) and Kimi K2.7 Code (82.18%). Qwen3.7-Max and DeepSeek-V4-Pro obtain lower intermediate scores of 75.78% and 75.23%, respectively, while MiniMax M3 achieves 68.58%. The 22.79 percentage-point gap between the highest and lowest scores indicates substantial variation in model capability. Second, this variation is also reflected in performance consistency across products and scenarios. Claude Opus 4.8 achieves at least 90% mean RPR in 13 of the 18 product–scenario cells, and GLM-5.2 reaches this level in 10 cells. Kimi K2.7 Code reaches the same level in 6 cells, while Qwen3.7-Max and DeepSeek-V4-Pro do so in 2 cells each. At the same time, even Claude Opus 4.8 falls to 65.73% on Advanced In-App Payment, whereas MiniMax M3 ranges from 46.54% on Advanced Authorization Hold to 98.00% on Basic QR Code Payment. These results show that model capability is expressed through both overall performance and consistency across payment products and scenarios. Third, product context shapes how model capability is expressed. App has the lowest overall score (58.65%), and only Claude Opus 4.8 exceeds 65% mean RPR on this product (71.30%). The source application originally supported online booking followed by offline settlement, rather than an existing online payment flow. Agents therefore had to introduce a payment entry point, backend payment state, and confirmation logic into a workflow not organized around online settlement. The other models score lower on App, including GLM-5.2 at 62.71%, Qwen3.7-Max at 55.39%, and DeepSeek-V4-Pro at 51.12%. Pre-Authorization exhibits a different profile. Unlike In-App Payment, which is challenging for nearly all models, Pre-Authorization shows substantial variation across models and scenarios. Its overall mean RPR is 69.76%, while individual scores range from 42.96% to 98.67%. These results show that some products are difficult for nearly every model, whereas others differentiate models by their ability to maintain payment state, confirmation, and risk-handling logic across a longer code path.
10
Claude Opus 4.8
76.86
96.19
91.11
93.85
98.67
100.00
96.67
96.67
83.17
GLM-5.2
59.11
78.33
83.70
96.92
90.52
100.00
97.33
97.33
80.16
Kimi K2.7 Code
51.04
84.52
91.11
92.31
68.96
71.67
96.67
96.67
69.84
Qwen3.7-Max
45.00
59.23
88.15
90.77
42.96
89.96
82.33
83.33
56.19
70.00
MiniMax M3
58.21
63.10
83.70
76.36
53.89
60.30
76.33
98.00
61.90
40.00
DeepSeek-V4-Pro
40.00
65.24
79.26
92.31
53.70
72.27
96.67
85.83
54.92
In-App
Usage
Mobile Web
JSAPI
Auth.
PC Web
Order QR
QR
Merchant Init.
100.00 mean RPR (%)
Model
Basic
Claude Opus 4.8
65.73
94.69
85.30
94.17
80.76
98.47
96.22
100.00
96.21
GLM-5.2
66.31
82.54
96.43
83.33
73.51
97.05
91.17
97.38
97.01
Kimi K2.7 Code
64.17
88.85
80.24
88.33
79.43
93.24
82.06
89.95
90.15
Qwen3.7-Max
65.78
85.77
83.15
85.00
78.15
92.14
80.74
80.41
74.88
73.27
DeepSeek-V4-Pro
62.24
83.29
81.85
80.00
70.02
89.49
81.06
83.69
82.37
46.54
MiniMax M3
49.40
74.23
70.48
62.50
46.54
88.55
77.38
60.10
73.55
In-App
Usage
Mobile Web
JSAPI
Auth.
PC Web
Order QR
QR
Merchant Init.
100.00 mean RPR (%)
Model
Advanced
Alipay payment product
Figure 3 Model capability across Alipay payment integration products and scenarios under the with-skill condition. Each cell reports mean RPR for one model, product, and scenario. The top and bottom panels show Basic and Advanced tasks, respectively, using separate color scales.
Finally, Basic and Advanced tasks provide complementary views of model capability. Basic tasks primarily test whether a model can establish the initial Alipay payment loop. Advanced tasks focus on fund safety, risk handling, and boundary conditions after a basic integration exists. The scenario split therefore reveals different model profiles rather than a single difficulty ordering. Claude Opus 4.8 remains strong in both scenarios, with scores of 92.58% on Basic tasks and 90.17% on Advanced tasks. GLM-5.2 is similarly stable, scoring 87.05% and 87.19%, respectively. By contrast, Qwen3.7Max scores lower on Basic tasks (70.88%) than on Advanced tasks (80.67%), whereas MiniMax M3 performs better on Basic tasks than on Advanced tasks (70.20% versus 66.97%). These results indicate that constructing a payment flow and preserving safety-related payment behavior should be examined as distinct capabilities. RQ1 Summary. Model capability explains much of the observed performance spread and is expressed through overall performance, cross-product consistency, and adaptation to scenariospecific requirements. Alipay-PIBench therefore supports a fine-grained characterization of current models, showing where their Alipay payment integration capability is strong and 11
Table 2 Main results for skill intervention: mean RPR (%) with and without alipay-payment-integration across models, products, and scenarios. The task, repository, and evaluator are fixed within each paired setting; only access to alipay-payment-integration is varied. For each scenario–model block, No and With report the without-skill and with-skill conditions, and ∆ reports the percentage-point change, defined as 100× (with-skill mean RPR minus without-skill mean RPR). Positive and negative changes are shaded in blue and orange. Scenario
Model Claude Opus 4.8
GLM-5.2
Kimi K2.7 Code Basic DeepSeek-V4-Pro
MiniMax M3
Qwen3.7-Max
Claude Opus 4.8
GLM-5.2
Kimi K2.7 Code Advanced DeepSeek-V4-Pro
MiniMax M3
Qwen3.7-Max
Skill Mobile PC In-App Usage JSAPI Auth. Order QR Intervention Web Web without 50.3 92.4 91.1 80.8 89.2 86.7 96.7 with 76.9 96.2 91.1 93.8 98.7 100.0 96.7 +26.5 +3.8 0.0 +13.0 +9.4 +13.3 0.0 ∆ without 50.6 66.2 79.3 91.1 59.3 90.9 96.7 with 59.1 78.3 83.7 96.9 90.5 100.0 97.3 +8.5 +12.1 +4.4 +5.8 +31.3 +9.1 +0.7 ∆ without 44.3 72.4 74.8 65.4 60.3 60.6 75.0 with 51.0 84.5 91.1 92.3 69.0 71.7 96.7 +6.7 +12.1 +16.3 +26.9 +8.6 +11.1 +21.7 ∆ without 35.7 42.9 71.9 83.1 37.1 64.2 58.3 with 40.0 65.2 79.3 92.3 53.7 72.3 96.7 ∆ +4.3 +22.4 +7.4 +9.2 +16.6 +8.0 +38.3 without 43.2 60.7 73.3 71.8 47.9 59.5 52.3 with 58.2 63.1 83.7 76.4 53.9 60.3 76.3 ∆ +15.0 +2.4 +10.4 +4.5 +6.0 +0.8 +24.0 without 33.7 50.9 73.3 62.7 32.9 81.2 75.8 with 45.0 59.2 88.1 90.8 43.0 90.0 82.3 ∆ +11.3 +8.3 +14.8 +28.1 +10.0 +8.7 +6.5 without 60.3 84.1 82.4 93.3 71.6 97.9 90.7 with 65.7 94.7 85.3 94.2 80.8 98.5 96.2 ∆ +5.4 +10.6 +2.9 +0.8 +9.2 +0.6 +5.6 without 63.8 71.0 90.0 72.5 69.9 91.3 78.0 with 66.3 82.5 96.4 83.3 73.5 97.0 91.2 ∆ +2.5 +11.6 +6.4 +10.8 +3.6 +5.7 +13.2 without 45.4 73.5 83.9 65.8 31.4 91.7 63.5 with 64.2 88.8 80.2 88.3 79.4 93.2 82.1 ∆ +18.7 +15.4 -3.7 +22.5 +48.0 +1.6 +18.5 without 51.9 73.2 76.4 75.0 63.7 91.3 77.2 with 62.2 83.3 81.8 80.0 70.0 89.5 81.1 ∆ +10.3 +10.1 +5.5 +5.0 +6.3 -1.8 +3.8 without 33.7 67.5 72.4 55.8 51.8 78.7 67.0 with 49.4 74.2 70.5 62.5 46.5 88.5 77.4 +15.7 +6.8 -2.0 +6.7 -5.3 +9.8 +10.4 ∆ without 52.3 76.2 66.6 75.0 58.8 65.7 75.5 with 65.8 85.8 83.2 85.0 78.2 92.1 80.7 ∆ +13.5 +9.6 +16.5 +10.0 +19.4 +26.4 +5.2
QR 96.7 96.7 0.0 90.8 97.3 +6.5 90.0 96.7 +6.7 60.8 85.8 +25.0 96.5 98.0 +1.5 81.7 83.3 +1.7 96.4 100.0 +3.6 93.3 97.4 +4.1 69.7 89.9 +20.3 75.5 83.7 +8.2 45.4 60.1 +14.7 78.9 80.4 +1.5
Merchant Init. 71.7 83.2 +11.4 65.7 80.2 +14.4 55.9 69.8 +14.0 45.1 54.9 +9.8 48.3 61.9 +13.7 51.1 56.2 +5.1 89.9 96.2 +6.3 93.7 97.0 +3.3 76.4 90.2 +13.8 57.4 82.4 +25.0 64.9 73.5 +8.6 70.8 74.9 +4.0
Avg. 84.0 92.6 +8.6 76.7 87.0 +10.3 66.5 80.3 +13.8 55.5 71.1 +15.7 61.5 70.2 +8.7 60.4 70.9 +10.5 85.2 90.2 +5.0 80.4 87.2 +6.8 66.8 84.0 +17.2 71.3 79.3 +8.0 59.7 67.0 +7.3 68.9 80.7 +11.8
where it remains limited. 4.3
RQ2: Skill Intervention
RQ2 examines how access to alipay-payment-integration changes coding-agent performance on Alipay payment integration tasks. We compare paired without-skill and with-skill trials under the same task, repository, and evaluation procedure. Table 2 reports mean RPR for each model, product, and scenario, together with the percentage-point change associated with access to alipay-paymentintegration. Across all 108 model–product–scenario cells, access to alipay-payment-integration is associated with a 10.31 percentage-point increase in mean RPR. The paired change is positive in 101 cells, lower in 4 cells, and effectively unchanged in 3 cells. The gain is larger for Basic tasks than for Advanced tasks (+11.27 versus +9.35 points), but both scenarios benefit. Every evaluated model also 12
improves on average, with gains ranging from +6.81 points for Claude Opus 4.8 to +15.51 points for Kimi K2.7 Code. These results show a clear positive association under the paired evaluation protocol, with the magnitude varying by model, product, and scenario. The gain is larger when the without-skill baseline is low. Cells with a without-skill mean RPR below 40% improve by 17.64 points on average, compared with 4.52 points for cells with baseline scores between 80% and 100%. This pattern suggests that alipay-payment-integration is particularly useful when the baseline solution has not recovered the payment-flow structure, although the smaller gains at high baseline scores may also reflect a ceiling effect. Representative examples include Kimi K2.7 Code on Advanced Pre-Authorization, which improves from 31.43% to 79.43% (+47.99 points), and DeepSeek-V4-Pro on Basic Barcode, which improves from 58.33% to 96.67% (+38.33 points). These examples indicate that alipay-payment-integration is especially useful when models must recover product selection, payment-state transitions, payment confirmation, and callback or notification handling from repository context. The few lower-scoring cells are concentrated in Advanced tasks, where evaluation emphasizes safety and boundary behavior rather than simply adding a missing payment entry point. The remaining decreases are isolated across products and models, rather than indicating a consistent degradation under with-skill. Manual inspection of the corresponding execution traces indicates that these cases are associated with runtime interruptions or implementation failures in individual trials. As a secondary analysis, we examined output tokens normalized by mean RPR to assess generation efficiency. For each model and scenario, we averaged the paired results across the nine payment products, yielding 12 model–scenario averages in total (6 models × 2 scenarios). The relative output/RPR ratio was below 1.0 for 11 of the 12 averages, ranging from 0.72 to 1.03, with a mean of 0.86. This result indicates that alipay-payment-integration generally reduced the amount of generated output required for each unit of rubric completion, although the effect was not universal. Because the ratio captures only output tokens relative to rubric completion, it should be interpreted as a relative efficiency measure rather than an end-to-end cost estimate. RQ2 Summary. Access to alipay-payment-integration helps coding agents complete Alipay payment integration more effectively and efficiently. It improves mean RPR across models and products, with the largest gains occurring when models cannot reconstruct product-specific payment logic from repository context alone. It also tends to reduce output tokens per unit RPR, suggesting that structured Alipay guidance can reduce trial and error and support a more efficient integration process for developers using coding agents. 4.4
RQ3: Evaluation-Method Diagnostics
RQ3 examines what different evaluation methods reveal about Alipay payment integration capability. The methods target different verification layers: Static checks inspect source-level structure, Unit tests provide local behavioral evidence where available, Integration tests exercise backend interactions and state transitions, E2E tests inspect externally visible workflow behavior, and LLM-assisted assessment examines payment-domain semantics. Figure 4 reports the unweighted within-method RPR under the with-skill condition. The first pattern is a gap between structural completion and executable behavior. Static RPR averages 92.79% for Basic tasks, whereas Integration and E2E average 73.16% and 78.24%, respectively. For example, Basic In-App Payment obtains 83.0% Static RPR but only 38.6% Integration RPR. These results show that agents can introduce expected SDK usage, payment entry points, fields, and state models without fully connecting them into an executable payment workflow. 13
Table 3 Output-normalized effectiveness under skill intervention. No and With report output tokens per unit RPR, computed as output tokens in K divided by mean RPR expressed on a 0–1 scale, for the without-skill and with-skill conditions, respectively. The relative row reports the paired ratio (O/RPR)with−skill /(O/RPR)without−skill for the same model–product–scenario cell. Values below 1 indicate fewer generated output tokens per unit RPR under with-skill. Blue and orange shading mark lower and higher relative output/RPR, respectively. Scenario
Model Claude Opus 4.8
GLM-5.2
Kimi K2.7 Code Basic DeepSeek-V4-Pro
MiniMax M3
Qwen3.7-Max
Claude Opus 4.8
GLM-5.2
Kimi K2.7 Code Advanced DeepSeek-V4-Pro
MiniMax M3
Qwen3.7-Max
In-App Usage without with relative without with relative without with relative without with relative without with relative without with relative without with relative without with relative without with relative without with relative without with relative without with relative
133.1 95.5 0.72 85.1 75.0 0.88 84.8 66.3 0.78 87.2 76.9 0.88 88.1 58.7 0.67 98.1 64.3 0.66 134.7 116.5 0.86 96.3 89.1 0.93 81.1 63.6 0.78 72.8 36.1 0.50 128.5 77.1 0.60 65.5 53.6 0.82
31.2 36.2 1.16 21.8 24.5 1.12 18.7 21.5 1.15 28.9 19.9 0.69 42.3 48.3 1.14 28.6 27.4 0.96 36.4 52.0 1.43 36.9 26.3 0.71 36.1 29.1 0.81 18.0 14.2 0.79 57.8 44.5 0.77 18.0 14.2 0.79
Mobile PC Merchant JSAPI Auth. Order QR QR Avg. Web Web Init. 40.0 79.6 82.0 40.1 46.0 55.9 73.4 64.6 35.8 76.9 79.6 25.6 51.4 53.7 87.6 60.3 0.90 0.97 0.97 0.64 1.12 0.96 1.19 0.93 36.6 56.0 68.7 22.2 39.6 37.5 42.4 45.5 49.2 43.4 56.8 18.8 33.4 41.4 43.0 42.8 1.34 0.78 0.83 0.85 0.85 1.10 1.01 0.94 36.3 76.1 58.6 34.8 35.2 37.9 70.5 50.3 36.9 52.9 34.8 23.4 33.8 30.8 57.5 39.8 1.02 0.70 0.59 0.67 0.96 0.81 0.82 0.79 35.1 61.1 85.9 24.8 28.2 46.1 52.1 49.9 27.7 30.0 59.1 23.1 19.7 22.4 41.1 35.6 0.79 0.49 0.69 0.93 0.70 0.49 0.79 0.71 47.9 90.5 68.8 53.0 74.4 52.3 72.6 65.5 49.2 57.9 76.9 32.9 46.5 37.5 66.6 52.7 1.03 0.64 1.12 0.62 0.63 0.72 0.92 0.80 33.7 60.2 105.3 23.9 34.9 39.7 72.3 55.2 25.9 43.8 81.7 25.2 26.5 38.3 45.8 42.1 0.77 0.73 0.78 1.06 0.76 0.96 0.63 0.76 40.9 73.0 64.0 56.9 85.4 88.0 116.7 77.3 46.5 71.4 67.8 68.4 89.9 65.2 89.2 74.1 1.14 0.98 1.06 1.20 1.05 0.74 0.76 0.96 36.4 59.3 62.5 70.8 59.0 55.3 86.6 62.5 34.2 48.8 57.2 44.0 69.7 53.3 67.6 54.5 0.94 0.82 0.92 0.62 1.18 0.96 0.78 0.87 42.6 67.0 125.1 35.4 75.3 66.1 80.8 67.7 42.6 47.2 39.0 37.1 52.6 52.3 55.8 46.6 1.00 0.70 0.31 1.05 0.70 0.79 0.69 0.69 23.6 32.8 35.7 25.0 52.8 41.2 67.2 41.0 22.5 32.4 31.4 20.7 36.8 31.3 33.9 28.8 0.95 0.99 0.88 0.83 0.70 0.76 0.50 0.70 51.8 88.8 54.1 33.6 82.6 122.6 118.6 82.0 54.6 81.0 94.8 43.7 47.0 86.6 55.9 65.0 1.05 0.91 1.75 1.30 0.57 0.71 0.47 0.79 29.1 47.4 67.1 42.8 48.0 82.7 83.6 53.8 25.2 40.9 46.6 32.4 57.6 64.7 54.3 43.3 0.86 0.86 0.69 0.76 1.20 0.78 0.65 0.80
The second pattern is a gap between executable paths and payment-domain requirements. In Advanced tasks, E2E RPR averages 97.03%, whereas LLM-assisted domain assessment averages 61.05%. Advanced Order QR, for example, reaches 100% E2E RPR but only 58.6% LLM-assisted RPR. This result indicates that an implementation can execute through the tested entry points while still lacking evidence for payment-safety and state-consistency requirements. The reverse pattern also occurs. Basic Pre-Authorization obtains 96.4% under LLM-assisted assessment but only 39.3% under E2E testing. An implementation may therefore appear semantically consistent with product requirements while failing through the required application-facing workflow. Together, these patterns show that structural, executable, and domain-level evaluation expose different forms of incomplete payment integration.
14
Static
Unit
E2E
Integ.
LLM
Basic 96.6
100 83.0
mean RPR (%)
100.0
99.3
98.8
96.4
100.0
81.8
93.3
95.0 86.7
82.7
74.1 76.5
75.0
75
95.0
91.4
92.1 79.3
76.4
100.0
78.2
92.8 78.2
100.0
92.9
73.2
87.8
82.0
60.0 50.6
50
44.8 41.1
39.3
38.6
25 0
Advanced
mean RPR (%)
100 75
99.4
66.1
72.2
100.0
100.0 95.0
93.6
97.0
94.3
92.9
76.8
83.5 78.6
73.5
80.0
77.4 66.7 58.6
58.3
61.0 61.3
67.3
48.6
50 25 0
86.9
84.2 70.7
100.0 94.9
92.8 90.0
87.6
84.9 67.3
100.0
99.6
99.2
16.7
In-App
Usage
Mobile Web
JSAPI
Auth.
PC Web
Order QR
QR
Merchant Init.
Avg.
Figure 4 Evaluation-method diagnostics: method-specific RPR (%) by product and scenario under the with-skill condition. Bars report evaluator-method RPR before method weighting, averaged over models; missing evaluator methods are omitted. The Avg. group reports the across-product mean, with vertical lines indicating the standard deviation across products. The top and bottom panels show Basic and Advanced tasks, respectively.
Because the methods cover different rubric subsets, their scores should not be interpreted as direct comparisons of evaluator accuracy or strictness. Their value lies in failure localization: the combined signals help determine whether an integration is missing expected structure, fails during execution, or remains incomplete in payment safety and business-state consistency. RQ3 Summary. Payment integration cannot be reliably assessed by checking only whether the expected code is present or whether a tested path executes. Static, execution-based, and LLM-assisted evaluations reveal structural completion, executable workflow behavior, and payment-domain correctness from different perspectives. Multi-method evaluation is therefore necessary not only to determine whether an integration fails, but also to identify where it fails.
5
Conclusion
Taken together, the results show that Alipay payment integration requires an agent to select an appropriate product, modify a repository across components, preserve payment state, and satisfy payment-specific safety requirements. We developed Alipay-PIBench to measure these demands in realistic project repositories. The benchmark connects payment products with business workflows and existing codebases, and separates Basic functional completion from Advanced risk-aware hardening. Its rubric-derived framework combines deterministic checks with supplementary LLM-assisted assessment, while the paired without-skill/with-skill protocol measures the effect of structured payment guidance. In our evaluation of 18 task instances and six models, mean RPR under the with-skill setting ranged from 68.58% to 91.37%. Relative to the corresponding without-skill condition, mean RPR increased by 10.31 percentage points on average, with gains in 101 of 108 model–product– scenario comparisons. The average improvement was larger for Basic tasks than for Advanced tasks (+11.27 versus +9.35 percentage points). The method-specific results also separated source-level completion from executable payment behavior and payment-domain requirements, supporting the use of progressive scenarios and complementary rubric-grounded signals. Our results are grounded in the Alipay Open Platform products, task instances, and coding-agent 15
configurations studied here. Future work can extend the benchmark with additional models and agent frameworks, broader repository coverage, and longer-horizon executable payment workflows, allowing the robustness of these findings to be examined across a wider range of settings.
References A2M Recipes. AI Payment Collection Pilot project site. https://babyeat.coze.site/, 2026. Accessed June 2026. Alipay Open Platform. Alipay open platform documentation. https://open.alipay.com/, 2026. Accessed June 2026. Anthropic. Introducing claude opus 4.8. https://www.anthropic.com/news/claude-opus-4-8, 2026. Accessed July 2026. James Beswick and Peter Epsteen. You can’t whisper at an ai agent. Stripe Developer Blog, May 2026. https://stripe.dev/blog/ ai-steering-experiments. Accessed: 2026-07-13. Guoxin Chen, Fanzhe Meng, Jiale Zhao, Minghao Li, Daixuan Cheng, Huatong Song, Jie Chen, Yuzhi Lin, Hui Chen, Xin Zhao, Ruihua Song, Chang Liu, Cheng Chen, Kai Jia, and Ji-Rong Wen. BeyondSWE: Can current code agent survive beyond single-repo bug fixing? arXiv preprint arXiv:2603.03194, 2026. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. DeepSeek. DeepSeek V4 preview release. https://api-docs.deepseek.com/news/news260424, 2026. Accessed July 2026. Jingzhe Ding et al. NL2Repo-Bench: Towards long-horizon repository generation evaluation of coding agents. arXiv preprint arXiv:2512.12730, 2026. Ahmed Elassas. BookCars project repository. https://github.com/aelassas/bookcars, 2026. Accessed June 2026. Endor Labs. Ai coding agent security benchmark. https://www.endorlabs.com/research/ai-code-security-benchmark, 2026. Accessed June 2026. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. NeurIPS, 2021. HiEventsDev. Hi.Events project repository. https://github.com/HiEventsDev/hi.events, 2026. Accessed June 2026. Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks. arXiv preprint arXiv:2607.07946, 2026. https://arxiv.org/abs/2607.07946. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. Smriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali, Shubham Ugare, and Satish Chandra. ProdCodeBench: A production-derived benchmark for evaluating ai coding agents. arXiv preprint arXiv:2604.01527, 2026. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? International Conference on Learning Representations, 2024. Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. FEA-Bench: A benchmark for evaluating repository-level code generation for feature implementation. arXiv preprint arXiv:2503.06680, 2025. linlinjava. litemall project repository. https://github.com/linlinjava/litemall, 2026. Accessed June 2026. Pengrui Lu, Shiqi Zhang, Yunzhong Hou, Lyumanshan Ye, Chaoyi Huang, Zixi Chen, Ji Zeng, Hantao Jiang, Pengfei Liu, Yiwei Wang, and Ming-Hsuan Yang. ProjDevBench: Benchmarking ai coding agents on end-to-end project development. arXiv preprint arXiv:2602.01655, 2026. lubusIN. laravel-gymie project repository. https://github.com/lubusIN/laravel-gymie, 2026. Accessed June 2026.
16
MiniMax. MiniMax M3: Frontier coding, 1m context, native multimodality. https://www.minimax.io/blog/minimax-m3, 2026. Accessed July 2026. Moonshot AI. Kimi K2.7 Code quickstart. https://platform.kimi.ai/docs/guide/kimi-k2-7-code-quickstart, 2026. Accessed July 2026. opensourcepos. Open Source Point of Sale project repository. https://github.com/opensourcepos/opensourcepos, 2026. Accessed June 2026. Yangda Peng, Yunjia Qi, Hao Peng, et al. Can LLM-as-a-Judge reliably verify rubrics in agentic scenarios? arXiv preprint arXiv:2606.29920, 2026. Arafay Saleem. ez_tickets_app project repository. https://github.com/arafaysaleem/ez_tickets_app, 2026. Accessed June 2026. Stripe. Can ai agents build real stripe integrations? we built a benchmark to find out. can-ai-agents-build-real-stripe-integrations, 2026. Accessed June 2026.
https://stripe.com/blog/
UniBee-Billing. UniBee API project repository. https://github.com/UniBee-Billing/unibee-api, 2026. Accessed June 2026. Z.AI. GLM-5.2 overview. https://docs.z.ai/guides/llm/glm-5.2, 2026. Accessed July 2026. Yiming Zhang, Yingfan Ma, Yanmei Gu, Zhengkai Yang, Yihong Zhuang, Feng Wang, Zenan Huang, Yuanyuan Wang, Chao Huang, Bowen Song, Cheng Lin, and Junbo Zhao. ABench-Physics: Benchmarking physical reasoning in llms via high-difficulty and dynamic physics problems. arXiv preprint arXiv:2507.04766, 2025.
17