Does the Judge See What the Evaluation Requires? Input Assurance for LLM Agent Evaluation

Does the Judge See What the Evaluation Requires? Input Assurance for LLM Agent Evaluation Boming Xia∗ , Qinghua Lu† , Runze Li‡ , Dino Sejdinovic§ and Liming Zhu† ∗ Responsible AI Research Centre, Adelaide University, Australia † CSIRO, Australia ‡ UNSW Sydney, Australia

arXiv:2611.02676v1 [cs.SE] 2 Nov 2026

§ Adelaide University, Australia

reads the bytes last checked. Each requires distinct evidence: comparison with source records for content, violating cases for requirements coverage, integration tests for failure handling and byte checks at use for integrity. The pipeline compares judgements on trajectory prefixes (ending at the judged step) with judgements on full trajectories, to assess how later context changes error flags. Its renderer kept tool results but omitted every structured tool call, leaving 53 of 60 judged steps blank in the original packets. We compared these with call-restored packets, which restore calls throughout each packet, under five judge configurations (model, prompt and delivery settings). With the original packets, moving from prefix to full packets raised error flags under the first configuration, J0, by 15 on error steps and by 2 on correct steps, a contrast of 13. With the calls restored, this contrast was 2. Research software can produce reproducible results despite bugs [3], and direct data validation can expose problems not apparent from model quality [4]. Work on evaluation harnesses addresses implementation choices and reproducibility [5], while benchmark audits identify flaws in task setup and grading [6]. AgentRewardBench shows that trajectory representation affects judge agreement with experts [1]. Judge-robustness studies examine deliberately changed inputs [7], and content-free responses can draw favourable verdicts [8], [9]. Fault-injection studies examine agent failures [10], [11], while SPADE I. I NTRODUCTION synthesises and selects assertions for LLM pipeline outputs [12]. LLM judges are used to assess individual steps in recorded Unlike these studies, we examine whether the judge’s input runs (trajectories) of tool-using agents [1], [2]. In the pipeline contract is preserved and enforced from rendering to use, and we study, a renderer turns structured trajectory records into what the pipeline’s own checks establish about the bytes the text packets for the judge. Two software checkers test source judge reads. We assess the two separately written checkers as records and packets against written requirements. An agent test oracles [13] and use a recording stub in place of the judge may leave its message text empty and place its action only in to observe the bytes read in integration tests. a structured tool-call field. If the renderer copies only message On the real packets both checkers passed, and staging text, it omits the call and makes the judged step appear blank. (copying packets for the judge and checking their hashes) The packet can still match its recorded hash, because a hash succeeded. Yet mutation cases showed that both accepted shows only that the bytes are unchanged. a tool result with no matching call, and code inspection We ask what evidence establishes that a judge’s input meets found a hash comparison that could never fail. Constructed the evaluation’s requirements, and call such evidence input integration tests showed that failed or changed packets could assurance. The content and validation rules that this input still reach the point of use. These defects reflect established must satisfy form the judge’s input contract. We propose four software-engineering concerns: incomplete test oracles [14], obligations for the software that prepares that input: (1) required shared failures in separately written programs [15], redundant content survives rendering, (2) each checker covers its stated assertions [16], build failure handling [17] and time-of-checkrequirements, (3) a failed check stops later use and (4) the judge to-time-of-use gaps [18]. Output-based testing requires a fault’s

Abstract—An LLM judge assesses agent steps from text packets. A renderer converts recorded runs into these packets, while software checkers test records and packets against requirements. We ask what evidence shows a packet is what the evaluation requires and propose four input-assurance obligations: required content survives rendering, checkers cover their requirements, a failed check stops later use and the judge reads the bytes last checked. In one pipeline with 30 trajectories, matching hashes, passing checkers and successful staging (copying packets for the judge) coexisted with violations of all four obligations, the last two in integration tests. The renderer omitted every structured tool call, leaving 53 of 60 judged steps blank despite matching hashes. For one judge configuration, later context raised error flags by 13 more on error than on correct steps. Restoring every call reduced this contrast to 2, a reduction of 11 (0.37 per trajectory, 95% cluster-bootstrap interval 0.17 to 0.53). In packets ending at the judged step, that configuration abstained at least once on every blank step, while the other four sometimes abstained but also flagged blank error and correct steps. Both separately written checkers accepted an unmatched tool result. In integration tests, a stub replacing the judge read a failed build’s packet and a copy changed after staging. Three added controls stopped 18 of 19 constructed fault cases before the stub read any packet. The remaining stale script never ran. We turn the obligations into four tests and plan to evaluate their transfer to independent pipelines. Index Terms—AI engineering, LLM agents, LLM-as-judge, evaluation software, input validation, test oracles, fault injection

effects to reach an observed result [19], but the judges’ answers, although affected by the omission, did not identify the missing calls. In prefix packets, whether a blank step drew abstentions or error flags depended on the judge configuration. The paper makes three contributions. It organises established assurance practices into four obligations for the boundary between recorded trajectories and judge inputs, with a direct test for each (Table I). It shows a rendering omission preserved by matching hashes, requirements missed by passing checkers and failed or changed packets read in integration tests, and measures how restoring the calls changed a judge-based contrast. It evaluates 3 controls on 19 constructed fault cases and proposes an evaluation of the tests’ transfer to independent pipelines.

G = (FE −PE )−(FC −PC ), where F and P count consensus flags in full and prefix packets, and E and C denote error and comparison targets. B. Pipeline and evidence

The builder creates the R packets and writes a manifest of expected hashes for all 240 O/R packets and a job list assigning each draw its packet. Staging copies the packets into a separate folder and checks hashes of their text against the manifest. Emission turns the job list into scripts identifying each draw’s packet, and dispatch runs them. Two software checkers validate source records and packets, rather than judge agent behaviour. The inline checker runs inside the builder, before staging, while the disk checker later rereads source records and archived O/R II. C ASE AND S TUDY D ESIGN packets. This study’s written plan assigned both checkers ten A. Targets and renderings requirements, including call placement and staged-file hashes Each judgement concerns one agent step, the target step, (supplementary Section II). The two checkers were written marked in a text packet containing task context and recorded from the same plan but separately, and the disk checker’s trajectory messages. A prefix packet ends at that step, before writer did not read the builder’s code. Both checkers reported that all ten requirements were met on its tool result, while a full packet continues to the trajectory’s the real packets, and J0 judged all 240 packets in the diagnostic end. We use the BFCL (Berkeley Function-Calling Leaderrun (Section V). We read both checkers’ code and ran them board) [20] and tau2 [21] (customer service with simulated-user on valid inputs and on 38 constructed mutation cases, each tool calls) subsets of AgentProcessBench [2]. The benchmark intended to violate a requirement by changing source records, labels agent steps +1 (correct and effective), 0 (neutral) or in-memory packets or saved files. We counted a miss when −1 (incorrect or harmful). Steps depending on an error remain a checker reported no problem while the mutation was still −1 until it is corrected or an independent subtask begins. The present. The shared mutation cases are the 34 cases whose labels select targets but are not included in judge packets. changes reached inputs read by both checkers (supplementary Each selected trajectory contributes two target steps. The Section II). error target is the earliest tool-calling step that begins a run Twelve integration tests examined whether packets could of −1 labels. The comparison target is the closest earlier toolstill reach later steps after a failed check or file change. We calling step labelled +1. Of 500 trajectories, 99 remained after separately ran 19 constructed fault cases under four cumulative applying these rules and excluding 32 used in earlier runs. An conditions: P0 without added controls and P1 to P3 with earlier study sampled 30 of them at random with a fixed seed, controls added successively (Section IV). Eight cases reused 14 BFCL and 16 tau2, and we reuse that cohort (supplementary mutation cases or integration tests, and eleven used new Table S1 and Section I). injection points chosen by a fixed rule. A recording stub The original renderer kept message roles and text but omitted replaced the judge in these tests and hashed any packet it the structured tool-call field (messages[].tool_calls). read, showing which bytes reached the point of use. A target with empty message text therefore appeared blank: 26 of 30 error targets and 27 of 30 comparison targets (53 of III. W HAT PASSING C HECKERS D ID N OT S HOW 60). The other seven had text but still lost their calls. A checker may pass without covering every assigned requireWe compared original (O) and call-restored (R) packets ment (obligation 2). Coverage depends on which properties it (Fig. 1). O reuses the earlier study’s archived packets byte checks, which files it reads and when it runs. Both checkers for byte, and each still matched its recorded hash. R inserts accepted the real packets, but code inspection and mutation each missing call as [TOOL CALL] name(arguments) cases showed that neither checked for tool results without after the text of its originating message, preserving call order matching calls or inspected the staged copies. and arguments. R restores calls throughout each packet, not only at the target, so the O/R comparison concerns the whole A. A requirement checked in only one direction rendering. In both renderings, tool results omit their tool names One requirement called for one-to-one matching between and the ids linking them to calls, and tau2 packets omit tool tool calls and results in the source records. Both checkers definitions. R therefore restores call lines but is not a complete verified that each call had a result, but not that each result had rendering (supplementary Section I). The two target types, a call. The inline checker’s loop starts from calls and skips two packet extents and two renderings give eight cells of 30 messages without them: packets, or 240 packets in total. Each judge configuration tcs = m.get("tool_calls") or [] scheduled two separate judgements (draws) per packet, or if not tcs: 480 per configuration, and a consensus error flag required continue both draws to answer yes. For each rendering, the contrast is

Inline checker: source, in-memory packets, O archive Structured trajectory

Build R packets

Disk checker: source + archived files

O/R archive

Staging: copy + hash

Manifest + job list

Staged packet files

Dispatch: judge reads copy

Emit scripts

Source: selected fields

O: original packet

R: calls restored

"content": "", "tool_calls": [{"function": { "name": "cd", "arguments": "{\"folder\":\"..\"}"}}]

[BEGIN TARGET STEP] [assistant]

[BEGIN TARGET STEP] [assistant] [TOOL CALL] cd({"folder":".."}) [END TARGET STEP]

[END TARGET STEP]

Fig. 1. Pipeline before the added controls (top) and one error target, bfcl:527, in its source record and prefix packets (bottom). The original (O) packet omits the target’s tool call, and the call-restored (R) packet includes it. In both draws, J0 abstained on O and answered yes on R. Dashed arrows locate the checkers, and their boxes list the inputs they read. Downstream steps did not enforce checker outcomes. TABLE I O BLIGATIONS FOR INPUT ASSURANCE WITH THEIR TESTS (S ECTION VI), WHAT THE CASE SHOWED AND THE CONTROLS . ROW 1 USES THE REAL PACKETS , AND ROWS 2 TO 4 CODE INSPECTION AND CONSTRUCTED CASES . T ESTED : EXERCISED IN THIS STUDY. R ECOMMENDED : NOT EXERCISED . Obligation and test

What the case showed

Control and status

1. Required content survives rendering. Test: render an action held only in a structured field 2. Checkers cover their requirements. Test: violate each requirement on the object and at the pipeline step it covers 3. A failed check stops later use. Test: continue after a failure 4. The judge reads the bytes last checked. Test: change a checked file

Packets omitted every call yet matched their hashes

Restored call lines (R): tested

Both accepted an injected unmatched result, and neither read the staged copies

Reverse matching (P3): tested on three cases. Disk-checker fixes: recommended

A failed build’s packet was staged and read

Stop on failure (P1): tested

The stub read a copy changed after staging and a packet with line endings changed before staging

Checks before use (P2): tested after staging. Build-time exact-byte hashes checked at staging: recommended

When we added a tool result with no matching call to a source record and rebuilt the packets, both checkers exited successfully. This was the only one of the 34 shared mutation cases that both accepted. Across the 34 cases, the inline checker exited successfully despite two violations and the disk checker despite one. Their error output pointed to the violated requirement in 23 and 30 cases, respectively, and their other nonzero exits named other failures or crashed (supplementary Table S3). The disk checker rejected an invalid JSON argument string that the inline checker accepted. The shared omission illustrates why separately written checkers based on one plan cannot be assumed to fail independently [15]. B. A requirement checked on the wrong files The plan required 240 staged files, each with a hash matching its manifest entry, and assigned this requirement to both checkers. The inline checker runs before staging creates those files, so it cannot inspect them. The disk checker runs later but reads archived packets, not the staged copies or the written manifest. Only staging’s own hash check compared staged files with the manifest. The disk checker’s own hash comparison built its expected hash from archive text it had loaded, then hashed the same unchanged text again. Here O and R hold that loaded text, and m is an expected-hash entry built from it:

"sha256": sha256_hex(txt), ... txt = (O if rend == "O" else R)[(rid, cond)] if sha256_hex(txt) != m["sha256"]: sha_bad += 1

Since both hashes come from the same unchanged text, this comparison cannot detect a changed file. It has the same failure mechanism as a redundant assertion in software tests [16]. To cover its assigned staged-file requirement, the disk checker must inspect the staged file set and compare each copy’s hash with its previously recorded manifest entry (Table I). IV. W HAT S UCCESSFUL S TAGING D ID N OT S TOP Before the added controls, downstream steps did not require earlier steps to have succeeded, and nothing checked a staged packet’s bytes when the judge read it (obligations 3 and 4). A. Later steps read failed and changed packets In an integration test, the builder wrote a packet missing the target’s tool call and exited with an error. The test deliberately continued through staging and emission. Both succeeded, and the recording stub read the altered packet in both scheduled draws. Staging passed because the failed build had written the packet and its manifest hash from the same altered text. In another test, we changed a staged packet after staging passed, and the stub read the changed bytes: a time-of-check-to-timeof-use gap [18]. A third test changed a packet’s line endings to CRLF after build but before staging. Staging hashed the packet

TABLE II O UTCOMES OF 19 CONSTRUCTED FAULT CASES UNDER CUMULATIVE CONTROLS , WITH READS BY THE RECORDING STUB . P0 CONTINUES AFTER A FAILURE , P1 STOPS ON FAILURE , P2 ALSO CHECKS FILES BEFORE USE AND P3 ALSO MATCHES EVERY RESULT TO A CALL . Fault and injection point

Cases

Outcome under P0

First stop or refusal

Restored call deleted or changed during build Restored call deleted after build Invalid argument JSON in a source record Unmatched result added to a source record Manifest hash corrupted after build Failed build with previous outputs present Packet changed after staging Extra file present before staging Extra stale script

5 Altered packet read 1 Altered packet read 1 Altered packet read 3 Altered packet read 3 Unaltered packet read after staging failed 1 Previous build’s output read 3 Altered packet read 1 Run completed with an extra staged file 1 Script unused

Total

19

P1: build P1: staging P1: disk check P3: reverse matching P1: staging P1: build P2: before dispatch P2: before dispatch None: unused under P0–P3

Where a first stop or refusal is listed, earlier conditions retain the P0 outcome and later conditions retain that stop or refusal.

as text, normalising CRLF to LF, so its check passed although the stub read changed bytes (supplementary Section III).

TABLE III C ONSENSUS ERROR FLAGS ( BOTH DRAWS Y E S ) AMONG 30 TARGETS PER CELL , ORIGINAL (O) → CALL - RESTORED (R) PACKETS .

B. Different controls stop different cases We ran 19 constructed fault cases under four cumulative conditions (Table II). P0 invokes later steps despite earlier failures and dispatches only scripts emitted for that run. P1 stops when a step exits with an error or a required checker does not report success. P2 records each staged packet’s exact-byte hash at staging and checks the archived packets against those hashes before and after the disk check. Before dispatch, it requires successful validation for this run and the expected, unchanged staged files, manifest, job list and scripts. The recording stub also checks each packet’s hash at read time (supplementary Section IV). P3 adds reverse matching: every tool result in the source records must answer a call, including calls by simulated users. P1 stopped 11 cases. P2 refused dispatch in four more: three packets changed after staging and one extra staged file. The three unmatched-result cases passed P2 because neither checker enforced reverse matching and their files remained unchanged, and P3 stopped them. The remaining case, an extra stale script, was unused under every condition. Under P0, 14 of the 18 cases stopped by P1 to P3 had led to altered or stale packet reads. Both cases without injected faults (the unaltered cohort and one with an unrelated file outside the staged folder) completed under all conditions, and every packet the stub read matched its hash in the diagnostic run’s manifest. In a separate test, the stub under P2 and P3 refused a packet changed after the last check before dispatch, in both draws. The line-ending change made before staging was not among the 19 cases. By code inspection, P2 and P3 would accept it: staging hashes normalised text, and exact-byte packet hashes start only at staging.

Error target Key Model alias J0 J1 J2 J3 J4

Prefix

Comparison (+1) Full

Prefix

Full

claude-opus-5 2 → 20 17 → 22 0 → 2 gpt-5.6-terra 22 → 19 13 → 18 15 → 9 gpt-5.6-luna 22 → 20 12 → 21 23 → 7 claude-sonnet-5 8 → 21 9 → 15 7 → 9 qwen3.6:27b 21 → 19 18 → 16 17 → 7

2→2 6→3 8→5 4→3 15 → 4

Each arrow compares cell totals (per-target changes: supplementary Table S9).

counterproductive for the user’s task. The permitted answers were yes, no and cannot_decide. J0’s O and R draws ran in a randomly interleaved order to limit time-related confounding, each in a fresh session that read its packet from a file. The recorded reads returned every packet line. J1 to J4, run later to examine variation across configurations, received packet text in their prompts without tools. For each J1 to J4 request, the recorded hash matched the hash computed from the manifest-listed packet bytes followed by one separating line break. We interpret differences at the configuration level because model, prompt wrapper, output-schema enforcement, reasoning settings, client and serving software, packet delivery and date changed together (supplementary Sections V and VI). For J0, the error-prefix flag count rose from 2 with O to 20 with R: 18 targets gained a consensus flag and none lost one (supplementary Table S9). With O packets, full packets had 15 more flags than prefix packets on error targets and 2 more on comparison targets, giving G = 13. Fourteen of the 17 error-full flags concerned targets blank in their original prefix packets, where J0 had abstained. With R packets, G = 2, a reduction of 11 (0.37 per trajectory). Resampling the 22 V. W HAT J UDGE R ESPONSES D ID N OT S HOW clusters formed by shared user requests, with recorded verdicts Restoring calls changed J0’s contrast, but the judges’ fixed, gave 95% bootstrap intervals (per trajectory) of 0.17 responses did not identify the missing calls (obligation 1, to 0.53 for the reduction and −0.05 to 0.21 for the restored Table III). Each judge was instructed to use only the supplied contrast (supplementary Table S7). With O packets, J1 to J4 packet to decide whether the target step was incorrect or had contrasts of 0, 5, 4 and −1, compared with J0’s 13. In J0’s original prefix packets, both draws abstained on all

26 blank error targets, and at least one draw abstained on each of the 27 blank comparison targets. Such abstentions could not produce consensus error flags. No J0 draw abstained on these targets in R prefix packets. On the same blank O prefix targets, J1 to J4 flagged 20, 20, 7 and 19 of the 26 error targets, and 13, 22, 7 and 17 of the 27 comparison targets (supplementary Table S8). A consensus error flag does not reveal whether the target action was present in the packet, and an abstention does not identify the missing call. At least one draw abstained on 17 of 30 original error-prefix packets under J3 and 3 to 6 under J1, J2 and J4, all at blank targets (supplementary Section VI). The benchmark labels were assigned using complete trajectories, so a flag on a blank prefix error target does not establish that the judge detected the omitted action’s error. Direct validation of packet content against its source can identify the missing calls without relying on judge responses [4].

and a matcher that took calls from assistant messages only flagged all 40 of their results as unmatched. Test 3: continue after a failure. Make a required check fail while packets from the failed or an earlier run remain, then invoke the normal downstream entry point. Require that no output of the failed run, or stale output substituted for it, reaches the judge or recording stub. As in build failure handling [17], failed prerequisites must block dependent work. A manifest written by a failed build can match its defective output exactly, so a matching hash does not authorise use. Test 4: change a checked file before use. For file-based inputs, change a staged packet after staging and, separately, after the last check before dispatch. Require refusal before use in both cases. P2’s hash comparisons from staging to use resemble the file-integrity checks of the in-toto framework [24]. Exact-byte hashes recorded at build and checked at staging would also detect changes made between build and staging, VI. I MPLICATIONS including the line-ending change. J0’s file-read tool returned the An LLM judge, whether assessing agent steps or annotating packet without checking its hash. For such a judge, verification software artifacts [22], is one part of a measurement process: at read time or delivery of the validated bytes without another source selection, rendering, checking and response aggregation unchecked read can close the gap. For other delivery modes, also shape its results. In this case, the answer schema had test the boundary between the last check and use. no category for defective input, and J0’s abstentions on blank Scope. This exploratory case study examines one pipeline original prefix targets counted as no error flags. Full packets selected after its rendering omission was found, with 30 then produced flags on some of those targets, contributing to trajectories from 22 distinct user requests. The controls were the measured contrast even though their tool calls remained developed on the same cohort, tested on two cases without omitted. The engineering implication for construct validity [23] injected faults and evaluated on constructed fault cases written is that evaluations should report evidence that judge inputs with knowledge of the code. met their requirements. The four tests below provide direct Future evaluation. We plan to test transfer on two targets: evidence for the input contract without live judge calls, using a AgentRewardBench’s trajectory-to-judge pipeline [1], whose recording stub where needed, and can serve as regression tests screenshot and accessibility-tree inputs require adapting test 1, when input representations or pipeline components change. and independently written judge adapters for held-out AgentEach test includes a valid input that must stay accepted. ProcessBench trajectories. For each target, an assessor who has Test 1: render required content. Derive required and not seen its checkers will derive the input contract, applicable excluded fields from the input contract for each packet type. obligations and valid and faulty cases from its documentation Render a message with empty text and an action only in a and evaluation goal. We will adapt the tests before seeing those structured field, then check the action’s name, arguments and cases. Without seeing the test results, the assessor will classify position against the source. Check that excluded information, observed pipeline behaviour against the required behaviour for such as benchmark labels or the target’s tool result in a each applicable obligation. These classifications will provide prefix packet, remains absent. For screenshots or accessibility the reference for counting detected violations and false alarms, trees, check contract-required information against the source, both with denominators. We will also record adaptation effort allowing permitted transformations. As secondary diagnostics, and whether repairs change judge results under the same report abstentions separately from no and inspect error flags configurations. On the predefined cases, a test transfers if on steps labelled correct. it detects every confirmed violation of its obligation without Test 2: violate each requirement. For each requirement, flagging compliant behaviour. Where no case or no violation specify its object, pipeline step and expected behaviour, then is confirmed for an applicable obligation, detection remains test its predicates with cases expected to pass and fail. For unassessed. one-to-one relations, test both directions: a call without a VII. C ONCLUSION result and a result without a call. A content check should take expected values from source records, while an integrity LLM-based evaluation depends not only on the judge but check should compare the current bytes’ hash with a reference also on its input. In this case, omitted tool calls persisted recorded before any change it must detect. The contract must despite matching hashes, and restoring calls throughout the match the evaluation goal: valid JSON was required here, but packets changed J0’s later-context contrast from 13 to 2. Both malformed calls should be preserved when their behaviour checkers accepted an unmatched tool result in a mutation case, is being judged. Requirements and checks must cover every while constructed integration tests exposed failure-handling and participant: simulated users made 40 of the cohort’s 311 calls,

check-to-use gaps. These findings motivate four input-assurance obligations, each with a direct test. Artifact. The artifact repository provides the study plans, released observations, source code and scripts for reconstruction, replay and reanalysis: https://anonymous.4open.science/r/ evaluation-artifact-342B/. R EFERENCES [1] X. H. Lù, A. Kazemnejad, N. Meade, A. Patel, D. Shin, A. Zambrano, K. Stańczak, P. Shaw, C. J. Pal, and S. Reddy, “AgentRewardBench: Evaluating automatic evaluations of web agent trajectories,” in Proceedings of the Second Conference on Language Modeling (COLM), 2025, arXiv:2504.08942. [2] S. Fan, X. Ye, Y. Huo, Z.-Y. Chen, Y. Guo, S. Yang, W. Yang, S. Ye, J. Chen, H. Chen, X. Cong, and Y. Lin, “AgentProcessBench: Diagnosing step-level process quality in tool-using agents,” in Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, 2026. [3] S. Papi, M. Gaido, A. Pilzer, and M. Negri, “When good and reproducible results are a giant with feet of clay: The importance of software quality in NLP,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 3657– 3672. [4] E. Breck, N. Polyzotis, S. Roy, S. E. Whang, and M. Zinkevich, “Data validation for machine learning,” in Proceedings of the 2nd SysML Conference, Palo Alto, CA, USA, 2019. [5] S. Biderman, H. Schoelkopf, L. Sutawika et al., “Lessons from the trenches on reproducible evaluation of language models,” arXiv preprint arXiv:2405.14782, 2024. [6] Y. Zhu, T. Jin, Y. Pruksachatkun et al., “Establishing best practices for building rigorous agentic benchmarks,” arXiv preprint arXiv:2507.02825, 2025. [7] S. Dev, A. Sloan, J. Kavner, N. Kong, and M. Sandler, “Judge Reliability Harness: Stress testing the reliability of LLM judges,” arXiv preprint arXiv:2603.05399, 2026, accepted at the ICLR 2026 Agents in the Wild Workshop. [8] X. Zheng, T. Pang, C. Du, Q. Liu, J. Jiang, and M. Lin, “Cheating automatic LLM benchmarks: Null models achieve high win rates,” in Proceedings of the Thirteenth International Conference on Learning Representations (ICLR), 2025. [9] Y. Zhao, H. Liu, D. Yu et al., “One token to fool LLM-as-a-judge,” arXiv preprint arXiv:2507.08794, 2025. [10] G. Tan, Z. Sun, J. Shi, T. Zhang, Z. He, Q. Wu, S. Liang, W. Sun, J. He, P. Chen, C. Zhang, L. K. Shar, and D. Lo, “AgentChaos: Chaos engineering for agent systems via programmatic fault injection,” arXiv preprint arXiv:2608.06790, 2026, accepted at ASE 2026. [11] Z. Seyedghorban, E. Klimov, A. van Deursen, A. Panichella, and B. Kulahcioglu Ozkan, “Observability and fault injection for LLM-based multi-agent systems in software engineering,” in Proc. IEEE Int. Conf. Software Testing, Verification and Validation (ICST), 2026, pp. 211–215. [12] S. Shankar, H. Li, P. Asawa, M. Hulsebos, Y. Lin, J. D. ZamfirescuPereira, H. Chase, W. Fu-Hinthorn, A. G. Parameswaran, and E. Wu, “SPADE: Synthesizing data quality assertions for large language model pipelines,” Proceedings of the VLDB Endowment, vol. 17, no. 12, pp. 4173–4186, 2024. [13] G. Jahangirova, D. Clark, M. Harman, and P. Tonella, “Test oracle assessment and improvement,” in Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA), 2016, pp. 247–258. [14] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015. [15] J. C. Knight and N. G. Leveson, “An experimental evaluation of the assumption of independence in multiversion programming,” IEEE Transactions on Software Engineering, vol. SE-12, no. 1, pp. 96–109, 1986. [16] A. Peruma, K. Almalki, C. D. Newman, M. W. Mkaouer, A. Ouni, and F. Palomba, “On the distribution of test smells in open source Android applications: An exploratory study,” in Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering (CASCON). IBM Corp., 2019, pp. 193–202.

[17] R. M. Stallman, R. McGrath, and P. D. Smith, GNU Make: A Program for Directing Recompilation, for GNU make version 4.4.1 ed., Free Software Foundation, 2023, section 5.5, Errors in Recipes. [18] D. Dean and A. J. Hu, “Fixing races for fun and profit: How to use access(2),” in 13th USENIX Security Symposium (USENIX Security 04). USENIX Association, 2004. [Online]. Available: https://www.usenix.org /legacy/events/sec04/tech/full_papers/dean/dean_html/index.html [19] J. M. Voas, “PIE: A dynamic failure-based technique,” IEEE Transactions on Software Engineering, vol. 18, no. 8, pp. 717–727, 1992. [20] S. G. Patil, H. Mao, F. Yan et al., “The Berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models,” in Proceedings of the 42nd International Conference on Machine Learning, ser. PMLR, vol. 267, 2025, pp. 48 371–48 392. [21] V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan, “τ 2 -bench: Evaluating conversational agents in a dual-control environment,” arXiv preprint arXiv:2506.07982, 2025. [22] T. Ahmed, P. Devanbu, C. Treude, and M. Pradel, “Can LLMs replace manual annotation of software engineering artifacts?” in Proceedings of the IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR), 2025. [23] P. Ralph and E. Tempero, “Construct validity in software engineering research and software metrics,” in Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering (EASE), 2018, pp. 13–23. [24] S. Torres-Arias, H. Afzali, T. K. Kuppusamy, R. Curtmola, and J. Cappos, “in-toto: Providing farm-to-table guarantees for bits and bytes,” in Proceedings of the 28th USENIX Security Symposium, 2019, pp. 1393– 1410.

Related documents

• Correct Is Not Governed: Provenance Integrity in Agentic Workflows#451245
• A Psycholegal Deskbook for Bench and Bar: Book Review of Forensic Assessments in Criminal and Civil Law#1115427
• JEV as a Judge for Agent Trace Security: An Empirical Comparison with Generative LLM Judges#1129475
• JEV as a Judge for Agent Trace Security: An Empirical Comparison with Generative LLM Judges#1108630
• Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation#1202249