Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation

Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation Kefeng Duan1 , Dewu Zheng1 , Yanlin Wang1∗ , Xiwen Wang1 , Ensheng Shi2 , Xilin Liu2 , Yuchi Ma2 , Jiachi Chen3 , Mingwei Liu1 , Zibin Zheng1 1

arXiv:2609.01603v1 [cs.SE] 1 Sep 2026

School of Software Engineering, Sun Yat-sen University, Zhuhai, China 2 Huawei Cloud Computing Technologies Co., Ltd., China 3 College of Computer Science and Technology, Zhejiang University, Hangzhou, China {duankf, zhengdw5, wangxw86}@mail2.sysu.edu.cn, {wangylin36, liumw26, zhzibin}@mail.sysu.edu.cn {shiensheng, liuxilin3, mayuchi1}@huawei.com, chenjiachi@zju.edu.cn Abstract

and use limited observations to estimate full-benchmark performance [3–6]. IRT-based methods fit historical response matrices to estimate item measurement characteristics and to infer a model’s latent ability from a limited set of observed responses. IRT therefore provides a promising framework for budgeted SWE agent evaluation. However, prior IRT methods for efficient benchmarking typically represent each task solely through its final pass/fail response, while a SWE agent’s outcome is produced through a multi-step trajectory of reasoning, context exploration, tool use, code editing, and verification. Reducing a trajectory to a binary outcome discards crucial process-level measurement signals about how an agent reaches success or failure, including what context it explores, which actions or edits it attempts, and where its problem-solving process breaks down. To address this limitation, we propose PTA-IRT (Privileged Trajectory-Aware Item Response Theory), which incorporates historical agent trajectories as privileged information into IRT-based data selection and evaluation. PTA-IRT first converts trajectories into structured semantic summaries and uses them to learn trajectory-aware fourparameter logistic (4PL) item measurement characteristics. It then selects a difficulty-stratified calibration subset using trajectory-aware Fisher information and transfers measurement knowledge to an ability estimator through Learning Using Privileged Information (LUPI). Given a new agent, PTA-IRT executes only the selected calibration tasks and uses limited calibration observations to estimate its fullbenchmark performance and ranking. To validate the effectiveness of PTA-IRT, we conduct experiments on four SWE benchmarks and compare it with prior IRT baselines. Our results show that:

Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT.

1

Introduction

As Large Language Models (LLMs) are increasingly deployed as software engineering agents, more challenging benchmarks emerge to evaluate their ability to solve realworld software engineering tasks. However, these agentic benchmarks make evaluation substantially more expensive than conventional single-turn settings. Repository-level benchmarks require agents to reason over long horizons, explore codebases, invoke tools, edit source files, and execute tests. These multi-step interactions result in substantial inference costs and long evaluation cycles. For example, SWEbench contains more than 2,000 programming tasks; a single full-benchmark SWE-agent run has an estimated upperbound cost exceeding $8,000 under a $4-per-task limit, while the average cost on successfully resolved instances is $1.59 for SWE-agent with GPT-4 Turbo [1, 2]. This motivates the central question of our work: can we evaluate a new agent on only a small budgeted subset of tasks while reliably recovering its performance and relative ranking on the full benchmark? To address this question, prior work on efficient benchmarking introduces Item Response Theory (IRT) to select a small but representative subset of benchmark instances ∗

• Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery. • Privileged trajectory information is the source of these gains: outcome-only modeling or corrupted trajectory summaries weaken recovery. • Structured trajectory summaries carry process signals beyond issue text and submitted patches, making budgeted evaluation more informative. Our contributions are as follows:

Corresponding author.

1

Dataset curation Dimension

Lite

Verified

Budgeted evaluation (subset→full) Classical IRT

Deep/PSN-IRT

Target domain SWE SWE General Observation signal — — Pass/Fail Modeling method Heuristic rules Human judgment Parametric IRT LLM involvement — — — Human involvement — Heavy —

General Pass/Fail Neural IRT — —

AutoJudger

PTA-IRT

Multimodal SWE Pass/Fail Pass/Fail+Agent Traj. IRT + LLM selection Traj.-aware IRT Selects tasks Summarizes traj. — —

Table 1: Comparison of efficient benchmarking for SWE evaluation. Dataset curation: SWE-bench Lite and Verified are fixed smaller subsets filtered from SWE-bench (Lite: rule-based; Verified: human annotation). Budgeted evaluation recovers fullbenchmark scores from a calibration subset. “—” denotes not applicable / not involved. • We formulate efficient SWE benchmarking as a budgeted subset-to-full evaluation problem under a realistic protocol in which a new agent is executed only on the calibration subset, while historical trajectories are used offline for training.

In natural language processing, IRT was initially used to diagnose fundamental properties of datasets and test items, including item difficulty and discriminability [44]. More recently, its principles have been applied to efficient evaluation. Deep-IRT and PSN-IRT extend conventional IRT with neural architectures and more flexible response functions to model complex model–item interactions [3, 6]. In a different direction, AutoJudger proposes an agent-driven framework for efficient benchmarking of multimodal large language models [45]. However, IRT remains underexplored for SWE agent evaluation specifically, where agent–instance response data are sparse and trajectories are long. Instead, current SWE benchmarks still rely on rule-based filtering or human verification to reduce evaluation cost, as in the curated SWE-bench Lite and Verified subsets [27]. Table 1 contrasts these approaches along target domain, observation signal, modeling method, and human/LLM involvement.

• We introduce PTA-IRT, which incorporates historical agent trajectories as privileged information into IRTbased item selection and full-benchmark performance estimation. • Extensive experiments on four SWE benchmarks show that PTA-IRT improves budgeted score and ranking recovery over prior IRT baselines.

2 2.1

Background

Related Work

Large language models and LLM-based agents have reshaped a wide range of software engineering tasks [7–9]. On code generation, they now cover function-level synthesis as well as repository-level completion, secure generation, and token-efficient decoding [10–18]. Code search and retrievalaugmented generation further connect natural-language intent to relevant code context [19–23]. Code summarization and comment generation recover intent from source context [24, 25], while retrieval-augmented commit message generation links edits to natural-language descriptions [26]. Issue resolution has become a central testbed for agentic software engineering, with systems that explore repositories, apply patches, and verify fixes against tests [1, 27–33]. Related directions include debugging, repository-level reasoning, code translation, and code review [34–37]. These advances have driven more demanding repositorylevel and agentic benchmarks, which require long-horizon reasoning, tool use, editing, and test execution [12, 13, 17, 18, 27, 29, 30, 36–39]. Such evaluations are substantially more expensive than conventional single-turn settings [10, 11]. Recent research therefore examines the efficiency and reliability of LLM benchmarks, with a growing interest in reducing evaluation cost through a smaller set of test instances [40– 42]. IRT provides a data-driven, item-level framework for this purpose [43]. By modeling agents’ responses across instances, IRT estimates item properties, such as difficulty and discriminability, together with the latent abilities of evaluated models.

2.2

Preliminary: Item Response Theory

IRT models the response behavior between respondents and items through respondents’ latent abilities and items’ measurement properties [46, 47]. In SWE agent evaluation, agents correspond to respondents, while benchmark instances correspond to items. IRT uses an Item Characteristic Curve (ICC) to model the conditional probability of a correct response. Under the one-parameter logistic model, the probability that agent i successfully solves instance j is defined as P (Xij = 1 | θi ) =

1 , 1 + exp[−(θi − bj )]

(1)

where θi denotes the latent ability of agent i and bj denotes the difficulty parameter of instance j. As IRT research advances, subsequent work incorporates additional item parameters, explores more expressive ICC functions, and adopts deep-learning approaches to improve its modeling capacity in complex evaluation settings [3, 6].

2.3

Preliminary: Learning Using Privileged Information

LUPI leverages additional information that is available during training but unavailable at test time [48, 49]. Formally, each training example is represented as (x, x∗ , y), where x 2

SWE Benchmark Data Task / Repository

Prior Work and Its Limitation

Agent Trajectories

Outcomes

>_

Repository files

$ ls -la $ pytest -q $ grep -R "TODO" $ vim src/app.py $ git diff

…

✓

×

Pass

Fail

Outcome-only IRT

Select Subset

Estimate Ability θ

Issue description Dependencies

+ +

Environment

— —

Limitations

…

…

Signal: only final pass/fail Selection and Estimation: trajectory-agnostic item choice

Multi-step execution signals beyond binary outcomes

PTA-IRT

Parse Trajectories

Summarize Trajectories LLM

$ ls -la $ pytest -q $ grep -R "TODO" $ vim src/app.py $ git diff

Task Goal Context Explored

>_

Edits Executed + +

— —

Path Overview

Embed Summaries

● ● ● ● ⋮

Stage 2: Select Informative Calibration Items Input

Output

Y: Response Matrix Q: Task Identifiers Z*: Trajectory Summary ESS Reliability

Z*

Trajectory-aware

Calibration Subset

4PL IRT 1

Easy

P(correct)

Stage 1: Construct Agent Trajectory Representations

Training (Offline) Teacher (Q + Z*)

■ ■ ■ … 0

×

Medium

Deployment (Online)

New Agent

Run on Subset

Fit ability θ*

Medium

Ability θ

■ ■ ■ …

Fisher Information

Hard

■ ■ ■ …

ⓘ

distillation

Student (Q only)

Select Top-K tasks across different difficulty bins

Difficulty-Stratified Top-k Easy

Stage 3: Estimate Agent Ability via LUPI

θ*

Predict full-benchmark score & ranking

Hard

Figure 1: Limitations of prior work and an overview of PTA-IRT. denotes the standard input, x∗ denotes privileged information, and y is the supervision signal. The goal of LUPI is to use x∗ during training to improve a predictor that relies only on x at inference time. A neural realization of LUPI uses a teacher–student paradigm, in which the teacher accesses privileged information and transfers its knowledge to a student through distillation. The student then relies only on the standard input, without requiring privileged information at inference time [50–52]. In our setting, agent trajectories serve as privileged information, as they provide process-level evidence beyond binary responses. They are available for historical agent–item interactions, but are not required when estimating a new agent’s performance on the full benchmark from its responses on a calibration subset.

3 3.1

into comparable structured process summaries, preserving process-level evidence that binary pass/fail labels cannot capture. • Stage 2: Select Informative Calibration Items. We fit a trajectory-aware 4PL IRT model, score tasks by Fisher information weighted by trajectory reliability, and select a difficulty-stratified calibration subset that is both informative and representative. • Stage 3: Estimate Agent Ability via LUPI. On the same trajectory-aware 4PL backbone, a teacher–student objective transfers privileged process knowledge to a deployable student. At test time, we fit only a scalar ability on the calibration subset and extrapolate to the full benchmark.

3.2

Construct Agent Trajectory Representations

Raw agent trajectories contain lengthy commands, tool outputs, and intermediate messages that vary across agents and frameworks. We therefore build a trajectory parser to extract action steps from heterogeneous logs, and design a prompt protocol that compresses each historical trajectory into a structured process summary. The summary has four fields: Task Goal, Context Explored, Edits Executed, and Path Overview. The summarization model receives the task description, extracted action steps, the benchmark reference patch, and the agent’s submitted patch. The reference patch serves to align task-relevant files, APIs, and change scope across heterogeneous trajecto-

PTA-IRT

Overview

Evaluating a SWE agent on a full benchmark is costly. We study how to construct a small calibration subset from a full benchmark under a limited evaluation budget, and how to accurately estimate an agent’s full-benchmark performance using only its outcomes on this subset. As illustrated in Figure 1, PTA-IRT consists of three consecutive stages: • Stage 1: Construct Agent Trajectory Representations. We compress heterogeneous and lengthy execution logs 3

Aggregating over A with offline quality weights ωij yields P i∈A ωij Iij (θi ) P Fisher(j) = , X i∈A ωij (5) ESS(j) = mij ωij , i∈A
Info(j) = Fisher(j) · log 1 + ESS(j) .

ries, and is not treated as an outcome label. The protocol further instructs the summarizer to describe observable actions and edits, without inferring correctness or unobserved reasoning. This compression preserves process-level evidence absent from binary pass/fail labels. We encode process summaries as Z = {zij } and use them together with the response matrix Y = {yij } during offline training. Under our protocol, a new agent is evaluated on the calibration subset to estimate full-benchmark performance, so process summaries are available for historical interactions during training but unavailable for unevaluated tasks at test time.

3.3

Here mij ∈ {0, 1} marks a usable process summary, and ωij ∈ [0, 1] down-weights incomplete or malformed trajectory summaries. ESS(j) is the effective sample size of privileged evidence on item j. Multiplying Fisher(j) by log(1 + ESS(j)) therefore reduces the score of items whose privileged evidence P is of low quality. 1 Let rj = |A| i∈A yij be the pass rate of task j on historical agents only. We partition tasks into difficulty bins by rj , allocate the budget k across bins in proportion to bin size, and within each bin select the tasks with the largest Info(j). The resulting subset S therefore favors high-information items while covering easy-to-hard regimes, which is essential for recovering full-benchmark scores from a small calibration budget.

Select Informative Calibration Items

This stage constructs a calibration subset S that reduces uncertainty about agent ability while remaining representative of full-benchmark difficulty. Purely maximizing item information tends to select overly hard, mutually similar tasks and harms score recovery. We therefore select items by a trajectory-aware information score under difficulty stratification. Trajectory-aware scorer. We fit a 4PL IRT model on historical agents A only. Each agent i has latent ability θi , and each task j has global parameters (aj , bj , cj , dj ) for discriminability, difficulty, guessing, and feasibility ceiling. Under these parameters, the response probability is
pij = cj + (dj − cj ) σ aj (θi − bj ) , (2)

3.4

A new agent is evaluated only on the calibration subset S, and process summaries arise only after the agent is actually executed. Full-benchmark summaries are therefore unavailable at scoring time, which matches the LUPI setting. We design a teacher–student estimator on the same trajectory-aware 4PL backbone: privileged summaries supervise the teacher offline, while the student uses only identifiers and shared item parameters, estimates ability from S, and extrapolates to all tasks.

with 0 ≤ cj < dj ≤ 1. Architecturally, an agent encoder maps a one-hot agent identifier to θi , an item encoder maps a one-hot item identifier to (aj , bj , cj , dj ), and a summary encoder maps each privileged summary zij to bounded residuals (∆aij , ∆bij ) = α tanh(g(zij )). Each encoder is a twolayer multilayer perceptron (MLP). On cells with a usable summary (mij = 1), we form effective parameters aij = aj + mij ∆aij ,

bij = bj + mij ∆bij ,
pij = cj + (dj − cj ) σ aij (θi − bij ) .

Estimate Agent Ability via LUPI

Teacher–student training. The teacher and student share the agent, item, and summary encoders. The student predicts with global item parameters (aj , bj , cj , dj ). The teacher predicts with trajectory-adjusted parameters (aij , bij ), so the training objective depends on process-summary information. We minimize

(3)

Here (aj , bj , cj , dj ) remain shared item parameters for the agent population. A process summary provides agent–item interaction evidence, which we inject as residuals on discriminability and difficulty, i.e., the parameters that govern the ability-linked transition of the item characteristic curve. In contrast, the asymptotes cj and dj encode structural bounds of the task and evaluation setting, namely chance success and feasibility limits. These bounds are properties of the item and harness rather than of a single solving path, so we do not make them trajectory-dependent.

L = BCE(pSij , yij ) + λt mij ωij BCE(pT ij , yij ) S + λKL mij ωij KL pT ij ∥ pij

(6)

+ λ∆ mij ∥∆aij ∥2 + ∥∆bij ∥2 , where pSij and pT ij are the student and teacher pass probabilities. The teacher and KL terms distill summary-conditioned measurements into the student. The residual penalty keeps (∆a, ∆b) small.

Information-aware stratified selection. We score each task by how much its outcomes inform ability estimation, then select under difficulty-stratified budget allocation. The contribution of interaction (i, j) is the 4PL Fisher information [53]
2
2 a2ij pij − cj dj − pij Iij (θi ) = . (4) (dj − cj )2 pij (1 − pij )

Test-time estimation. We freeze all network weights and fit a scalar ability θ∗ on S by L-BFGS [54] from the observed outcomes, optionally with teacher distillation on S. With θ∗ and the shared item parameters (aj , bj , cj , dj ), the student predicts pass probabilities on every benchmark task and aggregates them into overall performance and ranking. 4

Benchmark SWE-bench Lite SWE-bench Verified SWE-bench Full SWE-bench Pro

# Tasks

# Evaluated Models

300 500 2,294 730

35 70 14 14

4.4

To ensure reliability, we use four-fold cross-validation: in each fold, 75% of the models are used for training and the remaining 25% for testing. Process summaries are generated with DeepSeek-V4-Flash1 and embedded with all-MiniLML6-v2.2

Table 2: Overview of the benchmarks and evaluated models.

5 5.1

4 4.1

Experimental Setup

Datasets

Baselines

We compare PTA-IRT against a representative set of prior IRT baselines spanning classical IRT, neural IRT, and agent-driven selection. Classical baselines include Maximum Likelihood Estimation (MLE), Markov Chain Monte Carlo (MCMC) [55], Variational Inference (VI) [56], and VIBO [57]. Advanced methods include Deep-IRT [3] and PSN-IRT [6]. We additionally include AutoJudger [45], which combines IRT-based difficulty estimation with an LLM agent that adaptively selects calibration items.

4.3

Main Results

RQ1: Effectiveness of PTA-IRT

RQ1 examines how PTA-IRT performs under limited calibration budgets relative to prior IRT baselines. Table 3 reports the main comparison at a 10% budget on four SWE benchmarks, and Figure 2 evaluates budget sensitivity on SWE-bench Lite under 5–25% calibration. Finding 1. With only 10% calibration, PTA-IRT consistently outperforms prior IRT baselines on both score recovery and ranking. As shown in Table 3, PTA-IRT is best on every MAE/τ /ρ column across the four SWE benchmarks, averaging MAE 0.041±0.015, τ = 0.888, and ρ = 0.973. This advantage holds under two complementary regimes. On Lite and Verified, where agent pools are dense, PTA-IRT attains the lowest MAE and the highest τ /ρ. On Full and Pro, where task sets are large but agents are few, PTA-IRT likewise achieves the best score and ranking recovery. Finding 2. PTA-IRT is robust across calibration budgets, and a small budget already yields practically useful ranking agreement. As shown in Figure 2, 5% calibration already gives τ = 0.768, within the practical range reported in prior analyses [58, 59] (τ ≥ 0.7–0.8). Raising the budget from 5% to 20% further improves ranking on Lite (τ : 0.768 → 0.886) and tends to lower MAE. PTA-IRT remains the best method across the full 5–25% range, underscoring the robustness of our approach under varying calibration budgets.

As shown in Table 2, we evaluate on four SWE-bench versions (Lite, Verified, Full, and Pro). Lite and Verified are curated subsets of Full [27]: Lite uses rule-based filtering toward more self-contained functional fixes, while Verified uses human annotation to remove underspecified or problematic instances. SWE-bench Pro [39] further extends SWEbench to long-horizon, multi-file enterprise-style tasks across broader repositories and languages. Because Lite and Verified are the community’s default reporting targets, they attract substantially more public agent submissions with parseable trajectories than Full or Pro. We therefore obtain denser model coverage on Lite and Verified, while Full and Pro emphasize larger task collections under sparser agent pools. We use each official release and retain all models with complete, successfully parsed trajectories.

4.2

Implementation Details

5.2

RQ2: Ablation Analysis of PTA-IRT

RQ2 examines the contribution of each component of PTAIRT to recovering full-benchmark scores and rankings. Table 4 reports ablations at a 10% calibration budget. w/o Traj. scorer removes the summary encoder from the 4PL IRT used for informative selection. w/o LUPI drops teacher– student LUPI and estimates ability with the student model alone, without privileged summary supervision. + Top-K and + Clustering replace stratified information selection with global Top-K by information and pass-rate clustering, respectively. Finding 3. Each component of PTA-IRT contributes to recovering full-benchmark scores and rankings. As shown in Table 4, the full configuration achieves the best average MAE/τ /ρ. Removing the trajectory-aware scorer (w/o Traj. scorer) or LUPI (w/o LUPI) raises Avg MAE and lowers ranking agreement; the only local exception is a marginal MAE improvement on Verified without LUPI, which does not overturn the average. Replacing stratified selection with Top-K or clustering likewise worsens the average on both score and ranking.

Metrics

We adopt three metrics: Mean Absolute Error (MAE), Kendall’s τ , and Spearman’s ρ. • MAE: the average deviation between predicted and ground-truth full-benchmark performance, MAE = PN 1 i=1 |ŷi − yi |, where ŷi and yi are the predicted and N ground-truth scores of model i, and N is the number of evaluated models. • Kendall’s τ : pairwise ordinal agreement between the predicted and ground-truth rankings, τ = C−D , where C (N2 ) and D are the numbers of concordant and discordant model pairs. • Spearman’s ρ: the global monotonic correlation between the predicted and
ground-truth rankings, ρ = corr rank(ŷ), rank(y) , where ŷ and y are the vectors of predicted and ground-truth performance.

1

https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash https://huggingface.co/sentence-transformers/all-MiniLML6-v2 2

5

Lite Method

Verified

τ↑

MAE↓

ρ↑

τ↑

MAE↓

Full ρ↑

Pro

τ↑

MAE↓

ρ↑

Avg τ↑

MAE↓

ρ↑

MAE↓

τ↑

ρ↑

MLE .164±.022 .718 .894 .235±.045 .648 .831 .120±.056 .319 .424 .165±.039 .429 .675 .171±.041 .528 .706 MCMC .217±.048 .647 .805 .130±.041 .769 .923 .073±.050 .780 .921 .084±.041 .846 .952 .126±.045 .761 .900 VI .239±.027 .731 .898 .142±.012 .855 .967 .281±.140 .934 .982 .226±.028 .868 .952 .222±.052 .847 .950 VIBO .206±.031 .691 .866 .291±.036 .583 .737 .157±.069 .890 .960 .211±.057 .692 .864 .216±.048 .714 .857 Deep-IRT .305±.145 .682 .863 .191±.049 .724 .909 .362±.120 .934 .982 .289±.144 .692 .868 .287±.114 .758 .905 PSN-IRT .114±.035 .650 .817 .216±.045 .786 .941 .198±.133 .758 .921 .073±.013 .692 .886 .150±.057 .722 .891 AutoJudger .128±.014 .620 .826 .200±.011 .673 .871 .151±.051 .626 .736 .120±.047 .626 .807 .150±.031 .637 .810 PTA-IRT

.045±.004 .836 .950 .043±.015 .872 .976 .048±.025 .956 .991 .029±.017 .890 .974 .041±.015 .888 .973

Table 3: Comparison at a 10% calibration budget across four SWE benchmarks. Lower MAE and higher Kendall’s τ / Spearman’s ρ are better. Best values in each column are bolded (all ties are bolded). Values are rounded to three decimals with leading zeros omitted.

0.30 0.25 0.20 0.15 0.10 0.05

Spearman

Kendall

0.9

0.95 0.8 0.7 0.6 5

10

15

20

25

MLE MCMC VI VIBO Deep-IRT PSN-IRT AutoJudger PTA-IRT

0.90

Spearman

Kendall

MAE

MAE

0.85 0.80 0.75

5

10

15

20

Calibration budget (%)

25

5

10

15

20

25

Figure 2: Budget sensitivity on SWE-bench Lite under 5–25% calibration. Each panel shows one metric (MAE, Kendall’s τ , and Spearman’s ρ).

5.3

RQ3: Contribution of Trajectory Summaries

weight corrupted cells. At extreme dropout, both degrade as usable process signal disappears.

RQ3 examines the contribution of historical trajectory summaries to PTA-IRT. We apply summary dropout: a controlled fraction of train-summary embeddings is set to zero before retraining the trajectory-aware scorer and rebuilding the stratified calibration set S. We compare With ESS, which downweights corrupted cells, against Without ESS, which still feeds corrupted embeddings at full weight. Figure 3 reports MAE, Kendall’s τ , and Spearman’s ρ on four SWE benchmarks under a 10% calibration budget. Finding 4. Trajectory summaries improve recovery quality; their benefit comes from summary content rather than a vacuous privileged channel. As shown in Figure 3, increasing summary dropout generally weakens score recovery and ranking relative to the clean setting, though the curves are not strictly monotonic across benchmarks. At full dropout without ESS, Lite approaches the RQ2 ablation that removes the trajectory-aware scorer (MAE 0.056, τ = 0.778), indicating that the gains of PTA-IRT mainly come from summary content rather than merely having an unused summary pathway. Finding 5. ESS effectively protects PTA-IRT when trajectory summaries are unreliable. Under the same dropout levels, With ESS stays closer to the clean setting than Without ESS on most benchmark–metric pairs. The protective gap is clearest at low-to-mid dropout (e.g., on Lite, Without ESS already falls to τ ≈ 0.735 at 50% dropout), where ESS can still down-

5.4

RQ4: Characteristics of Trajectory Summaries

RQ4 examines what distinctive structure trajectory summaries capture relative to issue text and submitted patches, and how that structure is organized across summary fields and same-item outcome pairs. We answer this with two complementary views: cross-channel similarity (Table 5) and sameitem outcome geometry (Table 6). Cross-channel similarity. For each (agent, item) cell we compute cosine similarity between the process summary (and its four fields) and (i) the question embedding qj and (ii) the submitted-answer embedding. As a control for question alignment, we also report ∆ vs. randQ: the gap between similarity to the true question and the mean similarity to a randomly sampled question. Table 5 summarizes the results. Finding 6. Trajectory summaries exhibit a clear division of labor: Task Goal tracks the issue text, Edits Executed tracks the submitted patch, while Context Explored and Path Overview remain process-specific. Whole-summary similarity to the question (0.714) exceeds a random-question baseline by a large margin (∆ = +0.473). Field-level scores reveal complementary roles rather than redundancy. Task Goal is closest to the question (0.776), which largely explains the high Summary↔Question score. Edits Executed is closest 6

Lite Method

MAE↓

Verified

τ↑

ρ↑

τ↑

MAE↓

Full ρ↑

MAE↓

Pro

τ↑

ρ↑

Avg τ↑

MAE↓

ρ↑

τ↑

MAE↓

ρ↑

PTA-IRT .045±.004 .836 .950 .043±.015 .872 .976 .048±.025 .956 .991 .029±.017 .890 .974 .041±.015 .888 .973 w/o Traj. scorer .055±.021 .778 .930 .065±.008 .860 .969 .048±.029 .912 .982 .047±.010 .890 .969 .054±.017 .860 .963 w/o LUPI .085±.004 .573 .755 .042±.013 .848 .964 .053±.032 .802 .912 .092±.040 .868 .965 .068±.022 .773 .899 + Top-K + Clustering

.196±.031 .812 .942 .193±.012 .752 .917 .228±.163 .934 .982 .086±.025 .802 .921 .176±.058 .825 .941 .182±.037 .819 .947 .189±.015 .757 .921 .225±.162 .912 .974 .090±.027 .758 .908 .171±.060 .812 .937

Table 4: Ablation of PTA-IRT at a 10% calibration budget. Lower MAE and higher Kendall’s τ / Spearman’s ρ are better. Best values in each column are bolded (all ties are bolded). Values are rounded to three decimals with leading zeros omitted.

0.05

Kendall

MAE

0.06

0.04 0.03

0 10 20 30 40 50 60 70 80 90 100

Kendall

1.00 0.95 0.90 0.85 0.80 0.75 0.70

Spearman

MAE

0 10 20 30 40 50 60 70 80 90 100

Summary dropout (%)

1.00 0.98 0.96 0.94 0.92 0.90 0.88 0.86

Spearman Lite · With ESS Lite · Without ESS Verified · With ESS Verified · Without ESS Full · With ESS Full · Without ESS Pro · With ESS Pro · Without ESS 0 10 20 30 40 50 60 70 80 90 100

Figure 3: Contribution of trajectory summaries under summary dropout (10% calibration). Solid lines: With ESS; dashed lines: Without ESS. Colors denote benchmarks.

Channel

↔Q ↔Ans. ∆ randQ

Summary Task Goal Context Explored Edits Executed Path Overview

0.714 0.776 0.502 0.530 0.468

0.638 0.532 0.533 0.670 0.441

Channel

+0.473 +0.587 +0.314 +0.333 +0.272

Answer Summary Task Goal Context Explored Edits Executed Path Overview

Table 5: Cross-channel cosine similarity on SWE-bench Verified. ∆ vs. randQ applies to the ↔ Question column.

PP

FF

0.175 0.143 0.127 0.337 0.366 0.363

0.290 0.163 0.147 0.387 0.422 0.387

PF ∆out 0.265 0.158 0.141 0.372 0.414 0.389

0.033 0.005 0.003 0.010 0.020 0.013

Table 6: Same-item pair distances on SWE-bench Verified. PP/FF/PF: both-pass / both-fail / pass–fail.

to the submitted answer (0.670), whereas Context Explored and Path Overview stay lower on both axes (≤ 0.533). A trajectory summary therefore combines question-facing goal understanding with edit-level content and independent exploration/path structure.

plored, Edits Executed, and Path Overview keep large absolute distances (≥ 0.337), and Edits Executed shows the largest ∆out (0.020), consistent with its stronger answer alignment in Table 5. Thus pass/fail is expressed most directly in the final patch, whereas summaries retain heterogeneous solving paths that are not collapsed by binary labels and patches alone.

Same-item outcome geometry. We next compare representations of successful and failed solving paths on the same task. For every item with multiple agents, we form agent pairs and report mean embedding distance (1 − cosine) for bothpass (PP), both-fail (FF), and pass–fail (PF)
pairs, together with ∆out = d(PF) − 12 d(PP) + d(FF) . A larger ∆out indicates stronger outcome-related structure. Table 6 reports Answer, Summary, and the four summary fields. Finding 7. Submitted answers concentrate outcome geometry, while trajectory summaries preserve process diversity. For answers, same-outcome successes are much closer than failures (PP 0.175 vs. FF 0.290), and ∆out = 0.033. For full summaries, PP and FF distances remain close (0.143 vs. 0.163) and ∆out is only 0.005; among fields, Context Ex-

6

Conclusion

Evaluating SWE agents on full benchmarks is expensive because each task can require long-horizon exploration, editing, and testing. We present PTA-IRT, which treats historical execution trajectories as privileged information for efficient IRT-based evaluation. PTA-IRT builds structured trajectory summaries, selects a difficulty-stratified calibration subset with trajectory-aware measurement features, and estimates a new agent’s full-benchmark score and ranking from limited calibration observations via LUPI. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT base7

[9] Yanlin Wang, Kefeng Duan, Dewu Zheng, Ensheng Shi, Fengji Zhang, Yanli Wang, Jiachi Chen, Xilin Liu, Yuchi Ma, Hongyu Zhang, Qianxiang Wang, and Zibin Zheng. Towards an understanding of context utilization in code intelligence. ACM Comput. Surv., 58(11), April 2026. ISSN 0360-0300. doi: 10.1145/3797261. [10] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. [11] Naman Jain, Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 58791– 58831, 2025. [12] Dewu Zheng, Yanlin Wang, Ensheng Shi, Xilin Liu, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. Top general performance= top domain performance? domaincodebench: A multi-domain code generation benchmark. arXiv preprint arXiv:2412.18573, 2024. [13] Dewu Zheng, Yanlin Wang, Ensheng Shi, Ruikai Zhang, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. Humanevo: An evolution-aware benchmark for more realistic evaluation of repository-level code generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 1372–1384. IEEE, 2025. [14] Yifan Li, Ensheng Shi, Dewu Zheng, Kefeng Duan, Jiachi Chen, and Yanlin Wang. Repomincoder: Improving repository-level code generation based on information loss screening. In Proceedings of the 15th Asia-Pacific Symposium on Internetware, pages 229– 238, 2024. [15] Sicong Liu, Yanxian Huang, Mingwei Liu, Jiachi Chen, Ensheng Shi, Yuchi Ma, Hongyu Zhang, Yin Zhang, and Yanlin Wang. Shortcoder: Knowledge-augmented syntax optimization for token-efficient code generation, 2026. [16] Ziyao Zhang, Chong Wang, Yanlin Wang, Ensheng Shi, Yuchi Ma, Wanjun Zhong, Jiachi Chen, Mingzhi Mao,

lines on score and ranking recovery across four SWE benchmarks under our evaluation protocol, showing that privileged trajectory information makes budgeted SWE agent evaluation more informative.

References [1] John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 50528–50652. Curran Associates, Inc., 2024. doi: 10.52202/0790171601. [2] Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter. Transactions on Machine Learning Research, 2025. [3] Emiko Tsutsumi, Ryo Kinoshita, and Maomi Ueno. Deep item response theory as a novel test theory based on deep learning. Electronics, 10(9), 2021. doi: 10.3390/electronics10091020. [4] Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: evaluating LLMs with fewer examples. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 34303–34326. PMLR, 21–27 Jul 2024. [5] Alex Kipnis, Konstantinos Voudouris, Luca Schulze Buschoff, and Eric Schulz. metabench a sparse benchmark of reasoning and knowledge in large language models. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 31734–31770, 2025. [6] Hongli Zhou, Hui Huang, Ziqing Zhao, Lvyuan Han, Huicheng Wang, Kehai Chen, Muyun Yang, Wei Bao, Jian Dong, and Bing Xu. Lost in benchmarks? rethinking large language model benchmarking with item response theory. Proceedings of the AAAI Conference on Artificial Intelligence, 40(41):35085–35093, March 2026. doi: 10.1609/aaai.v40i41.40814. [7] Yanlin Wang, Wanjun Zhong, Yanxian Huang, Ensheng Shi, Min Yang, Jiachi Chen, Hui Li, Yuchi Ma, Qianxiang Wang, and Zibin Zheng. Agents in software engineering: Survey, landscape, and vision. Automated Software Engineering, 32(2):70, 2025. [8] Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. Large language model-based agents for software engineering: A survey. ACM Trans. Softw. Eng. Methodol., March 2026. ISSN 1049-331X. doi: 10.1145/3796507. Just Accepted. 8

on code context expansion. ACM Transactions on Software Engineering and Methodology, 33(1):1–30, 2023. [26] Ensheng Shi, Yanlin Wang, Wei Tao, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. Race: Retrieval-augmented commit message generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5520–5530, 2022. [27] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024. [28] Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. Magis: Llmbased multi-agent framework for github issue resolution. Advances in Neural Information Processing Systems, 37:51963–51993, 2024. [29] Lianghong Guo, Wei Tao, Runhan Jiang, Yanlin Wang, Jiachi Chen, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, and Zibin Zheng. Omnigirl: A multilingual and multimodal benchmark for github issue resolution. Proceedings of the ACM on Software Engineering, 2(ISSTA):24–46, 2025. [30] Lianghong Guo, Yanlin Wang, Caihua Li, Wei Tao, Pengyu Yang, Jiachi Chen, Haoyu Song, Duyu Tang, and Zibin Zheng. Swe data construction, automatically! Proceedings of the ACM on Software Engineering, 3 (FSE):525–546, 2026. [31] Tianyue Jiang, Yanlin Wang, Xin He, Daya Guo, Jiachi Chen, Ming Wen, Ensheng Shi, Xilin Liu, Yuchi Ma, and Guanbin Li. Phoenixrepair: Rethinking repair strategy exploration in software agents, 2026. [32] He Ye, Aidan ZH Yang, Chang Hu, Yanlin Wang, Tao Zhang, and Claire Le Goues. Adverintent-agent: Adversarial reasoning for repair based on inferred program intent. Proceedings of the ACM on Software Engineering, 2(ISSTA):1398–1420, 2025. [33] Dewu Zheng, Ruizhe Ye, Yanlin Wang, Yang Ye, Hongyu Zhang, Ensheng Shi, Xilin Liu, Yuchi Ma, Jianxing Yu, and Zibin Zheng. Swe-prime: Fewer trajectories, better performance, 2026. [34] Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, and Maosong Sun. DebugBench: Evaluating debugging capability of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 4173–4198, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.247. [35] Yanlin Wang, Suiquan Wang, Yanli Wang, Bowen Zhang, Daya Guo, Jiachi Chen, and Zibin Zheng. Reporeasoner: Evaluating repository-level code reasoning ability of long-context language models. Proceedings

and Zibin Zheng. Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation. Proceedings of the ACM on Software Engineering, 2 (ISSTA):481–503, 2025. [17] Yanlin Wang, Bowen Zhang, Yanli Wang, Daya Guo, Terry Yue Zhuo, Jiachi Chen, Mingwei Liu, Xingong Zhang, and Zibin Zheng. Arkrepobench: A repositorylevel code completion benchmark for harmonyos development. In Findings of the Association for Computational Linguistics: ACL 2026, pages 19409–19429, 2026. [18] Yanlin Wang, Ziyao Zhang, Chong Wang, Xinyi Xu, Mingwei Liu, Yong Wang, Jiachi Chen, and Zibin Zheng. Realsec-bench: A benchmark for evaluating secure code generation in real-world repositories. In Findings of the Association for Computational Linguistics: ACL 2026, pages 35866–35883, 2026. [19] Jing Gong, Yanghui Wu, Linxi Liang, Yanlin Wang, Jiachi Chen, Mingwei Liu, and Zibin Zheng. Cosqa+: Enhancing code search evaluation with a multi-choice benchmark and test-driven agents. IEEE Transactions on Software Engineering, 52(1):206–220, 2025. [20] Yanlin Wang, Lianghong Guo, Ensheng Shi, Wenqing Chen, Jiachi Chen, Wanjun Zhong, Menghan Wang, Hui Li, Hongyu Zhang, Ziyu Lyu, et al. You augment me: Exploring chatgpt-based data augmentation for semantic code search. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 14–25. IEEE, 2023. [21] Wenchao Gu, Juntao Chen, Yanlin Wang, Tianyue Jiang, Xingzhe Li, Mingwei Liu, Xilin Liu, Yuchi Ma, and Zibin Zheng. What to retrieve for effective retrievalaugmented code generation? an empirical study and beyond. arXiv preprint arXiv:2503.20589, 2025. [22] Tianyue Jiang, Yanli Wang, Yanlin Wang, Daya Guo, Ensheng Shi, Yuchi Ma, Jiachi Chen, and Zibin Zheng. Aligncoder: Aligning retrieval with target intent for repository-level code completion. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 971–982. IEEE, 2025. [23] Yanlin Wang, Jiadong Wu, Tianyue Jiang, Mingwei Liu, Jiachi Chen, Chong Wang, Ensheng Shi, Xilin Liu, Yuchi Ma, and Zibin Zheng. Draincode: Stealthy energy consumption attacks on retrieval-augmented code generation via context poisoning. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 778–790. IEEE, 2025. [24] Yanlin Wang, Ensheng Shi, Lun Du, Xiaodi Yang, Yuxuan Hu, Yanli Wang, Daya Guo, Shi Han, Hongyu Zhang, and Dongmei Zhang. Context-aware code summarization with multi-relational graph neural network. Automated Software Engineering, 32(1):19, 2025. [25] Hanyang Guo, Xiangping Chen, Yuan Huang, Yanlin Wang, Xi Ding, Zibin Zheng, Xiaocong Zhou, and Hong-Ning Dai. Snippet comment generation based 9

of the ACM on Software Engineering, 3(FSE):2790– 2812, 2026. [36] Yanli Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, et al. Repotransbench: A real-world multilingual benchmark for repositorylevel code translation. IEEE Transactions on Software Engineering, 2025. [37] Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, and Zibin Zheng. From static to dynamic: Benchmarking realworld code review with mcr-bench, 2026. [38] John Yang, Carlos E Jimenez, Alex Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik Narasimhan, Diyi Yang, Sida Wang, and Ofir Press. Swe-bench multimodal: Do ai systems generalize to visual software domains? In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 2794–2829, 2025. [39] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. Swe-bench pro: Can ai agents solve longhorizon software engineering tasks?, 2025. [40] Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. Efficient benchmarking (of language models). In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2519–2536, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.139. [41] Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. Anchor points: Benchmarking models with much fewer examples. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1576–1601, St. Julian’s, Malta, March 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.eacl-long.95. [42] Peiwen Yuan, Yueqi Zhang, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, and Kan Li. Beyond one-size-fits-all: Tailored benchmarks for efficient evaluation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15591– 15615, Vienna, Austria, July 2025. Association for

Computational Linguistics. ISBN 979-8-89176-2510. doi: 10.18653/v1/2025.acl-long.759. [43] Frederic M Lord and Melvin R Novick. Statistical theories of mental test scores. IAP, 2008. [44] John P. Lalor, Hao Wu, and Hong Yu. Building an evaluation scale using item response theory. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 648–657, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1062. [45] Xuanwen Ding, Chengjun Pan, Zejun Li, Jiwen Zhang, Siyuan Wang, and Zhongyu Wei. AutoJudger: An agent-driven framework for efficient benchmarking of MLLMs. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15009–15034, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acllong.685. [46] Ronald K Hambleton and Hariharan Swaminathan. Item response theory: Principles and applications. Springer Science & Business Media, 2013. [47] Bryce B Reeve and Peter Payers. Applying item response theory modelling for evaluating. Assessing quality of life in clinical trials: Methods and practice, 55, 2005. [48] Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Neural Networks, 22(5):544–557, 2009. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2009.06.042. Advances in Neural Networks Research: IJCNN2009. [49] Viktoriia Sharmanska, Novi Quadrianto, and Christoph H. Lampert. Learning to rank using privileged information. In Proceedings of the 2013 IEEE International Conference on Computer Vision, ICCV ’13, page 825–832, USA, 2013. IEEE Computer Society. ISBN 9781479928408. doi: 10.1109/ICCV.2013.107. [50] Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: Similarity control and knowledge transfer. Journal of Machine Learning Research, 16(61):2023–2049, 2015. [51] David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information, 2016. [52] Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. Learning with privileged information for efficient image super-resolution. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 465– 482, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58586-0. [53] Frederic M Lord. Applications of Item Response Theory to Practical Testing Problems. ERIC, 1980. 10

[54] Dong C. Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical Programming, 45(1):503–528, Aug 1989. ISSN 14364646. doi: 10.1007/BF01589116. [55] W. K. Hastings. Monte carlo sampling methods using markov chains and their applications. Biometrika, 57 (1):97–109, 04 1970. ISSN 0006-3444. doi: 10.1093/ biomet/57.1.97. [56] Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183–233, Nov 1999. ISSN 1573-0565. doi: 10.1023/A:1007665907178. [57] Mike Wu, Richard L Davis, Benjamin W Domingue, Chris Piech, and Noah Goodman. Variational item response theory: Fast, accurate, and expressive. International Educational Data Mining Society, 2020. [58] Nelson F. Liu, Tony Lee, Robin Jia, and Percy Liang. Do question answering modeling improvements hold across benchmarks? In Anna Rogers, Jordan BoydGraber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13186–13218, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 2023.acl-long.736. [59] Kaiser Sun, Adina Williams, and Dieuwke Hupkes. The validity of evaluation results: Assessing concurrence across compositionality benchmarks. In Jing Jiang, David Reitter, and Shumin Deng, editors, Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 274–293, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.conll-1.19.

11

Related documents

• Benchmarking Autonomous Agents against Temporal, Spatial, and Semantic Evasions#216717
• PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives#455597
• Agentic Performance at the Edge: Insights from Benchmarking#175159
• UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities#820473
• SEE: Structure-aware Exploring & Exploiting for Long-horizon GUI Agent Trajectory Synthesis#630083