Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments

arXiv:2606.14397v1 [cs.LG] 12 Jun 2026

Mykola Vysotskyi2,7∗ Runqi Lin1∗ Grzegorz Biziel2 Michal Zakrzewski2 Sebastian Montagna2 Damian Rynczak2 Shreyansh Padarha1 Kumail Alhamoud3 Zihao Fu4 William Lugoloobi1 Kai Rawal1 Hanna Yershova2 Xander Davies1,5 Taras Rumezhak2 Guohao Li6 Fazl Barez1 Baoyuan Wu4 Arkadiusz Drohomirecki2 Yarin Gal1 Chris Russell1 Christopher Summerﬁeld1 Adam Mahdi1 Volodymyr Karpiv2 Philip Torr1 Adel Bibi1† 1 4

University of Oxford 2 SoftServe 3 Massachusetts Institute of Technology The Chinese University of Hong Kong, Shenzhen 5 UK AI Security Institute 6 Eigent.AI 7 Ukrainian Catholic University

Abstract As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabilities. However, current benchmarks are typically built on popular applications with relatively simple tasks and focus on a narrow set of capabilities while overlooking broader dimensions, resulting in saturated performance on modern agents and failing to probe their limitations. To this end, we introduce GauntletBench, a web-based benchmark for evaluating agent generalisation in challenging scenarios, focusing on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning), across ﬁve less-covered professional applications (Video Editor, Workflow Builder, 3D Modeller, Flight Analyser, and Circuit Designer), each with 20 vision-intensive tasks (100 in total). Our benchmark provides a modular pipeline that comprises an environment compatible with both open- and closed-source agent frameworks, a controlled web-based application, a well-structured task suite, and an automated evaluation engine with diverse metrics. Contrary to widespread expectations, our empirical results reveal that frontier agentic systems remain far from achieving human-level performance. Even the state-of-the-art agent achieves only a 19.1% success rate on our GauntletBench, highlighting the limitations in these overlooked capabilities and generalisation. By comparison, non-expert human annotators achieve over 80% success on our challenging yet feasible tasks, revealing the substantial gap between current agent capabilities and those required for complex real-world scenarios.1 “The greatest obstacle to discovery is not ignorance — it is the illusion of knowledge.” — Daniel J. Boorstin

Introduction

The rapid advancement of multimodal large language models (MLLMs) has signiﬁcantly improved their cross-modal perception and reasoning capabilities [3, 12, 24], establishing a strong foundation ∗ Equal contribution

† Corresponding author: [email protected] 1 The homepage, code, and tasks of GauntletBench are available publicly.

Preprint.

Workflow Builder

3D Modeller

Flight Analyser

Circuit Designer

Tasks

Applications

Video Editor

Split a video at its midpoint and insert a second video between the two . . .

Modify an existing linear workflow so that active and inactive users are . . .

Open a single-object scene, double the cube’s scale on all axes, and change . . .

Among several flights near an airport, determine which one is the last to remain . . .

Modify an existing linear workflow so that active and inactive users are . . .

Figure 1. Overview of GauntletBench. Our benchmark contains ﬁve controlled web-based applications with 100 vision-intensive tasks. Detailed application descriptions are provided in Table 1, and full application screenshots are shown in Appendix A. for the transition from passive assistants to intelligent agents [42, 28, 32]. Agentic systems built on these models are now widely deployed for autonomous interactions and execution across diverse real-world domains, spanning from consumer-facing [19, 41, 45] to enterprise applications [30, 22]. As modern agents evolve, they are expected to handle increasingly complex scenarios, necessitating more challenging evaluations to reﬂect their capabilities in such settings [5, 28]. While existing benchmarks provide valuable assessments, they often operate within highly homogeneous settings with relatively simplistic tasks, leading to saturated performance on modern agents and failing to reﬂect real-world complexity [5, 19]. Speciﬁcally, current benchmarks face two fundamental limitations: (i) they are built on popular applications or near-identical replicas (e.g., Amazon, Booking, or CRM systems [11, 22]), with the tasks built upon them exhibiting high similarity and simplicity, making them insufﬁcient to probe the limits of current agentic systems; and (ii) the task design typically targets a narrow set of capabilities, such as UI understanding, tool use, and longterm reasoning [45, 37], while failing to capture broader yet critical aspects of agentic performance. Moreover, these benchmarks often struggle to continuously support new agentic frameworks due to intricate pipelines, limited compatibility, and rigid conﬁgurations [19]. These limitations lead to a mismatch between the evaluations provided by current benchmarks and the challenges of complex real-world scenarios, resulting in an inaccurate assessment of agentic capabilities. To bridge this gap, we propose GauntletBench, a web-based benchmark targeting underrepresented capabilities in less-covered applications via visually grounded tasks to evaluate agents generalisation in challenging scenarios. Speciﬁcally, our benchmark focuses on three previously overlooked capabilities, namely temporal perception, graphical understanding, and 3D reasoning, which are increasingly important as agentic systems operate in more complex settings and expand across broader modalities. In our benchmark, we build ﬁve professional applications, including Video Editor, Workflow Builder, 3D Modeller, Flight Analyser, and Circuit Designer, which are well-suited for evaluating our target capabilities and provide less familiar interfaces for assessing generalisation. Each application includes 20 challenging tasks across well-calibrated difﬁculty levels, covering a spectrum of capabilities from vision-based UI understanding to the targeted aspects of our benchmark. To support compatibility, controllability, and reproducibility, our platform adopts a modular design consisting of four components: environments, applications, tasks, and the evaluation engine. The environments provide high-level observation and action interfaces, supporting both open-source and API-based MLLM agents, as well as closed-source agent frameworks. Built on these environments, we develop our applications using modern web technologies to support natural interaction patterns and ﬂexible conﬁguration. We provide tasks across three difﬁculty levels: easy, medium, and hard, following a standardised structure comprising application background, task description, and result format. Finally, we provide an automated evaluation engine that integrates tailored objective evaluators to assess success rate, MLLM-judged progress rate, and efﬁciency measures, supporting automated evaluation even for closed-source agent frameworks. Our evaluations indicate that most open-source MLLM agents, such as Qwen-3-VL [4] and Llama4-Maverick [20], achieve near-zero performance, and no API-based MLLM agents achieve success rates above 13.2% on GauntletBench. Even the state-of-the-art (SOTA) closed-source agentic frameworks like GPT Computer Use [26], Gemini Enterprise [23], and Claude Opus Computer Use [1] achieve success rates of only 4.3%, 13.7%, and 19.1%, respectively. In contrast, human annotators without speciﬁc domain knowledge of these applications achieve a success rate of over 80% on our 2

benchmark while requiring 30% fewer steps than the frontier agentic system. This substantial gap in our challenging yet feasible benchmark indicates that current agentic systems still have critical limitations in these previously overlooked capabilities and generalisation, which are not faithfully reﬂected by existing benchmarks. We believe our benchmark provides a new perspective on evaluating agent generalisation in complex real-world scenarios, enabling a more accurate assessment to meet the growing demands of such settings. Our major contributions are summarised as follows: • We introduce the GauntletBench, a challenging benchmark designed to evaluate agents generalisation by targeting three underexplored capabilities: temporal perception, graphical understanding, and 3D reasoning. • Our benchmark provides a web-based environment, comprising ﬁve less-covered professional applications, 100 challenging yet human-feasible tasks, and an automated evaluation engine. • Empirical results show that SOTA agent achieve only a 19.1% success rate on GauntletBench, indicating that they remain far from human-level performance in complex real-world scenarios.

GauntletBench

In this section, we introduce GauntletBench, a challenging web-based benchmark targeting agent generalisation in complex real-world scenarios. Our platform follows a modular pipeline comprising a uniﬁed web-based environment (Section 2.1), where agents interact with controlled applications (Section 2.2) to execute well-structured tasks (Section 2.3), and outcomes are evaluated by an automated engine (Section 2.4). 2.1 Environments Following standard formulations [22, 11], we deﬁne a uniﬁed environment as E = (S, A, O, T , R), where S denotes the state space, A and O indicates the action and observation space, T is the transition function, and R refers to the reward function. At each time step t, the agent receives an observation ot ∈ O, executes an action at ∈ A, transitions to the next state according to st+1 = T (st , at ), and use a binary reward rt ∈ {0, 1} to evaluate task success. Observation Space As the capabilities targeted by GauntletBench and its task design inherently require visual understanding, we restrict agent interaction to high-level interfaces (e.g., Playwright) without exposing low-level browser APIs (e.g., Chrome DevTools Protocol), thereby grounding observations in visual perception. Consequently, the 3D Modeller, the Flight Analyser, and the Circuit Designer applications provide only canvas-based interfaces, forcing agents to rely on Screenshots to understand the current web page, as alternative observation channels are unavailable under these constraints. For the Video Editor and the Workflow Builder applications, we additionally provide the Accessibility Tree to supply semantic information about UI elements. Notably, relying solely on structured elements is insufﬁcient for completing our vision-intensive tasks; thus, visual information remains necessary for successful execution. Action Space Agents act by selecting at ∈ A from a set of high-level commands that emulate standard user interactions, such as click, type, select, and scroll, as well as navigation operations. These actions are executed via a browser automation layer (e.g., Playwright) and, when available, are optionally guided by elements identiﬁed from the accessibility tree. Agent Interface We provide a uniﬁed interface that supports open-source and API-based MLLM agents, as well as closed-source agent frameworks. For MLLM agents, we leverage components of the REAL framework [11] to instantiate agents via MLLM-webpage interactions. For closedsource agent frameworks, the standardised interface we provide enables their execution without requiring access to the internal invocation process. This design decouples agent implementation from environment execution, enabling consistent evaluation across different systems. 2.2 Applications To faithfully evaluate agent generalisation in complex real-world scenarios, our benchmark is designed to assess previously underexplored capabilities while introducing less familiar interfaces. To achieve these objectives, we identify ﬁve less-covered professional applications spanning diverse domains: Video Editor, Workflow Builder, 3D Modeller, Flight Analyser, and Circuit Design. 3

Table 1. Applications in GauntletBench. We provide anonymous links for these applications, along with their corresponding descriptions and targeted capability categories. Application

Description

Target Capabilities

Video Editor

A timeline-based media editing interface for previewing clips and performing common editing operations, including trimming, audio overlay, colour adjustment of video segments, transition effects, and text insertion. A node-based workﬂow builder in which users build automation pipelines by adding, connecting, and conﬁguring nodes on a visual canvas, including trigger, branching, data, API, system, and AI nodes. An interactive 3D scene workspace for navigation, object inspection, and spatial manipulation, including translation, rotation, scaling, and foreground property adjustment. A ﬂight radar map for searching aircraft, replaying historical trafﬁc, inspecting trajectories, and extracting telemetry from detailed ﬂight panels. A browser-based circuit environment, adapted from CircuitJS1 [31], for composing and editing circuits consisting of logical and electrical components.

Temporal perception; Graphical understanding

Workﬂow Builder

3D Modeller

Flight Analyser

Circuit Designer

Graphical Understanding

Graphical understanding; 3D reasoning Temporal perception; Graphical understanding Graphical understanding

As shown in Table 1, these applications not only support the evaluation of basic capabilities, such as UI understanding, tool use, and long-term reasoning, but are also well-suited for assessing our targeted capabilities: temporal perception, graphical understanding, and 3D reasoning. Moreover, the interfaces of our applications differ substantially from those of popular applications, further supporting the evaluation of generalisation, as illustrated in Figure 1. These characteristics make our applications an effective testbed for assessing the limitations of modern agentic systems. Website Technology Stack All applications in our benchmark are implemented using modern web technologies to ensure realistic interaction patterns. Speciﬁcally, the websites are implemented using JavaScript frameworks (e.g., React or Next.js), while their visual interfaces are rendered using canvas-based techniques. We provide only minimal structured interfaces in two of the ﬁve applications, where the exposed information alone remains insufﬁcient for task completion, while restricted low-level browser interaction prevents access to canvas-based content. This implementation not only provides consistent interaction logic, but also intentionally restricts agents from directly accessing structured observations (e.g., DOM or accessibility tree) heavily relied upon by previous LLM-based agents [11], thereby ensuring that the targeted capabilities are genuinely evaluated through visual perception. 2.3 Tasks Our benchmark provides 20 challenging yet human-feasible tasks for each application, totalling 100 tasks. Each task is designed to evaluate one or more agent capabilities, either basic ones or those targeted by GauntletBench. We further support customised task design by enabling users to deﬁne tasks and upload ground truth, allowing seamless integration with our automated evaluation engine. Task Complexity We categorise tasks into three levels: easy, medium, and hard. For each application, we construct 2 easy, 9 medium, and 9 hard tasks. Easy tasks require agents to follow explicit and straightforward instructions, whereas medium tasks involve composing multiple features or functionalities within the environment, requiring agents to coordinate several operations to achieve the goal. In contrast, hard tasks demand more complex capabilities, including multi-step reasoning, ﬁne-grained understanding, and the combination of multiple capabilities. We provide detailed descriptions of all 100 tasks in Appendix G. Task Structure Each task follows a standardised three-level structure: application background, task description, and result format. The application background consists of three components: an application overview (describing the applications purpose), features & capabilities (describing the available functionalities), and environment architecture & interaction paradigms (deﬁning agent interaction with the interface). The task description speciﬁes the objective through a clearly deﬁned goal (describing the high-level objective) and steps (enumerating the conditions required for successful task completion). The result format deﬁnes the expected output format and evaluation target. We consider two types of output formats: information retrieval and action-based. For information re4

Figure 2. Tailored Evaluator. Each application uses a dedicated comparator that aligns agent output with ground truth via task-aware rules rather than strict string equality. Flight Analyser (top) combines exact property matching with order-invariant set equality on list ﬁelds, while Video Editor (bottom) applies ∆ ≤ 100 ms timing tolerance and format-normalised fuzzy colour matching, enabling human-aligned yet rigorous scoring. trieval tasks, it speciﬁes the exact output format and required content. For action-based tasks, agents are required to export the application state as a JSON ﬁle or return a completion signal (e.g., a done JSON object) upon task completion. The detailed three-level structure and background information for each application are provided in Appendix F. 2.4 Evaluation Engine and Metrics Our evaluation engine supports fully automated evaluation for both open- and closed-source agent frameworks with diverse evaluation metrics. To achieve this, we embed a tailored evaluator for each application, enabling the system to automatically execute the corresponding evaluation once an agent signals task completion. This design not only improves compatibility by enabling evaluation even for closed-source agent frameworks, but also enables more ﬂexible and sophisticated evaluation metrics. As shown in Figure 2, the evaluator for Video Editor allows a 100 ms tolerance when matching start times and video durations between agent outputs and ground truth, as even human annotators cannot perform such edits with frame-level precision. Our evaluator also supports fuzzy matching for order-invariant tasks and colour differences. More importantly, this tailored evaluator provides more ﬁne-grained localisation and detection of actual errors and their variants, such as property mismatches in Flight Analyser. This design makes success rate assessment more reasonable and human-aligned without compromising rigour, and similarly extends to progress rate and efﬁciency metrics. More case analyses of tailored evaluators on different applications can be found in Appendix E. Success Rate We deﬁne the success rate (SR) as the fraction of tasks in which the agent successfully completes the task according to the tailored objective evaluator. Formally, let D = {d1 , . . . , dN } denote the set of N tasks and let 1[·] be the indicator function. For each task di , the objective evaluator returns a binary label si ∈ {0, 1}, where si = 1 if and only if the agent output matches the ∑N ground truth across all checkpoints. The SR is then given by SR = N1 i=1 1[si = 1]. This metric is a strict all-or-nothing criterion that reﬂects the agents ability to fully satisfy all the requirements of the given tasks. Progress Rate To capture meaningful intermediate progress on tasks that the agent partially completes, we additionally report the progress rate (PR). For each task di , an MLLM-based judge assigns 5

Table 2. Performance of Agents on GauntletBench. The success rate (%) and progress rate (15) are reported in the left and right panels, respectively. “-” indicates applications that provide screenshot-only access and are therefore unsuitable for LLMs or vision-disabled MLLM agents. Results for each application are computed over 20 tasks and averaged over 3 independent runs. Bold and underlined entries denote the best and second-best results, respectively. Success Rate (↑) Agent

Progress Rate (↑)

Video Workﬂow 3D Flight Circuit Editor Builder Modeller Analyser Designer

Avg.

Video Workﬂow 3D Flight Circuit Avg. Editor Builder Modeller Analyser Designer

0.0 0.0 0.0 0.0 0.9 0.3

2.0 1.9 1.7 1.9 2.1 2.0

1.8 1.8 1.4 1.9 2.3 1.9

2.2 1.7

0.3 0.3 1.7 2.3 12.3 13.2

2.0 2.2 2.3 2.7 2.9 2.5

Open-source MLLM Agent Gemma-3-27B Mistral-Large-3 Qwen3-32B (LLM) Qwen3-VL-32B DeepSeek-V3.2 (LLM) Llama-4-Maverick

0.0 0.0 0.0 0.0 0.0 0.0

0.0 0.0 0.0 0.0 1.7 0.0

0.0 0.0

0.0 0.0 1.7 6.7 13.3 5.0

1.7 0.0 1.7 1.7

0.0 0.0

0.0

1.7

0.0

0.0 1.7

0.0 0.0

3.3 16.7

0.0 0.0

5.0

0.0 15.0 25.0

15.8

15.0

0.0 0.0 13.3

15.3

8.3 11.7 11.7

5.0 3.3 11.7

4.3 13.7

43.3

19.1

85.0

75.0

81.7

77.5

80.8

1.9 1.9

1.0 1.4

1.8 1.7 1.6 1.7 2.2 1.7

1.8

2.0

1.1

1.8

1.9

1.0

2.7 2.1 2.1 2.4 3.1 2.9

2.4 1.7

2.0 2.1

1.0 1.0

2.0 2.4 3.1

2.3 2.8 2.8

1.0 1.0

2.8 3.1

2.4

2.5

2.3

3.3

3.6

3.1

3.2

2.9 2.9

1.1 2.2 2.7

2.2 3.0

3.5

4.2

3.8

3.6

4.4

4.1

4.0

API-based MLLM Agent Qwen-Max OpenAI o3-pro GPT-5.4 (w/o vision) GPT-5.4 Claude-Opus-4.6 Gemini-3.1-Pro

16.7

2.9

2.0 1.8 2.2 2.1 2.4 2.8

Closed-source Agent Framework GPT-5.4 CU Gemini Enterprise Claude-Opus-4.6 CU

8.3 13.3

0.0 40.0

3.1

Human Annotation Human

a score pi ∈ {1, 2, 3, 4, 5} based on visible ﬁnal state and trajectory evidence, where 1 denotes no meaningful progress and 5 denotes fully successful completion. The detailed scoring criteria are ∑N described in Appendix C. The PR is computed as the mean judge score PR = N1 i=1 pi . Unlike the SR, the PR rewards agents that make substantial progress toward the goal, providing a more ﬁne-grained view of agent capability. Efﬁciency Metrics In addition to effectiveness, efﬁciency is another important aspect of agent performance. We report the following efﬁciency metrics: consumed tokens and consumed steps. Let D = {d1 , . . . , dN } denote the set of N tasks. For each task di , let ci be the total number of tokens consumed by the agent, ai the number of interaction steps, and si ∈ {0, 1} the binary success ∑N indicator returned by the objective evaluator. consumed tokens (CT) is deﬁned as CT = N1 i=1 ci . ∑N consumed steps (CS) is deﬁned as CS = N1 i=1 ai .

Experiment

3.1

Evaluation Setup

We evaluate GauntletBench using 14 frontier agents, spanning open-source and API-based MLLM agents as well as closed-source agent frameworks, assessing both task performance and efﬁciency. The details of human annotation are provided in Appendix A. Open-Source MLLM Agents We evaluate a diverse set of widely used open-source MLLMs, including Gemma-3 [34], Mistral-Large-3 [21], Qwen3-VL [4], and Llama-4-Maverick [20]. To further verify the vision-intensive nature of our tasks, we additionally evaluate text-only LLM agents, including Qwen3 [40] and DeepSeek-V3.2 [8], on the Video Editor and Workflow Builder, where minimal structured information is exposed. API-Based MLLM Agents We also provide the result from proprietary MLLMs, including QwenMax [35], OpenAI o3-pro [25], GPT-5.4 [27], Gemini-3.1-Pro [14], and Claude-Opus-4.6 [2]. Closed-Source Agent Frameworks Finally, we evaluate advanced closed-source agent frameworks to further demonstrate the challenge and compatibility of our benchmark, including GPT Computer Use (GPT CU) [26], Gemini Enterprise [13], and Claude Computer Use (Claude CU) [1]. 6

Success Rate (%)

Easy Medium Hard

56.7

43.3

46.7

33.3 26.7 23.0

14.8

13.3 3.3 0 0.7 0

0.7 0

0 0

8.1

8.9

3.02.2

0.71.5

em .6

k ic

4 s-

i-3

6 4. s- e pu s O U e- er t e ud u is la p pr C om er C nt iE in em G se U .4 er -5 ut PT p G om ro C -P .1

V k-

.4 -5 PT ) G M LL n, .4 io -5 is PT v G /o ro (w -p o3 I

O -M en

0 0.7 0

9.6 7.4

8.98.9

3.3

)

Figure 3. Agent Success Rate Across Task Difﬁculty Levels. Our benchmark contains 10, 45, and 45 tasks in the easy, medium, and hard difﬁculty levels, respectively. Results are averaged over three independent runs.

Tasks Solved

Easy Medium Hard

8 6 3.3

4 2

3.3

2.7

0.7 0

0 Video Editor

2.3

1.7

2 0.7

Workﬂow Builder

0.3 0

3D Modeler

Flight Analyser

Circuit Designer

Figure 4. Number of Solved Tasks Across Difﬁculty Levels. Our benchmark contains 2, 9, and 9 tasks for the easy, medium, and hard difﬁculty levels of each application, respectively. Results are generated by Claude-Opus-4.6 Computer Use, and averaged over three independent runs. 3.2

Performance Evaluation

Frontier Agentic Systems Remain Far from Human-Level Performance As shown in Table 2 (left panel), all open-source MLLM agents fail to surpass a 1.7% success rate on any single application, while most evaluated systems achieve 0.0% across all applications. The strongest agentic system, Claude-Opus-4.6 Computer Use, achieves only a 19.1% average success rate, followed by Gemini-3.1-Pro at 13.7% and Claude-Opus-4.6 at 13.2%. In contrast, non-expert human participants achieve between 75% and 85% across all applications, indicating that the benchmark is challenging yet feasible. This substantial gap revealed by GauntletBench not only more faithfully reﬂects agent capabilities in complex real-world scenarios, but also exposes the limitations of existing benchmarks in evaluating the challenges and capabilities required for modern agents. How Far Current Agents Remain from Reliable Performance? From Table 2 (right panel), we can observe that nearly all agents exhibit meaningful intermediate progress toward task completion, as reﬂected by greater than 1 progress rates. Claude-Opus-4.6 Computer Use even achieves an average progress rate exceeding 50% across all tasks, suggesting that failures often stem not from an inability to make progress, but from difﬁculty reliably handling the accumulating complexity of longhorizon task execution. This observation is further supported by Figure 3, where agents perform substantially better on easy tasks than on medium and hard ones, highlighting the limited robustness of current agents under increasing task complexity. These results provide an optimistic perspective: current agents are not entirely incapable of handling our targeted capabilities and challenging tasks, but their existing abilities remain insufﬁcient for reliable performance. 7

Figure 5. Efﬁciency of Agents on GauntletBench. The consumed tokens and consumed steps are reported in the top and bottom panels, respectively. Results for each application are computed over 20 tasks and averaged over three independent runs. What Capabilities Do Current Agents Lack Most? As illustrated in Figure 4, the Flight Analyser and Circuit Designer are the most challenging applications for current agents, with success rates at 11.7%. These results suggest that current agents may already possess basic recognition of graphical elements and structures, yet still lack a deeper understanding of their relationships, interactions, and resulting behaviours. Does the Vision Modality Help Agents Complete Tasks? To understand whether MLLMs genuinely rely on vision to solve our intentionally vision-intensive tasks, we compare text-only LLM agents on the Video Editor and Workflow Builder applications, where minimal structured information is exposed. From Table 2 (right panel), we can observe that relying solely on structured information enables agents to complete only a very small subset of tasks. In contrast, enabling visual inputs consistently improves agent performance, increasing average progress rates on these two applications by 43.5% and 15.5% for the Qwen and GPT families, respectively. Which Reasoning Strategies Are Most Efﬁcient? As shown in Figure 5, Qwen-Max incurs the highest inference cost in terms of token consumption, yet does not achieve a correspondingly strong success rate. In contrast, Claude Opus 4.6 Computer Use achieves the best overall performance with surprisingly low inference cost. For a horizontal comparison among token-similar models, Claude Opus 4.6 Computer Use consumes signiﬁcantly more steps than Qwen-3-VL-32B and OpenAI o3pro. While we cannot fully exclude the inﬂuence of model scale, intrinsic capabilities, and closedsource framework design, our results suggest that taking more ﬁne-grained and incremental steps, a divide-and-conquer strategy, may be more effective and efﬁcient for challenging tasks. 3.3 Ablation Studies The Impact of Model Scale Table 3 shows that increasing model scale consistently improves both success rate and progress rate. Notably, the largest model in each family also achieves the lowest token consumption (despite the higher cost of individual inference steps), while maintaining a comparable number of interaction steps; consumed tokens and steps are reported on the mediumdifﬁculty subset, where the comparison across scales is most informative. These results suggest that, for our challenging tasks, improvements in performance cannot be achieved solely through additional steps, but instead require fundamentally stronger models. The Role of Reasoning As shown in Table 4, enabling extended reasoning generally improves success rates for sufﬁciently capable base models, but also introduces a trade-off in token consumption and interaction steps, reported here on the medium-difﬁculty subset. We additionally evaluate Qwen-3-VL with and without extended reasoning, but observe no improvement in performance. These results suggest that extended reasoning is most effective when the underlying model has sufﬁcient capacity to beneﬁt from additional deliberation; for weaker models, the extra computation translates into only limited performance gains. 8

Table 3. The Impact of Model Size. We compare two model families that share an architecture but differ in scale: GPT-5.4 (Nano, Mini, full) and Claude-4.6 (Haiku, Sonnet, Opus). Success rate (%) and progress rate (1–5) are computed over all 100 tasks. Consumed tokens (in thousands, K) and consumed steps are reported on the 45 medium-difﬁculty tasks, where most agents make meaningful but not yet saturated progress and the efﬁciency comparison across scales is therefore most informative. All values are mean ± standard deviation over three independent runs. Bold and underlined entries denote the best and second-best results, respectively. Results for every difﬁculty tier tasks are present in Appendix B. Model

Success Rate (↑)

Progress Rate (↑)

Consumed Tokens (↓)

Consumed Steps (↓)

GPT-5.4-Nano GPT-5.4-Mini GPT-5.4

0.0 ± 0.0 0.0 ± 0.0 2.3 ± 2.4

1.50 ± 0.04 1.35 ± 0.16 2.14 ± 0.84

521 ± 819 449 ± 807 330 ± 396

36.0 ± 40.7 27.3 ± 42.7 35.7 ± 34.3

Claude-Haiku-4.5 Claude-Sonnet-4.6 Claude-Opus-4.6

1.7 ± 2.4 8.7 ± 1.2 12.3 ± 3.3

2.01 ± 0.08 2.21 ± 0.10 2.44 ± 1.17

1204 ± 1317 1496 ± 3448 888 ± 633

74.3 ± 38.6 84.8 ± 31.1 85.1 ± 25.2

Table 4. The Role of Reasoning. We compare pairs of models that share the same base model but employ different levels of reasoning effort: GPT-5.4 with and without extended reasoning, Gemini3.1-Pro at low versus high reasoning, and Claude-Opus-4.6 with and without thinking. Success rate (%) and progress rate (1–5) are reported over all 100 tasks. Consumed tokens (in thousands, K) and consumed steps are reported on the 45 medium-difﬁculty tasks, the regime where the impact of extra deliberation is most cleanly observable. All values are mean ± standard deviation over three independent runs. Bold and underlined entries denote the best and second-best results, respectively. Results for every difﬁculty tier tasks are present in Appendix B. Model

Success Rate (↑) Progress Rate (↑) Consumed Tokens (↓)

Steps (↓)

GPT-5.4 w/o reasoning GPT-5.4

0.7 ± 0.3 2.3 ± 2.4

1.50 ± 0.68 2.14 ± 0.90

132 ± 226 330 ± 396

10.3 ± 19.6 35.7 ± 34.3

Gemini-3.1-Pro Low-reasoning Gemini-3.1-Pro High-reasoning

6.3 ± 1.2 13.2 ± 0.7

2.38 ± 0.93 2.84 ± 1.00

326 ± 485 815 ± 652

28.7 ± 29.0 59.7 ± 33.6

Claude-Opus-4.6 Claude-Opus-4.6 Thinking

12.3 ± 3.3 10.3 ± 1.5

2.44 ± 1.17 2.37 ± 1.12

888 ± 633 947 ± 603

85.1 ± 25.2 92.9 ± 21.2

Accuracy of MLLM-Judged Progress Rate To validate the accuracy of the MLLM-judged Progress Rate, we collect human-in-the-loop annotations using a 15 rubric on ﬁve uniformly sampled trajectories per application (25 in total), and compare them with the stage-2 (ﬁnal-outcome) judge. As shown in Table 5, all evaluated MLLM judges produce results highly consistent with human annotations, suggesting that they provide reliable assessments. Moreover, GPT-5.1 achieves the strongest overall agreement and is therefore used for large-scale evaluation. 3.4 Error Analysis We manually inspected a sample of failed trajectories and identiﬁed four recurring error categories, detailed in Appendix D: (1) task-completion misjudgment, where agents either corrupt an otherwise correct result by continuing to act or terminate prematurely; (2) grounding failures, where actions are applied to incorrect UI elements without being detected by the agent; (3) protocol violations, ranging from outputting an entire plan without executing actions to batching multiple steps into a single response; and (4) systematic failures in open-source models, including syntactically invalid outputs and inﬁnite action loops, likely caused by the lack of agentic browser-control data during ﬁne-tuning.

Related Work

Prior work on computer-use agents has developed benchmarks for web, mobile, and desktop interaction, ranging from synthetic browser tasks to realistic websites, operating-system environments, and enterprise workﬂows [33, 18, 45, 17, 15, 9, 38, 29, 6, 10, 11]. These benchmarks have been central to measuring progress, but much of the current evaluation landscape still emphasizes consumerstyle information seeking, navigation, and form-based workﬂows. As agents improve, such settings 9

Table 5. Human Agreement with MLLM-Judged Progress Rate. We evaluate agreement using ﬁve uniformly sampled trajectories for each application. For each application, we report the mean PR, linearly weighted Cohen’s κ, and macro F1, averaged over three independent runs. ⋆ Human self-agreement is 1.00 by deﬁnition and shown as an upper bound. Video Editor

Judge PR

Workﬂow Builder F1

3D Modeller F1

Flight Analyser F1

Circuit Designer F1

Avg. F1

GPT-5.1 (w/ stage-1) 2.87 0.94 0.71 2.73 0.65 0.41 1.87 0.43 0.24 2.40 0.44 0.20 1.40 1.00 0.67 2.25 0.73 0.58 GPT-5.1 (w/o stage-1) 2.53 0.70 0.48 2.47 0.56 0.29 1.87 0.56 0.33 2.47 0.56 0.29 1.40 1.00 0.67 2.05 0.62 0.48 Claude Opus 4.6 2.13 0.53 0.21 2.07 0.36 0.04 1.33 0.46 0.35 1.87 0.43 0.31 1.27 0.74 0.44 1.73 0.54 0.35 Claude Sonnet 4.6 2.00 0.41 0.08 2.13 0.40 0.21 1.40 0.09 0.17 1.80 0.38 0.29 1.20 0.62 0.33 1.71 0.44 0.25 Human Evaluation 2.80 1.00⋆ 1.00⋆ 3.00 1.00⋆ 1.00⋆ 1.80 1.00⋆ 1.00⋆ 2.60 1.00⋆ 1.00⋆ 1.40 1.00⋆ 1.00⋆ 2.32 1.00⋆ 1.00⋆

increasingly risk saturating or underrepresenting the perceptual and spatial reasoning demands of professional software. Web Agent Benchmarks A growing number of work has focused on evaluating autonomous agents in web-based environments. Early benchmarks such as MiniWoB [33] and MiniWoB++ [18] introduced simpliﬁed web interaction tasks where agents had to perform basic operations like clicking buttons, ﬁlling forms, and navigating menus. While these benchmarks were instrumental in establishing the ﬁeld, they relied on synthetic, toy-like interfaces that did not capture the complexity of real websites. More recent efforts are aiming to close this gap. WebArena [45] proposed a suite of realistic, self-hosted web environments spanning e-commerce, forums, and content management systems, enabling agents to be evaluated on tasks that more closely resemble real-world usage. VisualWebArena [17] extended this idea by emphasizing visually grounded tasks that require agents to reason over rendered page content rather than relying solely on HTML structure. WebVoyager [15] took a different approach by deploying agents on live websites, which introduces non-determinism but better reﬂects practical deployment conditions. Mind2Web [9] contributed a large-scale dataset of real-world web tasks annotated with action sequences across diverse domains. REAL [11] proposed deterministic simulations of real websites to combine realism with reproducibility. AssistantBench [43] evaluated agents on open-ended web tasks requiring information synthesis. Despite this progress, most existing web benchmarks focus on consumer-level scenarios such as online shopping or information lookup, and they primarily test basic UI navigation and form-ﬁlling skills. Our work differs in that we target professional-level tasks that require higher-order capabilities like dynamic perception, graphical understanding, and 3D spatial reasoning, which remain largely unexplored in prior benchmarks. OS Agent Benchmarks A neighbouring line of work looks at agents that operate at the operating system level rather than inside a web browser. While this is not directly related to our work, the challenges are similar in many ways, and progress in one area often signals progress in the other. OSWorld [38] sets up a benchmark where agents control full desktop environments (Ubuntu, Windows, macOS) through keyboard and mouse actions across many different applications. AndroidWorld [29] and AndroidEnv [36] focus on mobile systems, asking agents to navigate apps and complete tasks on Android. WindowsAgentArena [6] targets productivity tasks inside the Windows operating system. CRAB [39] further broadens this direction by evaluating multimodal agents across desktop and mobile environments with graph-based ﬁne-grained evaluation. OS-level benchmarks cover a very wide range of agent skills, but they can be harder to reproduce and control than web environments, since desktop state depends on many moving parts. Our benchmark goes in a different direction: we stay inside the browser, where we get cleaner control and reproducibility, while still picking tasks that require reasoning well beyond simple point-and-click interactions. Agentic Evaluation A key challenge in evaluating autonomous agents is designing evaluation pipelines that are both fair and informative. Traditional approaches rely on exact-match comparisons between agent outputs and predeﬁned ground truths [10], which works reasonably well for simple retrieval tasks but becomes problematic for complex, open-ended scenarios where multiple valid solutions may exist. REAL [11] addressed this by introducing deterministic website simulations paired with step-wise trajectory tracing, implementing reproducible evaluation of agents through screenshots and action logs. Their framework decouples agent implementation from environment execution, which we adopt and extend in our work to support evaluation of both open-source models and closed-source agent frameworks through a uniﬁed interface. More recently, the use of LLMs as automated judges has gained popularity as a way to assess partial task completion [7]. Instead 10

of only checking whether the ﬁnal output is exactly correct, an LLM judge can inspect the agent’s trajectory and the ﬁnal visible state to assign a graded score that reﬂects how much progress was actually made. This is especially useful for tasks where an agent might get most of the way to the goal but fail on a small detail. However, LLM-based judges can be inconsistent and may hallucinate progress that did not actually occur [44], so careful prompt design and calibration are needed. Some works have also explored hybrid evaluation strategies that combine rule-based checks with modelbased assessment [17], taking the best of both worlds. In our benchmark, we implement a two-stage evaluation pipeline: a structured exact-match evaluator that checks domain-speciﬁc constraints, and an LLM-based judge that provides partial success scores based on visual trajectory evidence. We additionally report efﬁciency-oriented metrics including token consumption, latency, and number of interaction steps to give a more complete view of agent performance beyond just task accuracy. Saturation of Computer-Use Agent Benchmarks Agent systems have improved a lot in a short time, and many of the earlier benchmarks are starting to look too easy. On MiniWoB++ [16], modern agents score close to perfect on most tasks. Even on harder benchmarks like WebArena [45], the gap between top agents and humans is getting smaller every year. This trend suggests that soon current benchmarks might become insufﬁcient for differentiating performance of strong systems. Some recent work attempted to address it. For example, WorkArena [10] targets enterprise software tasks on the ServiceNow platform, and REAL [11] focuses on deterministic simulations of real websites that require longer interaction traces. But most of these benchmarks still test mostly text-heavy, form-based interactions. They do not really assess whether an agent can understand dynamic visual content, read graphical layouts, or reason about 3D scenes. Our benchmark targets exactly these underexplored skills, and our experiments show that even the strongest frontier models still perform far below human evaluators on these tasks. GauntletBench is complementary to this line of work. We keep the reproducibility advantages of browser-based environments while shifting the task distribution toward visually dense, dynamic, and specialist interfaces, including graphical editing, circuit design, ﬂight tracking, and 3D manipulation. Our evaluation also follows recent work on richer agentic assessment by combining structured checks with model-based partial-credit judging and efﬁciency metrics [7, 44].

Conclusion and Discussion

In this work, we introduce GauntletBench, a challenging web-based benchmark for evaluating agent generalisation in complex real-world scenarios. Our benchmark targets previously overlooked capabilities, including temporal perception, graphical understanding, and 3D reasoning, through diverse professional applications and carefully designed vision-intensive tasks. Empirical results across a broad range of frontier agentic systems reveal that current agents still remain far from reliable real-world performance. By focusing on challenging yet feasible tasks grounded in realistic environments, GauntletBench exposes substantial limitations of existing agentic systems while providing a more faithful assessment of their real-world capabilities. We hope GauntletBench can facilitate future research on robust perception, reasoning, planning, and interaction for next-generation agentic systems. Limitations Despite providing a challenging and professional benchmark for evaluating agent generalisation, GauntletBench covers only ﬁve specialised domains and a subset of interaction paradigms, which may not fully capture the diversity of real-world scenarios and could potentially introduce evaluation bias. In addition, our benchmark constrains agents’ capabilities in visually grounded interaction to ensure faithful evaluation, which may limit their full potential when combined with more observational challenges. Future Works We plan to expand GauntletBench to more diverse professional applications, richer multimodal interaction settings, and longer-horizon tasks. We also aim to incorporate additional evaluation dimensions, such as robustness, safety, and failure recovery, to better reﬂect the requirements of real-world deployment. Broad Impacts We believe GauntletBench can support the development of more robust, reliable, and trustworthy agentic systems by providing realistic and challenging evaluation settings for assessing agent generalisation. By promoting more rigorous evaluation practices, our benchmark may facilitate future research on safer and more dependable autonomous agents in real-world applications. 11

Acknowledgments and Disclosure of Funding The authors express their gratitude to Igor Ivanyshyn for his participation in the human annotation. The authors also thank Svyatoslav Marutyak for his valuable comments on the task design. Runqi Lin was funded by the Horizon Europe grant 101213369 (DVPS). Adel Bibi and Philip Torr acknowledge the UKRI Turing AI Fellowship (EP/W002981/1). Adel Bibi also acknowledges support from Coefﬁcient Giving and is afﬁliated with the Institute for Decentralized AI.

References [1] Anthropic. Developing a computer use model. developing-computer-use, 2024.

https://www.anthropic.com/research/

[2] Anthropic. System Card: Claude Opus 4.6. https://www-cdn.anthropic.com/ 14e4fb01875d2a69f646fa5e574dea2b1c0ff7b5.pdf, February 2026. [3] AI Anthropic. System card: Claude opus 4 & claude sonnet 4. Claude-4 Model Card, 2025. [4] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report, 2025. [5] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. [6] Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale, 2024. [7] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. [8] DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Erhang Li, Fangqi Zhou, Fangyun Lin, Fucong Dai, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Li, Haofen Liang, Haoran Wei, Haowei Zhang, Haowen Luo, Haozhe Ji, Honghui Ding, Hongxuan Tang, Huanqi Cao, Huazuo Gao, Hui Qu, Hui Zeng, Jialiang Huang, Jiashi Li, Jiaxin Xu, Jiewen Hu, Jingchang Chen, Jingting Xiang, Jingyang Yuan, Jingyuan Cheng, Jinhua Zhu, Jun Ran, Junguang Jiang, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Kexin Huang, Kexing Zhou, Kezhao Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Wang, Liang Zhao, Liangsheng Yin, Lihua Guo, Lingxiao Luo, Linwang Ma, Litong Wang, Liyue Zhang, M. S. Di, M. Y Xu, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Panpan Huang, Peixin Cong, Peiyi Wang, Qiancheng Wang, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du, Ruiling Xu, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runqiu Yin, Runxin Xu, Ruomeng Shen, Ruoyu Zhang, S. H. Liu, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoyuan Chen, Shengding Hu, Shengyu Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Songyang Zhou, Tao Ni, Tao Yun, Tian Pei, Tian Ye, Tianyuan Yue, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjun Gao, Wentao Zhang, Xi Gao, Xiangwen Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xingyou Li, Xinyu Yang, Xinyuan Li, Xu Chen, Xuecheng Su, Xuehai Pan, Xuheng Lin, Xuwei Fu, Y. Q. Wang, Yang Zhang, Yanhong Xu, Yanru Ma, Yao Li, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Qian, Yi Yu, Yichao Zhang, Yifan Ding, Yifan Shi, Yiliang Xiong, Ying He, Ying Zhou, Yinmin Zhong, Yishi Piao, Yisong Wang, Yixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyuan Liu, Yonglun Yang, Yongqiang Guo, Yongtong Wu, Yu Wu, Yuan Cheng, Yuan Ou, Yuanfan Xu, Yuduan Wang, Yue Gong, Yuhan Wu, Yuheng Zou, Yukun Li, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehua Zhao, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhixian Huang, Zhiyu Wu, Zhuoshu Li,

Zhuping Zhang, Zian Xu, Zihao Wang, Zihui Gu, Zijia Zhu, Zilin Li, Zipeng Zhang, Ziwei Xie, Ziyi Gao, Zizheng Pan, Zongqing Yao, Bei Feng, Hui Li, J. L. Cai, Jiaqi Ni, Lei Xu, Meng Li, Ning Tian, R. J. Chen, R. L. Jin, S. S. Li, Shuang Zhou, Tianyu Sun, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xinnan Song, Xinyi Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Dongjie Ji, Jian Liang, Jianzhong Guo, Jin Chen, Leyi Xia, Miaojun Wang, Mingming Li, Peng Zhang, Ruyi Chen, Shangmian Sun, Shaoqing Wu, Shengfeng Ye, T. Wang, W. L. Xiao, Wei An, Xianzu Wang, Xiaowen Sun, Xiaoxiang Wang, Ying Tang, Yukun Zha, Zekai Zhang, Zhe Ju, Zhen Zhang, and Zihua Qu. Deepseek-v3.2: Pushing the frontier of open large language models, 2025. [9] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023. [10] Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024. [11] Divyansh Garg, Shaun VanWeelden, Diego Caples, Andis Draguns, Nikil Ravi, Pranav Putta, Naman Garg, Tomas Abraham, Michael Lara, Federico Lopez, James Liu, Atharva Gundawar, Prannay Hebbar, Youngchul Joo, Jindong Gu, Charles London, Christian Schroeder de Witt, and Sumeet Motwani. Real: Benchmarking autonomous agents on deterministic simulations of real websites, 2025. [12] Google. A new era of intelligence with gemini 3. Google. URL: https://blog. google/products-andplatforms/products/gemini/gemini-3/( : 16.01. 2026), 2025. [13] Google. Computer Use. https://ai.google.dev/gemini-api/docs/computer-use, 2026. Google AI for Developers. [14] Google DeepMind. Gemini 3.1 Pro Model Card. https://storage.googleapis.com/ deepmind-media/Model-Cards/Gemini-3-1-Pro-Model-Card.pdf, February 2026. [15] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models, 2024. [16] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 39648–39677. Curran Associates, Inc., 2023. [17] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024. [18] Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workﬂow-guided exploration. CoRR, abs/1802.08802, 2018. [19] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In The Twelfth International Conference on Learning Representations. [20] Meta. Llama 4. https://www.llama.com/models/llama-4/, 2026. [21] Mistral AI. Introducing Mistral 3. https://mistral.ai/news/mistral-3, 2025. [22] Ying Mo, Yu Bai, Dapeng Sun, Yuqian Shi, Yukai Miao, Li Chen, and Dan Li. Entworld: A holistic environment and benchmark for veriﬁable enterprise gui agents. arXiv preprint arXiv:2601.17722, 2026. [23] OpenAI. Chatgpt atlas - release notes. https://help.openai.com/en/articles/ 12591856-chatgpt-atlas-release-notes, 2025. Accessed: 2026-05-06. [24] OpenAI. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. [25] OpenAI. OpenAI o3-pro. https://platform.openai.com/docs/models/o3-pro, 2025. [26] OpenAI. Computer Use. tools-computer-use, 2026.

https://developers.openai.com/api/docs/guides/

[27] OpenAI. GPT-5.4 Thinking System Card. https://deploymentsafety.openai.com/ gpt-5-4-thinking/gpt-5-4-thinking.pdf, March 2026.

[28] Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023. [29] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents, 2025. [30] Divyanshu Saxena, Rishikesh Maurya, Xiaoxuan Ou, Gagan Somashekar, Shachee Mishra Gupta, Arun Iyer, Yu Kang, Chetan Bansal, Aditya Akella, and Saravan Rajmohan. Continuous benchmark generation for evaluating enterprise-scale llm agents. arXiv preprint arXiv:2511.10049, 2025. [31] Iain Sharp. CircuitJS1: Electronic circuit simulator in the browser. https://github.com/sharpie7/ circuitjs1, 2015. GWT port of Paul Falstad’s original Java applet. Open-source, GPL v2.0. [32] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36:38154–38180, 2023. [33] Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An opendomain platform for web-based agents. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3135–3144. PMLR, 06–11 Aug 2017. [34] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Pluciska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, JeanBaptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. [35] Qwen Team. Qwen3-max: Just scale it, September 2025. [36] Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android, 2021. [37] Weixuan Wang, Dongge Han, Daniel Madrigal Diaz, Jin Xu, Victor Rühle, and Saravan Rajmohan. Odysseybench: Evaluating llm agents on long-horizon complex ofﬁce application workﬂows. arXiv preprint arXiv:2508.09124, 2025.

[38] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. [39] Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Anjie Yang, Zhaoxuan Jin, Jianbo Deng, Philip Torr, Bernard Ghanem, and Guohao Li. Crab: Cross-environment agent benchmark for multimodal language model agents, 2025. [40] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. [41] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022. [42] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022. [43] Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Oﬁr Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks?, 2024. [44] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-asa-judge with mt-bench and chatbot arena, 2023. [45] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024.

Appendix: Table of Contents A Human Evaluation

B Additional Ablation Studies on Efﬁcient

C Progress Rate Criteria

C.1 LLM-as-a-Judge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.1.1

Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.2 LLM-as-a-Judge Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.2.1

Stage 1 System Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.2.2

Stage 1 Task Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.2.3

Stage 2 System Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.2.4

Stage 2 Task Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D Error Analysis

E Tailored Evaluation

F Task Format

F.1

Prompt Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

F.2

Per-Application Background Blocks . . . . . . . . . . . . . . . . . . . . . . . . .

F.2.1

Video Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

F.2.2

Workﬂow Builder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

F.2.3

3D Modeller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

F.2.4

Flight Analyser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

F.2.5

Circuit Designer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

G GauntletBench Task Illustrations

G.1 Video Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

G.2 Workﬂow Builder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

G.3 3D Modeller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

G.4 Flight Analyser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

G.5 Circuit Designer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Human Evaluation

Harness Overview We built a custom web-based Human Evaluation Harness that mirrors the agent’s interaction surface: the participant works directly inside the same web application an agent would control. The harness only adds an instrumentation layer around the application — task delivery, screenshot capture, and answer collection — so human trajectories are structurally same as agent trajectories. The participant ﬂow has three stages: • Authentication Each participant enters an application-speciﬁc access code that allows a single evaluation session. • Pre-session screen Before the ﬁrst task, the participant sees written instructions and a short demo video of the application (Figure 6; the per-application videos are listed in Table 6). Both remain available during every task via Recall Instructions and Recall Demo Video buttons; screenshot recording is paused while either is open. • Session workspace During each task, the participant sees the application in the center, a header with the timer, task index, and recording status, and a right-hand sidebar with the task prompt and answer ﬁelds (Figure 7). The participant ﬁlls in the answer ﬁeld(s) and clicks Finish to proceed.

Figure 6. Human Evaluation Harness: Pre-Session Brieﬁng. Before starting the ﬁrst task, each participant is shown written instructions describing the application and the evaluation procedure, alongside an embedded demo video that illustrates basic interactions with the application. Both resources remain accessible during every task via the Recall Instructions and Recall Demo Video buttons; screenshot recording is paused while either is open to avoid leaking instructional content into the captured trajectory.

Table 6. Demo Videos Used in Human-evaluation. Each video provides a short walkthrough of the corresponding application interface, helping participants understand the available controls without revealing solutions to benchmark tasks. Application Demo Video Video Editor Workﬂow Builder 3D Modeller Flight Analyser Circuit Designer

https://www.youtube.com/watch?v=zS_8YdKwsx8 https://www.youtube.com/watch?v=8yWhj0ivqPY https://www.youtube.com/watch?v=Yr1hH7tWp28 https://www.youtube.com/watch?v=CQsKjizsID4 https://www.youtube.com/watch?v=bU5zztXSnjk

(a) Video Editor

(b) Workﬂow Builder

(d) Flight Analyser

(e) Circuit Designer

Figure 7. Human Evaluation Harness: Per-Task Workspace Across All Five Applications. Each panel shows the workspace for one application; the layout is identical across applications and mirrors the interaction surface presented to agents. The center panel embeds the live application that the participant directly operates. The right sidebar shows the task prompt (using the same uniﬁed template sent to agents, see Appendix F.1) and the answer ﬁeld(s) the participant must ﬁll in before clicking Finish to submit and advance to the next task. The header displays the countdown timer (40-minute hard limit), the current task index within the session, and the screenshot recording status. Recruitment For this round, we have 3 participants per application (15 participants total, 300 tasks completed in total). Each participant has a CS background and prior hands-on experience with the type of tool their assigned application represents. The harness is mainly built to scale to multiple annotators per application. Task Presentation Participants see the same prompt sent to agents, formatted with the uniﬁed template from Appendix F.1 (without application background part). No paraphrasing or extra hints are given. Time Limit Each task has a hard 40-minute limit. The timer keeps running while instructions or the demo video are reviewed mid-task. 19

No Retries Once Finish is clicked, the answer is ﬁnal and the session goes to next task. Recording With participant consent the harness captures screenshots of actions performed inside the tested application window only. Scoring Human outputs are scored through the same pipeline as agent outputs, with no humanspeciﬁc adjustments.

Additional Ablation Studies on Efﬁcient

In this section, we report the consumed tokens and consumed steps for the two ablation studies of the main paper (the impact of model scale and the role of reasoning) broken down by difﬁculty bucket (easy, medium, hard) as well as the pooled overall value. These per-difﬁculty efﬁciency numbers are the source from which the medium-difﬁculty columns in Tables 3 and 4 of the main paper are drawn. Table 7. The Impact of Model Size: Consumed Tokens by Difﬁculty. Consumed tokens (in thousands, K) for the two model families compared in Table 3: GPT-5.4 (Nano, Mini, full) and Claude4.6 (Haiku, Sonnet, Opus). Values are reported per difﬁculty bucket (Easy: 10 tasks, Medium: 45 tasks, Hard: 45 tasks) and pooled overall (100 tasks), as mean ± standard deviation over three independent runs. Best results in each column are highlighted in bold, second-best are underlined. Model

Easy

Medium

Hard

Overall

GPT-5.4-Nano GPT-5.4-Mini GPT-5.4

369 ± 736 366 ± 735 183 ± 249

521 ± 819 449 ± 807 330 ± 396

516 ± 796 415 ± 757 310 ± 438

504 ± 802 425 ± 778 307 ± 406

Claude-Haiku-4.5 Claude-Sonnet-4.6 Claude-Opus-4.6

697 ± 528 974 ± 613 680 ± 595

1204 ± 1317 1496 ± 3448 888 ± 633

1196 ± 1312 2322 ± 5786 995 ± 683

1150 ± 1267 1815 ± 4548 915 ± 659

Table 8. The Impact of Model Size: Consumed Steps by Difﬁculty. Consumed steps for the same models as Table 7, reported per difﬁculty bucket and pooled overall, as mean ± standard deviation over three independent runs. Best results in each column are highlighted in bold, second-best are underlined. Model

Easy

Medium

Hard

Overall

GPT-5.4-Nano GPT-5.4-Mini GPT-5.4

26.2 ± 38.5 20.3 ± 39.8 21.5 ± 25.3

36.0 ± 40.7 27.3 ± 42.7 35.7 ± 34.3

41.9 ± 42.9 26.4 ± 42.0 30.1 ± 33.4

37.7 ± 41.8 26.2 ± 42.1 31.8 ± 33.4

Claude-Haiku-4.5 Claude-Sonnet-4.6 Claude-Opus-4.6

72.8 ± 40.9 85.0 ± 29.8 60.2 ± 40.2

74.3 ± 38.6 84.8 ± 31.1 85.1 ± 25.2

73.2 ± 40.7 84.6 ± 30.6 86.1 ± 23.1

73.7 ± 39.8 84.7 ± 30.7 83.0 ± 27.3

Table 9. The Role of Reasoning: Consumed Tokens by Difﬁculty. Consumed tokens (in thousands, K) for the three reasoning pairs compared in Table 4: GPT-5.4 with and without extended reasoning, Gemini-3.1-Pro at low versus high reasoning, and Claude-Opus-4.6 with and without thinking. Values are reported per difﬁculty bucket (Easy: 10 tasks, Medium: 45 tasks, Hard: 45 tasks) and pooled overall (100 tasks), as mean ± standard deviation over three independent runs. Best results in each column are highlighted in bold, second-best are underlined. Model

Easy

Medium

Hard

Overall

GPT-5.4 w/o reasoning GPT-5.4

70 ± 108 183 ± 249

132 ± 226 330 ± 396

158 ± 280 310 ± 438

138 ± 245 307 ± 406

Gemini-3.1-Pro Low-reasoning Gemini-3.1-Pro High-reasoning

220 ± 366 483 ± 424

326 ± 485 815 ± 652

396 ± 539 870 ± 731

347 ± 503 806 ± 680

Claude-Opus-4.6 Claude-Opus-4.6 Thinking

680 ± 595 806 ± 573

888 ± 633 947 ± 603

995 ± 683 985 ± 685

915 ± 659 950 ± 640

Table 10. The Role of Reasoning: Consumed Steps by Difﬁculty. Consumed steps for the same models as Table 9, reported per difﬁculty bucket and pooled overall, as mean ± standard deviation over three independent runs. Best results in each column are highlighted in bold, second-best are underlined. Model

Easy

Medium

Hard

Overall

GPT-5.4 w/o reasoning GPT-5.4

4.7 ± 6.2 21.5 ± 25.3

10.3 ± 19.6 35.7 ± 34.3

10.5 ± 20.8 30.1 ± 33.4

9.8 ± 19.4 31.8 ± 33.4

Gemini-3.1-Pro Low-reasoning Gemini-3.1-Pro High-reasoning

21.5 ± 25.9 40.8 ± 30.6

28.7 ± 29.0 59.7 ± 33.6

32.8 ± 29.1 61.9 ± 34.1

29.8 ± 29.0 58.8 ± 34.1

Claude-Opus-4.6 Claude-Opus-4.6 Thinking

60.2 ± 40.2 74.4 ± 33.6

85.1 ± 25.2 92.9 ± 21.2

86.1 ± 23.1 89.0 ± 24.8

83.0 ± 27.3 89.4 ± 24.9

Progress Rate Criteria

C.1

LLM-as-a-Judge

For clarity, in this appendix we refer to the two prompted LLM calls as Stage 1 and Stage 2. In our implementation, Stage 1 uses gpt-4o-mini and Stage 2 uses gpt-5.1. C.1.1

Details

Given a trajectory of screenshots and actions, we ﬁrst construct consecutive screenshot pairs. Before invoking the judge model, we remove pairs with no meaningful visual change using a lightweight image-difference check. Each screenshot is resized to 320 × 200 and converted to grayscale, and we compute three metrics between adjacent frames: RMSE, perceptual hash distance, and the fraction of changed pixels. A pair is forwarded to the LLM judge if at least one threshold is exceeded (RMSE ≥ 0.03, pHash ≥ 5.0, or changed-pixel fraction ≥ 0.01). This ﬁltering step reduces unnecessary model calls and focuses the judge on visually informative transitions. Stage 1: Pairwise Trajectory Parsing For each retained screenshot pair, the judge receives the task instruction, the success condition when available, the action executed between the two frames, the previous screenshot, the current screenshot, and a compact running summary of the visible state accumulated from earlier judged steps. We use gpt-4o-mini for this step because the task is local: it only requires identifying visible changes between adjacent screenshots and updating a concise state summary. The model is instructed to rely on visual evidence ﬁrst and to use the action text only as supporting context. It outputs a structured record containing: (i) the visible action, (ii) a short summary of the visible change, (iii) a change type, (iv) task relevance, (v) whether the step represents positive, neutral, or negative progress, (vi) a conﬁdence label, (vii) an uncertainty note, and (viii) an updated visible-state summary. Stage 2: Final Outcome Judgment After processing the trajectory, we compress the Stage 1 outputs into a sequence of trajectory events and combine them with the ﬁnal screenshot. The ﬁnal judge also receives the task instruction, the success condition, the agent’s ﬁnal textual answer, and, when available, auxiliary reference signals such as llm_judge_gt and a binary objective evaluation result. We use gpt-5.1 for this step because it requires integrating evidence across the trajectory and assigning a calibrated ﬁnal score. The model outputs a score from 1 to 5, where 1 denotes no meaningful visible progress and 5 denotes clearly successful completion with the required details present and no important visible errors or redundant artifacts. The judge is instructed to be conservative: the score must be grounded primarily in the visible ﬁnal state, while intermediate trajectory events and textual signals serve only as supporting evidence. C.2

LLM-as-a-Judge Prompts

C.2.1

Stage 1 System Prompt

You are a careful visual trajectory parser for computer-use agent benchmarks.

Your task is to compare TWO consecutive screenshots from an agent trajectory and ,→ determine what visibly changed in the interface. You must be conservative and evidence-based: - Only report changes that are visually supported by the screenshots. - Do not assume hidden state changes. - Do not infer success from the action alone. - If the change is ambiguous, say so. - Prefer "unknown" or "no clear change" over guessing. You are given: 1. The task instruction 2. The action taken between the screenshots 3. The previous screenshot 4. The current screenshot 5. Optionally, a compact prior visible-state summary from earlier confirmed steps Your goals: - Identify the visible UI change between the screenshots - Judge whether that change is relevant to task completion - Judge whether the step made positive progress, no progress, or negative progress - Update the visible-state summary only when the evidence is sufficient - Keep the visible-state summary compact and cumulative, focused only on durable facts ,→ that are still visible - If the evidence is too weak to update the summary, carry the prior summary forward ,→ unchanged Important: - The prior visible-state summary is only contextual memory, not ground truth. - If the screenshots contradict the prior summary, trust the screenshots. - Never force consistency with prior results. - Count something as completed only if it is visibly supported. Return valid JSON only.

C.2.2

Stage 1 Task Prompt

Task instruction: {TASK_PROMPT} Action performed between screenshots: {ACTION_TEXT} Prior visible-state summary: {PRIOR_STATE_SUMMARY_JSON} Please compare the screenshots and return a strict JSON object with this schema: { "visible_action": "<short textual description of the actual action if visually ,→ supported or consistent, otherwise 'unknown'>", "visible_change_type": "navigation | ui_interaction | content_update | object_added | ,→ object_removed | selection_changed | text_changed | dialog_opened | dialog_closed | ,→ error | no_clear_change | other", "task_relevance": "high | medium | low", "progress": "positive | neutral | negative", "confidence": "high | medium | low", "uncertainty_note": "<brief note if ambiguous, otherwise empty string>", "updated_visible_state_summary": "<compact visible-state summary to carry into later ,→ steps; if no reliable update is possible, repeat the prior summary>" } Rules: - Base the answer on visual evidence from the screenshots first, action text second. - If the action suggests something happened but the screenshots do not confirm it, say ,→ "no_clear_change" or mark confidence low. - If the screenshots show a likely change but the exact object is unclear, describe it ,→ conservatively.

- Do not claim task success. - Keep the updated visible-state summary short and durable. Include only facts still ,→ visible after the current screenshot. - Do not output markdown.

C.2.3

Stage 2 System Prompt

You are a careful final-outcome judge for computer-use agent benchmarks. Your task is to judge how much of the task was visibly completed by the agent at the end ,→ of the trajectory. You must be conservative, thorough, and evidence-based: - Judge only from the provided screenshot-based events, the final screenshot, and the ,→ final agent answer. - Do not assume hidden state changes unless they are supported by visible evidence. - Do not treat the final answer as ground truth. - If the final answer claims success but the screenshots do not support it, score based ,→ on the screenshots. - If the evidence is ambiguous, prefer the lower score. - Inspect fine-grained visible details that matter to task completion. - Penalize clearly wrong, extra, duplicated, or redundant visible elements when the task ,→ requires a precise final state. - Penalize missing required details even when the overall page or app looks close. You are given: 1. The task instruction 2. The success condition 4. An objective evaluation result (`OBJECTIVE EVALUATION RESULT`) with binary value `0` ,→ or `1` 5. A sequence of screenshot-based events extracted from the trajectory 6. The final screenshot 7. The final agent answer Scoring rubric: - 5 = Full success clearly visible, including essential details; no meaningful visible ,→ mistakes or redundant artifacts remain - 4 = Near success; most required state is achieved, but a minor non-essential issue ,→ remains - 3 = Important partial progress, but at least one major requirement is missing, wrong, ,→ duplicated, or visibly inconsistent - 2 = Limited progress; only a small portion of the task is visibly completed, or the ,→ final state contains major visible problems - 1 = No meaningful visible progress at all Important: - Score visible task completion, not effort. - Use the trajectory events as supporting context, but trust direct visual evidence from ,→ the screenshots most. - Treat `OBJECTIVE EVALUATION RESULT` as supporting context. A value of `0` does not ,→ erase clearly visible success. - If you cannot verify something - lean towards a `OBJECTIVE EVALUATION RESULT`. If it is ,→ `0`, be skeptical of claimed success. If it is `1`, assume success. - However, if `OBJECTIVE EVALUATION RESULT` is provided and equals `0`, you must not ,→ assign score `5`. - A correct-looking final answer without visible support should not receive a high score. - A visibly successful final state can still score highly even if the final answer is ,→ weak or incomplete. - High scores require correct details, not just approximate intent. - If extra uploaded items, duplicate objects, wrong selections, leftover dialogs, ,→ incorrect quantities, or other redundant/wrong visible artifacts remain, lower the ,→ score accordingly. Return valid JSON only.

C.2.4

Stage 2 Task Prompt

Task instruction: {TASK_PROMPT} An OBJECTIVE EVALUATION RESULT (binary 0 or 1): {OBJECTIVE_EVALUATION_RESULT} Sequence of screenshot-based events extracted from the trajectory: {TRAJECTORY_EVENTS_JSON} Final agent answer: {FINAL_AGENT_ANSWER} Please inspect the final screenshot and judge partial success. Return a strict JSON object with this schema: { "score": <k>, "reason": "<brief explanation>" } Rules: - Base the judgment on visible evidence from the trajectory events and the final ,→ screenshot first. - Be strict about required details in the final visible state. - Penalize clearly redundant, duplicated, extra, or wrong visible elements when they ,→ conflict with the intended final result. - Use `OBJECTIVE EVALUATION RESULT` only as supporting context, not as proof by itself. - If `OBJECTIVE EVALUATION RESULT` is provided and equals `0`, do not assign score `5`. - Use the final agent answer only as supporting context, not as proof of success. - If the evidence supports only partial completion, use the lower partial score that best ,→ matches the rubric. - If the trajectory shows useful progress but the final screenshot does not preserve it, ,→ score the visible final state plus clearly supported trajectory progress ,→ conservatively. - In the reason, mention the most important satisfied requirement and the most important ,→ missing, wrong, or redundant detail when applicable. - Keep the reason short, concrete, and tied to the visible evidence. - Do not output markdown.

Error Analysis

Beyond the aggregate numbers, we also went through a sample of failed trajectories by hand to see what was actually going wrong. A few patterns kept showing up across almost all agents, and we group them into four categories below. Agents Do Not Know When To Stop A common failure mode we observed is that agents struggle to correctly judge when a task is actually done. On one side, we saw cases where the agent completes the task correctly, but then keeps going and ends up corrupting its own result. For example, on a 3D Editor task, an agent might set an object’s position to the requested coordinates and at that point the task is essentially done but instead of stopping, it continues to "check" the change by clicking around, dragging the object, or modifying some unrelated property, and the ﬁnal state no longer matches the ground truth. On the other side, we also saw the opposite problem of early stopping, where the agent declares the task ﬁnished after only completing a part of the required steps, leaving the ﬁnal state clearly incomplete. Both behaviours happen across most models we tested, but the issue is especially noticeable with GPT-based agents: they often declare in their textual output that the task is ﬁnished, while the actual ﬁnal state is still far from satisfying the requirements. So there is a real mismatch between what the agent thinks it did and what actually ended up on the screen. Grounding Failures A second big source of errors is grounding, i.e. mapping a high-level intent like "click the Map node" to the correct low-level action on the page. We observed agents clicking close to the right element but missing it, selecting a visually similar but functionally different element (like a neighbouring toolbar button), or typing the correct value into the wrong input ﬁeld. What makes grounding errors especially bad is that the agent itself usually does not realise anything went wrong the action executes ﬁne from the browser’s point of view, so the agent assumes everything is on track and keeps building on a state that is already broken. Instruction-following Issues A third pattern, which affects both open- and closed-source models, is that agents do not follow the execution-loop instructions of the prompt. Our prompt template asks the agent to emit one action at a time and wait for the next observation, but several models simply do not respect this. Llama-4-Maverick, for instance, often emits its entire plan as one big block of text and then ends the trajectory without actually executing any browser action, which essentially gives zero progress regardless of how reasonable the plan looked. Closed-source models do milder versions of the same thing: sometimes they batch multiple actions into one step, or drop the structured action format and answer in free-form prose. These are not really perception or reasoning failures they are protocol failures but they still cap how well an agent can do, no matter how strong the underlying model is. Open-source Models Failure Finally, the near-zero FSR of all open-source models is not just about model size. Most of them produced trajectories that were syntactically broken, looped on the same action over and over, or gave up after only a few steps. Our guess is that this is mostly because these models were probably never trained on agentic, multi-step browser-control data their ﬁne-tuning seems to be focused on single-turn instruction following and tool use in static settings, rather than long-horizon interaction with a stateful environment. Closing this gap likely needs not only bigger models but also targeted post-training on agent trajectories.

Tailored Evaluation

Each application domain uses a dedicated objective evaluator. A test case is scored 1 (pass) only when all checked properties match; partial credit is not awarded. Video Editor The evaluator decomposes both the ground-truth and predicted timelines into individual media blocks and matches them by content: media name, type, and timing. Matching is order-independent—blocks are sorted and paired globally rather than by position in the track list. Each matched block must have the correct start time, duration, and trim offsets within ±1 s. Track placement is only enforced for text blocks that are layered on top of a video clip; standalone text is considered track-agnostic. For scenarios that additionally require visual effects (light adjustment or fade-out), the effect type, timing, and parameters are veriﬁed as extra constraints. Workﬂow Builder The evaluator compares the node-and-edge structure of the exported workﬂow. Because agents assign their own node identiﬁers, nodes are matched by type and conﬁguration payload using a greedy minimum-cost assignment. List-valued ﬁelds such as switch cases are 26

Figure 8. Examples of tailored evaluators across the ﬁve environments. The evaluator compares agent outputs against ground-truth JSON using application-speciﬁc matching rules: set equality and property matching for Flight Analysis, tolerance-based timestamp and colour matching for Video Editor, UUID remapping and edge matching for Workﬂow Builder, tolerance and symmetry-aware matching for 3D Modeller, and truth-table equivalence for Circuit Designer.

compared order-independently. Edges are then veriﬁed using the resulting ground-truth-to-predicted ID mapping: an edge passes if the correct (source, target, output handle) triple exists in the predicted graph. Any extra or missing nodes or edges cause the test to fail. 3D Modeller Objects are matched by content rather than by name, again using greedy minimumcost assignment. For each matched pair, object type and wireframe ﬂag are checked exactly; position, rotation, and scale must agree within ±0.15; color and opacity are compared exactly. A key special case: Y-axis rotation is not checked for rotationally symmetric shapes (sphere, cylinder, cone), since any rotation about their symmetry axis produces an identical visual result. Scene-level settings— background colour, ﬁeld of view, and grid/axes visibility—are veriﬁed separately. Flight Analyser The agent’s answer is compared ﬁeld-by-ﬁeld against the ground truth. For tasks involving multiple aircraft, the aircraft objects are matched order-independently (all permutations are tried). All ﬁelds must match exactly (case-insensitive); time-of-day values are given a ±1 minute tolerance to account for minor display rounding. Circuit Designer For tasks that require building a logic circuit, correctness is determined by truthtable equivalence: both the ground-truth and predicted circuits are simulated over all input combinations, and the test passes only if their outputs agree on every row. For tasks that require reading 27

analogue measurements, the agent’s reported values are compared against the ground truth with a 5 % relative tolerance.

Task Format

F.1 Prompt Template To enable standardised, fair comparisons across various applications, we provide a uniﬁed prompt template for our tasks. You are interacting with a web-based application. # APPLICATION BACKGROUND ## Application Overview [APPLICATION_NAME] is a web application used for [PRIMARY PURPOSE]. The main workspace/interface displays [MAIN UI SURFACE], where users can [CORE USER ,→ ACTIONS]. ## Features & Capabilities - [FEATURE 1]: [DESCRIPTION OF FEATURE 1] - [FEATURE 2]: [DESCRIPTION OF FEATURE 2] - [FEATURE 3]: [DESCRIPTION OF FEATURE 3] - [CONTINUE AS NEEDED] - **General Rule:** Follow the task statement exactly. Only perform actions necessary to ,→ complete the requested task. Do not make unrelated changes. ## Environment Architecture & Interaction Paradigms To successfully complete tasks in this environment, adhere to the following UI behaviors ,→ and state mechanics: - [UI MECHANIC 1]: [DESCRIPTION OF UI MECHANIC 1] - [UI MECHANIC 2]: [DESCRIPTION OF UI MECHANIC 2] - [UI MECHANIC 3]: [DESCRIPTION OF UI MECHANIC 3] - [CONTINUE AS NEEDED]

# TASK # [TASK_TITLE] ## GOAL [HIGH-LEVEL OBJECTIVE] ## STEPS 1. [STEP 1] 2. [STEP 2] 3. [STEP 3] 4. [CONTINUE AS NEEDED] # RESULT FORMAT ```json {"[OUTPUT_KEY_1]": "[FINAL_VALUE_1]", "[OUTPUT_KEY_2]": "[FINAL_VALUE_2]", "..."} ```

F.2 Per-Application Background Blocks F.2.1 Video Editor # APPLICATION BACKGROUND ## Application Overview Video Editor is a web application used for arranging and editing video, audio, and text.

The main workspace/interface displays a media panel on the left, a preview player in the ,→ center, and a timeline at the bottom, where You can drag, drop, cut, and enhance ,→ media clips to assemble a finished project. ## Features & Capabilities - **Media Management**: You can select pre-loaded media from the "Sample media" dropdown. - **Timeline Editing**: You can drag media to the timeline and use tools to Delete, ,→ Split, and Duplicate clips. A snapping feature helps align items seamlessly. - **Text & Effects**: You can insert text blocks using the "T" tool or add and adjust ,→ visual effects (like Light Adjustment) using the "Add Effects" ("+" icon) and "Tweak ,→ Selected Effect" features. - **Playback & View Control**: You can preview their edits, step through frames, and ,→ manipulate their view of the timeline using Zoom In, Zoom Out, and Fit To Screen ,→ tools (button on the right side above the timeline). - **Exporting**: Once a project is finished, You can render and download the final video ,→ file. - **General Rule:** Follow the task statement exactly. Only perform actions necessary to ,→ complete the requested task. Do not make unrelated changes. ## Environment Architecture & Interaction Paradigms To successfully complete tasks in this environment, adhere to the following UI behaviors ,→ and state mechanics: - **Drag-and-Drop Workflow**: Media items must first be loaded into the left panel and ,→ then physically dragged onto the bottom timeline area to be included in the edit. - **Playhead Dependency**: Editing actions, specifically the "Split" tool, execute ,→ precisely at the current position of the playhead indicator on the timeline. Ensure ,→ the playhead is scrubbed to the correct timestamp before making a cut. - **Selection State**: Clips and effect layers on the timeline must be explicitly clicked ,→ and highlighted before applying contextual actions like "Delete" or "Tweak Selected ,→ Effect". - **Modal Interactions**: Adjusting effects opens a secondary modal window ("Tweak ,→ effect") containing sliders and save/cancel actions that must be completed before ,→ returning to the main workspace.

F.2.2 Workﬂow Builder # APPLICATION BACKGROUND ## Application Overview Workflow Builder is a web application used for designing automation workflows and ,→ API-driven agents. The main workspace/interface displays a visual node-based canvas, where users can drag, ,→ connect, and configure various functional nodes to build data pipelines and resilient ,→ agent behaviors. ## Features & Capabilities - **Workflow Hub**: A central dashboard allowing users to create new agent workflows from ,→ scratch, import/paste JSON recipes, or utilize pre-built quick start templates (e.g., ,→ "API Data Pipeline", "Enterprise Data Pipeline"). - **Node Library**: A searchable, categorized left-hand sidebar containing functional ,→ blocks such as Triggers, Flow control & branching, Data shaping, APIs & connectivity, ,→ and Security. - **Node Properties Editor**: A contextual right-hand sidebar that dynamically populates ,→ with a selected node's specific settings, allowing for parameter adjustments and ,→ advanced JSON configuration. - **General Rule:** Follow the task statement exactly. Only perform actions necessary to ,→ complete the requested task. Do not make unrelated changes. ## Environment Architecture & Interaction Paradigms To successfully complete tasks in this environment, adhere to the following UI behaviors ,→ and state mechanics:

- **Click-based Instantiation**: New workflow steps are added by searching for the ,→ desired node in the left-hand "Nodes" panel and clicking on it - this makes node ,→ appear directly on the canvas. - **Edge Routing (Connections)**: The flow of execution is dictated by edges (lines). You ,→ can connect nodes by clicking and dragging from the output port (right side) of a ,→ preceding node to the input port (left side) of a subsequent node. - **Edge Deletion**: To break a connection, you must select the existing edge on the ,→ canvas to open its menu in the right-hand panel, then click the "Delete edge" button.

F.2.3 3D Modeller # APPLICATION BACKGROUND ## Application Overview 3D Modeller is a web application used for viewing and manipulating 3D scenes, ,→ specifically designed for completing objective-based tasks. The main workspace/interface displays a central 3D viewport, a scene hierarchy panel, an ,→ inspector panel, and a task prompt sidebar, where users can select 3D objects, modify ,→ their transformations and appearance, and follow specific instructional steps. ## Features & Capabilities - **Scene Management Dashboard:** A home screen allowing users to browse, open, and ,→ manage various pre-configured 3D environments (e.g., "Four Objects Scene", "Basic ,→ Shapes"). - **Scene Hierarchy Panel:** A left-sided menu that lists all active objects within the ,→ current 3D space, providing a straightforward way to locate and select specific ,→ items. - **Inspector Panel:** A detailed right-sided menu that displays the properties of the ,→ currently selected object. Users can modify the object's `Name`, `Transform` ,→ attributes (Position, Rotation, Scale across X, Y, and Z axes), and `Appearance` ,→ (Color, Opacity, Wireframe toggle). - **Precise Color Picker:** An appearance tool allowing users to alter object colors ,→ using either precise HEX codes or specific RGB (Red, Green, Blue) integer values ,→ ranging from 0-255. - **General Rule:** Follow the task statement exactly. Only perform actions necessary to ,→ complete the requested task. Do not make unrelated changes. ## Environment Architecture & Interaction Paradigms To successfully complete tasks in this environment, adhere to the following UI behaviors ,→ and state mechanics: - **Selection-Dependent Editing:** The Inspector panel remains blank ("No object ,→ selected") until an object is actively clicked either in the 3D viewport or from the ,→ Scene list. Selecting a new object immediately updates the Inspector to reflect that ,→ specific object's current state. - **Direct Input Manipulation:** While objects can visually be seen in the viewport, ,→ structural and aesthetic changes (like moving an object to an exact coordinate or ,→ setting a specific color) are achieved by typing specific numeric values directly ,→ into the input fields within the Inspector panel.

F.2.4 Flight Analyser # APPLICATION BACKGROUND ## Application Overview Flight Analyser is a web application used for tracking, visualizing, and replaying ,→ aircraft flight data. The main workspace/interface displays a map populated with aircraft icons alongside a ,→ task panel, where users can search for specific flights, adjust historical playback ,→ times, view flight trajectories, and extract detailed telemetry. ## Features & Capabilities

- **Temporal Navigation**: Adjust the temporal state of the map using the playback ,→ controls and timeline slider located at the top center of the screen to set exact UTC ,→ times. - **Flight Search**: Locate specific aircraft by typing their callsign into the Search ,→ bar situated in the top left corner. - **Flight Details View**: Selecting an aircraft highlights its flight path and opens a ,→ detailed panel overlaying the map, detailing its route, origin/destination, current ,→ status, and specific Flight Data (Altitude, Geo Altitude, Ground Speed, Vertical ,→ Rate). - **Radius Tool**: Draw or clear dashed boundaries around target airports using the "Draw ,→ Radius Circle" tool (accessed via the target icon next to the pause button). - **General Rule:** Follow the task statement exactly. Only perform actions necessary to ,→ complete the requested task. Do not make unrelated changes. The benchmark dataset is derived from the publicly-released OpenSky Network historical ,→ state-vector samples \cite{schafer2014opensky}, specifically a 12-hour window ,→ covering 2022-06-27 06:0018:00 UTC retrieved from ,→ s3.opensky-network.org/data-samples/states/. Raw ~1 Hz observations are downsampled ,→ to 60 s buckets by retaining the most recent state per aircraft per bucket, yielding ,→ 720 timesteps over 12 hours of global air traffic. The full 12-hour window is exposed ,→ by the replay API and the UI scrubbable timeline; auto-play defaults to the final 2 ,→ hours (16:0018:00 UTC) so that users immediately see aircraft with substantial trail ,→ history. ## Environment Architecture & Interaction Paradigms To successfully complete tasks in this environment, adhere to the following UI behaviors ,→ and state mechanics: - **Sequential Data Retrieval**: The interface displays detailed data for one flight at a ,→ time. - **Panel Scrolling**: Specific telemetry metrics are located under the "Flight Data" ,→ section of the details panel. You may need to scroll down within the panel to locate ,→ fields like Geo Altitude or Ground Speed.

F.2.5 Circuit Designer The Circuit application is adapted from CircuitJS1 [31], an open-source browser-based circuit simulator ported from Paul Falstad’s original Java applet by Iain Sharp. # APPLICATION BACKGROUND ## Application Overview Circuit Designer is a web application used for designing and simulating analog and ,→ digital circuits. The main workspace displays a circuit canvas, where users can place components, wire them ,→ together, and observe live simulation behavior. ## Features & Capabilities - **Component Placement:** Add parts from the **Draw** menu, top search button, or by ,→ right-clicking empty canvas space. - **Circuit Wiring:** Connect terminals by clicking and dragging between component ,→ endpoints. Crossing wires do not connect unless joined at endpoints. - **Component Editing:** Right-click a placed component to edit values, duplicate, or ,→ delete it. - **Measurement:** Hover over a component to view live values in the bottom-right ,→ information panel. - **General Rule:** Follow the task statement exactly. Only perform actions necessary to ,→ complete the requested task. Do not make unrelated changes. ## Environment Architecture & Interaction Paradigms To successfully complete tasks in this environment, adhere to the following UI behaviors ,→ and state mechanics:

- **Right-Click Menus:** Most components are added through nested right-click menus on ,→ empty canvas space. - **Property Dialogs:** Values such as resistance or voltage are changed through popup ,→ dialogs after selecting **Edit**. - **Drag Wiring:** Connections are made by dragging from one terminal to another. - **Logic Input Placement:** Draw logic inputs from right to left for proper orientation. - **Live Updates:** Simulation runs continuously, so allow time for values to stabilize ,→ before measuring.

GauntletBench Task Illustrations

This appendix lists all 100 tasks in GauntletBench across our ﬁve environments. Each table contains No. indicating the task identiﬁer, Task, Ground Truth, and Initial State, where applicable. indicates easy testcases; Difﬁculty: hard testcases. G.1

indicates medium-difﬁculty testcases;

indicates

Video Editor

No.

Task

# Trim a Video to First 5 Seconds ## GOAL Add a video and trim it so that only the first 5 seconds remain. ## STEPS 1. Open the "Sample Media" dropdown and select "White Candle" and add it to timeline. 2. Trim the clip so that only the first 5 seconds remain. 3. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Apply Light Correction and Boost Contrast ## GOAL Add a video, apply light correction, and increase only the contrast. ## STEPS 1. Open the "Sample Media" dropdown, select "Flower Video" and add it to the timeline. 2. Apply light correction to the entire clip by maxing out only the contrast value. 3. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Apply Fade-Out Transition ## GOAL Add a video and apply a fade-out transition to its last few seconds. ## STEPS 1. Open the "Sample Media" dropdown, select "Flower Video" and add it to the timeline. 2. Apply a fade-out transition targeting the last 3 seconds of the clip. 3. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Trim a Video to a Specific Portion ## GOAL Add a video and trim it to keep only the segment from 4s to 11s. ## STEPS 1. Open the "Sample Media" dropdown, add "White Candle" video and drag to the timeline. 2. Trim the video so that only the 4.00-11.00 seconds chunk remain. 3. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Add Background Audio to End of Video ## GOAL Add a video and an audio clip, positioning the audio as background for the end of the video. ## STEPS 1. Open the "Sample Media" dropdown, add "Flower Video" and audio clip to the timeline. 2. Position the audio clip so that it plays as background audio during the end portion of the video clip. 3. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Combine Three Videos in Sequence ## GOAL Place three videos on the timeline in a specific order and export. ## STEPS 1. Open the "Sample Media" dropdown and add "Tuning a Radio", "White Candle" and "Flower Video". 2. Drag all the videos to timeline and place them in same order. 3. Trim ends of all videos so length of each clip matches length of "Flower Video" clip. 4. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# The Quote ## GOAL Lift a middle segment out of a video, play it on its own first, then play the full original video. ## STEPS 1. Open the "Sample Media" dropdown and add "Tuning a Radio". 2. On the timeline, first play only seconds 3 through 6 of the clip, then immediately after play the full original clip from start to end. 3. Add a 1-second text block reading "Earlier..." with white text on black background, placed immediately before the full playthrough begins. 4. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Split Video and Insert Another in the Middle ## GOAL Split a video at its midpoint and insert a second video between the two halves. ## STEPS 1. Open the "Sample Media" dropdown and select "Flower Video". 2. Drag "Flower Video" to the timeline. 3. Split the clip at its midpoint. 4. Open the "Sample Media" dropdown and select "Tuning a Radio". 5. Drag "Tuning a Radio" to the timeline. 6. Insert "Tuning a Radio" between the two halves of "Flower Video". 7. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Reverse Order, Light Correction, and Fade-Out ## GOAL Arrange two videos in reverse order, apply light correction to a portion of the first, and apply a fade-out to the second. ## STEPS 1. Open the "Sample Media" dropdown and add "Flower Video". 2. Open the "Sample Media" dropdown and add "Tuning a Radio". 3. Drag "Tuning a Radio" to the timeline first. 4. Drag "Flower Video" to the timeline, placing it right after "Tuning a Radio". 5. Select "Tuning a Radio" and apply light correction to the first 4 seconds of the clip: max out contrast only. 6. Select "Flower Video" and apply a fade-out effect with ease-out easing to the last 3 seconds of the clip. 7. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Symmetric Audio Bridge ## GOAL Place an audio clip so its midpoint aligns with the transition between two videos, then trim both videos so the audio has equal silent runway on both sides. ## STEPS 1. Open the "Sample Media" dropdown and add "Flower Video", "Tuning a Radio" and audio clip. 2. Drag "Flower Video" and "Tuning a Radio" to the timeline back-to-back in that order. 3. Place the audio clip on a separate track so that the middle of the audio clip aligns exactly with the cut between the two videos. 4. Trim "Flower Video" from its start and "Tuning a Radio" from its end so that the audio clip begins exactly 1 second after the video timeline starts and ends exactly 1 second before the timeline ends. 5. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Split a Video, Swap Halves, and Insert a Midpoint Title ## GOAL Split a video at its midpoint, swap the two halves, and insert a styled text block between them. ## STEPS 1. Open the "Sample Media" dropdown and select "Tuning a Radio". 2. Drag "Tuning a Radio" to the timeline. 3. Split the clip at its midpoint. 4. Move the second half to the beginning of the timeline. 5. Add a text block with the text "Midpoint". 6. Set the font color to white. 7. Set the background color to black. 8. Set the text block duration to 1 second. 9. Place the text block immediately after the second half. 10. Move the first half to immediately after the text block. 11. Verify the final timeline order is: second half of "Tuning a Radio" -> "Midpoint" text block -> first half of "Tuning a Radio". 12. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Trim and Duplicate a Video ## GOAL Trim a video, then duplicate several times. ## STEPS 1. Open the "Sample Media" dropdown, add "Flower Video" and drag it to timeline. 2. Trim the clip so that 4 first seconds remain, duplicate the video 4 times, but trim even numbered video to 2 last seconds. 4. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Three Videos with Empty Gaps Between Them ## GOAL Arrange three videos in sequence with specific empty gaps between them. ## STEPS 1. Open the "Sample Media" dropdown and add "Flower Video", "White Candle" and "Tuning a Radio" videos. 2. Drag the videos to timeline in the same order. 3. Place the videos with empty gaps of 1 second and 2 seconds between them. 4. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Different Light Corrections on Three Videos ## GOAL Arrange three videos in sequence and apply a different light correction to each. ## STEPS 1. Open the "Sample Media" dropdown and add "Flower Video", "White Candle" and "Tuning a Radio". 2. Drag the video to timeline in asked above order. 3. Apply light correction one each video clip: max out contrast for first, brightness for second and saturation for last video respectively. 4. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Two Videos with Audio at Transition ## GOAL Arrange two videos in sequence and place an audio clip so it spans the transition between them. ## STEPS 1. Open the "Sample Media" dropdown and add "Flower Video", "Tuning a Radio" and audio clip. 2. Drag the videos to the timeline placing "Tuning a Radio" right after "Flower Video". 3. Drag the audio clip to the timeline. 4. Position the audio clip so that it spans the transition point between the two videos (equally overlapping the end of the first and the beginning of the second). 5. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Three Colored Text Blocks ## GOAL Create an intro video with different color scenes. ## STEPS 1. Add a half-second text block with white "Second!" text. 2. Duplicate the text block two times, setting new blocks texts, so that the timeline reads: "Second!" -> "Third!" -> "First!", resulting in three consecutive blocks on the timeline. 3. Change text blocks background colors to be "yellow", "red" and "green" accordingly to their timeline order. 4. Change durations of text blocks, so that each one is 2 times longer than previous one, leaving first text block duration unchanged. 5. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Trimmed Video with Closing Titles and Fade-Out ## GOAL Add a video, trim it, append a closing title card, and apply a fade-out to the title. ## STEPS 1. Open the "Sample Media" dropdown and select "White Candle". 2. Drag "White Candle" to the timeline. 3. Trim the end of the clip so that only the first 10 seconds remain. 4. Add a text block with the text "The End", black font color, and white background. 5. Set the text block duration to 2 seconds. 6. Place the text block on the timeline immediately after the trimmed video. 7. Select the text block and apply a fade-out effect to it. 8. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Overlay Text and Audio on a Video ## GOAL Add a video to the timeline, overlay a text block on the first 2 seconds, and add an audio clip playing in parallel for the same 2 seconds. ## STEPS 1. Open the "Sample Media" dropdown and add "Flower Video" and audio clip. 2. Drag "Flower Video" to the timeline. 3. Add a 2-second "Intro" text block with font size of 72 on a track above the video track so it overlays the first 2 seconds of "Flower Video". 4. Drag the audio clip to the timeline. 5. Position the audio clip so it starts at the beginning of the timeline and plays for the first 2 seconds, in parallel with the video and the text overlay. 6. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Alternating Split Segments with Per-Segment Corrections ## GOAL Split two videos into quarters, interleave them on the timeline, and apply a different light correction to each odd-indexed segment. ## STEPS 1. Open the "Sample Media" dropdown and add "Tuning a Radio" and "White Candle". 2. Drag "Tuning a Radio" to the timeline and split it into 3 equal segments (T1, T2, T3). 3. Drag "White Candle" to the timeline after "Flower Video" and split it into 3 equal segments (W1, W2, W3). 4. Reorder the segments on the timeline so the final order is: T1 -> W1 -> T2 -> W2 -> T3 -> W3. 5. Apply light correction to first two odd-positioned segment on the timeline: max out contrast for first segment, max out brightness for last segment. 6. Trim each of the segments so that all segments on the timeline have the same duration, equal to the shortest segment. 7. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Act Titles Before Each Video ## GOAL Arrange three videos in sequence, each preceded by a titled text block. ## STEPS 1. Open the "Sample Media" dropdown and add "Flower Video". 2. Open the "Sample Media" dropdown and add "White Candle". 3. Open the "Sample Media" dropdown and add "Tuning a Radio". 4. Add a text block with text "Act I", black font color, white background, and 1-second duration. Place it on the timeline. 5. Drag "Flower Video" to the timeline, placing it immediately after the "Act I" text block. 6. Add a text block with text "Act II", black font color, white background, and 1-second duration. Place it on the timeline immediately after "Flower Video". 7. Drag "White Candle" to the timeline, placing it immediately after the "Act II" text block. 8. Add a text block with text "Act III", black font color, white background, and 1-second duration. Place it on the timeline immediately after "White Candle". 9. Drag "Tuning a Radio" to the timeline, placing it immediately after the "Act III" text block. 10. Verify the final timeline order is: "Act I" -> "Flower Video" -> "Act II" -> "White Candle" -> "Act III" -> "Tuning a Radio". 11. Export the result. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

G.2

Workﬂow Builder

No.

Task

# Remove a Node and Reconnect in a Pipeline ## GOAL Remove a node from a multi-step pipeline and reconnect the upstream node to the downstream node. ## STEPS 1. Open the "API Data Pipeline" workflow. 2. Delete the "Sort" node. 3. Reconnect the "Map" node directly to the next downstream node. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

No.

Task

# Create a Simple Three-Node Workflow ## GOAL Create a simple three-node workflow from scratch with a Manual Trigger, a configured HTTP Request, and a No Operation node. ## STEPS 1. Create a new workflow named "Example Todo Fetch". 2. Add these nodes and connect them in sequence: ‘Manual Trigger -> HTTP Request -> No Operation‘. 3. Configure the "HTTP Request" node: - Method: "GET" - URL: ‘https://jsonplaceholder.typicode.com/todos/1‘ # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

# Insert a Wait Node into a Workflow ## GOAL Insert a Wait node with a 1-second delay between two existing nodes in a workflow and reconnect the flow. ## STEPS 1. Open the "Range -> Map -> Table (scaffolding)" workflow. 2. Insert a "Wait" node between the "Map" and "Table Preview" nodes. 3. Configure the delay to "1 second". 4. Reconnect the workflow correctly. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

No.

Task

# Delete a Node and Disable Default Port in Switch ## GOAL Delete a branch target node from a Switch-based pipeline and disable the corresponding default port in the Switch settings. ## STEPS 1. Open the "Data Categorization Pipeline" workflow. 2. Delete one of the "Edit Fields (Set)" node responsible for "Default" branch of a "Switch" node. 3. Disable the "Default" port in the "Switch" node settings. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Add a Node Using Search ## GOAL Use the node search to find and add a Send Email node, configure its subject, and connect it to the end of a pipeline. ## STEPS 1. Open the "API Data Pipeline" workflow. 2. Use the node search box to find "Send Email". 3. Add the "Send Email" node to the canvas after the "Sort" node. 4. Set "Send Email" subject to "Users list". 5. Connect the "Sort" node output to the "Send Email" node input. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

# Insert a Formula Node into a Workflow ## GOAL Insert a Formula node into a workflow, configure it to compute a derived field, and reconnect the data flow. ## STEPS 1. Open the "Range -> Map -> Table (scaffolding)" workflow. 2. Insert a "Formula" node between the "Map" and "Table" nodes. 3. Configure the "Formula" node to name a new field ‘double‘ with value ‘item.value * 2‘. 4. Reconnect the workflow so the final path is: ‘Range -> Map -> Formula -> Table‘. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

No.

Task

# Insert and Configure a Filter Node ## GOAL Add and configure a Filter node in an existing workflow, wiring it into the correct position in the data flow. ## STEPS 1. Open the "Range -> Map -> Table (scaffolding)" workflow. 2. Delete the rightmost "Table" node. 3. Add a "Filter" node and connect "Map" node output to the "Filter" node input . 4. Configure the filter to pass only values where ‘item.value > 10‘. 5. The final path must be: ‘Range -> Map -> Filter‘. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

# Insert a Filter Node into an Existing Pipeline ## GOAL Insert a Filter node between two existing nodes in a pipeline and configure it with a specific condition. ## STEPS 1. Open the "Order Processing Pipeline" workflow. 2. Add a "Filter" node between the "Database Connector" and the "If" node. 3. Configure the filter to pass only orders where ‘item.status === ’completed’‘. 4. Reconnect the workflow so the path through this section is: ‘Database Connector -> Filter -> If‘. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

No.

Task

# Rewire a Branch Input to a Different Node ## GOAL Rewire a branch input on a downstream node so it feeds into a different merge point in the pipeline. ## STEPS 1. Open the "Enterprise Data Pipeline" workflow. 2. Find the "Data Table" node that receives inputs from the priority branches. 3. Disconnect the input coming from the lower "Edit Fields (Set)" node. 4. Reconnect that lower "Edit Fields (Set)" node into the "HTTP Request" node located on the right, below "Send Email" node. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

# Add an Side Branch Without Breaking Existing Output ## GOAL Add a side branch that records processed records while keeping the original main pipeline output unchanged. ## STEPS 1. Open the "Order Processing Pipeline" workflow. 2. Find the node immediately before the final "Aggregate" output node. 3. Add a "No Operation" node. 4. Add a "Data Table" node after the "No Operation" node. 5. Connect the node immediately before the final output node to the new "No Operation" node, leaving all other existing connections as is. 6. Connect "No Operation" to the new "Data Table" node. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Split a Pipeline into Two Conditional Branches ## GOAL Modify an existing linear workflow so that active and inactive users are processed through separate branches. ## STEPS 1. Open the "API Data Pipeline" workflow. 2. Insert an "If" node between the "Map" node and the next downstream node. 3. Configure the "If" condition to check: ‘item.active === true‘ 4. Add a new "Send Email" node for the true branch. 5. Set the new "Send Email" subject to: ‘Active user report‘ 6. Connect the workflow so that: - ‘HTTP Request -> Map -> If‘ - The ‘true‘ output of "If" goes to the new "Send Email" node. - The ‘false‘ output of "If" goes to the existing downstream path that originally followed "Map". # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Add a New Output Branch to a Switch Node ## GOAL Add a new output case to a Switch node in a complex pipeline and wire it to an existing downstream node. ## STEPS 1. Open the "Enterprise Data Pipeline" workflow. 2. In the main "Switch" node, add a new case labeled ‘priority_urgent‘, do not include "expr" field. 3. Connect the new ‘priority_urgent‘ output to the same "Edit Fields (Set)" node currently connected to the "default" output. 4. Do not change any other routes or node settings. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

No.

Task

# Rewire Edges in a Pipeline ## GOAL Rewire existing connections in a complex pipeline without adding, deleting, or reconfiguring any nodes. ## STEPS 1. Open the "Enterprise Data Pipeline" workflow. 2. Disconnect the edge from the top-row "No Operation" node to the top-row "Merge" node. 3. Connect that same "No Operation" node to the "Switch" node. 4. Rewire the ‘priority_low‘ output of the "Switch" to the lower "Edit Fields (Set)" node. 5. Do not create, delete, move, or reconfigure any nodes. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

# Insert a Filter and Modify Sort in a Pipeline ## GOAL Make multiple changes to an existing pipeline: delete a trailing Limit node.

insert a Filter, reconfigure a Sort, and

## STEPS 1. Open the "API Data Pipeline" workflow. 2. Insert a "Filter" node between the "Map" and "Sort" nodes. 3. Configure the "Filter" to pass only items where ‘item.email.includes(".org")‘. 4. Change the "Sort" node to use field ‘email‘ with direction ‘desc‘. 5. Delete last "Limit" node. 6. The final path must be: ‘HTTP Request -> Map -> Filter -> Sort‘. 7. Do not add any other nodes. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

No.

Task

# Add a New Priority Branch to a Switch Node ## GOAL Extend a Switch node with a new priority branch, add a dedicated processing node for it, and wire it into the main pipeline. ## STEPS 1. Open the "Enterprise Data Pipeline" workflow. 2. Find the "Switch" node that routes items by priority. 3. Add a new output branch labeled ‘priority_urgent‘, do not include "expr" field. 4. Add a new "Edit Fields (Set)" node for the urgent branch. 5. Connect the ‘priority_urgent‘ output to the new "Edit Fields (Set)" node. 6. Connect the new "Edit Fields (Set)" node back into the "Data Table" node. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

# Create a Simple Loop Workflow ## GOAL Create a new workflow where data passes through a loop before reaching the final output. ## STEPS 1. Create a new workflow named "Simple Loop Pipeline". 2. Add exactly these nodes: - "Manual Trigger" - "Map" - "If" - "Data Table" 3. Configure the "Map" node to add or update the field: ‘counter = (item.counter || 0) + 1‘ 4. Configure the "If" node with the condition: ‘item.counter < 3‘ 5. Connect the workflow as follows: - ‘Manual Trigger -> Map -> If‘ - The ‘true‘ output of "If" loops back to the "Map" node. - The ‘false‘ output of "If" goes to "Data Table". # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Create a Branching Workflow with Merge ## GOAL Build a branching workflow with an If condition that routes data to two different nodes, then merges the branches back together. ## STEPS 1. Create a new workflow named "ICML Project". 2. Add these nodes and connect them: ‘Manual Trigger -> Database Connector -> If‘. 3. Add two more nodes: "Crypto" and "Guardrails". 4. From the "If" node: - Connect the "true" output to a "Crypto" node. - Connect the "false" output to a "Guardrails" node. 5. Connect both branches into a "Merge" node: ‘Crypto -> Merge‘ and ‘Guardrails -> Merge‘. 6. Set the "If" condition to check: ‘item.country === "UK"‘. 7. Do not add any extra nodes. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

No.

Task

# Create a Simple Decision Tree Workflow ## GOAL Create a small tree-shaped workflow with 3 If nodes and 4 final outputs. ## STEPS 1. Create a new workflow named "Simple Decision Tree Pipeline". 2. Add: - 1 Manual Trigger - 3 If nodes - 2 Data Table nodes 3. Configure the If nodes: - Root If: ‘item.score >= 50‘ - Upper-branch If: ‘item.score >= 80‘ - Lower-branch If: ‘item.score >= 20‘ 4. Connect: - ‘Manual Trigger -> Root If‘ - Root If ‘true‘ -> Upper-branch If - Root If ‘false‘ -> Lower-branch If - Upper-branch If ‘true‘ -> first Data Table - Upper-branch If ‘false‘ -> second Data Table - Lower-branch If ‘true‘ -> first Data Table - Lower-branch If ‘false‘ -> second Data Table # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Create a Conditional Email Workflow ## GOAL Create a conditional email workflow where the If node routes to two separate Send Email nodes with different subjects. ## STEPS 1. Create a new workflow named "ICML Project". 2. Add these nodes and connect them in sequence: ‘Chat Trigger -> Database Connector -> If‘. 3. Add two "Send Email" nodes. 4. Connect first "Send Email" node only to the "false" output of the "If" node. 5. Connect second "Send Email" node only to the "true" output of the "If" node. 6. Set the "If" condition to check: ‘item.country === "FR"‘. 7. Set the first "Send Email" subject to: ‘ICML Project notification‘. 8. Set the second "Send Email" subject to: ‘Notification de projet ICML‘. 9. Do not add extra nodes. # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

No.

Task

# Create a Multi-Node Workflow with Configuration ## GOAL Create a five-node workflow from scratch with HTTP, mapping, filtering, and email nodes, each configured with specific settings. ## STEPS 1. Create a new workflow named "Newsletter System". 2. Add exactly these 5 nodes and wire them it the following in sequence: ‘Manual Trigger -> HTTP Request -> Map -> Filter -> Send Email‘. 3. Configure the nodes: - "HTTP Request" method: "GET" - "HTTP Request" URL: ‘https://jsonplaceholder.typicode.com/users‘ - "Filter" condition: pass only items where ‘item.address?.city === "Gwenborough"‘ - "Send Email" From: ‘[email protected]‘ - "Send Email" Recipients: ‘[email protected]‘ - "Send Email" Subject: ‘Local Newsletter‘ # RESULT FORMAT “‘json {"answer": "done"} “‘ Ground truth

G.3

3D Modeller

No.

Task

# Double the Cube’s Scale ## GOAL Open a single-object scene, double the cube’s scale on all axes, and change the scene background color. ## STEPS 1. Open the "Single Cube Scene". 2. Double its current scale along all three axes, and set the scene’s background color to white (#ffffff). # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

Ground truth

No.

Task

# Position Elements Relative to Reference ## GOAL Open a scene and reposition two elements at a fixed distance from a reference object along the X-axis, preserving their orientations. ## STEPS 1. Open the "Basic Shape Scene". 2. Use the blue sphere as the reference point. Position the other two elements at a distance of 3 units from the reference along the X-axis, while preserving their original orientations. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

Ground truth

No.

Task

# Place Cubes in All Octants ## GOAL Create a new scene and place one cube in each of the eight 3D octants at symmetric coordinates. ## STEPS 1. Create a new scene named "Octant Cubes". 2. Generate cubes by assigning one default cube to each octant in 3D space, positioned at coordinates with absolute values of (1, 1, 1). Name all the cubes as Cube. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Reorder Elements Along X-Axis ## GOAL Rearrange four objects into a specified order along the positive X-axis with uniform spacing between centers. ## STEPS 1. Open the "Row of Objects Scene". 2. Reorder the elements as sphere, cone, cube, and cylinder along the positive X-axis. Final distance between centers of consecutive objects must be equal to 1. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

No.

Task

Ground truth

# Find and Recolor Nearest Elements ## GOAL Identify the two elements closest to a reference cube and change their color to red. ## STEPS 1. Open the "Geometric Garden". 2. Use the stone block cube as the reference. Identify the two remaining elements whose centers are closest to that of the stone block cube. Finally, change their color to red (#ff0000). # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

No.

Task

Ground truth

# Spheres Touching Cube Faces From Inside ## GOAL Create unit spheres that touch the faces of an imagined cube from inside. ## STEPS 1. Open the "Empty Scene". 2. Imagine a cube of side 6 laying on the XZ plane. 3. Add one unit sphere per cube face. 4. Place each sphere inside the imagined cube so it touches, but does not intersect, its assigned face at the center of that face. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Place a Torus on the Missing Axis ## GOAL Identify the unoccupied axis in the XZ plane and place a new torus there, matching the distance of existing elements from the center. ## STEPS 1. Open the "Four Objects Scene". 2. Add a torus and place it along the only axis in the XZ plane that does not contain any existing element. Set its distance from the central point to be the same as that of the other elements. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

Ground truth

No.

Task

# Adjust Cube (Cuboid) Scale and Stack Cone ## GOAL Modify the cuboids Y-axis scale and reposition it to rest on the XZ plane, then place the cone on top of the cuboid with proper surface contact. ## STEPS 1. Open the Four Objects Scene. 2. Set the cuboids scale along the Y-axis to 20, and adjust its position so that its bottom surface is in contact with the XZ plane. 3. Adjust the position of the cone so that its bottom surface is in contact with the top surface of the cuboid, without overlap. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

Ground truth

No.

Task

# Symmetric Scene Around the Origin Object ## GOAL Make the scene symmetric using the object at the origin as the scene center. ## STEPS 1. Open the "Two Objects Scene". 2. Treat the object located at the origin as the center of the scene. Do not move or modify this center object. 3. Change the existing cube’s color to red (#ff0000). 4. Add exactly one new object. 5. Change the new object’s properties so that the final scene is symmetric around the center object. 6. Do not add any guide objects or extra objects. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

Ground truth

No.

Task

# Complete the Top-Down Plus ## GOAL Look at the scene from above. The four existing objects form an incomplete plus sign one arm of the plus is missing. Add a single cube that visually completes the plus as seen from the +Y (top-down) direction. ## STEPS 1. Open the "Four Objects Scene". 2. From top down view figures form a plus sign without one arm. Identify which arm of the top-down plus shape is missing. 3. Add a new object named "Plug" in the missing arm position so that, viewed from directly above (+Y), the five objects form a symmetric plus sign. The new object’s shape must match the existing object on the opposite arm of the plus, while keeping default rotation and unit scale. 4. Place the "Plug" object on the same Y-axis level as all the other objects and color it purple (#aa00ff). # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

Ground truth

No.

Task

# Fill the Missing End ## GOAL Add one object that visually extends the row pattern, without disturbing the existing row. ## STEPS 1. Open the "Row of Objects Scene". 2. In the default camera, observe the four objects as a left-to-right row of distinct silhouettes. 3. Add exactly one new object named "Row Extension" at the next equally spaced position beyond the current rightmost visible silhouette, continuing the same straight row and center-to-center spacing. 4. The new object’s must: * match the shape of the second (starting from the origin) object in the row, * match the row’s Y and Z levels, use default rotation and unit scale, * have a purple (#aa00ff) color. 5. Preserve every original row object’s position, rotation, scale, color, name, visibility, opacity, and wireframe setting. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

Ground truth

No.

Task

# Minimize Total Exposed Volume ## GOAL Reposition objects in the XZ plane to minimize total exposed volume by maximizing overlap, keeping Y positions and all rotations/scales fixed. ## STEPS 1. Open the "Geometric Garden". 2. Use the stone block as the reference. Adjust the positions of the other objects along the X- and Z-axes while keeping their Y-axis positions fixed, with all rotations and scales unchanged. The objective is to minimize the total exposed volume of all objects, excluding any overlapping volume from the calculation. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

Ground truth

No.

Task

# Create Cube (Cuboid) with Scale Combinations ## GOAL Find all possible integer scale combinations of a cube using the values 1, 2, and 4 on the x, y, and z axes. The three scale values must be assigned in all possible permutations, and each axis may be either positive or negative. ## STEPS 1.

Open the "Empty Scene".

2. Create cubes (cuboids) for all possible combinations of scale values [1, 2, 4] across the x, y, and z axes, allowing each axis to be positive or negative. 3.

Set the position of each cube to (0, 0, 0) and assign all of them the same name, Cube 1.

## RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Match Cone Volume to Cylinder and Stack ## GOAL Adjust a cone’s Y-axis scale to match the cylinder’s volume, place the cone on top of the cylinder, and recolor both to red. ## STEPS 1. Open the Four Objects Scene. 2. Adjust the cone so that its volume matches that of the cylinder by modifying only the scale along the Y-axis. The volume of the cylinder is given by V = πr 2 h, and the volume of the cone is given by V = (1/3)πr 2 h. Determine the required Y-axis scaling factor to achieve equal volumes. Next, position the cone on top of the cylinder without any overlap. Finally, treat the cylinder and cone as a single object and change its color to red (#ff0000). # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

Ground truth

No.

Task

# Create a Minimum-Surface-Area Cube (Cuboid) ## GOAL Create a cube with a volume of 8 and the smallest possible surface area, then set specific position, rotation, color, opacity, and background. ## STEPS 1. Open the Empty Scene. 2. Create a cube (cuboid) with integer side lengths a, b, and c such that the volume constraint abc = 8 is satisfied. The surface area is given by the formula S = 2(ab + bc + ac). Determine the integer dimensions that minimize the surface area, and construct the cuboid accordingly. 3. Set the cuboids position to (1, 1, 1), rotation to (45, 45, 45), color to blue (#0000ff), and opacity to 0.5. 4. Set the background color to white (#ffffff). # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

# Stack Three Cubes into a Larger Cube (Cuboid) ## GOAL Add two cubes (cuboids) to an existing scene so that all three cubes together form a single larger cube stacked along the Y-axis with no overlap. ## STEPS 1. Open the "Single Cube Scene". 2. Add two cubes (cuboids) that, together with the given Cube 1, form a unified larger cube (cuboid) with the same rotation. The bottom cubes (cuboids) should have its lower surface in contact with the XZ plane, and the total height along the Y-axis should be 3. Ensure that there is no overlap among the three individual cubes (cuboids). Name the bottom cube Cube 2 and set its color to blue (#0000ff). Name the top cube Cube 3 and set its color to green (#00ff00). # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

No.

Task

Ground truth

# Iterative Volume-Based Filtering and Scaling ## GOAL Iteratively remove the object with the smallest volume, double the Y-axis scale of the remaining objects, and repeat until only one object remains, then set its color to white. ## STEPS 1. Open the Row of Objects Scene. 2. Compute the volumes of all objects (cuboid, cone, ellipsoid, cylinder). surface area formulas are: * Cube (Cuboid): V = a * b * c * Cylinder: V = pi * rˆ2 * h * Sphere (Ellipsoid): V = 1/6 * pi * a * b * c * Cone: V = 1/3 * pi * rˆ2 * h 3. Identify and delete the object with the smallest volume. 4. Double the Y-axis scale of all remaining objects. 5. Repeat Steps 24 until only one object remains. 6. Set the remaining objects color to white (255, 255, 255). # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

The relevant

No.

Task

# Reorder by Surface Area Plus Volume ## GOAL Sort four elements along the positive X-axis by the sum of their surface area and volume (smallest to largest), preserving the original color order. ## STEPS 1. Open the Row of Objects Scene. 2. Reorder the given elements (cube (cuboid), sphere, cylinder, cone) along the positive X-axis, arranging them from smallest to largest based on the sum of their surface area and volume. The relevant formulas are: * Cube (cuboid): surface area S = 2(ab + bc + ac), volume V = abc * Sphere: surface area S = 4πr 2 , volume V = (4/3)r * Cylinder: surface area S = 2πr 2 + 2rh, volumeV = πr 2 h * Cone: surface area S = πr 2 + rl, volumeV = (1/3)πr 2 h (where l is the slant height) Preserve the original color order while reordering. 3. Distance between centers of consecutive object must me 2. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Initial state

Ground truth

No.

Task

# Build a Ball on Table (Open-Ended) ## GOAL Construct a table from a tabletop cube and four cylinder legs, then place a ball on the tabletop, computing positions so all parts are in contact. ## STEPS 1. Create a new scene named Ball on Table. Then, construct a 3D table and place a ball on the tabletop. The table consists of the following components: a cube named Tabletop, which serves as the reference point, with its position set to (0, 4.2, 0) and scaled to (4, 0.4, 4); four cylinders of equal size named TableLeg, each scaled to (0.5, 4, 0.5); and a sphere named Ball with default settings. These components constitute all the elements required to construct the Ball on Table scene. Your task is to determine their positions (except for the Tabletop, which is fixed as the reference). Each element should be connected to or tangent to the others. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

No.

Task

# Build a 3D Snowman ## GOAL Build a 3D snowman from scratch using multiple primitives (cylinder body, sphere head, torus eyes, cone nose, torus mouth, cube arms) with precise transforms. ## STEPS 1. Create a new scene named "Snowman". 2. Construct a 3D snowman. First, add a cylinder and name it Body. Set its position to (0, 0.5, 0), using the default rotation, scale, color, and opacity. Second, add a sphere named Head and set its position to (0, 1.4, 0). Third, add a torus named Right Eye. Set its position to (0.45, 1.6, 0.2), rotation to (0, 90, 0), scale to (0.2, 0.2, 0.2), and color to black (#000000). Fourth, duplicate the Right Eye and rename it Left Eye. Set its position symmetrically along the Z-axis opposite to the right eye. Fifth, add a cone named Nose. Set its position to (0.5, 1.5, 0), rotation to (0, 0, -90), scale to (0.1, 0.2, 0.1), and color to yellow (#ffff00). Sixth, add another torus named Mouth. Set its position to (0.5, 1.3, 0), rotation to (0, 90, 0), scale to (0.3, 0.1, 0.1), and color to red (#ff0000). Seventh, add a cube named Right Arm. Set its position to (0, 0.5, 0.6), rotation to (-45, 0, 0), and scale to (0.3, 0.8, 0.3). Finally, duplicate the Right Arm and rename it Left Arm. Set its position symmetrically along the Z-axis opposite to the right arm, and apply the opposite rotation along the X-axis. # RESULT FORMAT “‘json {"answer": “‘

"done"}

Ground truth

G.4

Flight Analyser

No.

Task

Ground truth

# Count Visible Aircraft at a Specific Zoom Level

{

## GOAL Count how many aircraft icons are visible on screen after centering on an airport and zooming in.

}

"count": 2

## STEPS 1. Set the playback time to 14:00 UTC. 2. Search for airport ICN to center the map on it. 3. Zoom in exactly 3 steps. 4. Count the total number of aircraft icons currently visible on the screen. # RESULT FORMAT “‘json { "count": } “‘

<N>

{

# Read Telemetry at a Specific Time ## GOAL Look up a flight’s geo altitude and ground speed at a precise moment. ## STEPS 1. Set the playback time to 14:30 UTC. 2. Pause the playback. 3. Search for flight VVC5776 using the search bar. 4. Open its telemetry data. 5. Read and report the exact geo altitude and ground speed values. # RESULT FORMAT “‘json { "geo_altitude": "ground_speed": } “‘

"<value>", "<value>"

"geo_altitude": "30,407 ft", "ground_speed": "386 kts" }

No.

Task

Ground truth

# Count Landings at an Airport Over a Time Window

{

## GOAL Observe an airport and count how many aircraft land during a specified period.

}

"count": 5

## STEPS 1. Set the playback time to 16:14 UTC. 2. Navigate to Budapest Ferenc Liszt Airport (BUD). 3. Observe the airport for 10 minutes of playback time. 4. Count how many new aircraft land at the airport during this period. # RESULT FORMAT “‘json { "count": } “‘

<N>

No.

Task

Ground truth

# Count Aircraft Within a Circle at a Specific Time

{

## GOAL Count how many aircraft are located inside a defined radius around an airport.

}

"count": 4

## STEPS 1. Set the playback time to 11:41 UTC. 2. Search for Thiruvananthapuram International Airport (TRV) to center the map on it. 3. Draw a 450 km radius circle centered on the airport. 4. Count the number of aircraft located within this circle. # RESULT FORMAT “‘json { "count": } “‘

<N>

# Identify the First Departure After a Given Time

{

## GOAL Find the first flight that departs from an airport after a specific time.

}

"callsign": "AAL1535"

## STEPS 1. Set the playback time to 11:30 UTC. 2. Zoom into John F. Kennedy International Airport (JFK) in New York. 3. Observe departures from the airport. 4. Identify the first flight that departs after this time. 5. Report that flight’s callsign. # RESULT FORMAT “‘json { "callsign": } “‘

"<callsign>"

No.

Task

Ground truth

# Determine a Border Crossing Time (Canada to US)

{

## GOAL Track a flight and report the exact time it crosses the CanadaUnited States border.

}

"crossing_time_utc": "16:35 UTC"

## STEPS 1. Set the playback time to 16:00 UTC. 2. Search for flight SAS907 using the search bar. 3. Locate the flight on the map and begin tracking it forward in time. 4. Report the UTC time at which it crosses the CanadaUnited States border. # RESULT FORMAT “‘json { "crossing_time_utc": } “‘

"<HH:MM UTC>"

No.

Task

Ground truth

# Identify the First Channel Crossing (UK to France)

{

## GOAL Find the first flight to cross from England into France over the English Channel after a given time.

}

"callsign": "TOM7KW"

## STEPS 1. Set the playback time to 16:08 UTC. 2. Zoom into the narrowest part of the English Channel. 3. Identify all flights that were over England at 16:08 UTC. 4. Observe which of those flights is the first to cross into France. 5. Report that flight’s callsign. # RESULT FORMAT “‘json { "callsign": } “‘

"<callsign>"

# Find the Last Flight to Leave a Circle

{

## GOAL Determine which flight is the last to exit a radius around a given airport.

}

"callsign": "TVS5AD"

## STEPS 1. Set the playback time to 16:00 UTC. 2. Navigate to Karlovy Vary Airport. 3. Draw a 40 km radius circle around the airport. 4. Identify all flights currently inside the circle. 5. Track them forward in time and determine which flight leaves the circle last. 6. Report that flight’s callsign. # RESULT FORMAT “‘json { "callsign": } “‘

"<callsign>"

No.

Task

Ground truth

# Count Airborne Aircraft Near an Airport

{ "count": 5, "callsigns": [ "CTM3813", "TVF3828", "EJU54MG", "TVF3124", "FMY8055" ]

## GOAL Count how many aircraft within a radius of a given airport have an "Airborne" status. ## STEPS 1. Set the playback time to 16:00 UTC. 2. Search for airport ORY to center the map on it. 3. Zoom in exactly 3 steps. 4. Within a 20 km radius of ORY, open the details of each aircraft in view. 5. Count how many have an "Airborne" status at 16:00 UTC and collect their callsigns. # RESULT FORMAT “‘json { "count": <N>, "callsigns": ["<CS1>", "<CS2>", "..."] } “‘

}

No.

Task

Ground truth

# Check Whether a Flight Path Enters a Circle

{"callsign": N213AL}

## GOAL Determine whether a specific flight’s path ever enters a defined radius around an airport. ## STEPS 1. Set the playback time to 11:15 UTC. 2. Search for Missoula Montana Airport (MSO) to center the map on it. 3. Draw a 500 km radius circle centered on the airport. 4. Search for flight ASA1217 and identify the flight closest to ASA1217 at that moment whose track crosses the 500 km radius circle. 5. Report the callsign of that closest flight. # RESULT FORMAT “‘json { "callsign": } “‘

"<callsign>"

# Detect a Border Crossing (Poland to Germany)

{

## GOAL Find the first aircraft whose track crosses from Poland into Germany after a given time.

}

"callsign": "LOT335"

## STEPS 1. Set the playback time to 16:00 UTC. 2. Navigate to the GermanyPoland border region. 3. Observe aircraft tracks crossing the border. 4. Identify the first aircraft that crosses from Poland into Germany (not the reverse direction) and does not re-enter Polish airspace afterward. 5. Confirm the crossing direction by comparing two successive positions one before and one after the border. 6. Report that aircraft’s callsign. # RESULT FORMAT “‘json { "callsign": } “‘

"<callsign>"

No.

Task

Ground truth

# Count Airports Passed Along a Flight Path

{ "count": 3, "airports": [ "RUT", "ALB", "SWF" ]

## GOAL Track a flight over a period and count how many airports it passes within close proximity. ## STEPS 1. Set the playback time to 17:15 UTC. 2. Search for flight SAS907 using the search bar and locate it on the map. 3. Track the flight for the next 30 minutes of playback time. 4. Count how many airports the flight passes within a 30 km radius and list their airport codes. # RESULT FORMAT “‘json { "count": <N>, "airports": ["<airport1>", "<airport2>", "..."] } “‘

}

No.

Task

Ground truth

# Detect the First Stable East-to-West Ordering of Three Flights

{ "order_start_utc": "16:27 UTC" }

## GOAL Determine the first time three aircraft reach a specified west-to-east ordering and keep it for a sustained period. ## STEPS 1. Set the playback time to 16:20 UTC. 2. Search for flights AFR24NL, RYR61XN, and BEL0GA and track them forward in time. 3. Find the first UTC minute when the west-to-east order of the flights becomes BEL0GA -> AFR24NL -> RYR61XN. 4. Report that UTC time. # RESULT FORMAT “‘json { "order_start_utc": } “‘

"<HH:MM UTC>"

# Count Country Borders Crossed from Track History

{ "num_country_borders": 4, "countries": [ "Belgium", "Germany", "Austria", "Slovenia" ]

## GOAL Determine how many country borders a flight has crossed based on its visible track history. ## STEPS 1. Set the playback time to 16:00 UTC. 2. Search for flight MSR726 using the search bar. 3. Examine the flight’s visible track history (trajectory) as shown at the current moment. 4. Count how many country borders the track crosses and list each country entered. # RESULT FORMAT “‘json { "num_country_borders": <N>, "countries": ["<country1>", "<country2>", "..."] } “‘

}

No.

Task

Ground truth

# Find the Closest Aircraft and Its Overtake Time

{

## GOAL Locate the aircraft nearest to a given flight and determine when it overtakes that flight. ## STEPS 1. Set the playback time to 07:50 UTC. 2. Search for flight VOE25HV using the search bar. 3. Identify the aircraft currently closest to VOE25HV. 4. Track both aircraft forward in time. 5. Report the closest aircraft’s callsign and the UTC time at which it overtakes (passes ahead of) VOE25HV. # RESULT FORMAT “‘json { "closest_callsign": "<callsign>", "overtake_time_utc": "<HH:MM UTC>" } “‘

"closest_callsign": "N656FN", "overtake_time_utc": "07:54 UTC" }

No.

Task

Ground truth

# Find the Closest Aircraft Pair Near an Airport

{

## GOAL Among airborne aircraft near Zurich, find the two that are closest to each other. ## STEPS 1. Set the playback time to 16:00 UTC. 2. Zoom into Zurich, Switzerland. 3. Identify all aircraft within 40 km of the airport at the time. Exclude any aircraft on the ground in the airport area. 4. Determine which two airborne aircraft are closest to each other. 5. Report both callsigns. List the aircraft with the higher geographical altitude first. # RESULT FORMAT “‘json { "aircraft_1_callsign": "aircraft_2_callsign": } “‘

"<callsign>", "<callsign>"

"aircraft_1_callsign": "AFR33SD", "aircraft_2_callsign": "SWR60L" }

No.

Task

Ground truth

# Detect the First Landing and First Departure from an Airport

{

## GOAL Monitor one airport over a short time window and identify the first aircraft that lands and the first aircraft that departs after the start time.

}

## STEPS 1. Set the playback time to 10:05 UTC. 2. Find Vienna Airport (VIE). 3. Observe aircraft activity for the next 10 minutes of playback time. 4. Identify: - the first aircraft that departs from the airport after the start time. - the aircraft that is second to land at the airport after the start time. 5. Report both callsigns and their UTC times. ## RESULT FORMAT “‘json { "second_landing_callsign": "<callsign>", "first_departure_callsign": "<callsign>" } “‘

"second_landing_callsign": "AUA87", "first_departure_callsign": "AUA7F",

No.

Task

Ground truth

# Find the Previous Departure Before a Target Flight

{

## GOAL Monitor departures from Manchester Airport within a defined radius and identify which flight departed immediately before a given target flight. ## STEPS 1. Set the playback time to 14:22 UTC. 2. Search for Manchester Airport (MAN). 3. Draw a 10 km radius circle centered on the airport. 4. Monitor departures from the airport. 5. Locate the departure of flight EZY14GQ. 6. Determine which flight departed from Manchester Airport immediately before EZY14GQ. 7. Report: - the previous flight callsign, - the UTC time when the previous flight left the radius circle. ## RESULT FORMAT “‘json { "previous_callsign": "<callsign>", "previous_left_time": "<HH:MM UTC>" } “‘

"previous_callsign": "RUK6797", "previous_left_time": "14:25 UTC", }

No.

Task

Ground truth

# Identify the Last Remaining Flight Inside a Zone

{

## GOAL Among several flights near an airport, determine which one is the last to remain inside a monitored radius. ## STEPS 1. Set the playback time to 16:21 UTC. 2. Search for airport XCR and draw a 50 km radius circle around it. 3. Locate flights SWR441R, UAE4PV, THY6FE, and S5BBC in the area. 4. Track all four flights forward in time. 5. Determine the moment when all of those flights left the circle. 6. Report: - the UTC time when all 4 initial flights left the circle, - the callsign of the first new flight entering the circle, - the first UTC time when the circle contains 4 new flights. # RESULT FORMAT “‘json { "initial_left_utc": "<HH:MM UTC>", "first_new_callsign": "<callsign>", "first_time_with_four_new_utc": "<HH:MM UTC>" } “‘

"initial_left_utc": "16:25 UTC", "first_new_callsign": "EJU79XU", "first_time_with_four_new_utc": "16:28 UTC" }

No.

Task

Ground truth

# Track a Two-Stage Zone Transition for Two Flights

{

## GOAL Determine a multi-step temporal relationship between two flights around the same airport. ## STEPS 1. Set the playback time to 11:43 UTC. 2. Search for the target airport, Ostend-Bruges International Airport (OST). 3. Identify the two approaching flights that will be the first new flights to enter a 35 km radius zone around the airport. 4. Find the first UTC time when exactly one of the two flights enters the 35 km zone around the airport. 5. Continue tracking until both flights are outside the circle. 6. Report: - the first time either of the two planes enters the zone, - which flight is the second to be inside the zone, - the UTC time when both flights finally exit the zone. # RESULT FORMAT “‘json { "any_enters_utc": "<HH:MM UTC>", "last_inside_callsign": "<callsign>", "final_exit_time_utc": "<HH:MM UTC>" } “‘

G.5

Circuit Designer

"any_enters_utc": "11:44 UTC", "last_inside_callsign": "RYR496K", "final_exit_time_utc": "11:50 UTC" }

No.

Task

Ground truth

# Build a NOT Gate Using a NAND Gate ## GOAL Implement a logical NOT function using only a universal NAND gate. ## STEPS 1. Add the following components: - One input switch: A - One output switch: Y - Logical gates as needed 2. Wiring: - Make a NOT gate using only NAND gate. 3. Define output: - The output of the scheme must be the circuit output Y. # RESULT FORMAT “‘json {"answer": “‘

"done"}

# Build a 3-Input Odd-Parity Circuit ## GOAL Construct a circuit with three binary inputs (A, B, C) and one output (Y) such that: - Y = 1 when an odd number of inputs are 1 - Y = 0 when an even number of inputs are 1 ## STEPS 1. Add the following components: - Three input switches: A, B, C - Two XOR gates - One output switch: Y 2. Wiring: - Connect A and B to the inputs of the first XOR gate. - Connect the output of the first XOR gate and C to the inputs of the second XOR gate. 3. Define output: - The output of the second XOR gate must be the circuit output Y. # RESULT FORMAT “‘json {"answer": “‘

"done"}

No.

Task

Ground truth

# Build a Full-Adder Using Logic Gates ## GOAL Construct a circuit that computes the 1-bit full-adder functions: SUM = A XOR B XOR CIN COUT = (A AND B) OR (A AND CIN) OR (B AND CIN) ## STEPS 1. Add the following components: - Three input switches: A, B, CIN - Two output switches: SUM and COUT - Logical gates as needed 2. Build the circuit so that: - SUM is computed as the XOR of A, B, and CIN - COUT is computed as the carry bit of the full adder 3. Define outputs: - The output of the second XOR gate must be the circuit output SUM. - The output of the second OR gate must be the circuit output COUT. # RESULT FORMAT “‘json {"answer": “‘

"done"}

# Build a 1-Bit ALU With 2 Operations ## GOAL Construct a simple 1-bit ALU that takes two data inputs A, B and one operation-select bit OP, then produces one output RESULT. The ALU must support two operations OR and AND ## STEPS 1.

Add three input switches:

Add one output switch:

A, B and OP

RESULT

3. Build both candidate operation wires: - W_AND = A AND B - W_OR = A OR B 4. Use OP to select which operation appears at RESULT: - When OP = 0, RESULT = W_AND - When OP = 1, RESULT = W_OR 5. Verification: - For OP = 0, the circuit must behave like an AND gate. - For OP = 1, the circuit must behave like an OR gate. # RESULT FORMAT “‘json {"answer": “‘

"done"}

No.

Task

Ground truth

# Build a BCD-to-7-Segment Decoder for Segment a ## GOAL Implement a circuit that takes a BCD input D = (D3, D2, D1, D0) and drives output SEG_A high when segment a of a 7-segment display must be lit. Use the following Boolean expression: SEG_A = D3 + D1 + (D2 AND D0) + (NOT D2 AND NOT D0) ## STEPS 1. Add the following components: - Four input switches: D3, D2, D1, D0 - One output switch: SEG_A - Logical gates as needed 2. Build the circuit: - Treat D3 as the most significant bit. - Implement ‘SEG_A‘ exactly as given in the goal equation. - Use NOT gates where negated signals are required. - Combine the terms using the necessary AND and OR gates. 3. Define the output: - The single circuit output must be SEG_A. # RESULT FORMAT “‘json {"answer": “‘

"done"}

# Build a 1-to-4 Demultiplexer ## GOAL Implement a circuit with one data input D, two select bits S1 and S0, and four outputs Y0..Y3, such that D is routed to exactly one output chosen by (S1, S0) while the others stay low: Y0 = (NOT S1 AND NOT S0 AND D), Y1 = (NOT S1 AND S0 AND D), Y2 = (S1 AND NOT S0 AND D), Y3 = (S1 AND S0 AND D) ## STEPS 1. Add the following components: - One data input switch: D - Two select input switches: S1, S0 - Four output switches: Y0, Y1, Y2, Y3 - Logical gates as needed 2. Build the circuit so that: - For each Yi, build the 3-input AND of D and the select pattern for i. - Exactly one Yi can be 1 at any time, and only when D = 1. 3. Define outputs: - The four circuit outputs must be Y0, Y1, Y2, Y3. # RESULT FORMAT “‘json {"answer": “‘

"done"}

No.

Task

Ground truth

# Build a 1-Bit Comparator with Three Outputs ## GOAL Construct a circuit that compares two 1-bit inputs A and B and produces three outputs: A_GT_B = 1 when A > B A_EQ_B = 1 when A = B A_LT_B = 1 when A < B ## STEPS 1. Add the following components: - Two input switches: A and B - Two NOT gates - Two AND gates - One XNOR gate - Three output switches: A_GT_B, A_EQ_B, and A_LT_B 2. Wiring: - Connect input B to the first NOT gate. - Connect input A and the output of the first NOT gate to the inputs of the first AND gate. - Connect input A to the second NOT gate. - Connect the output of the second NOT gate and input B to the inputs of the second AND gate. - Connect inputs A and B to the inputs of the XNOR gate. 3. Define outputs: - The output of the first AND gate must be the circuit output A_GT_B. - The output of the second AND gate must be the circuit output A_LT_B. - The output of the XNOR gate must be the circuit output A_EQ_B. # RESULT FORMAT “‘json {"answer": “‘

"done"}

No.

Task

Ground truth

# Inverting Summing Amplifier ## GOAL Construct an inverting summing amplifier using an ideal op-amp and measure its output voltage. ## STEPS 1. Add and configure components: - Add an Ideal Op-Amp. - Add a 1-terminal DC Voltage Source (V1) and set it to 1.5 V. - Add a Resistor (R1) and set its resistance to 2 k ohms. - Add a 1-terminal DC Voltage Source (V2) and set it to -3.0 V. - Add a Resistor (R2) and set its resistance to 5 k ohms. - Add a Feedback Resistor (Rf) and set its resistance to 10 k ohms. - Add a Ground connection. 2. Wiring: - Connect the non-inverting (+) input of the Op-Amp directly to Ground. - Connect V1 to one end of R1. - Connect V2 to one end of R2. - Connect the free ends of both R1 and R2 to the inverting (-) input of the Op-Amp. - Connect Rf between the Op-Amp’s output terminal and its inverting (-) input terminal. 3. Once the circuit reaches a steady state, hover your mouse over the Op-Amp and report the exact Output Voltage (Vout) displayed in the information box at the bottom right corner of the screen. # RESULT FORMAT “‘json { "opamp_output_voltage": } “‘

"value"

{ "opamp_output_voltage": }

"-1.5 V"

No.

Task

Ground truth

# Wheatstone Bridge Voltage Measurement ## GOAL Construct a Wheatstone bridge circuit and measure the voltage drop across one of the bridge resistors. ## STEPS 1. Add and configure components: - Add a 2-terminal DC Voltage Source and set its voltage to 10 V. - Add a Resistor (R1) and set its resistance to 1 k ohms. - Add a Resistor (R2) and set its resistance to 2.2 k ohms. - Add a Resistor (R3) and set its resistance to 3.3 k ohms. - Add a Resistor (R4) and set its resistance to 4.7 k ohms. - Add a Ground connection. 2. Wiring: - Ground the negative terminal of the voltage source. - Connect one end of R1 to the positive terminal of the voltage source. - Connect one end of R2 to the positive terminal of the voltage source. - Connect the free end of R1 to one end of R3. - Connect the free end of R2 to one end of R4. - Connect the free ends of R3 and R4 together and to Ground. 3. Once the circuit reaches a steady state, hover your mouse over Resistor R4 (the 4.7 k ohm resistor) and report the exact Voltage Drop (Vd) displayed in the information box at the bottom right corner of the screen. # RESULT FORMAT “‘json { "voltage_drop_R4": } “‘

"value"

{ "voltage_drop_R4": }

"6.812 V"

No.

Task

Ground truth

# RLC DC Steady-State Trick ## GOAL Construct an RLC circuit driven by DC and measure the capacitor voltage at steady state. ## STEPS 1. Add and configure components: - Add a 2-terminal DC Voltage Source and set its voltage to 24 V. - Add a Resistor (R1) and set its resistance to 1 k ohms. - Add an Inductor and set its inductance to 100 mH. - Add a Resistor (R2) and set its resistance to 2.2 k ohms. - Add a Capacitor and set its capacitance to 10 uF. - Add a Resistor (R3) and set its resistance to 4.7 k ohms. - Add a Ground connection. 2. Wiring: - Ground the negative terminal of the voltage source. - Connect one end of R1 to the positive terminal of the source. Call the other end of R1 "Node A". - From Node A, create a path to Ground containing the Inductor in series with R2. - From Node A, create a second parallel path to Ground containing the Capacitor in series with R3. 3. Hover your mouse over the Capacitor. Wait for the simulation to charge the capacitor and reach a complete DC steady state (capacitor current displayed in the information box is less than 20 nA) and report the exact Voltage Drop (Vd) displayed in the information box at the bottom right corner of the screen. # RESULT FORMAT “‘json { "capacitor_voltage_drop": } “‘

"value"

{ "capacitor_voltage_drop": }

"16.5 V"

No.

Task

Ground truth

# 555 Timer Astable:

Output Frequency

## GOAL Configure a 555 timer in astable (free-running) mode and measure the exact output frequency and duty cycle on the oscilloscope. ## STEPS 1. Add and configure components: - Add a 555 Timer chip. - Add a Voltage Source 1-terminal (Vcc) and set it to 5 V. - Add a Resistor (Ra) and set its resistance to 10 k ohms. - Add a Resistor (Rb) and set its resistance to 22 k ohms. - Add a Capacitor (Ct) and set its capacitance to 100 nF. - Add a Ground connection. 2. Wiring: - Connect Vcc to Pin 8 (V+) of the 555. - Connect Pin 1 (GND) of the 555 to Ground. - Connect Pin 4 (RESET) of the 555 directly to Vcc (keep reset inactive). - Connect Ra between Vcc and Pin 7 (DISCH). - Connect Rb between Pin 7 (DISCH) and Pin 6 (THRESH). - Connect Pin 6 (THRESH) directly to Pin 2 (TRIG). - Connect Ct between Pin 6 (THRESH) and Ground. 3. Right-click Pin 3 (OUT) of the 555 and select "View in New Scope". In the scope’s properties, enable BOTH "Show Frequency" AND "Show Duty Cycle". Let the simulation run for at least 30 output cycles to let the measurements settle, then report the exact Frequency and Duty Cycle values displayed on the scope. # RESULT FORMAT “‘json { "output_frequency": "value", "duty_cycle": "value" } “‘ 12

# Build a 4-bit Monotonic Sequence Detector ## GOAL Construct a combinational circuit with four binary inputs A, B, C, D (read left-to-right as a sequence in that order) and one output Y, such that Y = 1 if and only if the bit sequence A, B, C, D is non-decreasing i.e., a 1 is never followed by a 0 anywhere in the sequence. ## STEPS 1. Add the following components: - Four input switches: A, B, C, D - One output switch: Y - AND, OR, NOT, NAND, NOR, XOR, XNOR gates as needed 2. Build the circuit so that: - Y must produce the correct value for all 16 possible input combinations. # RESULT FORMAT “‘json {"answer": “‘

"done"}

{ "output_frequency": "260.417 Hz", "duty_cycle": "58%" }

No.

Task

Ground truth

# Build a 2-bit Binary Squarer ## GOAL Construct a combinational circuit with two binary inputs A1, A0 (A1 the most significant bit) representing the unsigned integer N = 2ůA1 + A0, and four binary outputs P3, P2, P1, P0 (P3 the most significant bit) representing the unsigned binary value of Nš, such that for every one of the 4 input combinations: Nš = 8ůP3 + 4ůP2 + 2ůP1 + P0 ## STEPS 1. Add the following components: - Two input switches: A1, A0 - Four output switches: P3, P2, P1, P0 - AND, OR, NOT, NAND, NOR, XOR, XNOR gates as needed 2. The circuit must produce correct values for all 4 input combinations. # RESULT FORMAT “‘json {"answer": “‘

"done"}

# Build a 4-to-2 Priority Encoder with Valid Flag ## GOAL Construct a combinational circuit with four request inputs R3, R2, R1, R0 (R3 is the highest priority) and three outputs Y1, Y0, V such that: - V = 1 if at least one request is high; V = 0 otherwise. - When V = 1, (Y1, Y0) is the binary index of the highest-priority asserted input: R3 high (Y1, Y0) = (1, 1) R3 low, R2 high (1, 0) R3 low, R2 low, R1 high (0, 1) only R0 high (0, 0) - When V = 0, (Y1, Y0) is don’t-care. Implement the outputs using these expressions (priority is enforced by masking lower-priority requests with NOT of higher-priority ones): Y1 = R3 OR (NOT R3 AND R2) = R3 OR R2 Y0 = R3 OR (NOT R2 AND R1) V = R3 OR R2 OR R1 OR R0 Outputs must be correct for all 16 input combinations. ## STEPS 1. Add the following components: - Four input switches: R3, R2, R1, R0 - Three output switches: Y1, Y0, V - AND, OR, NOT, NAND, NOR, XOR, XNOR gates as needed 2.

Build Y1, Y0, V exactly as given above.

3. Define outputs: - The three circuit outputs must be Y1, Y0, V. # RESULT FORMAT “‘json {"answer": “‘

"done"}

No.

Task

Ground truth

# Build a 3-Input Majority and Minority Detector ## GOAL Construct a combinational circuit with three binary inputs A, B, C and two outputs MAJ and MIN such that: - MAJ = 1 when at least two of the three inputs are 1 - MIN = 1 when at least two of the three inputs are 0 Exactly one of MAJ and MIN is high for every input combination, and they are logical complements. Implement the outputs using these expressions, sharing the three pairwise-AND terms across both outputs: T_AB = A AND B T_AC = A AND C T_BC = B AND C MAJ = T_AB OR T_AC OR T_BC MIN = (NOT T_AB) AND (NOT T_AC) AND (NOT T_BC) The three pairwise-AND wires (T_AB, T_AC, T_BC) must each be computed by a single AND gate and fan out to both the MAJ OR-tree and the MIN NOT/AND-tree. ## STEPS 1. Add the following components: - Three input switches: A, B, C - Two output switches: MAJ, MIN - AND, OR, NOT, NAND, NOR, XOR, XNOR gates as needed 2. Build T_AB, T_AC, T_BC exactly as given. Then build MAJ and MIN by reusing those three wires. 3. Define outputs: - The two circuit outputs must be MAJ and MIN. - Outputs must be correct for all 8 input combinations. # RESULT FORMAT “‘json {"answer": “‘

"done"}

No.

Task

Ground truth

# BJT Differential Pair with Resistor Loads ## GOAL Build a BJT differential pair with resistor collector loads and an NPN current-mirror tail. Apply a small differential input and measure the peak-to-peak output voltage at the collector of the right-side transistor. ## STEPS 1. Add and configure components: - Add three supply nodes: Vcc = +10 V, Vee = -10 V, and Ground. - Add two default NPN transistors for the input pair: Q1 and Q2. - Add two collector-load resistors: - Rc1 = 10 k ohms - Rc2 = 10 k ohms - Add two default NPN transistors for the tail current mirror: Q5 and Q6. - Add a reference resistor Rref and set its resistance to 19 k ohms. - Add a 1-terminal A/C Voltage Source (Vin) and set: Max Voltage = 1 mV, Frequency = 1000 Hz, DC offset = 0 V, phase = 0 deg. 2. Wiring: - Connect the emitters of Q1 and Q2 together. Call this node TAIL. - Connect Rc1 from Vcc to the collector of Q1. - Connect Rc2 from Vcc to the collector of Q2. Call the collector of Q2 OUT. - Connect the emitters of Q5 and Q6 to Vee. - Diode-connect Q5 by shorting its collector to its base. - Connect Rref from Vcc to the collector/base node of Q5. - Connect the base of Q6 to the base of Q5. - Connect the collector of Q6 to the TAIL node. - Connect Vin to the base of Q1. - Connect the base of Q2 directly to Ground. 3. Measurement: - Run the simulation. - Confirm that the DC operating point places OUT between the rails, approximately around +5 V DC. - Scope the OUT node. - In steady state, read the Peak-to-Peak value shown in the scope overlay. - Report the measured peak-to-peak AC voltage at OUT. # RESULT FORMAT “‘json { "diff_pair_output_peak_to_peak": }

"value"

{ "diff_pair_output_peak_to_peak": }

"191.615 mV"

No.

Task

Ground truth

# Build a 3-bit Two’s-Complement Negator with Overflow Flag ## GOAL Construct a combinational circuit that takes a 3-bit two’s-complement signed integer A = (A2, A1, A0), where A2 is the sign bit and the represented value is val(A) = -4ůA2 + 2ůA1 + A0 (range:

-4 ..

+3)

and produces: - A 3-bit two’s-complement output Y = (Y2, Y1, Y0) such that val(Y) = -val(A) whenever -val(A) is representable in 3-bit two’s complement. - A 1-bit overflow flag OVF that is 1 if and only if A = 100 (i.e. val(A) = -4), because +4 cannot be represented in 3-bit two’s complement. When OVF = 1, the value of Y is unconstrained (don’t-care). The negation must be computed as Y = (NOT A) + 1 using bitwise inversion followed by a 3-bit ripple-carry incrementer. ## STEPS 1. Add the following components: - Three input switches: A2, A1, A0 - Four output switches: Y2, Y1, Y0, OVF - AND, OR, NOT, NAND, NOR, XOR, XNOR gates as needed 2. Build the circuit so that: - First stage: compute B0 = NOT A0, B1 = NOT A1, B2 = NOT A2 using NOT gates. - Second stage: ripple-carry increment B by 1 using three half-adders chained by carry: - Bit 0: Y0 = B0 XOR 1 = NOT B0 (equivalently, Y0 = A0); carry C1 = B0 AND 1 = B0. - Bit 1: Y1 = B1 XOR C1; carry C2 = B1 AND C1. - Bit 2: Y2 = B2 XOR C2. (The final carry-out is discarded it does not feed OVF.) - Overflow: OVF = A2 AND (NOT A1) AND (NOT A0). This is the only input pattern for which the negation overflows. - Y must produce the correct value for all 7 non-overflow input combinations, and OVF must be correct for all 8 input combinations. 3. Define outputs: - The four circuit outputs must be Y2, Y1, Y0, OVF. # RESULT FORMAT “‘json {"answer": “‘

"done"}

No.

Task

Ground truth

# 4-Bit Asynchronous Ripple Counter: Frequency

MSB Output

## GOAL Build a 4-bit asynchronous (ripple) counter from JK flip-flops driven by a 1 kHz clock, and measure the output frequency at the MSB (Q3). Each flip-flop divides its clock input by 2, so a 4-bit ripple counter divides the master clock by 2^4 = 16. ## STEPS 1. Add and configure components: - Add a Clock source (Inputs/Outputs menu Clock, or use a Square-wave Voltage Source). Set: frequency = 1000 Hz, duty cycle = 50%, high level = 5 V, low level = 0 V. - Add four JK Flip-Flops (Chips menu JK Flip-Flop). Label them FF0 (LSB) through FF3 (MSB). - Add a Logic Input set permanently to high (5 V) used as a constant logic-1 source for the J and K pins. - Add a Logic Output for visualising Q3. - Add Ground. 2. Wiring (asynchronous toggle counter): - Tie BOTH J and K of every flip-flop (FF0FF3) to the constant-high Logic Input this configures each one to toggle on every clock edge. - Connect the Clock source to the CLK input of FF0. - Connect Q (output) of FF0 CLK of FF1. - Connect Q of FF1 CLK of FF2. - Connect Q of FF2 CLK of FF3. - Connect Q of FF3 the Logic Output indicator. - Leave the !Q (inverted) outputs unconnected, and leave any RESET/SET pins unconnected (or tie them to the inactive level if the chip block exposes them). 3.

Click RUN.

4. Right-click the wire at Q3 (the output of FF3) View in New Scope. In the scope’s properties, enable "Show Frequency". Let the simulation run for at least 1 second of simulated time so the counter cycles through all 16 states many times and the frequency overlay locks in. 5. Report the frequency value displayed on the scope at Q3. # RESULT FORMAT The value must be a string containing the numeric frequency in Hz with units, e.g. "62.5 Hz". Do NOT emit the literal placeholder text "value". Do NOT omit units. Round to 1 decimal place if needed. “‘json { "msb_output_frequency": } “‘

"<frequency>"

{ "msb_output_frequency": }

"62.5 Hz"

No.

Task

# Common-Emitter BJT Amplifier: Peak-to-Peak Output

Ground truth Mid-Band

## GOAL Build a classic common-emitter BJT voltage amplifier with a voltage-divider bias network and a bypassed emitter resistor, and measure the peak-to-peak output voltage at the collector for a small AC input. ## STEPS 1. Add and configure components: - Add a Voltage Source 1-terminal (Vcc) and set it to 12 V. - Add a Transistor (bipolar, NPN). Leave default model parameters. - Add a Resistor (R1) and set it to 47 k ohms (top bias). - Add a Resistor (R2) and set it to 10 k ohms (bottom bias). - Add a Resistor (Rc) and set it to 3.3 k ohms (collector). - Add a Resistor (Re) and set it to 1 k ohms (emitter). - Add a Capacitor (Cin) and set it to 1 uF (input coupling). - Add a Capacitor (Ce) and set it to 100 uF (emitter bypass). - Add a Capacitor (Cout) and set it to 1 uF (output coupling). - Add a Resistor (R_load) and set it to 10 k ohms. - Add a 1-terminal A/C Voltage Source (Vin) and set: Max Voltage = 10 mV, Frequency = 10 kHz, DC offset = 0, phase = 0. - Add a Ground connection. 2. Wiring: - Connect Vcc to the top of R1 and to the top of Rc. - Connect the bottom of R1 and the top of R2 together call this node "Base_Bias". Connect the transistor’s base to Base_Bias. - Connect the bottom of R2 to Ground. - Connect the bottom of Rc to the transistor’s collector. Call this node "Collector". - Connect the transistor’s emitter to the top of Re. Call this node "Emitter". - Connect the bottom of Re to Ground. - Connect Ce between Emitter and Ground (parallel with Re). - Connect Cin between Vin and Base_Bias. - Connect Cout between Collector and one end of R_load. Call the other end of R_load "Node OUT". - Connect Node OUT to Ground through R_load (i.e., R_load is between Cout and Ground, and Node OUT is the junction with Cout). 3. Right-click the wire at Node OUT (the load side of Cout) and select "View in New Scope". In the scope’s properties, enable "Show Peak-to-Peak". Let the simulation run for at least 50 input cycles so all coupling capacitors fully charge and the amplifier settles into its operating point, then report the exact Peak-to-Peak value displayed on the scope. # RESULT FORMAT “‘json { "amplifier_output_peak_to_peak": } “‘

"value"

100

{ "amplifier_output_peak_to_peak": }

"2.605 V"

No.

Task

# Wien Bridge Oscillator with Diode AGC

Ground truth

## GOAL Build a Wien-bridge oscillator using an op-amp with positive-feedback RC network and a negative-feedback gain network. Use anti-parallel diodes for soft amplitude limiting (automatic gain control), let the loop self-start from a small initial-condition seed, and report both the steady-state oscillation frequency and the peak-to-peak output voltage at the op-amp output. ## STEPS 1. Add and configure components: - Add an Op Amp (use the default model, finite-GBW, rails at ś15 V). - Positive-feedback (Wien) network: - Resistor Rs = 10 k ohms (series arm). - Capacitor Cs = 10 nF (series arm). - Resistor Rp = 10 k ohms (parallel arm). - Capacitor Cp = 10 nF (parallel arm). - Negative-feedback (gain-setting) network with diode AGC: - Resistor Rg = 10 k ohms (from input to ground). - Resistor Rf_a = 12 k ohms (first half of the feedback resistor). - Resistor Rf_b = 10 k ohms (second half of the feedback resistor; the diodes shunt this part as the swing grows). - Two anti-parallel diodes (default Si model), each connected across Rf_b. - Add Ground connections. 2. Wiring: - Positive-feedback network (between op-amp OUTPUT and the + input): - Rs from op-amp OUTPUT to a node "Node_S". - Cs from Node_S to the op-amp + input (call this node "Node_Plus"). - Rp from Node_Plus to Ground. - Cp from Node_Plus to Ground (in parallel with Rp). - Negative-feedback network: - Rg from the op-amp input to Ground. - Rf_a from the op-amp input to a midpoint node "Node_FB_Mid". - Rf_b from Node_FB_Mid to op-amp OUTPUT. - Diode D1 from Node_FB_Mid (anode) to op-amp OUTPUT (cathode) i.e., across Rf_b. - Diode D2 from op-amp OUTPUT (anode) to Node_FB_Mid (cathode) i.e., the anti-parallel partner of D1, also across Rf_b. 3. Seed start-up: - Right-click Cp (the cap in the parallel arm) Edit set Initial Voltage = 0.1 V (or higher if needed). CircuitJS1’s components are noiseless, so without a seed the oscillator will not spontaneously start. 4. Run the simulation. 5. Right-click the op-amp OUTPUT terminal node and select "View in New Scope". In the scope’s properties, enable BOTH "Show Frequency" AND "Show Peak-to-Peak". Let the simulation run long enough for the amplitude to grow into the diode-clamped steady state (~50100 ms simulated). The frequency overlay needs at least 510 cycles visible per screen to lock in if it reads blank, zoom the scope’s TIME axis (not the voltage axis) until cycles are clearly resolved, then wait for the readout to stabilise. 6. Report the exact Frequency and Peak-to-Peak values displayed on the scope. # RESULT FORMAT “‘json { "oscillation_frequency": "value", "output_peak_to_peak": "value" } “‘

101

{ "oscillation_frequency": "1.589 kHz", "output_peak_to_peak": "1.422 V" }

Record · ID 271859 · SHA-256 452f1d63eec307eb
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.