CAPTCHAs in the Agentic Era: Solvers That Learn from Every Encounter

arXiv:2609.02393v1 [cs.CR] 2 Sep 2026

CAPTCHAs in the Agentic Era: Solvers That Learn from Every Encounter Oguzhan Salman

Kemal Bicakci

Istanbul Technical University salmano21@itu.edu.tr

Istanbul Technical University kemalbicakci@itu.edu.tr

object categories and later YOLO-family detectors [6] that can achieve high accuracy on trained classes much faster than earlier approaches. Specialized YOLO models have been shown to solve all visual reCAPTCHA types — classification (3×3 grids), segmentation (4×4 grids), and dynamic puzzles — effectively [7], [8] but these systems use browser automation frameworks such as Selenium, Playwright, Puppeteer, or Cypress to access the Document Object Model (DOM) to locate and interact with UI elements [9], [10], [11], [12]. Concurrently, vision-language models (VLMs) have emerged as powerful general-purpose systems capable of understanding and reasoning over diverse visual and language tasks. Building on the transformer architecture [13] that revolutionized sequence modeling through self-attention mechanisms, large language models [14] demonstrated zero-shot task completion. Today, multimodal models like GPT-4V, Claude 3, and QwenVL drive GUI agents such as OpenAI Operator, ChatGPT Atlas, Anthropic Computer Use, WebVoyager, and Perplexity Comet [15], [16], [17], [18], [19] and can interact with computers in diverse tasks through user commands. Unlike specialized YOLO models, these VLM systems are capable of understanding and navigating interfaces, interpret screenshots and execute complex workflows without task specific training, I. I NTRODUCTION which makes them primary candidate for automation. Despite CAPTCHAs (Completely Automated Public Turing tests this potential, recent benchmarks identify CAPTCHAs as a to tell Computers and Humans Apart) [1] remain a primary primary obstacle that frequently blocks these agents from defense against automated bot activity on the web. Among completing end-to-end workflows [20]. These challenges persist modern image-based systems, Google’s reCAPTCHA v2 [2] mainly because VLMs face fundamental difficulties with visual is still widely deployed despite the introduction of the implicit grounding and spatial localization without careful prompt behavioral v3 variant of reCAPTCHA [3]. Sites continue to engineering [21]. Just as prompting strategies significantly prefer v2 due to its explicit user verification and ease of influence model performance on reasoning tasks [22], visual integration into their products where visible challenge–response interactions similarly require grounding strategies. Without mechanisms are desirable. By presenting users with semantic grounding techniques that supply localized crops and structured image labeling tasks, these challenges are intended to ex- annotations, LLMs still struggle to identify and interact ploit the gap between human visual perception and current with small UI elements [23]. Equipping VLMs with explicit machine-vision capabilities to distinguish legitimate users from grounding methods such as Set-of-Mark prompting bridges automation. However, rapid advancements in computer vision much of this gap, enabling strong performance in visual are closing this gap, challenging the security assumptions of interaction tasks, including CAPTCHA solving [24]. CAPTCHAs. From the current landscape of automated solvers, we identify Early automated solvers relied on traditional optical character three critical observations that motivate a paradigm shift in recognition techniques for text-based CAPTCHAs [4], [5]. modern CAPTCHA solving. First, specialized classifiers rely As the CAPTCHA ecosystem shifted toward image-based heavily on browser automation frameworks (e.g., Selenium) from text-based challenges, deep learning approaches emerged, for element localization. This both prevents integration with initially using convolutional neural networks trained on specific fully visual GUI agents and exposes the solver to detection Abstract—Vision-language models (VLMs) can solve visual CAPTCHAs without task-specific training, but the agents built on them approach every challenge from scratch. For such an agent, the hundredth instance of a familiar puzzle costs as much time and compute as the first. Specialized detectors invert the trade-off, answering in milliseconds but only for categories they were trained on. Neither improves with exposure. We study what changes when a solver improves with use. Our system pairs a fine-tuned YOLOv8 detector with an open-weight VLM behind a confidence-based router, and runs entirely from screenshots and operating-system input events, with no browser automation or DOM access. It reaches 85.4% overall and 84.2% macro accuracy across 16 classes, exceeding either component alone. Every answer VLM produces also serves as a training label, so the detector absorbs categories it was never trained for, typically after one or two encounters and without human annotation. The same loop also repairs it. A CAPTCHA operator can perturb images against the publicly released detector and drive its accuracy to 0%, but the perturbations leave VLM untouched, and its labels let the detector recover. Under a yearlong simulated arms race in which the CAPTCHA operator re-crafts its perturbations each month, the solver recovers every round, and a cheap ∼70%-accurate open-weight teacher hardens it as effectively as a perfect oracle. Visual CAPTCHA defenses that assume a failing bot stays failing therefore understate how quickly an adaptive solver returns.

because such frameworks leave fingerprints in the browser demonstrated large-scale attack on reCAPTCHA v2, achieving that anti-bot systems routinely check for. Second, while VLMs 70.78% success rate on image challenges using deep learningdemonstrate superior reasoning capabilities and can decompose based image annotation services and 83.5% on Facebook’s visual puzzles into step-by-step solutions [25], their computa- image CAPTCHA. Subsequent work targeted specifically tional overhead prevents deployment as real-time solvers, with image-based reCAPTCHA systems using deep convolutional inference costs orders of magnitude higher than discriminative networks trained on specific object categories. Hossen et al. [7] alternatives. Third, CAPTCHA challenges represent determin- demonstrated that YOLOv3-based object detection can solve istic recognition problems once the underlying visual pattern Google’s image reCAPTCHA v2 with 83.25% success rate, is understood, creating an opportunity to distill expensive averaging 19.93 seconds per CAPTCHA. Their approach used semantic reasoning into lightweight, specialized detectors a bounding box to grid mapping algorithm to convert YOLOv3 through supervised learning. This opportunity extends beyond detections into cell selections for 3×3 classification puzzles. any single CAPTCHA system—Halligan [24] demonstrates While effective for COCO-trained object classes (bicycles, that VLMs can already solve 26 distinct CAPTCHA families buses, cars, traffic lights), their system required puppeteerwithout task-specific training, suggesting that a distillation firefox for DOM access to extract grid cell coordinates and paradigm proven on one family could generalize broadly. trigger click events. Hossen et al. also evaluated robustness In this work, we present a hybrid solver that improves against reCAPTCHA’s built-in image perturbations (noisy, itself with use, mirroring human dual-process cognition. YOLO distorted challenge images), achieving 73% object detection serves as the fast, reflexive pathway for familiar patterns, while on corrupted samples after adversarial training, though they an open-weight Qwen-7B-VL model [26] fine-tuned for web did not consider gradient-based attacks. Weng et al. [28] interactions [27] provides the slow, deliberate reasoning for demonstrated systematic attacks across three image CAPTCHA novel or ambiguous challenges. A confidence-based cascade types (selection-based, slide-based, click-based) using CNNs resolves 70% of challenges at YOLO’s reflex speed (4 ms per and Fast-RCNN, achieving 79–90% success on selectioncell), invoking VLM reasoning (225 ms) only when YOLO’s based puzzles similar to reCAPTCHA. Plesner et al. [8] fineconfidence is insufficient, reducing average latency by 3 times tuned YOLOv8 on 14,000 reCAPTCHA images across 13 compared to VLM-only inference. A separately fine-tuned object classes, achieving near-perfect solving rates on image YOLOv8 detector localizes the CAPTCHA area, grid cells, classification puzzles. However, Plesner et al. system also and buttons from the screenshot, with a finite-state controller relies on Selenium for browser automation and remains limited coordinating detection, solving, verification, and recovery. to trained object classes, achieving 0% coverage on unseen We make the following contributions. categories. These prior CAPTCHA solvers share a critical limitation: • A DOM-free hybrid solver. The system reads screenshots and acts through OS-level mouse and keyboard events, so reliance on browser automation frameworks (Selenium, Playit neither depends on browser automation nor leaves the wright) to access the Document Object Model (DOM) for element localization and interaction. This dependency matters fingerprints such frameworks expose. for two reasons. First, it is incompatible with the trend • A reasoning-to-reflex cascade. YOLO answers familiar cells at reflex speed and VLM is invoked only when toward fully visual GUI agents that interact with computers YOLO is uncertain, reaching 85.4% overall and 84.2% through screenshots and OS-level input events without DOM access [17]. A CAPTCHA solver that requires DOM access macro accuracy, higher than either model alone. cannot serve as a module within such agents. Second, browser • Autonomous self-extension and recovery. Because every automation frameworks inject detectable artifacts into the challenge VLM solves becomes training data, the solver browser environment — Selenium, for example, sets the absorbs object classes it was never shipped with, often navigator.webdriver property — and anti-bot systems from one or two puzzle encounters, and recovers on its routinely check for such fingerprints [31]. Plesner et al. [8] own when a PGD attack collapses YOLO from 83% to compensate for this detectability through VPN rotation, browser 0%, in both cases without human annotation. cookies, and Bézier-curve mouse trajectories; without these, • Adaptive-robustness findings. Under a year-long arms their solver degrades significantly (Section VIII). Our system race in which the CAPTCHA operator re-crafts its avoids this class of detection by operating at the OS level, perturbations each month, the solver re-hardens every capturing screenshots via system APIs and clicking at screen round, and a cheap ∼70%-accurate teacher proves as coordinates via PyAutoGUI, with no browser API or DOM effective for this hardening as a perfect oracle, since its access at any point. labeling errors make the solver harder to replicate. To overcome these limitations, research has shifted toward purely visual interface understanding, treating the browser as II. R ELATED W ORK a visual canvas rather than a structured document. In this Early automated CAPTCHA solvers focused on text-based broader context, YOLO-family detectors have been extensively challenges using OCR and segmentation techniques [29], used for UI element detection in mobile and web interfaces, [5], [4], which ultimately led to the development of imageachieving reliable localization of buttons, icons, and other based systems like reCAPTCHA v2. Sivakorn et al. [30] general components. OmniParser [23] further demonstrated that

2

TABLE I M ODERN CAPTCHA SOLVERS : MACRO ACCURACY, ROBUSTNESS UNDER ATTACK , AND OPERATIONAL COST. Paper

Primary Model

DOM-Free

Macro Acc.

Robustness

Cost

Weng et al. [28]

Specialized CNNs

×

N/A

Not evaluated

On-premises

73% on reCAPTCHA-perturbed images (corruption only; no PGD)

On-premises

Hossen et al. [7]

YOLOv3 (Detection)

×

60.74%∗

Plesner et al. [8]

YOLOv8 (Classification)

×

75.69%∗

0% under PGD (ε=4/255); no adversarial training evaluated

On-premises

OEDIPUS [25]

GPT-4 / Gemini

×

N/A

Not evaluated

$3–13 / 100

Halligan [24]

GPT-4o

Partial

N/A

−15–49 pp under visual distractors (no PGD)

$2.40 / 100

Our System

Qwen-7B-VL + YOLOv8

✓

84.17%

VLM: 75% under PGD (ε=4/255); YOLO recovers to 65% via VLM-labeled retraining (GT: 71%)

On-premises

∗

Calculated from per-class success rates provided in the original paper’s figures. N/A = not available.

pretrained YOLO detectors can accurately localize even small UI icons, but it did not consider CAPTCHA layouts or puzzlespecific grid structures. In parallel, multimodal vision–language models such as CLIP [32], Flamingo [33], LLaVA [34], and InstructBLIP [35] have greatly improved high-level screen understanding and visual reasoning. Yet prior evaluations of VLM-based GUI agents [20] (e.g., GPT-4V, Qwen-VL, Claude 3) report poor performance on CAPTCHAs, reflecting limited fine-grained grounding and a lack of structured output mechanisms needed for grid-style selection tasks. Together, these results suggest that neither generic UI detectors nor current VLM agents alone directly address CAPTCHA-specific perception and solution. Beyond browser-based automation frameworks, Zhang et al. [36] proposed UFO, a Windows UI-automation agent that interacts with applications via the Windows UI Automation API instead of Selenium. Although it removes the dependency on a web browser, UFO still assumes programmatic accessibility hooks and thus does not operate in a purely screenshot-based setting. While these agents focus on the mechanics of interaction, other recent works address the logical reasoning required to solve complex puzzles. Deng et al. [25] successfully employs a Domain Specific Language (DSL) to decompose complex reasoning CAPTCHAs into substeps and achieves an average success rate of 63.5% across multiple reasoning categories; however, it faces significant latency (>100 seconds) and cost constraints for every execution. Our work shifts this paradigm and, rather than using VLM as a standalone inference solver for every challenge, we propose a hybrid teacher-student model, addressing the fundamental economic and speed bottlenecks identified in the OEDIPUS framework. Most recently, Teoh et al. [24] introduced Halligan, a VLM-based CAPTCHA solver (GPT-4o [37]) that achieves a 60.7% success rate across 26 CAPTCHA families and 68.0% on reCAPTCHA v2 specifically. Halligan formulates CAPTCHA solving as a search problem where a visionlanguage model iteratively explores candidate actions guided by natural language objectives. While Halligan operates in a DOMfree environment, its reliance on iterative search results in a

median latency of 21.8 seconds per challenge and an operational cost of $2.40 per 100 solutions. Furthermore, Halligan reports that solve rates drop by 15.4% to 49.6% when faced with adversarial transformations or distractions such as gaussian noise and advertisement banners and assumes the challenge can be cleanly isolated from the surrounding interface. In contrast, our approach eliminates the assumption of clean isolation by performing end-to-end learned visual grounding via a finetuned YOLOv8 backbone, enabling robust handling of cluttered layouts and dynamic variants. Table I compares accuracy, robustness, and cost across modern reCAPTCHA v2 solvers and reasoning agents. Specialized classifiers [7], [8] are accurate on the classes they were trained for but have no coverage beyond them, and they rely on browser automation frameworks vulnerable to DOM-based detection, requiring substantial behavioural infrastructure to operate at scale. Robustness is largely unaddressed across prior work, with Plesner’s classifier dropping to 0% under PGD and Halligan losing 15–49 pp under visual distractors alone. Generalized agentic frameworks [24], [25] handle broader task distributions but at order-of-magnitude higher per-puzzle cost. Our system pairs Plesner’s published classifier with an openweight VLM cascade, raising macro accuracy across the full 16class distribution to 84.17%, retaining 75% under PGD through VLM path, and avoiding DOM-based detection without percall API costs. Because both systems use the same published YOLO classifier, these gains come from the hybrid design rather than a stronger detector. III. M ETHOD In this section, we present our hybrid architecture designed to solve visual CAPTCHA challenges without DOM access. The proposed system consists of a fine-tuned object detector for UI localization, a deterministic finite-state machine (FSM) for process control and a dual-path solving mechanism that leverages the complementary strengths of specialized YOLO classifiers and Vision-Language Models. The entire pipeline operates at the OS level: input is obtained by capturing screenshots via system APIs, and output is delivered by sending mouse and keyboard events to absolute screen coordinates. The browser is treated as a visual surface and is never accessed

3

programmatically. Figure 1 illustrates the complete system architecture with execution steps. Our architecture builds on two complementary model families. You Only Look Once (YOLO) [6] is a family of object detectors that, given an input image x ∈ RH×W ×3 , produces bounding box predictions {(bi , ci , pi )}Ki=1 in a single forward pass, where bi = (x, y, w, h) defines coordinates, ci ∈ {1, . . . ,C} is the predicted class, and pi ∈ [0, 1] is the confidence score. For classification, YOLO produces a probability distribution over C classes: f (x) = softmax(z),

z j = w⊤j φ (x) + b j

remain and then clicks verify. The complete solving procedure integrating FSM control with hybrid YOLO/VLM solving is presented in Algorithm 1 in the Appendix. C. Classification and Segmentation Pipelines After extracting all CAPTCHA-related UI elements using our trained YOLOv8 detector, we use EasyOCR to detect the instruction text within the CAPTCHA area and extract the target class. With the target identified, individual cells cropped from the detected captcha area and images are sent either to a YOLO classifier/segmentation [40] or to Qwen-7B-VL GUI agent [41]. For classification puzzles, VLM receives a numbered prompt listing all 16 classes (0–15) and is constrained to return a single class index, enforcing deterministic top-1 predictions and minimizing hallucinations (exact prompt in Appendix H). Each of the 9 cells is extracted from the 3×3 grid and independently classified by either YOLO or VLM (see Figure 7 in the Appendix). For segmentation puzzles, 4×4 detected cells are concatenated into a single image. YOLO produces masks or bounding boxes, while VLM receives a structured prompt requesting bounding box coordinates for all instances of the target object (exact prompt in Appendix H). Predictions are mapped back to individual cell coordinates using a 1% overlap threshold: any cell whose bounding box intersects the predicted region by ≥1% of its area is selected (see Figure 8 in the Appendix).

(1)

where φ (x) denotes the feature representation and w j , b j are the weights and biases for class j. In our system, we employ three separately trained YOLO models for UI element detection, tile classification, and instance segmentation respectively. Vision-Language Models (VLMs) complement YOLO’s discriminative classification with open-ended reasoning. VLM maps an image x to visual tokens v = gv (x) ∈ RNv ×d via a vision encoder, then generates output tokens autoregressively: T

P(y1 , . . . , yT | v,t) = ∏ Pθ (yi | y<i , v,t)

(2)

i=1

This token-by-token generation enables flexible reasoning but costs substantially higher latency than YOLO’s single-pass inference. We use Qwen-7B-VL [26], a 7-billion parameter VLM fine-tuned for GUI interaction tasks [27], constraining its output to a single class index for classification or bounding box coordinates for segmentation.

IV. DATA AND M ETRICS To accurately measure the performance of our hybrid solver, we constructed a comprehensive dataset reflecting real-world class distributions that contain challenging edge cases. This section details the composition of our dataset and defines the specific evaluation metrics used to measure performance across both classification and segmentation puzzle types.

A. Detection Backbone: YOLOv8 We fine-tune YOLOv8 from OmniParser’s [23] pretrained icon detection model [38] to detect five CAPTCHA-specific classes: captcha area, cell, robot checkbox, reload button, and submit button. OmniParser provides strong foundation model for small UI element localization, trained on diverse webpage screenshots. To create a diverse training dataset without exposing our model to live CAPTCHA services during development, we synthetically generated 1,200 training samples by programmatically overlaying CAPTCHA elements onto webpage backgrounds from a Kaggle screenshot dataset [39] (details in Appendix B.2). The model processes 512×512 RGB images and we train with AdamW optimizer, multi-scale training and early stopping following established fine-tuning practices.

A. Dataset Composition Our evaluation dataset comprises 15,172 samples in total, collected from public reCAPTCHA dataset [42] and CAPTCHA samples obtained during our experiments. The dataset covers 16 object classes: Bicycle, Boat, Bridge, Bus, Car, Chimney, Crosswalk, Hydrant, Motorcycle, Mountain, Other, Palm, Stairs, Taxi, Tractor, and Traffic Light. Among these classes, three of these (Boat, Taxi, Tractor) are not supported by the published YOLO classifier. We encountered these object categories during our experiments and include them in our evaluation to reflect real-world scenarios. Because the published YOLO classifier was itself trained on part of this dataset, we evaluate exclusively on held-out data that neither model has seen.

B. Finite-State Controller We coordinate detection, solving, and verification through a deterministic finite-state machine M = (S, s0 , Σ, δ ) with states S covering initialization, puzzle analysis, solving, verification, and recovery. Transitions δ : S×Σ → S are triggered by visual events (checkbox detected, grid appeared, CAPTCHA cleared). The FSM handles both static and dynamic puzzles. In static puzzles, the system solves once and then verifies. In dynamic puzzles, selected tiles refresh with new images after each click, requiring the FSM to iteratively re-analyze the grid until no target objects

B. Evaluation Metrics We use different primary metrics for classification and segmentation puzzles based on their task requirements. For 3×3 classification puzzles, we prioritize recall because these types of CAPTCHA puzzles require identifying all tiles that contain target object and missing a required tile increases risk score.

4

Fig. 1. Complete system architecture showing end-to-end CAPTCHA solving pipeline with execution steps. Step 1: Our system captures screenshots and uses a fine-tuned YOLOv8 detector to localize UI elements (captcha area, grid cells, verify/reload buttons). Step 2: Once UI elements are detected, OCR extracts the target object (e.g., "select all motorcycles") and determines the puzzle type based on the detected number of grid cells (classification vs. segmentation). Step 3: Based on puzzle type, the system loads appropriate solver models. The hybrid routing strategy assigns each class to YOLO or VLM based on the performance for the target class. If solver engine is YOLO and the target is unsupported by YOLO (e.g., Boat, Taxi, Tractor), the system reloads the puzzle. Step 4: YOLO returns probability arrays (classification) or masks/bounding boxes (segmentation); VLM returns cell indices or bounding box coordinates in JSON format. Step 5: The finite-state machine maps detected UI elements from screenshot pixel coordinates to absolute screen coordinates and sends OS-level mouse and keyboard events; the browser is never accessed programmatically. It then verifies success via screenshot capture, reloads the puzzle on failure, and handles dynamic puzzles where cells refresh after interaction.

We report both macro-averaged recall and overall accuracy to capture different aspects of solver robustness. Overall accuracy reflects performance on the natural class distribution in our dataset, where some classes appear more frequently than others. However, from a security perspective, macro-averaged metrics are more critical: CAPTCHA designers can adaptively bias puzzle distribution toward object classes where automated solvers are known to fail. For instance, if YOLO-based solvers cannot recognize Boats, Taxis, or Tractors, defenders could increase the frequency of these categories to maximize solver failure rate. Macro-averaged recall treats all classes equally regardless of frequency, providing a worst-case performance measure that is robust to such adversarial class selection. A solver with high overall accuracy but low macro-averaged recall remains vulnerable to targeted countermeasures, whereas our hybrid approach achieves strong performance on both metrics demonstrating resilience against adaptive defenses. Our evaluation for the classification task differs from live sessions of CAPTCHA puzzles in that we assess models on a static dataset of puzzle tiles from all 16 classes, requiring the

model to discriminate the dominant object across all possible categories. In contrast, real reCAPTCHA presents contextspecific puzzles where positive examples contain obvious instances of the target and negative examples contain clearly different objects. This makes our evaluation more challenging than live solving because models must handle images that contain multiple categories such as distinguishing dominant bridges from dominant cars in images where both may be present as occluded, out of context or secondary elements. In this static evaluation setting, recall reliably measures whether the model identifies the ground-truth dominant object, while precision can be artificially deflated when models detect genuine but secondary or occluded objects in the scene, (see Figure 6 in the Appendix). We report per-class precision, recall, and F1-score, along with overall accuracy (frequency-weighted, where frequent classes dominate) and macro-averaged accuracy (where all classes contribute equally). An alternative design would be to report top-k accuracy or probability-threshold metrics (e.g., treating a prediction as correct if Car appears in the top-3 classes with probability

5

at least 10%). We deliberately refrain from doing so in order work that evaluated only on the original dataset classes. On to maintain a fair evaluation between YOLO and VLM. The these 12 classes, YOLO achieves 83.1% top-1 classification YOLO classifier outputs a full softmax distribution over classes, accuracy, closely matching Plesner et al.’s reported 82.4% top-1 so top-k and probability thresholds are well defined. VLM, in accuracy on the same model (the small difference likely reflects contrast, does not expose class probabilities: in our setup, it additional evaluation samples from their live bot operations produces a single class index token (0–15), and when instructed that are not publicly available). Looking at the per-class to output probabilities, it returns unnormalized, inconsistent or performances in Table III, YOLO achieves high recall on even hallucinated values that often do not sum to 100%. These classes such as Bus (97.3%), Hydrant (100.0%) and texturevalues reflect the model’s language-generation behaviour rather heavy classes like Crosswalk (90.5%). However, it struggles than reliable posterior probabilities and there is no principled with structurally similar objects like Motorcycles (68.9% recall), way to construct a top-k metric for VLM that is comparable to which are frequently misclassified as Bicycles (Figure 2a), and YOLO’s softmax scores. To avoid this asymmetry, we discard scores 0% on unsupported classes. the additional confidence information provided by YOLO and evaluate both models under the shared metric which is single TABLE II C LASSIFICATION ACCURACY AND LATENCY COMPARISON ON top-1 label per tile. RE CAPTCHA OBJECT CATEGORIES . For 4×4 segmentation puzzles, we use F1-score as the primary metric, since boundary localization requires balancing Metric YOLO VLM (GUI Agent) Hybrid both under-selection (low recall) and over-selection (low preciOverall Acc. (16 classes) 0.7548 0.8052 0.8544 sion), making F1-score appropriate for measuring segmentation Overall Acc. (12 classes∗ ) 0.8314 0.7827 0.8498 Macro Acc. (16 classes) 0.5999 0.8101 0.8417 performances. V. E XPERIMENTS We evaluate discriminative and generative models through a series of experiments designed to highlight their distinct capabilities. We analyze classification performance in detail, summarize the extension to segmentation, and conclude with a latency analysis of each model.

Macro Acc. (12 classes∗ )

0.7521

0.7661

0.8160

VLM calls Avg. latency (per cell)

0% 4 ms

100% 225 ms

30.0% 71 ms

∗

Excludes four separately collected classes (Boat, Stairs, Taxi, Tractor).

b) VLM GUI Agent Performance: Next, we evaluate our Qwen-7B-VL GUI agent. VLM achieves an overall accuracy of 80.5% and macro accuracy of 81.0%. Unlike YOLO, VLM A. Classification Experiments demonstrates strong zero-shot generalization, achieving 100% To evaluate the solvers on classification-type puzzles, we recall on rare classes like Boat and Tractor and high recall compare the published YOLO classifier and our Qwen-7B- on semantically distinct categories like Traffic Light (93.2%) VL GUI agent on identical reCAPTCHA challenges. This and Stairs (94.7%). However, VLM underperforms on specific analysis focuses on cases where each grid cell is to be classified categories. For example, VLM often ignores the primary object independently. like Crosswalk to focus on secondary objects (classifying them Following the evaluation protocol described in Section IV-B, as Bridges or Others or hallucinates specific labels for generic we evaluate both models on a held-out test set spanning 16 backgrounds). object classes, requiring a single top-1 prediction per image. c) Hybrid Routing Strategy: Table II summarizes the Because the published YOLO classifier was trained on the performance of both models alongside our hybrid routing training split of the reCAPTCHA dataset [42], we construct strategy. the test set from the validation split plus separately collected Rather than statically assigning each class to a single model, samples for four classes (Boat, Stairs, Taxi, Tractor) that were we employ a confidence-based cascade that routes each image not part of YOLO’s training data. Neither model was trained independently. For unsupported classes (identified via OCR on any image in this test set. For YOLO, we take the argmax from the puzzle instruction), images are routed directly to over its softmax scores and for VLM, we use the numbered VLM. For supported classes, YOLO processes the image first 0–15 prompt described in Section III-C, which constrains it to and its top-1 softmax confidence determines whether to trust return exactly one class index. its prediction or fall back to VLM:  The test set exhibits class imbalance reflective of real / CYOLO  fVLM (x) c ∈ world reCAPTCHA distributions. Most classes contain 74 r(x, c) = fYOLO (x) c ∈ CYOLO ∧max fYOLO (x) ≥ τ (3) validation images, while rare classes (e.g., Boat: 11, Mountain:  f (x) c ∈ C ∧max f (x) < τ VLM YOLO YOLO 2, Chimney: 17) have fewer samples. a) YOLO Classifier Performance: We first evaluate the where c is the target class, CYOLO is the set of YOLO-supported published YOLO classifier [8]. As shown in Table II, the classes, and τ is the confidence threshold. model achieves an overall accuracy of 75.5% but a macro Table II functions as a controlled ablation over the same test accuracy of only 60.0% across all 16 classes. We also report set and metrics, isolating the contribution of each component: 12-class metrics excluding the four separately collected classes YOLO-only, VLM-only, and the hybrid are evaluated on (Boat, Stairs, Taxi, Tractor) for comparability with prior identical images so that the differences are attributable to

6

(a) YOLO normalized confusion matrix.

(b) GUI Agent (VLM) normalized confusion matrix.

Fig. 2. Comparison of per-class recall for YOLO vs VLM GUI agent. Darker diagonals indicate stronger recall; off-diagonals correspond to confusion. TABLE III P ER - CLASS P RECISION , R ECALL , AND F1 SCORES .

YOLO

Class

VLM (GUI Agent)

Prec

Rec

F1

Prec

Rec

F1

Bicycle Boat† Bridge Bus Car Chimney Crosswalk Hydrant Motorcycle Mountain Other Palm Stairs Taxi† Tractor† Traffic Light

81.5% 0.0% 82.7% 84.7% 41.0% 84.6% 85.9% 96.1% 89.5% 0.0% 45.0% 95.1% 91.5% 0.0% 0.0% 90.1%

89.2% 0.0% 83.8% 97.3% 77.0% 64.7% 90.5% 100.0% 68.9% 0.0% 66.2% 78.4% 57.3% 0.0% 0.0% 86.5%

85.2% 0.0% 83.2% 90.6% 53.5% 73.3% 88.2% 98.0% 77.9% 0.0% 53.6% 85.9% 70.5% 0.0% 0.0% 88.3%

97.2% 91.7% 59.4% 94.8% 47.7% 78.9% 91.2% 94.9% 94.5% 100.0% 84.2% 96.3% 91.0% 95.8% 100.0% 70.4%

94.6% 100.0% 85.1% 98.6% 82.4% 88.2% 41.9% 100.0% 93.2% 50.0% 21.6% 70.3% 94.7% 82.1% 100.0% 93.2%

95.9% 95.7% 70.0% 96.7% 60.4% 83.3% 57.4% 97.4% 93.9% 66.7% 34.4% 81.2% 92.8% 88.5% 100.0% 80.2%

Macro Avg.

60.5%

60.0%

59.3%

86.8%

81.0%

80.9%

†

Unsupported by YOLO Classification Model

the routing design alone. The two single-model configurations macro, and on VLM-only by +4.9 pp overall and +3.2 pp macro, are each insufficient in a distinct way. YOLO-only reaches while invoking VLM for only 30% of images. 75.5% overall but collapses to 60.0% macro accuracy because Importantly, this advantage is not merely an artifact of YOLO it scores 0% on the three unsupported classes; VLM-only lifts scoring 0% on the three unsupported classes. On the 12-class macro accuracy to 81.0% through zero-shot coverage but is subset that excludes the separately collected classes—where slower and less precise on the supported classes that dominate every class is one YOLO was trained to support—the hybrid the distribution. With τ = 0.70, the hybrid achieves 85.4% still exceeds both single models, reaching 85.0% overall (vs. overall and 84.2% macro accuracy, exceeding both single- YOLO’s 83.1% and VLM’s 78.3%) and 81.6% macro (vs. model baselines rather than merely interpolating between them: 75.2% and 76.6%). Here the gain comes entirely from the it improves on YOLO-only by +10.0 pp overall and +24.2 pp confidence-based fallback: when YOLO’s top-1 confidence is

7

confuses Motorcycles with Bicycles, with 15% of Motorcycle images misclassified as Bicycles. This confusion likely stems from both severe training imbalance (27 Motorcycle samples vs. 726 Bicycle samples) and genuine structural similarity between two vehicles. VLM distinguishes them through semantic reasoning about features such as engine blocks and exhaust pipes that discriminative training does not explicitly encode, achieving 93.2% recall compared to YOLO’s 68.9%. This reasoning capability extends naturally to novel categories without retraining. Secondly, YOLO outperforms VLM on pattern-based classes like Crosswalks (90.5%) and on the negative class Other (66.2%). VLM, however, exhibits two distinct failures on these categories: 1) Object-Centric Bias: VLM demonstrates a strong preference for discrete objects over surface features. It often prioritizes potentially distinct entities (even if secondary) and treats dominant texture patterns like Crosswalks (41.9% recall vs YOLO’s 90.5%) as contextual background, leading to false negatives. 2) Semantic Over-Interpretation: VLM struggles significantly with the Other class (21.6% recall), frequently hallucinating specific labels for generic scenes. Unlike YOLO, which learns a decision boundary for "none of the above," VLM’s pre-training forces it to converge to find meaningful concepts in every image. Consequently, it often over-interprets generic roads as Bridges or Crosswalks and background as Palms, unable to accept that an image contains "nothing of interest".

TABLE IV C OMPONENT L ATENCY ( MEDIAN ; MS PER INFERENCE ) Component YOLOv8 Detector YOLO Classifier (per cell) YOLOv8n-seg (per grid) YOLOv8x-seg (per grid) VLM Classification (per cell) VLM Segmentation (per grid)

Latency (ms)

FPS

1.1 4.4 23 26 225 2251

909 220 43.5 38.2 4.1 0.4

low on a supported class, routing that image to VLM corrects errors that neither model avoids alone. The average per-cell latency is 71.9 ms, a 3.1× reduction compared to VLM-only inference. Table III presents per-class Precision, Recall, and F1 scores. B. Segmentation The hybrid architecture also extends to 4×4 segmentation puzzles, where every cell overlapping the target object must be selected. As in classification, a hybrid (YOLOv8x-mask + VLM) performs best, reaching 88.6% accuracy (F1=0.850) by combining YOLO’s geometric precision on supported classes with VLM’s zero-shot coverage of unsupported ones. Unlike classification, however, the 7B VLM is not an adequate teacher here: its segmentation predictions (F1=0.769) are less precise than even a mid-sized YOLOv8m (F1=0.811), so distilling them would degrade rather than improve YOLO. We therefore restrict the teacher-student and adversarial experiments to classification (full per-model and per-class segmentation results in Appendix D-C).

VI. T EACHER -S TUDENT D ISTILLATION

C. Latency Analysis

The hybrid architecture described in the preceding sections Table IV presents the latency analysis for different models in demonstrates that VLM achieves high recall on categories different puzzle types. The YOLOv8 detector operates in real- where YOLO fails entirely, such as Boat, Taxi, and Tractor. time at 1.1 ms per image, enabling immediate UI localization. Since VLM already produces correct predictions for these A significant performance gap exists between specialized classes, its outputs can serve as labeled training data to fine-tune YOLO solvers and VLM. The YOLO-only pipeline maintains YOLO, converting expensive VLM reasoning into millisecondreal-time performance (≈26 ms for segmentation) because scale reflexes. Prior fine-tuning-based solvers [8] depend on YOLO is architecturally optimized for real-time inference a dataset that a human collected and annotated in advance, with lightweight size (<100M parameters). In contrast, VLM which fixes their vocabulary to the classes anticipated at exhibits a substantial latency cost (225 ms–2251 ms) driven training time. Here, the labels come from VLM already present by two factors. First, the Qwen-7B-VL model (7 billion in the pipeline, generated from the very puzzles the solver parameters) requires significantly higher compute and memory encounters at deployment, with no human annotation and no bandwidth per inference than the YOLO models. Second, unlike pre-collected dataset. The solver therefore extends its own YOLO’s near-instantaneous inference, VLM must generate text vocabulary to classes it was never deployed with, closing the tokens sequentially through autoregressive generation. This cost loop between reasoning and recognition at run time. Knowledge is particularly evident in the segmentation task (2251 ms), where distillation [43] transfers knowledge from a capable teacher to VLM must generate a long sequence of coordinate tokens for a smaller student by training the student to match the teacher’s every detected object, whereas classification requires generating outputs. In our setting, VLM teacher produces hard labels rather than soft probability distributions, reducing distillation only a single class index token (225 ms). to supervised learning: D. Discussion Ldistill = ∑ LCE ( fS (x), ŷT (x)) (4) Our results show that these models demonstrate fundamenx∈X tally different recognition strategies such that YOLO relies on learned silhouettes, while VLM leverages semantic reasoning where ŷT (x) is VLM’s predicted class for input x and LCE from internet-scale pre-training. For instance, YOLO frequently is the cross-entropy loss. Effective distillation requires a

8

(a) Taxi: 96% ± 8% recall at N=10.

(b) Tractor: 88% ± 10% recall at N=15.

(c) Boat: 48% ± 18% recall at N=6.

Fig. 3. New-class recall as a function of training images (N) for each unsupported class added via Experience Replay (10 seeds per point, shaded region shows standard deviation across seeds). Dashed vertical lines indicate approximate puzzle equivalents (e.g., N=5 ≈ 1 puzzle). TABLE V D ISTILLATION DATASET SUMMARY AND NEW- CLASS RECALL (%, MEAN ± STD OVER 10 SEEDS ) AFTER E XPERIENCE R EPLAY FINE - TUNING .

Dataset

Class Taxi Tractor Boat

Recall by number of training images (N)

Total

Val

Max Train

N=3

N=5

N=7

N=10

N=15

N=23

28 20 11

5 5 5

23 15 6

76±23 70±24 28±18

86±18 74±13 48±18

88±18 80±13 —

96±8 84±8 —

96±8 88±10 —

98±6 — —

teacher whose predictions are accurate enough to improve the student. For classification, VLM meets this criterion with strong per-class recall that clearly exceeds YOLO on unsupported categories. For segmentation, VLM is too imprecise teacher (Appendix D-C), so we focus distillation and adversarial recovery on classification.

datasets, performance can vary significantly depending on which specific images are selected, so running multiple seeds captures this variance. Table V summarizes the available data per class and presents the mean recall on held-out validation images across 10 seeds. These experiments use VLM labels exactly as produced, without any human correction or filtering. VLM labeling A. Method: Experience Replay Fine-Tuning accuracy on the three new classes is 100% for Boat and When extending a classifier to support new classes, catas- Tractor and 82.1% for Taxi (5 of 28 images misclassified; see trophic forgetting may occur such that the model forgets Appendix J). We deliberately retain the mislabeled Taxi samples previously learned classes when fine-tuned on new data. rather than filtering them, both because the datasets are already Experience Replay [44] (ER) mitigates this by combining extremely small (11–28 images) and because filtering would exemplars from old classes with new training data. Given an require ground-truth knowledge the autonomous pipeline does existing model trained on classes Cold and a new class with N not have. Consequently, the recall values reported in Table V training samples, ER constructs a balanced training set: already reflect realistic VLM label noise. We quantify the [ DER = Dnew ∪ Sample(Dc , N) (5) gap between VLM-labeled and ground-truth-labeled training directly in the adversarial recovery experiments (Section VII, c∈Cold Table VII), where it amounts to a 6–10 pp penalty. where Sample(Dc , N) selects N exemplars from each existing class, ensuring equal representation and preventing the model B. Results from being dominated by the new class. a) New-Class Recall.: To contextualize the N values We fine-tune the base 13-class YOLO classifier to sup- in terms of real-world CAPTCHA encounters, a single 3×3 port each unsupported class (Boat, Taxi, Tractor) as a 14th classification puzzle typically yields 3–5 positive examples of class using ER to prevent catastrophic forgetting. For each the target class (not all 9 tiles contain the target object). Thus, new class with N training images, we include N randomly N=3–5 reflects the data an agent could gather from a single sampled exemplars per existing class, maintaining balanced puzzle encounter, while N=7–10 corresponds to approximately representation. We freeze the first 5 backbone layers to preserve two puzzles. learned features, train for 50 epochs with a learning rate Taxi reaches 96% mean recall by solving 2 puzzles (10 of 5 × 10−4 , and apply standard data augmentation (color training samples) and exhibits low variance across seeds, jitter, rotation, scaling, horizontal flip, and random erasing) to indicating that any random subset of taxi images covers enough mitigate overfitting on small training sets. visual concept. Tractor reaches 88% at N=15 with moderate For each value of N, we run 10 independent trials with variance, reflecting moderate intra-class diversity. Boat plateaus different random seeds, each holding out 5 images for validation at ∼48% mean recall despite using all 6 available training and sampling different training subsets. With very small images (approximately one to two puzzle encounters), with

9

high variance depending on which specific images happen to its capabilities through VLM-guided distillation. A natural be selected. countermeasure for the defender is to neutralize this advantage The divergence in recall across classes is explained by with adversarial perturbations. We investigate whether such intra-class visual diversity. Taxi images are almost exclusively attacks succeed and whether the architecture recovers from it. yellow sedans on streets. The visual concept is consistent Threat Model: Throughout the paper, the solver is our and any random 5 of 28 images look nearly identical. Fine- system, which attempts to pass CAPTCHA challenges automattuning requires only small, focused weight adjustments to carve ically, and the defender is the CAPTCHA operator seeking to out a “yellow car” subregion within the existing Car feature block such automation. Note that the roles are inverted relative space. Tractor images are more diverse in color (green, red, to the usual convention, since it is the defender who crafts blue) but share a coherent visual structure—large vehicles with adversarial perturbations and the solver that must withstand proportionally big wheels, typically on farms or roads. The and recover from them. We assume the CAPTCHA defender concept is learnable from a moderate number of examples. By knows that the solver uses a published YOLO classifier [8] contrast, boat images span wildly different visual concepts such and can craft adversarial perturbations [45] against it. The as rowboats, speedboats, yachts, sailboats, fishing boats from defender is unaware whether a VLM is part of the solver’s distinct environments. A yacht shares almost nothing visually pipeline, and has no access to the model identity, prompt with a rowboat. With only 11 total images and a maximum of 6 template, or predictions. In the grey-box escalation experiments for training, any random subset fails to represent the full “Boat” (Section VII-B), we strengthen the defender further: they concept. This finding reveals that intra-class visual diversity know that the solver retrains on adversarial examples and relative to sample size is the primary predictor of distillation can reproduce its published model, training data, and training difficulty in few-shot class extension. For the teacher-student procedure. The only elements they cannot reproduce are the loop to work efficiently, either the new class must be visually order in which the solver shuffles its training batches and the consistent or substantially more samples are needed to cover labels its VLM assigns. We investigate whether the solver the visual variation. recovers under iterative attack, and how much its resistance b) Catastrophic Forgetting.: A critical concern in con- depends on teacher quality versus training independence. tinual learning is whether adding a new class degrades We generate untargeted adversarial examples using Projected performance on existing classes. Table VI reports the existing Gradient Descent (PGD) [46] with random initialization. Given 13-class accuracy after fine-tuning, compared to the baseline. a classifier f and input x with true label y, PGD iteratively Note that the baseline here (83.1%) is higher than the 16-class finds a perturbation δ such that f (x + δ ) ̸= y with ∥δ ∥∞ ≤ ε:    accuracy in Table II (75.48%) because it excludes four classes (k+1) (k) (k) δ = Π δ +α ·sign ∇ L ( f (x+δ ), y) (6) ∥·∥ ≤ε δ ∞ (Boat, Stairs, Taxi, Tractor) whose samples were not part of the original YOLO training data [8] and are therefore not included where α is the step size and Π denotes projection onto the in the Experience Replay. ℓ∞ -ball. We use ε = 4/255, step size α = 1/255, and 10 PGD steps with random initialization in [−ε, ε], crafting individual TABLE VI perturbations per image. We generate adversarial examples for E XISTING - CLASS ACCURACY (%) AFTER FINE - TUNING AT N=5 ( MEAN ± all images across the 13 YOLO-supported classes (excluding STD , 10 SEEDS ). BASELINE : 83.1% ON 13 SUPPORTED CLASSES . Boat, Taxi, and Tractor which YOLO does not support). New Class Accuracy After Forgetting The PGD attack achieves 100% misclassification, reducing YOLO accuracy from 83.1% to 0.0% on the validation set. This Taxi 76.3 ± 1.4 6.8% Tractor 75.9 ± 1.2 7.2% vulnerability is not specific to Plesner’s model — any YOLO Boat 65.9 ± 1.5 17.2% classifier without adversarial training is similarly susceptible, Taxi and Tractor fine-tuning produces approximately 7 percent- and could recover through adversarial fine-tuning with correct age points of forgetting at N=5, a modest cost for gaining a labels. The same adversarial images, however, have minimal previously unsupported class. Moreover, forgetting decreases as effect on VLM. Since the perturbations were optimized against more training samples become available: Taxi forgetting drops YOLO’s gradients, any effect on VLM represents unintentional from 6.8 pp at N=5 to 3.3 pp at N=23, and Tractor drops from transfer rather than a targeted attack. On the validation set, 7.2 pp to 5.3 pp at N=15. Boat, however, causes 17.2 percentage VLM accuracy drops from 78.7% to 75.0%, a decrease of points of forgetting at N=5 and remains high (15.0 pp) even at only 3.7 pp, consistent with the gradient misalignment between its maximum N=6. This mirrors the recall results: the model architecturally distinct models [47]. This asymmetry means must make broad, destructive weight updates to accommodate that even when YOLO is completely defeated, VLM still Boat’s visually dissimilar images under a single class, damaging classifies the adversarial images correctly. The contribution of our architecture is therefore not that it survives PGD while existing class boundaries in the process. prior work does not, but that this robust VLM serves as a builtVII. A DVERSARIAL ROBUSTNESS AND AUTONOMOUS in label source for adversarial retraining — without human R ECOVERY annotation or external data collection — extending the teacherThe preceding sections demonstrate that the hybrid architec- student paradigm from Section VI to adversarial robustness. ture achieves strong classification performance and can expand

10

TABLE VII AUTONOMOUS RECOVERY: YOLO ACCURACY AFTER RETRAINING ON ADVERSARIAL SAMPLES LABELED BY VLM OR GT. PGD-10, ε =4/255. Samples 0 (baseline) 442 897 1,794 4,498 10,790

YOLO (VLM labels)

TABLE VIII G REY- BOX ESCALATION , MEAN PRE - AND POST- RECOVERY ACCURACY OVER THE 12 MONTHLY ROUNDS . P RE - RECOVERY VARIES WITH THE SOLVER ’ S INDEPENDENCE FROM THE SURROGATE , WHILE POST- RECOVERY IS UNIFORM AT ∼60–64%.

YOLO (GT labels)

Clean

Adv

Clean

Adv

83.0% 72.1% 74.0% 72.5% 72.7% 73.1%

0.0% 27.8% 47.8% 57.0% 62.5% 65.0%

83.0% 80.4% 80.4% 80.1% 79.3% 77.7%

0.0% 32.6% 57.4% 66.8% 69.7% 71.0%

Teacher

Batch order

Pre

Post

Perfect (GT) Perfect (GT) VLM (∼70% acc.) VLM (∼70% acc.)

coupled independent coupled independent

28.4% 52.9% 58.2% 58.9%

60.1% 64.2% 62.3% 62.3%

“Coupled”: the solver shares the surrogate’s exact training order, making it a near-clone the defender can attack directly. “Independent”: the solver shuffles its batches in an order the defender cannot reproduce. Pre and Post are accuracy before and after that round’s retraining.

A. Autonomous Recovery To isolate the effect of VLM label noise on recovery quality, we retrain YOLO on balanced adversarial samples labeled either by our cheap open-weight VLM or by ground-truth (GT) labels, an upper bound a strong, expensive model such as GPT5, Gemini, or Claude would approach. Varying the training budget from a few hundred to ∼11,000 samples spread evenly across classes,1 we retrain on both adversarial and clean copies and evaluate on validation adversarial images crafted against the baseline YOLO. Table VII and Figure 4a show how adversarial accuracy evolves as the training budget increases. With GT labels, adversarial accuracy rises from 0% to 66.8% using only ∼1,800 samples (approximately 200 puzzle encounters). The majority of the gain occurs within the first ∼2,000 samples, after which returns diminish, suggesting that a deployed agent could achieve practical robustness from a modest retraining budget. VLM-labeled retraining incurs a 6–10 pp penalty in adversarial accuracy compared to GT labels across all training budgets (65.0% vs 71.0% at ∼11,000 samples), and clean accuracy drops from 83.0% to ∼72–73% (GT-labeled models retain ∼77–80%). This reflects a standard accuracy-robustness tradeoff [46], [48], where the retrained model allocates capacity to adversarial patterns at some cost to clean performance. B. Iterative Grey-Box Escalation The autonomous recovery experiments above evaluate a single round of retraining against fixed attacks. In practice, the defender re-crafts attacks against an updated surrogate each round, triggering an arms race. We simulate this over 12 rounds, one per month for a year, varying two factors that could shape the solver’s resilience, the teacher it learns from and whether the defender can reproduce its training. The two teachers span the cost spectrum. GT labels stand in for an expensive, highly capable model — the limit a strong proprietary VLM (e.g., GPT-5, Claude) would approach — while our open-weight 7B VLM, which labels the attacked training images at ∼70% accuracy, is the cheap variant. Every party, defender included, trains with label smoothing (0.1), a standard convention that keeps confidence calibrated (relevant for the cascade under 1 Of the 13 attacked classes, Stairs is excluded because our pipeline pairs each adversarial image with its clean training counterpart, and Plesner et al. [8] did not release the original Stairs training data, leaving 12 classes.

11

attack). Because the solver builds on the published YOLO classifier [8] trained on public data, the defender can replicate its starting point and retraining setup; the one thing they cannot replicate is the order in which the solver shuffles its training batches. Crossing the two teachers with whether the defender can reproduce that batch order gives four solver variants. Each round we record pre-recovery accuracy (before retraining) and post-recovery accuracy (after VLM relabels the attacked images and the solver retrains). The escalation shows two patterns, one about the solver’s recovery after retraining and one about its resistance to the attack before retraining. The first is that recovery is universal (Figure 4b, Table VIII). Each month the defender’s fresh attack degrades the solver, in some months cutting its accuracy on the attacked images to nearly zero before it retrains. Yet once the solver retrains on those images, every variant — including the coupled perfect-teacher solver, which is nearly identical to the surrogate the defender attacks — climbs back to 60–64% accuracy on average (Table VIII). The attack is therefore only a temporary setback. The defender can disrupt the solver each month but cannot prevent it from recovering. The second pattern concerns how well a solver resists an attack before retraining, which depends almost entirely on whether its training trajectory is independent of the surrogate’s. Averaged over the year, pre-recovery accuracy is 28.4% for the coupled perfect-teacher solver, which the defender attacks as a near-clone, but 52.9% for the same solver trained with an independent batch order — a 24 pp gain from trajectory independence alone. VLM-taught solver reaches 58.2% and is essentially unchanged whether its batch order is coupled (58.2%) or independent (58.9%), because its noisy labels already make its trajectory independent of the surrogate, so concealing the batch order provides no additional benefit. What resists a transferred attack is a training trajectory that has decorrelated from the surrogate’s, and two mechanisms each achieve this: an independent batch order, or VLM’s noisy labels. Attack transferability depends on gradient alignment between surrogate and target [47], and both mechanisms drive that alignment down. Measured against the surrogate at the final round, the coupled perfect-teacher solver stays aligned (0.42), an independent batch order drops it to 0.07, and noisy

(a) Single-round recovery: adversarial accuracy vs. training samples. VLM labels (red) incur a 6–10 pp penalty vs. GT (green). Robustness plateaus after ∼2,000 samples.

(b) Grey-box escalation over 12 monthly rounds. Pre-recovery accuracy (before each round’s retraining) for four solvers, each taught by a perfect oracle or a ∼70%-accurate VLM and trained either coupled to the surrogate (same batch order, which the defender can reproduce) or independent of it (an order the defender cannot reproduce). The dotted line is the defender’s white-box floor. Either an independent batch order or noisy labels decorrelates the solver from the surrogate, and all variants recover after retraining.

Fig. 4. Adversarial robustness dynamics. (a) Sample efficiency of autonomous recovery under transfer attacks. (b) Iterative grey-box escalation where the defender re-crafts attacks each month.

labels at the same batch order drop it further still, to 0.02. The the perturbations target YOLO rather than VLM. pre-recovery accuracies track this alignment. An independentGiven that VLM path stays robust, the confidence cascade batch-order perfect-teacher solver reaches 52.9% and a same- can serve as a second line of defense under attack, routing batch-order VLM solver 58.2%, both far above the aligned the perturbed cells YOLO misreads to VLM — provided clone’s 28.4%. The two mechanisms are redundant rather than the confidence gate still fires on them. This is where label additive — combining them adds almost nothing (58.9%), smoothing earns its place in the training convention. Without because once a trajectory is decorrelated by either route the it, adversarial retraining drives YOLO’s softmax to saturation surrogate’s perturbations no longer transfer. VLM’s noisy labels (28% of predictions at exactly 1.0), the τ = 0.70 gate rarely thus genuinely harden the solver, at least as effectively as hiding triggers, and only 4% of attacked cells escalate. With smoothing, the batch order would. under attack cascade routes 29–46% of cells to VLM and the This clarifies what the teacher does and does not buy, and the cascade lifts pre-recovery accuracy by +11 to +13 pp — for result is counterintuitive. Comparing the two independent-batch- VLM-taught solver, from 58.2% to 70.2%. order solvers isolates teacher quality: the perfect teacher yields VIII. L IVE RE CAPTCHA E VALUATION 52.9% pre-recovery robustness and the noisy VLM 58.9%, To verify how the hybrid solver’s performance transfers from so the cheap, imperfect teacher not only matches but slightly exceeds the expensive one. Its imperfection is not a liability for static evaluation to live deployment, we evaluated the hybrid robustness but, if anything, an asset — as the mechanism above solver against Google’s live reCAPTCHA v2 demo page over shows, noisy labels decorrelate the solver from the defender’s 100 sessions. We deploy the system in hybrid mode (τ = 0.70) using the surrogate more sharply than clean labels do (0.02 versus 0.07 gradient alignment). What the cheap teacher costs is clean same models, prompts, and FSM as in the static evaluation. accuracy: VLM-taught solver gives up ∼2–6 pp there while The browser was regular Chrome at 1920×1080 with linear surrendering no adversarial robustness. Teacher quality is thus mouse cursor between targets. A session begins at the “I’m a clean-accuracy lever, not a robustness lever, and a defender not a robot” checkbox and ends in a green tick, representing cannot treat the solver’s use of a cheap open-weight teacher successful verification. The FSM does not skip puzzles via as a weakness; for hardening, cheap is as good as expensive. pre-attempt reload — every puzzle served is attempted, and None of this gives the defender a lever. Grey-box knowledge reloads occur only after verification failures. Across 100 sessions comprising 206 distinct puzzles (140 helps only if the solver’s training can be reproduced exactly, which VLM labeling prevents at no cost. Deliberately hiding static, 66 dynamic), all 100 sessions passed successfully. 80 the training order would work equally well, but is a secret that sessions cleared without any reload; 18 required one, and 2 must be set up and maintained, whereas VLM labels come required multiple (three and four respectively). At the puzzle free with solving. Nor do the attacks persist, since retraining level, 181 of 206 puzzles were solved on first attempt (87.9%); restores the solver each round. Throughout, VLM path is itself the 25 failures were recovered through reload-and-retry within unaffected, holding ∼75% on the validation attacks because the same session. The aggregate VLM call rate of 37.1% is

12

within 7 pp of the 30% observed on the static benchmark (Table II), confirming that the cascade’s routing transfers to deployment. The per-session VLM call rate ranges from 0% to 75% (std 15.5%), reflecting the cascade adapting to per-session difficulty: easy sessions rely entirely on YOLO, hard sessions escalate more to VLM. Dynamic puzzles, which re-render selected tiles after each click, require multiple classification rounds per puzzle as the FSM iteratively re-analyses the grid until no targets remain. This iteration is a deployment-only cost the static evaluation cannot capture, since static images are classified once with no re-analysis. End-to-end, the median time was 17.1 s per puzzle and 35.2 s per session. Plesner et al. [8] report a 100% session pass rate on the same CAPTCHA family using a Selenium-controlled Firefox browser. However, their setup combines VPN rotation, realuser browser cookies and history, and Bézier-curve mouse trajectories. Without these helpers, their median required challenges per CAPTCHA degrades from 2 to 5 (cookies removed), 7 (Bézier replaced with straight lines), and 13 (no mouse simulation), and their no-VPN configuration was blocked entirely after 20 runs. Selenium-controlled browsers expose automation fingerprints that anti-bot systems can detect, which the additional behavioural infrastructure in Plesner’s setup compensates for. Their solver also explicitly skips puzzles whose target class their YOLO does not support (Boat, Taxi, Tractor, Stairs). Our screenshot-based approach operates at the operating-system level via mouse and keyboard events, with no DOM access and no browser automation framework, avoiding these fingerprints by design. We achieve a median of 2 distinct puzzles per session and 100% session pass rate from a privatemode Chrome session on a residential connection, with linear cursor paths, no cookies, and no VPN rotation — matching Plesner’s full-stack median while operating under conditions that block their setup entirely, and handles all 16 target classes via VLM fallback rather than skipping unsupported ones. Halligan [24], the closest comparable VLM-based solver, reports 68% on reCAPTCHA v2 under a single-attempt-perchallenge protocol. Under that same protocol our first-attempt rate is 87.9%. On latency, our 17.1 s median per puzzle is 21% faster than Halligan’s 21.8 s, which is achieved with selfhosted Qwen-7B-VL rather than proprietary GPT-4o ($2.40 per 100 puzzles). The cascade naturally limits VLM use to ∼37% of cells, avoiding the per-query API costs of a VLM-only approach. IX. C ONCLUSION This work demonstrates that GUI agents powered by VLMs can serve not only as visual navigators but also as teachers for specialized models. Neither YOLO nor VLM alone is the optimal CAPTCHA solver, but their complementary strengths mean they should collaborate rather than compete. YOLO delivers millisecond-scale inference but fails on unseen categories, while VLM handles any category but at prohibitive cost. By cascading YOLO’s confidence-aware predictions with VLM fallback and distilling VLM’s knowledge into YOLO over time, our DOM-free hybrid achieves +24.2 pp macro recall

13

over YOLO alone using VLM for only 30% of images, while eliminating the need for browser automation frameworks that expose solvers to detection. The teacher-student paradigm that enables this collaboration extends beyond classification. The same mechanism that teaches YOLO to recognize new object classes from a handful of VLMlabeled puzzle encounters also enables autonomous recovery from adversarial attacks. When PGD perturbations reduce YOLO accuracy to 0%, VLM relabels the attacked images and the solver retrains, recovering each round over a yearlong grey-box arms race in which it is never durably defeated. What determines this robustness is not the teacher’s quality. A cheap ∼70%-accurate open-weight VLM hardens the solver at least as well as a perfect oracle, and its noisy labels, far from weakening recovery, decorrelate the solver from the defender’s surrogate so that crafted perturbations no longer transfer. The loop we demonstrate is a general recipe, not a reCAPTCHA v2 tool. Whenever the fast specialist fails — on an unfamiliar class or an adversarial input — VLM’s reasoning supplies a label, and a handful of such labels is enough to fold that capability into the specialist’s reflexes. We show this concretely, extending YOLO to new object classes from as few as one or two puzzle encounters and restoring its adversarial robustness from a couple hundred. Nothing in the recipe depends on v2’s particular puzzles, and because a single VLM already reasons across dozens of CAPTCHA families without task-specific training [24], the same distillation applies to any of them — a stronger teacher would only widen its reach. More generally, whenever an expensive reasoning system operates alongside a fast but limited specialist on deterministic tasks, the reasoner’s outputs can progressively bootstrap the specialist’s capabilities, converting costly deliberation into hardened reflexes.

R EFERENCES [1] L. von Ahn, M. Blum, N. J. Hopper, and J. Langford, “Captcha: Using hard ai problems for security,” in Advances in Cryptology — EUROCRYPT 2003, E. Biham, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2003, pp. 294–311. [2] Google. (2024) reCAPTCHA v2 Documentation. [Online]. Available: https://developers.google.com/recaptcha/docs/display [3] ——. (2025) reCAPTCHA v3 Documentation. [Online]. Available: https://developers.google.com/recaptcha/docs/v3 [4] E. Bursztein, J. Aigrain, A. Moscicki, and J. C. Mitchell, “The end is nigh: Generic solving of text-based CAPTCHAs,” in 8th USENIX Workshop on Offensive Technologies (WOOT 14). San Diego, CA: USENIX Association, Aug. 2014. [Online]. Available: https://www. usenix.org/conference/woot14/workshop-program/presentation/bursztein [5] J. Yan and A. S. E. Ahmad, “A low-cost attack on a microsoft captcha,” in Proceedings of the 15th ACM Conference on Computer and Communications Security, CCS’08, ser. Proceedings of the ACM Conference on Computer and Communications Security, Oct. 2008, pp. 543–554, 15th ACM conference on Computer and Communications Security, CCS’08 ; Conference date: 27-10-2008 Through 31-10-2008. [6] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779–788. [7] M. I. Hossen, Y. Tu, M. F. Rabby, M. N. Islam, H. Cao, and X. Hei, “An object detection based solver for Google’s image reCAPTCHA v2,” in 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2020). San Sebastian: USENIX Association, Oct. 2020, pp. 269–284. [Online]. Available: https://www.usenix.org/conference/raid2020/presentation/hossen [8] A. Plesner, T. Vontobel, and R. Wattenhofer, “Breaking recaptchav2,” in 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, Jul. 2024, p. 1047–1056. [Online]. Available: http://dx.doi.org/10.1109/COMPSAC61105.2024.00142 [9] Selenium. (2025) Selenium webdriver. [Online]. Available: https: //www.selenium.dev/ [10] Microsoft. (2026) Playwright. [Online]. Available: https://playwright.dev/ [11] Google. (2025) Puppeteer. [Online]. Available: https://pptr.dev/ [12] Cypress.io. (2025) Cypress. [Online]. Available: https://www.cypress.io/ [13] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6000–6010. [14] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:160025533 [15] OpenAI. (2025) Introducing operator. [Online]. Available: https: //openai.com/index/introducing-operator/ [16] ——. (2025) Introducing chatgpt atlas. [Online]. Available: https: //openai.com/index/introducing-chatgpt-atlas/ [17] Anthropic. (2024) Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. [Online]. Available: https://www.anthropic.com/ news/3-5-models-and-computer-use [18] H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu, “WebVoyager: Building an end-to-end web agent with large multimodal models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L.-W. Ku, A. Martins, and V. Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 6864–6890. [Online]. Available: https://aclanthology.org/2024.acl-long.371/ [19] Perplexity AI. (2025) Introducing comet: Browse at the speed of thought. [Online]. Available: https://www.perplexity.ai/hub/blog/ introducing-comet [20] Y. Luo, Z. Li, J. Liu, J. Cui, X. Zhao, and Z. Shen, “Open captchaworld: A comprehensive web-based platform for testing and benchmarking multimodal llm agents,” 2025. [Online]. Available: https://arxiv.org/abs/2505.24878 [21] J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,” 2023. [Online]. Available: https://arxiv.org/abs/2310.11441 [22] J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing

14

Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 24 824–24 837. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/ 2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf [23] Y. Lu, J. Yang, Y. Shen, and A. Awadallah, “Omniparser for pure vision based gui agent,” 2024. [Online]. Available: https://arxiv.org/abs/2408.00203 [24] X. Teoh, Y. Lin, S. Li, R. Liu, A. Sollomoni, Y. Harel, and J. S. Dong, “Are captchas still bot-hard? generalized visual CAPTCHA solving with agentic vision language model,” in Proceedings of the 34th USENIX Security Symposium. USENIX Association, 2025. [Online]. Available: https://www.usenix.org/conference/usenixsecurity25/presentation/teoh [25] G. Deng, H. Ou, Y. Liu, J. Zhang, T. Zhang, and Y. Liu, “Oedipus: Llm-enchanced reasoning captcha solver,” in Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 6–20. [Online]. Available: https://doi.org/10.1145/3719027.3744872 [26] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,” 2023. [Online]. Available: https://arxiv.org/abs/2308.12966 [27] M. Andreux, B. B. Skuk, H. Benchekroun, E. Biré, A. Bonnet, R. Bordie, N. Bout, M. Brunel, P.-L. Cedoz, A. Chassang, M. Chen, A. D. Constantinou, A. d’Andigné, H. de La Jonquière, A. Delfosse, L. Denoyer, A. Deprez, A. Derupti, M. Eickenberg, M. Federico, C. Kantor, X. Koegler, Y. Labbé, M. C. H. Lee, E. L. J. de Kergaradec, A. Mahla, A. Manevich, A. Maret, C. Masson, R. Maurin, A. Mena, P. Modard, A. Moyal, A. N. Kerbel, J. Revelle, M. L. Richter, M. Santos, L. Sifre, M. Theillard, M. Thibault, L. Thiry, L. Tronchon, N. Usunier, and T. Wu, “Surfer-h meets holo1: Cost-efficient web agent powered by open weights,” 2025. [Online]. Available: https://arxiv.org/abs/2506.02865 [28] H. Weng, B. Zhao, S. Ji, J. Chen, T. Wang, Q. He, and R. Beyah, “Towards understanding the security of modern image captchas and underground captcha-solving services,” Big Data Mining and Analytics, vol. 2, pp. 118–144, 06 2019. [29] G. Mori and J. Malik, “Recognizing objects in adversarial clutter: breaking a visual captcha,” in Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, ser. CVPR’03. USA: IEEE Computer Society, 2003, p. 134–141. [30] S. Sivakorn, J. Polakis, and A. D. Keromytis, “I’m not a human: Breaking the google recaptcha,” Black Hat, vol. 14, pp. 1–12, 2016. [31] B. García, J. M. del Alamo, M. Leotta, and F. Ricca, “Exploring browser automation: A comparative study of selenium, cypress, puppeteer, and playwright,” in Quality of Information and Communications Technology, A. Bertolino, J. Pascoal Faria, P. Lago, and L. Semini, Eds. Cham: Springer Nature Switzerland, 2024, pp. 142–149. [32] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 8748–8763. [Online]. Available: https://proceedings.mlr.press/v139/radford21a.html [33] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. a. Bińkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan, “Flamingo: a visual language model for few-shot learning,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 23 716–23 736. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/ 2022/file/960a172bc7fbf0177ccccbb411a7d800-Paper-Conference.pdf [34] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 34 892–34 916. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/ 2023/file/6dcf277ea32ce3288914faf369fe6de0-Paper-Conference.pdf [35] W. Dai, J. Li, D. LI, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models

with instruction tuning,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 49 250–49 267. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/ file/9a6a435e75419a836fe47ab6793623e6-Paper-Conference.pdf [36] C. Zhang, L. Li, S. He, X. Zhang, B. Qiao, S. Qin, M. Ma, Y. Kang, Q. Lin, S. Rajmohan, D. Zhang, and Q. Zhang, “UFO: A UI-focused agent for windows OS interaction,” in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang, Eds. Albuquerque, New Mexico: Association for Computational Linguistics, Apr. 2025, pp. 597–622. [Online]. Available: https://aclanthology.org/2025.naacl-long.26/ [37] “Gpt-4v(ision) system card,” 2023. [Online]. Available: https: //api.semanticscholar.org/CorpusID:263218031 [38] Microsoft. (2025) Omniparser: Screen parsing tool for pure vision based gui agent. [Online]. Available: https://github.com/microsoft/OmniParser [39] P. Mostafapoor. (2025) Website screenshots dataset. [Online]. Available: https://www.kaggle.com/datasets/pooriamst/website-screenshots [40] A. Plesner. (2024) Breaking-recaptchav2. [Online]. Available: https: //github.com/aplesner/Breaking-reCAPTCHAv2 [41] HuggingFace. (2025) Holo1: Web agent model. [Online]. Available: https://huggingface.co/blog/Hcompany/holo1 [42] A. Mandourah, “recaptcha dataset,” 2024. [Online]. Available: https://github.com/ajmandourah/recaptcha-dataset [43] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015. [Online]. Available: https://arxiv.org/abs/1503.02531 [44] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato, “On tiny episodic memories in continual learning,” arXiv preprint arXiv:1902.10486, 2019. [45] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013. [46] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” arXiv preprint arXiv:1706.06083, 2017. [47] A. Demontis, M. Melis, M. Pintor, M. Jagielski, B. Biggio, A. Oprea, C. Nita-Rotaru, and F. Roli, “Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks,” in 28th USENIX security symposium (USENIX security 19), 2019, pp. 321–338. [48] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, “Robustness may be at odds with accuracy,” arXiv preprint arXiv:1805.12152, 2018.

A PPENDIX A O PEN S CIENCE To support transparency and reproducibility, we provide the following artifacts for evaluation: A. Available Artifacts CAPTCHA Solver Application: Complete implementation with finite-state machine controller, hybrid backend (YOLO/VLM/Hybrid) and pre-trained model weights. • UI Detection Model Dataset: Complete dataset generation scripts for the YOLOv8 UI element detector, with 220 CAPTCHA source images to produce 1,200 annotated training samples with YOLO-format annotations. Due to storage constraints, the generated dataset and background screenshots are not included but can be generated following the provided instructions. Background screenshots can be obtained from the publicly available Kaggle Website Screenshots Dataset [39]. Detailed setup instructions for downloading, preprocessing, and dataset generation are provided in the README of the artifact. • Evaluation Dataset: Due to storage constraints, we include samples only for three object classes (Taxi, Boat, Tractor) that are not available in the dataset by Mandourah [42]. Researchers can obtain samples for the remaining classes from the public dataset and run the provided inference scripts to reproduce experimental results. For verification purposes, we include raw prediction JSON files from our original experimental runs. • Evaluation Scripts and Notebooks: Scripts and jupyter notebooks to reproduce experimental results including: – Classification inference scripts, metrics notebook (classification_metrics.ipynb), and cascade routing analysis (cascade_routing_analysis.ipynb) – Segmentation evaluation across model sizes – Per-class metric computation • Distillation Notebooks (Section VI): Experience Replay fine-tuning experiments for each unsupported class: – Taxi: taxi_finetune_er.ipynb – Tractor: tractor_finetune_er.ipynb – Boat: boat_finetune_er.ipynb • Adversarial Robustness Notebooks (Section VII): Endto-end adversarial evaluation pipeline: – PGD attack generation: 01_generate_pgd_attacks.ipynb – VLM inference on adversarial images: 02_vlm_inference_on_adversarial.ipynb – Sample efficiency analysis: 03_sample_efficiency.ipynb – Grey-box escalation: 04_greybox_escalation.ipynb • VLM Backend: FastAPI server for Qwen-7B-VL model script with deployment instructions, ngrok integration. •

B. Access Instructions The artifact repository, including all source code and experiment notebooks, is available at:

15

https://oscilloscope.github.io/captchas-agentic-era/ C. System Requirements 1) Hardware Requirements: • Minimum: 16GB RAM, 20GB disk space. YOLO-only solver can run on CPU without GPU, though inference will be significantly slower • Recommended: 32GB RAM, 16GB+ GPU VRAM (NVIDIA RTX 3090 or equivalent), 100GB disk space for VLM inference and hybrid mode 2) Software Dependencies: • Operating System: Any (Windows, macOS, or Linux) • Python: 3.11 • Browser: Any (DOM-independent screenshot-based approach) • Display Resolution: The system performs optimally at 1920×1080 resolution. For other display resolutions, browser zoom adjustment may be necessary to ensure reliable UI element localization All Python dependencies with versions are listed in requirements.txt.

b) State Machine Controller: captcha_fsm.py: Implements the finite-state machine handling CAPTCHA workflow states (detection, checkbox click, puzzle analysis, solving, verification, recovery). c) Solver Models: unified_captcha_processor.py: Core processor supporting three backends: • yolo: Full YOLO-based detection and classification (default, fastest) • llm: YOLO UI detection with VLM-based classification (requires VLM backend) • hybrid: Confidence-based routing between YOLO and VLM (requires VLM backend) d) Models: The solver requires the following pre-trained model files: • detection_model.pt: YOLOv8 UI element detector (5 classes: captcha area, cell, reload button, submit button, robot checkbox) • classification_model.pt: YOLO classifier for 3×3 puzzles • yolov8x-seg.pt: YOLOv8x segmentation model for 4×4 puzzles (auto-download on first run)

D. Installation Instructions C. Running the Solver

1) Create Virtual Environment: python -m venv venv source venv/bin/activate

a) Basic Usage: cd complete_captcha_FSM python main.py

2) Install Dependencies: pip install -r requirements.txt

b) Command-line Options: {yolo,llm,hybrid}: Choose solver backend. • -no-mouse: Disable mouse movement animations for faster execution. In our experiments, mouse movements did not influence solving success rates c) Examples:

• -backend

3) Model Files: Fine-tuned YOLOv8 weights for UI detection, classification, and segmentation are included in the complete_captcha_FSM/ directory. For VLM inference, the Qwen-7B-VL model is automatically downloaded from HuggingFace on the first run. A PPENDIX B CAPTCHA S OLVER I MPLEMENTATION The complete_captcha_FSM/ directory contains CAPTCHA solving application that implements the finite-state machine controller and hybrid backend architecture.

# Default (YOLO backend) python main.py

A. System Architecture The solver implements a human-inspired dual-process approach which combines: • YOLO (Fast Reflexes): Handles frequent, routine patterns with millisecond-scale inference • VLM (Reasoning): Intervenes for novel or semantically complex challenges • Finite-State Machine: Coordinates detection, solving, verification, and recovery loops

# Hybrid mode python main.py --backend hybrid

# VLM backend python main.py --backend llm

B. Key Components a) Main Entry Point: main.py: Monitoring loop that continuously checks for CAPTCHA elements on the screen at 2-seconds intervals and invokes the finite state machine when CAPTCHAs are detected between these intervals.

16

d) Configuration: config.json: API URL for VLM backend (required for llm and hybrid modes) • config.py: Mouse movement behavior configuration •

D. Workflow The solver operates in a continuous monitoring loop at 2seconds intervals: 1) Detection: Continuously monitors screen for reCAPTCHA checkboxes using trained YOLOv8 2) Initialization: Clicks checkbox and waits for CAPTCHA dialog to appear

3) Puzzle Analysis: Captures CAPTCHA area, identifies grid structure, uses EasyOCR to extract target object and determines puzzle type (classification vs segmentation) 4) Backend Routing: Selects YOLO or VLM backend based on target object and configured strategy 5) Solving: Clicks cells identified by the selected model 6) Verification: Validates success by checking if CAPTCHA disappeared and distinguishes between completion, failure, or puzzle re-appearance 7) Recovery: Reloads puzzle on failure or handles dynamic puzzles where cells refresh after interaction A PPENDIX C YOLOV 8 UI D ETECTOR T RAINING Note on Datasets: Our system uses two distinct datasets for different purposes: (1) the UI Detection Training Dataset described in this section (1,200 samples for training YOLO to localize CAPTCHA UI elements on webpages), and (2) the CAPTCHA Evaluation Dataset described in Section 4 (11,908 classification samples and 204 segmentation puzzles with 16 cells each totaling 15,172 evaluation samples across 16 object classes). The UI detector trained here is used to extract CAPTCHA-related regions only.

3) Automated Overlay and Annotation: We programmatically overlaid each of the 220 CAPTCHA images on 4 unique background screenshots at varying positions and scales (0.6× to 1.5×). The script automatically detects grid type (3×3 or 4×4) and generates YOLO-format annotations for all UI elements including checkbox, captcha area, individual cells and buttons. 4) Manual Verification: The script also generates verification images with bounding boxes overlaid on the raw screenshots for manual verification. a) Dataset Composition: The final dataset comprises 1,200 total samples: • 880 CAPTCHA samples containing classes 0–3 (captcha_area, cell, reload_button, submit_button) • 200 robot checkbox samples containing class 4 (checkbox) • 120 negative samples with empty annotation files to reduce false positives on non-CAPTCHA webpages C. Training Configuration Hyperparameters: AdamW optimizer (weight decay 0.0005), learning rate 0.001→0.01 (cosine annealing), batch size 16, image size 512×512, 150 epochs with early stopping. Data augmentation included HSV adjustments, horizontal flip (p=0.5), scale variation (±30%) and mosaic augmentation (p=0.7).

A. Dataset

We fine-tuned OmniParser’s YOLOv8-based icon detection D. Results model [23] for CAPTCHA-specific UI element detection. Training converged at epoch 148 (Figure 5). Final validation The UI detection training dataset comprises 1,200 annotated metrics: Precision 0.999, Recall 1.000, mAP@0.5 0.995, screenshots across five classes: mAP@0.5:0.95 0.990. Inference speed: ∼1.1ms per image • captcha_area: Main CAPTCHA region where images (GPU). appear • cell: Individual clickable grid cells • reload_button: Refresh button • submit_button: Verification button • robot_checkbox: "I’m not a robot" checkbox The dataset includes 880 CAPTCHA screenshots, 200 robot checkbox samples and 120 negative samples (nonCAPTCHA images) to reduce false positives. We used a 80%/20% train/validation split. B. Dataset Generation Methodology To create a diverse training dataset without exposing our model to live CAPTCHA services during development, we synthetically generated training samples using the following procedure: 1) Background Collection: We collected diverse webpage screenshots from Kaggle Website Screenshots Dataset [39], which provides varied visual environments that simulate real-world CAPTCHA placement scenarios across different websites. 2) CAPTCHA Collection: We collected 220 reCAPTCHA screenshots spanning both 3×3 classification and 4×4 segmentation puzzle types across diverse object categories. These serve as source templates for synthetic generation.

17

Fig. 5. Training and validation metrics over 148 epochs.

A PPENDIX D M ETRIC G ENERATION AND E VALUATION M ETHODOLOGY To ensure reproducibility and transparency, we provide detailed documentation of our metric generation process. All evaluation scripts, notebooks, and raw inference results are organized under the experiments/ directory of the artifact repository. The distillation and adversarial experiments are provided as Google Colab notebooks so that all GPU-intensive experiments can be reproduced directly in a browser without requiring local GPU installation or environment configuration.

c) Per-Class Aggregation: The per-class metrics script aggregates cell-level metrics by target object class to compute the per-class F1-scores reported in Table X. For each class, we apply the following: • Sum TP, FP, FN, TN across all puzzles targeting that object • Calculate precision, recall, and F1-score from aggregated confusion matrix • Compare performance across different model sizes (n/m/x) and configurations (bbox vs mask)

A. Classification Metrics Generation

For classification experiments (Section 5.1), we collected model predictions across all 11,908 test samples and computed confusion matrices and per-class metrics as the following: a) Data Collection: We ran inference on both YOLO and VLM models to generate prediction files in JSON format. Each prediction file contains: • Image filename and ground truth label • Predicted class label (top-1 prediction) • Model confidence scores (for YOLO only) • Metadata (image dimensions, processing time) C. Segmentation Results b) Inference Execution: We ran separate inference scripts Table IX summarizes performance on segmentation puzzles for each model configuration: with 3,264 cell decisions from 204 puzzles. YOLOv8x-mask • test_yolo_classification_single.py: Runs YOLO classifier is the strongest single model (86.7% accuracy; F1=0.814). on individual grid cell images to produce top-1 class VLM achieves higher recall (80.1%) but significantly lower predictions precision (73.9%) than YOLO masks. As in classification, a • test_llm_classification_single.py: Runs VLM inference on hybrid of YOLOv8x-mask + VLM provides the best overall the same images with numbered class prompts (0–15) to results (88.6% accuracy; F1=0.850) by combining YOLO’s generate top-1 class predictions geometric precision with VLM’s coverage on rare and unseen c) Metrics Computation: The JSON prediction files from categories. both models are loaded by the classification_metrics.ipynb notebook, which filters predictions to the 893-image test set TABLE IX S EGMENTATION PERFORMANCE : YOLO, VLM, AND H YBRID . (759 validation images plus 134 collected-class images from Boat, Stairs, Taxi, and Tractor), builds confusion matrices, and Model Accuracy Precision Recall F1-Score computes per-class metrics using standard definitions: YOLOv8n-BBox 80.2% 78.0% 66.8% 0.720 • Precision = TP / (TP + FP) YOLOv8n-Mask 82.8% 87.0% 64.5% 0.741 YOLOv8m-BBox 82.6% 75.9% 79.8% 0.778 • Recall = TP / (TP + FN) YOLOv8m-Mask 86.4% 86.5% 76.4% 0.811 • F1-score = 2 × (Precision × Recall) / (Precision + Recall) YOLOv8x-BBox 83.9% 78.1% 80.5% 0.793 YOLOv8x-Mask VLM (BBox) Hybrid

B. Segmentation Metrics Generation

86.7% 81.6% 88.6%

87.2% 73.9% 85.6%

76.3% 80.1% 84.3%

0.814 0.769 0.850

For segmentation experiments (Section 5.2), we evaluated both YOLO models (in dual-mode: bounding box and mask) Table X presents per-class F1-score performances across and VLM (only bounding box) across 204 puzzles with 3,264 YOLO segmentation models and VLM. The results reveal that total cell decisions. For the evaluation: (1) larger YOLO models outperform smaller ones on supported a) Inference Execution: We ran separate inference scripts classes, with YOLOv8x-Mask achieving the highest F1-scores for each model configuration: (Bus: 0.901, Motorcycle: 0.898); (2) all YOLO models fail • test_yolo_dual_mode.py: Runs YOLOv8n/m/x in both completely (F1=0.000) on non-COCO classes (Crosswalk, bounding box and mask modes on all segmentation puzzles Stairs, Taxi) regardless of size; (3) VLM achieves competitive • test_llm_unified.py: Runs VLM inference with bounding performance with YOLOv8n but is significantly outperformed box output on the same puzzle set by larger YOLO models; (4) despite this, VLM provides full • Each puzzle’s results are saved in individual directories coverage across all 8 classes without any CAPTCHA-specific with ground truth masks and predicted outputs training with F1=0.857 on Taxi, 0.752 on Stairs and 0.685 on b) Cell-Level Metrics Calculation: For each puzzle, we Crosswalk—categories where YOLO has zero capability. computed cell-level confusion matrix by: The macro-averaged metrics favor VLM due to its zero-shot 1) Loading ground truth binary mask (which cells should coverage on unsupported classes. However, when aggregating across all 3,264 cell-level decisions (weighted by puzzle be selected) 2) Loading model predictions (bounding boxes or segmen- frequency), YOLOv8x-Mask achieves superior overall performance: 87.2% precision, 76.3% recall and F1=0.814, compared tation masks) 3) Mapping predicted regions to 4×4 grid cells using 1% to VLM’s 73.9% precision, 80.1% recall and F1=0.769. This demonstrates that, for segmentation tasks, while YOLO excels overlap threshold on supported classes which dominate the dataset, VLM provides 4) Computing TP, FP, FN, TN for each cell decision 5) Aggregating metrics across all puzzles for the same target coverage for rare and unseen categories. object class

18

TABLE X S EGMENTATION P ER -C LASS F1-S CORES Class

n-BBox

n-Mask

m-BBox

m-Mask

x-BBox

x-Mask

VLM

Bicycle Bus Crosswalk† Fire Hydrant Motorcycle Stairs† Taxi† Traffic Light

0.721 0.867 0.000 0.667 0.742 0.000 0.000 0.766

0.752 0.888 0.000 0.679 0.775 0.000 0.000 0.771

0.715 0.880 0.000 0.857 0.813 0.000 0.000 0.888

0.745 0.905 0.000 0.925 0.868 0.000 0.000 0.896

0.683 0.894 0.000 0.892 0.843 0.000 0.000 0.913

0.707 0.901 0.000 0.906 0.898 0.000 0.000 0.885

0.710 0.825 0.685 0.756 0.792 0.752 0.857 0.735

†

Unsupported by YOLO segmentation models n=YOLOv8n, m=YOLOv8m, x=YOLOv8x; BBox=bounding box, Mask=segmentation mask.

b) Public Deployment via Ngrok: For remote access, the backend supports ngrok tunneling:

A PPENDIX E VLM BACKEND I NFRASTRUCTURE

To enable VLM inference in our hybrid CAPTCHA solver, python gui_agent_backed.py \ we developed a dedicated API backend that serves fine-tuned --ngrok-token YOUR_TOKEN Qwen-7B-VL GUI Agent. This section documents the backend architecture, deployment options and configuration instructions. This creates a public HTTPS URL that forwards requests to the local server, enabling VLM inference from remote machines A. Architecture Overview without network configuration. This approach was used during VLM backend is implemented as a RESTful API server our experiments to separate GPU-intensive VLM runs from the main solver execution. using FastAPI. D. Integration with Main Solver

B. Implementation Details

The main CAPTCHA solver sends HTTP POST requests a) Model Loading: The backend uses the HuggingFace Transformers Library to load Qwen-7B-VL model fine-tuned to VLM backend’s /generate endpoint. For classification for GUI tasks [27]. On first run, the model is automatically tasks, our trained detection model crops individual grid cells downloaded from HuggingFace Hub (∼16GB) and cached and sends them with the classification prompt. For segmentation tasks, the individual cells are concatenated and sent to retrieve locally. Subsequent runs load the cached model. b) Image Processing Pipeline: The /generate end- bounding box predictions. All backend code, deployment instructions, and example point accepts: API calls are included in the llm_backend/ directory of • Image file (PNG, JPG, etc.) as multipart form data the artifact repository. • Messages JSON containing the prompt in OpenAIcompatible chat format For each request, the backend loads the uploaded image and records its original dimensions, then resizes it to meet model input requirements. The chat template is applied to format the prompt, followed by tokenization of both text and image inputs. Inference is executed and the generated text response is decoded, then returned to the client. c) Response Format: The API returns a JSON response containing: • model_output: Raw text generated by VLM • debug_info: Inference duration in seconds • original_dimensions: Width and height of uploaded image • resized_dimensions: Dimensions after smart resize C. Deployment Options a) Local Deployment: For local testing and development: python gui_agent_backed.py The API runs on http://0.0.0.0:8000 and accepts requests from localhost.

19

A PPENDIX F E VALUATION E XAMPLES

A PPENDIX G S OLVER A LGORITHM

(a)

(b)

(c)

(d)

Fig. 6. All four cells are labeled as Car in the ground truth, but our VLM predicts Boat in each case. In (b)–(c), a boat on the road is clearly visible. In (a)–(d), the cars are heavily occluded by environment or appear adjacent to water, making a prediction of Boat a semantically plausible interpretation. These examples illustrate how semantically reasonable predictions can still be counted as errors in our evaluation.

Algorithm 1 Hybrid CAPTCHA Solver Require: screenshot I 1: B ← detect_UI_elements(I) ▷ YOLOv8n 2: Igrid ← extract_captcha_area(I, B) 3: t,type ← OCR(Igrid ) ▷ Target & type 4: R ← detect_cells(Igrid ) 5: if type = classification then 6: for all r ∈ R do 7: if t ∈ / CYOLO then 8: yr ← VLM_classify(r,t) 9: else 10: yr , pr ← YOLO_classify(r,t) 11: if pr < τ then ▷ τ = 0.70 12: yr ← VLM_classify(r,t) 13: end if 14: end if 15: end for 16: C ← {r : yr = t} 17: else ▷ Segmentation 18: Imerged ← concatenate(R) 19: M ← YOLO_segment(Imerged ,t) or VLM_segment(Imerged ,t) 20: C ← map_to_cells(M, R) 21: end if 22: click(C), wait(200 ms) 23: if is_dynamic() then ▷ Dynamic puzzle detection 24: Inew ← capture_screenshot() 25: while didPuzzleChange(Igrid , Inew ) do 26: Igrid ← extract_captcha_area(Inew , B) 27: t,type ← OCR(Igrid ) 28: R ← detect_cells(Igrid ) 29: if type = classification then 30: for all r ∈ R do 31: yr ← route(r,t) ▷ Eq. 3 32: end for 33: C ← {r : yr = t} 34: end if 35: if C = 0/ then ▷ No targets found 36: break 37: end if 38: click(C), wait(200 ms) 39: Inew ← capture_screenshot() 40: end while 41: end if 42: click(verify_button) 43: return check_success() A PPENDIX H VLM P ROMPT T EMPLATES a) Classification Prompt (3×3 puzzles).: Choose ONLY ONE number from the following list: 0: Bicycle, 1: Bridge, 2: Bus, 3: Car, 4: Chimney, 5: Crosswalk, 6: Hydrant, 7: Motorcycle, 8: Mountain,

20

9: Other, 10: Palm, 11: Stairs, 12: Traffic Light, 13: Boat, 14: Taxi, 15: Tractor Answer with ONLY the number (0-15) of the class you see with the highest confidence. Do not include any explanation.

b) Segmentation Prompt (4×4 puzzles).: Find all {object_name} in this image and create bounding boxes that encompass the entire {object_name}. Requirements: - Include ALL visible parts of the {object_name} - Capture the complete {object_name} from top to bottom and side to side - Make sure the entire {object_name} is contained within the bounding box For each {object_name}, return coordinates: {Object_name} 1: [x1, y1, x2, y2] {Object_name} 2: [x1, y1, x2, y2] Where (x1, y1) is top-left corner and (x2, y2) is bottom-right corner.

A PPENDIX I P IPELINE D IAGRAMS

Fig. 7. Classification puzzle solving pipeline for 3×3 CAPTCHAs. The system extracts 9 individual cells from the detected CAPTCHA area, classifies each cell independently using either YOLOv8 (top-1) or VLM (top-1) and produces binary decisions indicating which cells contain the target object (cars in this example). Both models operate in top-1 prediction, committing to exactly one class label per cell.

Fig. 8. Segmentation puzzle solving pipeline for 4×4 CAPTCHAs. The system extracts and concatenates 16 cells into a unified image, runs either YOLOv8 segmentation (producing pixel-accurate masks or bounding boxes) or VLM (producing bounding boxes only) and maps detected regions back to cell coordinates. Any cell overlapping the predicted region by ≥1% is selected.

A PPENDIX J C OLLECTED C LASS S AMPLES Figures 9–11 show all collected samples for the three unsupported classes (Taxi, Tractor, Boat) used in the distillation experiments (Section VI). Each image is annotated with both YOLO and VLM predictions. Green indicates a correct prediction and red indicates a misclassification. YOLO predictions are always incorrect for these classes since they are outside its supported 13-class. VLM achieves 82.1% recall on Taxi (23/28), 100% on Tractor (20/20), and 100% on Boat (11/11).

21

Fig. 9. All 28 collected Taxi samples with YOLO and VLM predictions. YOLO misclassifies all images (predominantly as Car), while VLM correctly identifies 23 of these 28 samples (82.1% recall). The 5 VLM errors are predicted as Car (3) and Motorcycle (2).

22

Fig. 10. All 20 collected Tractor samples with YOLO and VLM predictions. YOLO misclassifies all images, while VLM achieves 100% recall.

Fig. 11. All 11 collected Boat samples with YOLO and VLM predictions. YOLO misclassifies all images, while VLM achieves 100% recall.

23

Related documents

• CAPTCHAs in the Agentic Era: Solvers That Learn from Every Encounter#924277
• CAPTCHAs in the Agentic Era: Solvers That Learn from Every Encounter#920046
• Generalizing the Turing Test to Interactive Agents#1144796
• Your Guide to Protecting Yourself Against Snitches, Informers, Informants, Agents Provocateurs, Narcs, Finks, and Similar Vermin#199077
• Calibrated Bait: Defensive Information Design under Adversarial Fingerprinting#1063008