arXiv CSopen access

Rehearsed Multi-Agent Live Product Demonstrations with Real-Time Voice Question Answering

Rehearsed Multi-Agent Live Product Demonstrations with Real-Time Voice Question Answering Rahul Khedar∗ , Mayank Malhotra∗ Avinash Karn, Mouli V, Prakhar Mehrotra

arXiv:2606.30294v1 [cs.AI] 29 Jun 2026

PayPal AI

Abstract

1

Live product demonstrations are a recurring, high-cost activity in software organizations: a human presenter must select features, dispatch the corresponding interactions on a running application, narrate them coherently, and answer questions in real time. Existing automation addresses only fragments—generalist browser agents target instruction-conditioned task completion, and demo-video tools produce fixed MP4 artifacts that cannot be questioned and silently break under interface drift. We propose R HETOR, a multi-agent system that takes a running web application and its source-code repository as input and produces a rehearsed live demonstration with segment-synchronized narration and real-time voice question answering. The architectural contributions are a cross-modal feature representation that merges UI exploration with source-code analysis into features tagged with discrete focus tiers, a grounded scripter constrained to UI elements observed during exploration and dispatched through multi-strategy semantic locators, a pre-presentation rehearsal loop with explicit convergence and graceful degradation to narration-only segments, and a runtime synchronization invariant that ties each browser action to the audio-end event of its narration segment. Across six pipeline sessions on four deployed applications—including the public-domain whiteboard application Excalidraw—the rehearser’s internal locator-firing rate σ̄ spans 0.31–1.00 over 147 scripted actions; on the substantial workload (53 actions, full tier differentiation), σ̄ ≈ 0.92, and on the public-domain reference point the locator-repair step drives convergence to σ̄ = 1.00 at iteration 2. We additionally define a benchmark protocol of ten metrics across six application categories that would establish, beyond the case study, whether each design choice contributes positively.

Live product demonstrations are a recurring activity in software organizations across sales engineering, developer relations, customer success, and engineering onboarding. The activity is structurally constrained: in a bounded time window, a human presenter selects a subset of features from an application, executes the corresponding interactions on the running product, narrates the interactions coherently, and responds to audience questions whose subject matter spans both user-facing behavior and architectural detail. The labor cost is substantial, the activity repeats indefinitely across audience rotations, and the produced artifact—an instance of internalized state in the recipient—is invalidated by the next interface change. Two classes of automation address subsets of this problem. Generalist browser agents [1, 2, 3, 4, 6, 5] target instruction-conditioned task completion: given a natural-language goal, the agent attempts to complete the task on a real website, with success measured by outcome. The demonstration problem differs in two respects: the system must select which features to demonstrate rather than be told, and it must verbalize what it is doing rather than only execute it. A second class of tools [10, 11, 12, 13] composes a vision LLM, a browserautomation library, and a text-to-speech model to render narrated MP4 videos from a target URL. The video format produces a fixed artifact: it cannot accept runtime questions, cannot adapt to user pacing, and fails silently when the underlying interface has drifted from its state at render time. Vardanyan [7], reporting on a year of production browser-agent operation, argues that the dominant factor governing real deployment is architectural rather than model-capability-driven; we adopt this observation as a working premise. We propose R HETOR, a multi-agent system for rehearsed product demonstrations of web applications under live runtime conditions. Given as input a running 1

Introduction

application and its source-code repository, R HETOR executes a five-phase pipeline—explore, code-read, understand, script, rehearse—and produces, as output, a rehearsed demonstration that is served to a client browser through a same-origin reverse proxy embedded in the user’s browser, with audio narration synchronized to browser actions through a per-segment handshake and user speech bridged to a real-time speech-to-speech endpoint grounded in a generated knowledge document.

case study. Sections 12 and 13 discuss limitations and design implications.

2

Related Work

Generalist browser agents. WebArena [2] provides reproducible Docker-sandboxed web applications with outcome-based evaluation; VisualWebArena [3] adds a vision dimension. Mind2Web [1] contributes 2,350 tasks across 137 websites with cross-task, cross-website, and cross-domain splits. WebVoyager [4] reports an end-toend multimodal agent and a benchmark of the same name. Recent work [5] re-evaluates state-of-the-art agents on Online-Mind2Web and reports substantially lower numbers than headline scores suggest. The open-weights MolmoWeb [6] releases a vision web agent with full training and evaluation tooling. These systems pursue task-completion under instruction; R HETOR pursues narrative selection and presentation. Action selection occurs at author time (rehearsal) rather than run time, so the live runtime consumes a validated trace rather than acting under uncertainty.

Contributions. (i) We formalize a cross-modal feature representation in which UI exploration and source-code analysis are jointly merged into features tagged with discrete focus tiers T = {HERO, SUPP, MENTION} and a continuous demo priority ρ ∈ [1, 10]; the focus tier operationalizes a narrative attention budget that downstream phases enforce. (ii) We constrain the scripter to ground every action in an element observed during exploration; each action is dispatched through an ordered locator tuple L = (ℓ1 , . . . , ℓ6 ) over the strategies role+name, text, label, placeholder, test-id, and CSS, tried in fixed priority order. (iii) We introduce a rehearse-then-present loop with explicit convergence and degradation: the script is executed in a real headless browser, an LLM proposes locator alternatives for failed actions, locator lists update by left-prepend Li+1 (a) ← Ri (a) ⊕ Li (a), iteration continues while σi < τ up to a fixed cap Imax , and any action that the loop cannot validate is converted to a narration-only segment rather than removed, preserving termination of the runtime trace. (iv) We specify a runtime synchronization invariant: the unit of authoring is the narration segment σ = ⟨text, action?⟩, and the run(σ) (σ) time guarantees taction = taudio_end via a server–client handshake, eliminating the word-offset drift characteristic of TTS-aligned automation under variable provider latency. Beyond the four core contributions, the paper specifies a real-time voice question-answering path that bridges browser PCM to a server-VAD speech-to-speech endpoint with the generated knowledge document injected as session instructions, and proposes a benchmark protocol of ten metrics across six application categories. A preliminary case study on four deployed applications, exercising multiple rehearsal regimes, is reported in Section 11. The remainder of the paper is organized as follows. Section 2 reviews prior work. Section 3 fixes notation. Sections 4 to 8 describe the five phases and the runtime. Section 9 reports implementation details. Section 10 defines the benchmark protocol. Section 11 reports the

Production browser-agent reports. Vardanyan [7] reports a year of production operation of a browser agent and concludes that hybrid accessibility-tree-plus-vision context, specialization over general autonomy, and programmatic safety boundaries determine reliability more than LLM scale does. We adopt this stance and contribute the additional pattern of rehearse-then-present: relocating the agent’s failure surface to a controlled offline phase with a defined degradation rule. Automated demo videos. A recent class of tools [10, 11, 12, 13] maps a URL or YAML specification to a narrated MP4 via the pattern: capture page → visionLLM analysis → script → record browser → TTS → ffmpeg merge. R HETOR differs in three respects: the output is a live, interactive demonstration in the target application served via a same-origin reverse proxy rather than a pre-rendered video; planning consumes both UI and source-code signals, so architectural content (integrations, scalability, data model) becomes addressable demo material; and a pre-presentation rehearsal phase converts a best-effort script into a validated one before deployment in a deployed setting. Speech-to-speech runtimes. Real-time audio APIs [9] expose server-side voice activity detection and direct speech-to-speech generation over WebSocket, eliminating the transcribe–think–synthesize round-trip. We use 2

such an endpoint as the carrier for live voice question answering, with a generated knowledge document injected as session instructions to ground responses in the demonstrated application. State-grounded multi-agent generation. Khedar et al. [8] introduce an authoritative state object that constrains LLM outputs across a multi-role generation loop in the setting of synthetic data generation. We adopt the same principle in a different setting: a non-LLM data structure—the merged SiteMap and CodeAnalysis—constrains LLM outputs across the pipeline phases.

3

i

Strategy ℓi

1 2 3 4

role+name getByRole(role,{name}) text visible text content label associated <label> placeholder input placeholder attr. test-id [data-testid="..."] CSS raw CSS selector (fallback)

5 6

Selector pattern

Table 1: The locator strategy tuple L. Strategies are tried in priority order; the first match becomes ℓ⋆ (t). The ordering favors semantic-role and text-based strategies, which are robust to incidental DOM changes, over CSS, which couples to internal structure.

Preliminaries and Notation

We fix notation used throughout the paper.

Script. A demo script S is a sequence of scenes S = (s1 , . . . , sn ). Each scene s comprises an act act(s) ∈ {HOOK, JOURNEY, HOOD, CLOSE}, an entry path, and a s ). sequence of narration segments Σ(s) = (σ1s , . . . , σm s Each segment is the pair σ = ⟨text, action?⟩, where the action is optional (allowing pure narration).

Application surface. Let G = (V, E) denote the navigation graph of the target application, where V is the set of pages discovered by exploration and E ⊆ V × V is the set of inter-page links. Each page v ∈ V carries a tuple ⟨path, title, type, summary, Ev , Fv , Mv ⟩ where Ev is the set of observed interactive elements, Fv the set of forms, and Mv the set of modals. An element e ∈ Ev has the structured signature ⟨role, text, aria, label , placeholder , testid , bbox ⟩.

Validated actions. After rehearsal, each action a is annotated with a status ϕ(a) ∈ {VERIFIED, FAILED, NARRATION _ ONLY}, the winning locator ℓ∗ (a), and a duration. A scene’s success rate is

Code surface. Let R denote the set of routes extracted from the repository, D the set of data models, and Frw the inferred web framework. Together, the CodeAnalysis object is C = ⟨R, D, Frw , α⟩ with α a free-text architecture summary.

σ(s) =

Objective. The system input is the pair (Uapp , Urepo ) of an application URL and a repository URL. The system output is a tuple ⟨S ∗ , Φ∗ , K ∗ , Uapp ⟩ where S ∗ is a rehearsed script, Φ∗ the validated-action annotation, K ∗ a Markdown knowledge document, and Uapp the same application URL, served live to a client browser with both text and voice question-answering grounded in K ∗ .

Locators. A locator strategy is a function ℓ : (page, element) 7→ {handle, ∅} that returns either a Playwright element handle or fails. We fix a priorityordered tuple of strategies L = (ℓ1 , . . . , ℓ6 ) summarized in Table 1. For an action target t, the winning locator is ℓ (t) = ℓ (t), i∗

∗

i = min{i : ℓi (t) ̸= ∅},

(2)

where A(s) = {a : ∃σ ∈ Σ(s), action(σ) = a}. The overall success script is the size-weighted P rate of the P mean σ̄ = s |A(s)|σ(s)/ s |A(s)|.

Features and tiers. A feature f ∈ F is a tuple ⟨name, Pf , Kf , τf , ρf , θf ⟩Swhere Pf ⊆ V are the pages over which f spans, Kf ⊆ v∈Pf Ev is the set of key elements, τf ∈ T = {HERO, SUPP, MENTION} is the focus tier, ρf ∈ [1, 10] the demo priority, and θf an optional architecture-level note.

∗

|{a ∈ A(s) : ϕ(a) = VERIFIED}| , |A(s)|

4 Phase 1: Exploration and Code Reading Phases 1a and 1b run concurrently. Phase 1a constructs G via a bounded breadth-first crawl of the running application; Phase 1b constructs C from the source repository. The two phases share no state and their wall-clock cost

(1)

or undefined if no strategy matches. 3

parallel (T1 = max(T1a , T1b ))

Phase 1a Explore G

Phase 1b Code Read

C

Phase 2 Understand

Phase 3 Script

F, K

S

Phase 4 Rehearse

⟨S ∗ , Φ∗ ⟩

Phase 5 Present

Figure 1: The five-phase R HETOR pipeline. Phases 1a and 1b execute concurrently and produce the navigation graph G and code analysis C consumed by Phase 2. Edge labels denote the typed artifact passed between phases; full definitions are in Section 3. differs in regime, so end-to-end Phase 1 latency is
T1 = max T1a , T1b ≤ T1a + T1b .

4.1

is performed and no source is written to disk. Without a token, a shallow clone (-depth 50) is used as fallback. The fetched files are partitioned into batches by a path-aware batcher with byte budget B ≈ 60 KB, cohorting files by directory. Each batch is analyzed by an LLM under a structured-output prompt and the per-batch results are merged with route deduplication keyed by method:path. Two additional one-shot calls produce the framework label Frw from the directory tree and the architecture summary α.

(3)

UI exploration

The crawl is bounded by |V | ≤ κp and depth(v) ≤ κd for κp = 30 and κd = 4 by default. Page perception is hybrid: deterministic JavaScript executed in the page context extracts Ev over a selector union (a, button, [role], [data-testid], headings, form controls, labeled inputs), restricted to the visible viewport; a multimodal LLM call then summarizes the page from its 1280×720 PNG screenshot, producing the structured triple ⟨type, summary, name⟩. Page-readiness combines Playwright’s network-idle signal with a JavaScript spinner detector polling at 250 ms intervals. Three behaviors extend the basic crawl. Modal probing clicks up to five candidate triggers per page in fresh contexts to discover dialogs absent from the static crawl. Vision-assisted login resolves authentication walls in the absence of configured credentials by issuing a vision LLM call that identifies a public-area click target, after which a JavaScript tree-walker performs the click; the same component identifies SSO entry points (OAuth, SAML, Auth0, Okta) and persona-card login flows characteristic of enterprise SaaS deployments, which are not handled by a generic password-field heuristic. App-type classification labels the application as SPA, MPA, or hybrid via URL-pattern heuristics; the runtime reverse proxy uses this label.

4.2

5 Phase 2: Cross-Modal Understanding The merge Ψ : (G, C) 7→ (F , K) is a single LLM call under a structured-output prompt; the prompt receives a serialized form of the navigation graph (pages with summaries and key elements) and the code analysis (routes, models, framework). Its outputs are the feature set F (with τf , ρf , Kf , Pf , θf populated for each f ) and the prompt for a second call that produces the Markdown knowledge document K with prescribed sections: Product Overview, Features, Architecture, Common Q&A, Talking Points. The Common Q&A section is generated anticipatorily: the model is prompted to enumerate ten to fifteen plausible audience questions and to produce reference answers grounded in F and C at generation time, rather than at runtime. Pre-computation of the question set is a structural advantage relative to a human presenter, who typically prepares answers reactively. Focus tier as attention budget. Let w(σ) ∈ N denote the word count of segment σ. The system enforces tierconditional bounds at narration time:   [100, 140] if τf (σ) = HERO w(σ) ∈ [80, 110] (4) if τf (σ) = SUPP   [40, 65] if τf (σ) = MENTION

Code reading

When a GitHub access token is configured, the system fetches the repository through the REST API: a single recursive tree request lists all blobs, files are filtered by extension and a directory deny list, and contents are fetched in parallel by a worker pool of size 10. No clone 4

7 Phase 4: Rehearsal with LLM Locator Repair

where f (σ) is the feature associated with σ via its scene. The tier is a coarse, prompt-enforceable signal that prevents an LLM scripter from producing flat, equallyweighted walkthroughs.

Iteration. At iteration i, every action a in S is dispatched in a real headless Chromium browser; the executor tries strategies in L in priority order and returns ℓ∗ (a) or ∅. Each a is annotated with ϕi (a) ∈ {VERIFIED, FAILED}, the winning locator, the postaction bounding box, the duration, and an error message if any. Let σi = σ̄ be the overall success rate after iteration i.

Knowledge index. K is partitioned into sections {c1 , . . . , cN } by splitting on level-2 headers. Each section is embedded once via an OpenAI-compatible embedding endpoint. Retrieval at query time uses cosine similarity eq · ec s(q, c) = , (5) ∥eq ∥2 ∥ec ∥2 with a keyword-overlap fallback when an embedding endpoint is unavailable. The retrieval context for query q is X R(q) = argmaxC⊆Topk (q), ∥C∥≤Bq s(q, c), (6)

Repair. For each a with ϕi (a) = FAILED, an LLM call under a structured-output prompt proposes new locator strategies Ri (a). The action’s locator list is updated by left-prepend

c∈C

Li+1 (a) ← Ri (a) ⊕ Li (a),

where Bq is a character budget (default 4000) and ∥C∥ is the total character length of the selected sections.

6

which preserves the original strategies as fallbacks. Iteration continues while

Phase 3: Grounded Demo Scripting

σi < τ ∧ i < Imax ,

The scripter produces S under the structural constraint act(s1 ) = HOOK, act(sn ) = CLOSE,

(10)

with default convergence threshold τ = 0.95 and iteration cap Imax = 3.

(7)

with act taking values JOURNEY for |act| = 3–6 middle scenes and HOOD for 1–2 technical scenes drawn from θf and α. The grounding constraint is the predicate [ [ ∀σ ∈ Σ(s), action(σ) ̸= ∅ ⇒ target(σ) ∈ Ev , s

(9)

Degradation. On termination at i∗ = Imax with σi∗ < τ , the residual failed actions are not removed from S. Instead the rule

v

(8) i.e., every action targets an element observed during exploration. The constraint is enforced both in the prompt (the LLM is given the element list) and in postprocessing: an unmatched target is downgraded to a single text-based locator and flagged for rehearsal scrutiny. The action vocabulary is

ϕ(a) := NARRATION _ ONLY

for all a with ϕi∗ (a) = FAILED (11) is applied, and the script is marked ready when σi∗ ≥ 0.7. At runtime, segments whose action is NARRA TION _ ONLY play the spoken text and skip the browser interaction. This ensures completion of the live presentation under any failure mode reachable via the rehearsal loop.

V = {navigate, click, fill, hover, scroll, wait, highlight}.

Action success at the script level. We use actionsuccess-at-1, mirroring the locator priority order:

highlight draws a temporary outline and scrolls the target into view; the rehearser uses it for narration alignment and the runtime uses it as a presenter pointer.

Succ@1(S) =

1 X 1[ϕ(a) = VERIFIED]. (12) |A| a∈A

Realistic data. For fill actions, the scripter invokes a small structured-output LLM call that generates plausible demo values (names, dates, currency, descriptive text) rather than placeholder strings; this is qualitatively observable in user perception even when not measurable in success-rate metrics.

?? 1 summarizes the full procedure.

8

Phase 5: Live Presentation Runtime

The runtime serves the rehearsed demo as an interactive experience. Server-side state is held by a per-session 5

Iteration i Run all actions

Algorithm 1: Rehearse-then-Present Input: script S, locator strategies L, threshold τ , cap Imax Output: rehearsed script S ⋆ with status map Φ⋆ 1 for i ← 1 to Imax do 2 for a ∈ A(S) do 3 h ← ℓ⋆ (a) // first matching locator in L 4 if h ̸= ∅ then 5 execute a on h 6 ϕi (a) ← VERIFIED 7 else 8 ϕi (a) ← FAILED 9 end 10 end 11 σi ← Succ@1(S | ϕi ) 12 if σi ≥ τ then 13 return ⟨S, ϕi ⟩ with ready ← T 14 end 15 foreach a with ϕi (a) = FAILED do 16 Ri (a) ← LLM-R EPAIR(a, err (a)) 17 Li+1 (a) ← Ri (a) ⊕ Li (a) // prepend 18 end 19 end 20 foreach a with ϕImax (a) = FAILED do 21 ϕ(a) ← NARRATION _ ONLY // degrade 22 end 23 ready ← [σImax ≥ 0.7] 24 return ⟨S, ϕ⟩

yes

ready save

no

yes, i ← i+1

Repair Ri (a) Li+1 ← Ri ⊕ Li

i < Imax ? no

ϕ(a) := NAR . ONLY

Figure 2: Rehearsal repair loop. Convergence at σi ≥ τ ; otherwise, repair and iterate up to Imax , then degrade residual failures to NARRATION _ ONLY. event and clears an internal narration_event. The client begins TTS playback while the server blocks on narration_event.wait(∆max ). The client’s audio onended fires a narration_done message that releases the wait, and the server then emits the action event. Provided ∆max exceeds the maximum permitted audio length, the invariant holds across variable TTS latency, network jitter, and prosody-induced length variation, yielding zero word-offset drift across an arbitrary number of segments.

8.2

Three-tier TTS fallback

The narration audio path uses a three-tier fallback chain to preserve Invariant 1 under provider variability. Let S1 denote streaming TTS over an OpenAI-compatible /audio/speech endpoint, S2 non-streaming TTS over the same endpoint, and S3 the browser-native speechSynthesis. The runtime selects the lowestindex source that emits an onended event; the handshake protocol is identical across all three. This decouples the synchronization argument from any specific TTS provider.

DemoSession containing ⟨S ∗ , Φ∗ , K ∗ , Uapp ⟩ and a DemoDriver state machine; a reverse proxy serves the original target application under a same-origin path so it can be embedded in an iframe and dispatched against from the browser DOM with the same locator strategies the rehearser validated.

8.1

σi ≥ τ ?

Synchronization invariant (σ)

For each segment σ = ⟨text, a⟩ with a ̸= ∅, let taudio_end denote the wall-clock time at which the segment’s nar(σ) ration audio finishes, and taction the wall-clock time at which the action is dispatched in the iframe. The runtime preserves:

8.3

Real-time voice question answering

User speech is bridged from the browser microphone to a server-VAD speech-to-speech endpoint via a WebSocket relay. The client opens a control channel to the server; the server retrieves a knowledge slice Kq∗ ⊆ K ∗ of bounded length and establishes a second WebSocket to the speech-to-speech endpoint with a session configured for PCM16 input and output, both text and audio modal-

Invariant 1 (Segment synchronization). For every seg(σ) (σ) ment σ with non-null action, taction = taudio_end . The invariant is maintained by a segment-completion handshake (Figure 3). The server emits a narration 6

Server

Multilingual interaction. The Realtime endpoint accepts arbitrary input languages via the configured input transcription model and produces output in the same language; narration generation is language-agnostic at the LLM interface. The runtime is therefore multilingual at the system level; localization to a specific language requires per-language voice and prompt selection but no architectural change.

Client narration event

emit narration

receive

clear event

play TTS

time

block on event audio ends

release event

done event

emit done action event

emit action

dispatch

8.4

The runtime mounts Uapp under the same origin as the demo client via an HTTP reverse proxy that strips X-Frame-Options and CSP headers from proxied responses and rewrites Location redirects to same-host paths. Same-origin embedding admits direct DOM dispatch into the iframe with the locator strategies validated during rehearsal.

Figure 3: Segment-completion handshake. The server emits a narration event, blocks until the client plays the TTS audio to completion and signals back, and only then emits the action event. The protocol encodes Invariant 1: action dispatch is anchored to the audio-end signal rather than to a predicted timestamp.

8.5

Recorded demo packages

The implementation supports an offline package format for asynchronous viewing: per-scene MP3 audio rendered through the same TTS chain, screenshot frames from rehearsal, and JSON metadata aligning frames to narration segments. The recorded path is structurally distinct from the live runtime; it preserves narrative content but loses dialog, which is the property that distinguishes a live demonstration from a video.

ities, server-side voice activity detection with a fixed threshold and silence-duration parameter, an input transcription model, a configured voice, and an instruction string I = I0 ∥ Kq∗ where I0 is a generic role preamble. Browser-captured audio frames are forwarded as appended buffer events; the streamed audio response is relayed back to the browser for playback.

9 Turn latency. Per audience turn, Tturn = Tup + Tvad + Tstream + Tdown ,

Iframe and reverse proxy

Implementation

The system is implemented in approximately 6,000 lines of Python with a Flask web layer and Playwright for browser control, plus a thin browser-side client. The agent layer is structured under a common contract: each agent extends a base class returning an AgentResult with success, typed payload, error, elapsed milliseconds, and a metadata dictionary; per-phase wall-clock latency is captured by a timing wrapper, and a token-usage field is reserved for cost accounting at the LLM-client boundary.

(13)

where Tup is end-of-speech to server VAD acceptance, Tvad the configured silence padding, Tstream the LLM-tofirst-audio latency, and Tdown the audio-stream playback start. Compared to a transcribe–think–synthesize REST pipeline, this path eliminates two full network roundtrips and one audio re-encode.

Provider-agnostic LLM client. The LLM client is a thin adapter over an OpenAI-compatible HTTP surface (/chat/completions, /audio/speech, /audio/transcriptions, /embeddings) plus a Realtime WebSocket. Endpoint base URL and bearer key are environment-driven; the client supports both standard chat parameters and reasoning-model parameters with automatic switching by model name; a configurable TLS-verify flag accommodates internal gateways. A deployment can route every LLM call to its preferred provider (direct OpenAI, self-hosted inference,

Because K ∗ is constructed from both G

Grounding. and C, audience questions over user-facing behavior and over architectural detail are answered from the same retrieval substrate. Let Q denote the set of questions issued during a presentation session. We partition Q into a UI-grounded subset Qu (questions answerable from G) and a code-grounded subset Qc (questions answerable only from C); a strict subset Qu∩c requires both. The cross-modal merge is a precondition for any grounded answer over Qc and Qu∩c . 7

or a multi-provider gateway routing to Claude, Gemini, or open-weight models) without modifying any agent. Table 2 reports an in-house benchmark of the same LLM client across six models and three tasks (narration, fast Q&A, code analysis), three runs per cell, with zero errors across 54 invocations and an inter-cell mean-latency spread of 1.7–7.9 s. The same model under two distinct endpoint base URLs (last two rows of Table 2) yields comparable latencies, supporting the claim that the client runs unmodified across endpoints.

Symbol Parameter

Default

κp Crawl page cap (Phase 1a) 30 κd Crawl depth cap (Phase 1a) 4 B Code batch byte budget (Phase 1b) 60 KB |T | Number of focus tiers 3 ρ Demo priority range [1, 10] |V| Action vocabulary size 7 |L| Locator strategies 6 τ Rehearsal convergence threshold 0.95 Imax Rehearsal iteration cap 3 τready Degraded-readiness threshold 0.70 Bq Q&A retrieval char budget 4000 Model Endpoint Narration Fast Q&A Code ∆max Narration handshake timeout 60 s anal. Per-phase model assignment (defaults) Gemini 2.0 Flash Gateway A 1.81 1.69 3.04 Code analysis, understanding, scripting, deep Q&A reasoning Gemini 2.5 Flash Gateway A 2.90 2.65 7.89 Vision summaries, action repair, narration, fast Q&A fast MM Claude Haiku 4.5 Gateway A 2.60 2.00 2.36 Text-to-speech, transcription, embeddings, realtime dedicated Claude Sonnet 4.6 Gateway A 3.80 3.58 4.42 GPT-4.1 Mini Gateway A 2.41 2.38 3.57 GPT-4.1 Mini Direct 2.75 2.05 3.60 Table 3: Default configuration of R HETOR. All values

are configurable via environment variables; the symbols correspond to the formal definitions in Sections 3, 5, 7 and 8.

Table 2: Mean latency (seconds) per LLM call across six models and three tasks, three invocations per cell (54 invocations total, zero errors). Gateway A is a multiprovider routing service; Direct is a direct OpenAIcompatible endpoint. The latency spread across models and tasks motivates the mixed-tier model assignment.

⟨phase, message, percent⟩ records, a demo page hosting the live viewer, and a player page for recorded packages. The REST surface covers scene retrieval, narration (blocking and streaming), text question answering with fast and deep mode selection, text-to-speech (blocking and streaming), transcription, and capability flags. Two WebSocket channels carry stateful flows: a playback channel between the runtime driver and the browser viewer, and a question-answering channel between the browser microphone and the speech-to-speech relay. Pipeline outputs are persisted under a per-project directory keyed by an MD5-derived slug, and a load endpoint restores any saved project as a new session.

Mixed-tier model assignment. Reasoning-heavy phases (code analysis, feature merging, deep Q&A, scripting) default to a reasoning model; latency-sensitive paths (vision summaries, action-loop decisions, locator repair, narration, fast Q&A) default to a fast multimodal model; speech and embeddings are served by dedicated endpoints. Reasoning-model detection is heuristic over a set of prefixes, automatically switching the request to use max_completion_tokens and reasoning_effort parameters. Table 3 lists the default configuration. Structured outputs, parallelism, streaming. The client exposes JSON-schema-enforced structured output, parallel batch evaluation through a configurable thread pool (default 4 workers, used by the code-batcher of Section 4.2), and three streaming modes: token streaming over chat completions, chunked TTS streaming, and event streaming for the Realtime session.

Tests. The unit-test suite contains 13 files and 129 tests covering the six agents (including success, repair, and degradation paths for the rehearser), the LLM client (including reasoning-model payload construction and JSON-fence stripping), the configuration loader, the GitHub API client (URL parsing for HTTPS / SSH / GitHub Enterprise, tree filtering, blob fetch, retry-onrate-limit), the auth parser, and the HTTP routes. External APIs and Playwright are mocked at the boundary, allowing the suite to run without network access.

Web surface. The web layer exposes a setup page with Server-Sent-Events progress streaming over 8

TTS API

playback

server browser

DemoDriver (state machine)

HTTP

ReverseProxy

Browser client

DemoSession ⟨S ∗ , Φ∗ , K ∗ ⟩

iframe (Uapp )

QAHandler (text + voice)

TTS audio + mic voice Q&A

Realtime API

Figure 4: Live runtime architecture. The target application is served back to the client under the same origin via the reverse proxy and embedded in an iframe so that the demo client can dispatch actions in its DOM. The driver controls playback over a WebSocket channel; a separate channel relays user speech to a speech-to-speech endpoint with a knowledge slice Kq∗ injected as session instructions.

10

Benchmark Protocol

A live demo has more than one thing to be good at, and a single aggregate score over it is misleading. The protocol below specifies ten metrics across four families and a six-category application corpus, and it includes two ablations designed to isolate the contributions of cross-modal merging and of the rehearsal phase. We have not yet run the full protocol; we describe it here so it is reproducible by others, and report what we have measured so far in Section 11.

10.1

Application

Auth

CRUD

Cal.com Ghost Plausible Supabase Studio NocoDB Directus Mattermost Excalidraw Penpot Appwrite

gated gated gated gated gated gated gated public gated gated

Dashboard CMS / records Communications Design Data platform

Application corpus

Table 4: Representative ten-application corpus stratified across six categories. The corpus targets coverage of authentication regimes and interaction surfaces rather than scale.

Let D be a corpus of open-source web applications stratified across six categories: form-heavy CRUD, dashboards, content management, communications, design tools, and data infrastructure. Table 4 lists a representative cohort. Each application is paired with its source repository; corpus items are selected to span (a) framework family (React/Next, Vue/Nuxt, Django, Rails templates), (b) authentication regime (public, gated with credentials, gated without), (c) interaction richness (formdriven, dashboard, canvas-heavy), and (d) repository size.

10.2

Category

tracted by Phase 1b and reference routes R∗ enumerated from the framework’s routing manifest, route recall is RR = |R̂ ∩ R∗ |/|R∗ |. Definition 3 (Feature-extraction precision and recall). For features F̂ extracted by Phase 2 and a humanannotated reference F ∗ keyed by name and entry point, Pf = |F̂ ∩ F ∗ |/|F̂| and Rf = |F̂ ∩ F ∗ |/|F ∗ |, with macro-average F1 over the corpus.

Pipeline-level metrics

Definition 1 (Crawl coverage). For a discovered navigation graph Gd and the reference set of user-reachable pages Gr (manually curated), crawl recall is CR = |V (Gd ) ∩ V (Gr )|/|V (Gr )| subject to the crawl budget κp .

10.3

Rehearsal-level metrics

Definition 4 (Action success at iteration i). Succi (S) = 1 P a∈A 1[ϕi (a) = VERIFIED ]. |A| Definition 5 (Repair return on iteration). RoI(i → i+ 1) = Succi+1 (S) − Succi (S).

Definition 2 (Code-extraction recall). For routes R̂ ex9

Definition

6

(Degradation

rate). DR(S)

=

Status. The corpus run over Table 4 has not been completed; we report only the six-session preliminary case study of Section 11. The corpus-level numbers and perapplication artifacts will appear in a follow-up revision.

|{a:ϕ(a)=NARRATION _ ONLY}| . |A|

10.4

Runtime-level metrics

Definition 7 (Live action latency). The wall-clock time from the segment-completion handshake to the dispatched DOM action, measured client-side, summarized by median and 95th-percentile across all segments in a presentation.

11

Independent of the corpus-level protocol of Section 10, we report a feasibility case study from running the pipeline end-to-end on four deployed web applications: Application A, an internal enterprise HR/talentmanagement tool (three sessions of the present implementation, varying crawl parameters); Application B, an internal multi-project synthetic-data-generation research framework with a Flask web dashboard (one session of the present implementation); Application C, an internal AI-platform governance tool (one session of an earlier version of the same multi-agent architecture, retained from saved artifacts); and Application D, the public open-source whiteboard application Excalidraw1 (one session of the present implementation, public-domain reproducible reference point). The case study establishes that the pipeline executes end-to-end across diverse deployed applications including a public-domain target and a canvas-rendered interaction surface, and that the rehearsal repair loop contributes empirically to reaching convergence in at least one observed run. It is not a validation of the individual design choices, which require the protocol of Section 10; we are explicit about this distinction below.

Definition 8 (Synchronization drift). For each segment (σ) (σ) σ, ∆(σ) = taction − taudio_end . Invariant 1 predicts ∆(σ) = 0 up to network jitter; we report the empirical distribution. Definition 9 (Voice Q&A turn latency). Tturn as decomposed in Eq. 13; reported per category and per question class (UI-grounded, code-grounded, or both). Definition 10 (Q&A grounding accuracy). For a curated set of questions Q per application with expert-graded reference answers, the proportion of system answers judged correct and grounded on a 3-point scale by a blind reviewer, partitioned by Qu , Qc , Qu∩c .

10.5

Ablations

The protocol includes two ablations. A1: Rehearsal on/off. Compare live Succ@1 when the runtime consumes the unrehearsed Phase 3 script versus the rehearsed Phase 4 script. Hypothesis: rehearsal raises live action success on every category and the gap widens with interaction richness.

11.1

Per-session observations

Table 5 reports the per-application aggregates; persession detail follows in prose, because the figures are heterogeneous and each requires context. We summarize the six sessions in prose below because the per-session figures are heterogeneous and each requires interpretation. Application D (Excalidraw) is the public-domain reference point and the only session in which the rehearsal repair loop is observed to contribute to convergence. The session produced a 7-scene script (1 HOOK + 4 JOURNEY + 1 HOOD + 1 CLOSE) on the canvas-rendered singlepage surface, with all three tiers populated (3 HERO + 3 SUPP + 1 MENTION ) and 14 scripted actions. Iteration 1 did not converge at τ ; the locator-repair step proposed alternatives and iteration 2 reached σ̄ = 1.00 with all 14 actions verified. Coherent demo structure on a canvasheavy SPA, full tier differentiation despite a one-URL

A2: Cross-modal vs UI-only. Compare Pf , Rf and the code-grounded Q&A accuracy on Qc ∪ Qu∩c between the full pipeline and a variant that drops Phase 1b. Hypothesis: removing code analysis collapses codegrounded Q&A accuracy and reduces feature-extraction recall on technical features (integrations, data model, scaling), while leaving UI-grounded Q&A approximately unchanged.

10.6

Preliminary Case Study

Reporting

For each application, we propose to release the full site_map.json, script.json, rehearsal.json, and knowledge.md, together with the per-iteration success rates, the action-by-action validation log, and the per-question Q&A grade. Aggregates are reported as macro-averages over D with 95% confidence intervals from bootstrap resampling.

1

https://excalidraw.com, source at //github.com/excalidraw/excalidraw.

10

https:

App

Description

D C A B

Excalidraw (public OSS) AI-platform governance HR/talent-management Synthetic-data dashboard

Sess.

|A|

σ̄

Iter

Tiers populated

Regime

1 1 3 1

14 53 16–22 ea. 22

1.00 0.92 0.31–0.55 1.00

2 3 3 1

H+S+M H+S+M varies H only

converged via repair near-convergence degraded (Imax ) small-surface easy case

Table 5: Per-application case-study aggregates over six rehearsal sessions on four deployed applications. |A|: scripted actions. σ̄: rehearser internal locator-firing rate (not an external success measure). Iter: rehearsal iterations to terminate. Tiers populated: H=HERO, S=SUPP, M=MENTION. Regime: which control-flow branch of Algorithm 1 the run took. Application D (public-domain Excalidraw, convergence reached via the repair step at iteration 2) and Application C (substantial workload with full tier differentiation, near-convergence) are the strongest evidentiary points; Application A (degraded regime) and Application B (small-surface easy case) are reported in context. NEY + 1 HOOD + 1 CLOSE ), all tagged HERO in the absence of feature differentiation under a small surface; all 22 scripted actions verified at iteration 1 (σ̄ = 1.00), and the repair loop and degradation rule were therefore not exercised in this run. The session is interpreted as the small-public-surface easy case rather than as evidence of reliable convergence.

crawl, and direct evidence that the repair loop drives convergence make this the strongest single session. Application C is the strongest substantial-workload session: a single run on an internal AI-platform governance tool produced an 11-scene script with 53 scripted actions and proper four-act distribution (2 HOOK + 7 JOURNEY + 1 HOOD + 1 CLOSE ) and proper tier differentiation (6 HERO + 2 SUPP + 3 MENTION). Of the 53 actions, 49 verified and 4 degraded under the rule of Section 7, yielding an internal locator-firing rate σ̄ ≈ 0.92 and a readiness flag of 0.92. The session is the most substantial workload in the case study and approaches the convergence threshold τ = 0.95 on a non-trivial action count.

11.2

Action latency

Across the 6 sessions and 147 scripted actions, the empirical action-duration distribution is bimodal: short actions (wait, scroll, in-DOM click) complete in milliseconds, while navigate and fill actions on slow pages occupy a long tail extending to the execution timeout. Per-session rehearsal wall-clock time was 10–41 s for Application A, approximately 24 s for Application B, and approximately 36 s for Application D over its two iterations, dominated by browser action latency rather than by LLM repair calls.

Application A comprises three sessions of the present implementation on an internal HR/talent-management tool, run with different crawl parameters. Across these sessions the pipeline reached 1, 4, and 29 pages respectively, produced 7, 9, and 11 scenes with 16, 22, and 20 actions, and recorded σ̄ values of 0.31, 0.41, and 0.55. All three sessions exited at Imax without converging at τ and entered the degradation path of Section 7. The 1-page session reflects a single-page-application crawl reaching one URL whose elements drove the script; we note this because the page count alone is not an exploration failure when the application is an SPA. The 4-page session produced a knowledge document of 125 sections, a ratio of about 31 sections per crawled page that suggests K ∗ extends beyond what exploration directly grounded; this is an open empirical question that Definition 10 would answer and is the kind of behavior the protocol is designed to surface.

11.3

Validated deployment

In addition to the runs reported above, an earlier integration of the same architecture operates within Application B as an in-tree component of its web layer. The integration provides the same runtime surface as the standalone system—script retrieval, narration, text and voice question answering, text-to-speech, and transcription— together with a real-time speech-to-speech path, and is reachable from the application home page through a toplevel navigation link. The deployment serves an authored knowledge document of 1,440 lines partitioned into 19 sections, and has been exercised in repeated presentation sessions on Application B in which voice queries were handled through the segment-completion synchronization protocol of Section 8.1 and the speech-to-speech path of Section 8.3. The standalone system described in this paper is the application-agnostic generalization

Application B comprises one session of the present implementation on a synthetic-data-generation research dashboard. The crawl reached 2 pages of the public surface; the script contained 8 scenes (1 HOOK + 5 JOUR 11

of that integration, sharing the same data structures, the same locator priority L, and the same synchronization invariant.

11.4

ment that would isolate the contribution of source-code analysis to feature extraction and to code-grounded Q&A accuracy. We have not run it. The case study therefore does not justify the cross-modal merge over a UI-only baseline.

What this case study establishes

The case study supports four narrow claims. First, the pipeline executes end-to-end without operator intervention on four deployed applications differing in framework, interaction surface, and authentication regime, including a public-domain canvas-rendered reference point (Application D, Excalidraw). Second, on a substantial workload (Application C, 53 actions, full tier differentiation), an internal locator-firing rate of σ̄ ≈ 0.92 is reached, with the residual handled by the degradation rule. Third, the locator-repair step empirically drives convergence in at least one observed run: Application D fails to converge at iteration 1, the repair step proposes alternatives, and iteration 2 reaches σ̄ = 1.00. Fourth, the wall-clock cost of rehearsal is dominated by browser action latency rather than LLM cost: action durations are bimodal, and per-session totals are 10–41 s for Application A, approximately 24 s for Application B, and approximately 36 s for Application D over its two iterations.

11.5

Output quality is unmeasured. σ̄ measures whether the rehearser’s locators fired; it does not measure whether the script presents the right features in the right order with appropriate narration. Feature-extraction precision and recall (Definition 3) require a human reference F ∗ which we have not yet annotated. Demo coherence and coverage are not measured. A session with low σ̄ on a well-covered crawl and a session with high σ̄ on a one-page crawl can in principle be ranked in opposite directions by the metric reported here and by a downstream quality measure; the case study cannot distinguish these. Q&A grounding accuracy is unmeasured. The voice and text Q&A paths retrieve from the generated knowledge document K ∗ but the case study does not measure whether retrieved answers are factually grounded in F and C. Definition 10 specifies the protocol; we have not curated the question set Q or rated answers. The high section count for Run A.3 (125 sections from 4 crawled pages) is a flag for this future work: it suggests K ∗ may extend beyond what exploration directly grounded, which is precisely what Definition 10 would catch.

What this case study does not establish

The case study does not validate the individual design contributions; we are explicit about each gap because each maps to a specific item in the protocol of Section 10.

Convergent-regime behavior is observed only on two sessions. Application B reaches σ̄ = 1.00 on a small public surface and Application C reaches σ̄ ≈ 0.92 on a substantial surface. These are encouraging single-session observations, not evidence that convergence is reliable across applications; the protocol of Section 10 is required to characterize the convergence rate and its dependence on application complexity over the corpus.

Repair-loop contribution is partially measured. Application D establishes one observed instance in which the repair step drives convergence (iteration 1 below τ , iteration 2 at σ̄ = 1.00). However, the per-iteration Succi trajectories were not retained for the case-study sessions, so the per-iteration return-on-iteration RoI(i → i + 1) (Definition 5) is not reported here, and no rehearsal-onvs-off comparison (Ablation A1) is reported either. Application A’s three sessions terminated at Imax without converging, establishing that repair did not reach τ on those workloads but not separating “no improvement per iteration” from “insufficient iterations.” Application B converged at iteration 1 (no repair exercised) and Application C terminated at Imax with σ̄ ≈ 0.92. Quantifying the magnitude of the repair contribution and characterizing its dependence on application complexity requires the runs specified in Section 10.5.

Summary. The case study demonstrates that the pipeline runs and that the artifacts described in Sections 3 to 7 are produced. Whether the produced artifacts are good demonstrations, whether each design choice contributes positively, and whether the reported σ̄ predicts downstream quality remain open and require the protocol of Section 10. We commit to running and reporting that protocol in subsequent revisions.

12

Limitations

Crawl budget. The default κp = 30, κd = 4 are calibrated for representative web-application surfaces. Deep

Cross-modal contribution is unmeasured. Ablation A2 (full pipeline vs. Phase 1b removed) is the experi12

enterprise applications with many gated workflows are systematically under-explored without per-application tuning.

those discussed in [7]. The system is intended for firstparty use on the operator’s own product; a hosted multitenant deployment requires an additional trust layer that we do not address.

Authentication. Vision-assisted login resolves unauthenticated public-area exploration. Authenticated demos require an injected session cookie, scripted credentials, or a bearer token; each requires operator setup.

13

Discussion

Rehearse-then-present as a design pattern. The architectural contribution of Phase 4 is the relocation of the agent’s failure surface from runtime to a controlled offline phase. The runtime consumes a validated trace, supplemented by an explicit degradation fallback for any action the rehearsal loop could not validate. The pattern generalizes beyond demonstrations to any browser-agent setting in which the marginal cost of a runtime failure exceeds the marginal cost of an offline retry: scheduled customer walk-throughs, training-environment scripted lessons, and reviewer-facing executive demonstrations are immediate examples. No new learning algorithms are required for the pattern; the implementation cost reduces to a pre-flight execution phase and a typed degradation rule.

Canvas-rendered surfaces. Applications whose primary interaction surface is HTML5 canvas (diagram editors, design tools) are difficult: the DOM does not expose interactive elements and the locator priority order in L is uninformative. Vision-only locators are a separate research direction. Repair under partial observability. The current locator-repair LLM call passes target metadata and an error message but not the post-failure DOM. Conditioning the repair prompt on a structured DOM extract is a low-cost extension expected to materially improve repair RoI.

Segment as the synchronization unit. Word-offset synchronization fails under variable TTS latency and under prosody-driven departures from the underlying text. The segment-completion invariant of Section 8.1 replaces a continuous timestamp-estimation problem with a discrete server–client handshake whose correctness follows from the unit of authoring rather than from real-time signal estimation. The invariant is independent of the specific TTS provider (Section 8.2) and applies to any system in which spoken narration is paired with discrete actions on a controllable surface.

Single-shot understanding. Phase 2 is a single noniterative LLM call without schema-validation re-prompt. Malformed or under-specified outputs degrade gracefully into missing demo content rather than surfacing as explicit errors. Multilingual surface area. The voice Q&A path is multilingual end-to-end and narration generation is language-agnostic at the model interface. The remaining English-specific surfaces—the REST transcription path’s default language and the default narration prompt—are reconfigurable per request, but a fully localized presentation surface (per-language voice presets, locale-specific narrative patterns) is a deployment task.

Cross-modal grounding for narrative planning. Prior work that combines code analysis with UI signals does so to inform task-time execution. R HETOR uses the same combination upstream, at planning time, to determine which features are demonstrated, what architectural content is foregrounded in Act 3, and which subset Qc ∪ Qu∩c of audience questions is answerable from the merged knowledge document. The contribution is the placement of the merge in the planning phase rather than in the execution phase.

Generalization. The pipeline assumes a serverrendered or SPA web application accessible at a URL with a parallel public source repository. Mobile, native desktop, and closed-source SaaS targets are out of scope; each is a separate adaptation of the same multi-phase pattern.

Knowledge document as a shared retrieval substrate. Both narration generation (Section 6) and runtime question answering (Sections 8.3 to 8) retrieve from the same Markdown document K ∗ . This sharing is the structural

Safety, privacy, and consent. A demo agent that crawls a target application and reads its source code raises consent and exfiltration concerns analogous to 13

reason that the runtime’s spoken responses are consistent with the script: the agent answers from the same representation it presents from. The voice path differentiates the runtime from a video player by converting the user-side interaction model from passive playback to grounded interruption.

14

[3] J. Y. Koh et al. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. arXiv:2401.13649, 2024. [4] H. He et al. WebVoyager: Building an end-to-end web agent with large multimodal models. arXiv:2401.13919, 2024. [5] T. Xue et al. An illusion of progress? Assessing the current state of web agents. arXiv:2504.01382, 2025.

Conclusion

[6] Allen Institute for AI. MolmoWeb: An open agent for automating web tasks. Technical report, 2026.

R HETOR is an end-to-end system for rehearsed live demonstrations of web applications. The pipeline executes UI exploration in parallel with source-code analysis, merges both signals into features tagged with discrete focus tiers, generates a four-act narrative whose actions reference observed UI elements, and validates each action in a real headless browser before deployment. At runtime, the rehearsed script is served through a sameorigin reverse proxy embedded in the user’s browser; narration audio is synchronized to browser actions through a per-segment handshake; and user speech is routed to a server-VAD speech-to-speech endpoint grounded in the generated knowledge document. The paper formalizes four design choices—cross-modal feature representation, grounded scripting with semantic locators, rehearse-thenpresent with explicit convergence and degradation, and a segment-completion synchronization invariant—and proposes a benchmark protocol comprising ten metrics across six application categories. A six-session case study spanning four deployed applications exercises multiple rehearsal regimes and supports the claim that the design preserves runtime completeness across the convergence threshold.

[7] A. Vardanyan. Building browser agents: Architecture, security, and practical solutions. arXiv:2511.19477, 2025. [8] R. Khedar et al. State-grounded multi-agent synthetic data generation for tool-augmented LLMs. arXiv:2606.16307, 2026. [9] OpenAI. Realtime API: speech-to-speech over WebSocket. Technical documentation, 2024. [10] Y. Madkour. DemoPilot: autonomous demo video agent. GitHub, 2026. [11] N. Holas. LooK: one-command product demo videos. GitHub, 2026. [12] NeuraScreen: JSON-driven demo video generator. GitHub, 2026. [13] F. Mathieu. DemoDSL. GitHub, 2026.

Reproducibility The implementation, configuration, and prompts are organized to be reproducible from pip install -e . plus playwright install chromium. Default models are configurable via environment variables; the system runs unmodified against any OpenAI-compatible API endpoint, including local inference servers and multiprovider gateways.

References [1] X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su. Mind2Web: Towards a generalist agent for the web. In NeurIPS Datasets and Benchmarks, 2023. arXiv:2306.06070. [2] S. Zhou et al. WebArena: A realistic web environment for building autonomous agents. arXiv:2307.13854, 2024.

14