Towards Expert-level Medical AI for Real-time Video Consultations

2026-08-11

Towards Expert-level Medical AI for Real-time Video Consultations Mahvish Nagda∗,1 , Jihyeon Lee∗,1 , Matthew Thompson1 , Chunjong Park2 , Tim Strother2 , Valentin Liévin2 , Roma Ruparel1 , Akshay Goel1 , Teya Bergamaschi1 , Suhana Bedi1 , Meet Shah2 , Pavel Dubov2 , Liviu Panait2 , Toshiyuki Fukuzawa2 , Sam Schmidgall2 , Craig Schiff1 , Joseph Xu1 , Aliya Rysbek2 , Yana Lunts2 , Jan Freyberg2 , Rebecca Hemengway1 , Sunny Virmani1 , David Racz2 , Carey Radebaugh2 , Joëlle Barral2 , Kavi Goel1 , Dale R. Webster1 , Katherine Chou1 , Avinatan Hassidim1 , Yossi Matias1 , James Manyika1 , Gregory Wayne2 , Tao Tu2 , Yun Liu1 , Ethan Goh1 , Christina Chen1 , Ryutaro Tanno2 , Po-Hsuan Cameron Chen1 , Mike Schaekermann†,1 and Anil Palepu†,1

arXiv:2608.09861v1 [cs.AI] 10 Aug 2026

* Equal contributions, † Equal leadership, 1 Google Research, , 2 Google DeepMind

Audio-visual interaction is the standard for patient-physician consultations, enabling natural communication and effective assessment of illness through non-verbal cues. While text-based AI has shown promise, it discards essential perceptual dimensions and limits patients who cannot articulate symptoms in writing. Early efforts to extend medical AI to audio-visual interaction have demonstrated feasibility, but not reached clinician-level performance. Here, we provide the first demonstration of expert-level AI in real-time clinical video consultations using AMIE (Articulate Medical Intelligence Explorer) in a video configuration. AMIE (Video) is a Gemini-based multi-agent system integrating low-latency dialogue, clinical reasoning, and real-time audio-visual perception. To guide development, we established a taxonomy and automated evaluations for clinical audio-visual cues in telehealth settings. In a randomized Objective Structured Clinical Examination (OSCE) study with 30 primary care physicians (PCPs), 15 patient actors and 100 clinical scenarios, we compared AMIE (Video), its text-only counterpart AMIE (Text), and PCPs consulting via video. Clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination. Patient actors preferred AMIE’s approach to assessing and explaining conditions, while PCPs were preferred for rapport and partnership building. In modality ablation, patient actors preferred AMIE (Video)’s interface over text chat for communicative effectiveness, convenience, and feeling understood. Limitations remain in fine anatomical precision, subtle affective nuances, and high-frequency movements. While further research is needed before real-world translation, these results mark an important milestone toward AI systems capable of augmenting care across the sensory complexity of clinical practice.

1. Introduction The information exchanged in a clinical encounter extends far beyond what words alone can convey. Throughout a consultation, skilled physicians continuously integrate non-verbal cues, including vocal tone, facial expression, posture, and the display of physical signs such as dyspnea or tremor [1], alongside the spoken history. These visual and auditory dimensions are foundational to clinical reasoning [2], shaping the trust, empathy, and rapport that underpin the therapeutic relationship [3], and are intrinsically linked to clinical outcomes including symptom resolution and patient adherence [4]. Evaluating physical appearance, behavior, and posture is a core component of clinical examination [5], and these cues remain important to patient satisfaction during telehealth encounters [6], where many physical examination maneuvers can be successfully adapted to video-based settings [7, 8]. Video-based consultations are the dominant modality for remote clinical care, making them an ecologically valid setting for evaluating clinical AI. Yet expert-level capabilities of AI for clinical

Corresponding author(s): {apalepu, mikeshake}@google.com © 2026 Google. All rights reserved

Towards Expert-level Medical AI for Real-time Video Consultations

A. AMIE for Real-Time Video Consultations

AMIE (Video) Planner Agent

Talker Agent

Video Interface

Patient Actor

Perception Agent

AMIE guiding a patient actor through a physical examination (pronator drift test) over a video call.

AMIE evolves towards real-time video consultations using a multi-agent architecture with three agents (Talker, Planner, Perception agents) whose design is guided by auto-evaluations.

C. Key Findings

B. Evaluation Study

Video Consultation Quality: AMIE and PCPs Primary Care Physician (Video) Randomized Order AI

Patient Actor

Scenario Pack

AMIE (Video)

Diagnostic Accuracy

Clinical Evaluator Ratings

100%

History Taking

90%

Perception & Exam

80%

Clinical Reasoning

70%

AMIE (Video) PCPs (Video)

60%

AI

Simulated Patient

AMIE (Text)

Treatment Planning Communication

50% 1

2

3

4

5

0%

Length of differential considered

Real-time Consultation via Text or Video Interface

50%

100%

% Points Earned

AMIE Modality Ablation: Video and Text Communicative effectiveness Feeling understood

AMIE (Video) Better

Evaluation Perspectives Patient Actors

AMIE (Text) Better

Ease and convenience

Clinical Evaluators 0%

25%

50%

75%

100%

% Consultations

Randomized study with text and video interfaces for remote Objective Structured Clinical Examination (OSCE)

AMIE matched PCPs in real-time video consultations across clinical rubrics. Patient actors preferred AMIE’s video configuration over text chat.

Figure 1 | Overview of Contributions. This work provides the first demonstration of expert-level AI performance in real-time clinical video consultations, enabled by three key contributions: (A) AMIE (Video), a Gemini-based asynchronous multi-agent architecture comprising a Talker, Planner, and Perception agent, guided by an automated evaluation framework derived from a taxonomy of clinical audio-visual competencies; (B) A multi-arm randomized video-based OSCE comparing AMIE (Video), AMIE (Text), and PCPs (Video) across 100 clinical scenarios enacted by professional patient actors, and evaluated by independent clinical evaluators using standard OSCE criteria and case-specific rubrics; (C) Clinical evaluators rated AMIE (Video) on par or better than PCPs across core clinical competencies, and patient actors preferred the video interface over text chat. reasoning and dialogue have been demonstrated exclusively in instant messaging settings, matching or exceeding human performance in text-based conversations [9–11] and in settings where additional information is shared through images [12]. Early prospective evidence suggests conversational safety and clinical utility of text-based patient-facing AI in real-world clinical studies [13–15]. Even with the addition of images, these interfaces cannot fully capture the dynamic audio-visual cues of a live clinical encounter, and patients still need to distill complex physical and behavioral observations into

2

Towards Expert-level Medical AI for Real-time Video Consultations discrete descriptions, a process that introduces biases [16] and risks discarding diagnostic information. Text-only interfaces also present barriers to equitable access, potentially excluding patients with lower digital or health literacy who may struggle to articulate symptoms in writing [17, 18]. Early efforts to extend medical AI to audio-visual interaction have demonstrated feasibility [19], but not reached clinician-level performance, and key challenges remain. An effective audio-visual clinical agent must balance low-latency dialogue with thorough clinical reasoning and real-time audio-visual perception while maintaining robust performance across diverse patient encounters. Fully characterizing these systems demands larger-scale evaluation, including modality ablations with text-based counterparts, and granular analyses of key clinical audio-visual capabilities. Here, we provide the first demonstration of expert-level AI performance in real-time clinical video consultations, presenting the Articulate Medical Intelligence Explorer (AMIE) in a video configuration. This advance is enabled by three key contributions (Figure 1): • AMIE for Real-Time Video Consultations: We introduce AMIE in its video configuration, AMIE (Video), a novel Gemini-based asynchronous multi-agent architecture that overcomes the latency, perception, and robustness challenges inherent in real-time, video-based clinical interactions. To characterize system capabilities and failure modes, we derived a taxonomy of clinical audio-visual competencies feasible in telehealth settings from the medical literature, and developed a corresponding automated evaluation suite comprising targeted single-turn assessments and multi-turn simulated consultations. Through this framework, we demonstrate how the individual components of AMIE (Video) contribute to overall system performance. • Multi-arm Video OSCE Study: In a randomized Objective Structured Clinical Examination (OSCE) study, we compared AMIE (Video), its text-only counterpart AMIE (Text), and 10 boardcertified primary care physicians (PCPs) conducting video consultations across 100 clinical scenarios with 15 professional patient actors, evaluated by an independent panel of 20 clinical evaluator PCPs. Consultation quality is evaluated from both the patient actor and clinical evaluator perspective, including standard OSCE assessment criteria as well as case-specific rubrics designed for each clinical scenario. • Key Findings: Clinical evaluators rated AMIE (Video) on par or better than PCPs in historytaking, diagnosis, management, and physical observation and examination. Patient actors preferred AMIE’s approach to assessing and explaining conditions, while PCPs were preferred for rapport and partnership building. In modality ablation, patient actors preferred AMIE’s video interface over text chat for communicative effectiveness, convenience, and feeling understood.

2. Methods We developed AMIE (Video) to conduct clinical consultations through a real-time video interface as is routine practice in many telehealth settings. The system coordinates several specialized agents (Figure 2, Section 2.1): the user-facing Talker Agent (Section 2.1.1) generates rapid patient-facing responses, informed by the background Planner Agent (Section 2.1.2) managing clinical goals, and the Perception Agent (Section 2.1.3) processing continuous audio and video streams. To guide development, we established an automated evaluation framework (Section 2.2) covering various audio-visual capabilities (Section 2.2.1) via both single-turn assessments (Section 2.2.2) and simulated multi-turn conversations (Section 2.2.3). Finally, we conducted a randomized multi-arm OSCE study (Section 2.3) comparing text and video configurations of AMIE with a group of primary care physicians (PCPs) (Section A.6.5) across 100 simulated patients (Section A.6.3). Consultations were assessed on general and case-specific rubrics (Section A.6.9) by a panel of clinical evaluators (Section A.6.7).

3

Towards Expert-level Medical AI for Real-time Video Consultations 2.1. System Architecture AMIE (Video) (Figure 2) is an experimental multi-agent conversational system designed to conduct real-time video-based clinical dialogue with the goal to generate clinically appropriate differential diagnoses and management plans. The system tested in this work is built upon Gemini 3 Flash [20] and Gemini 3.1 Pro [21]. The system relies on real-time orchestration of three specialized agents collaborating asynchronously to deliver fluent responses with low latency informed by thorough clinical reasoning and live audio-visual perception. AMIE (Video) Planner Agent

live

Patient Actor

Video Interface

async updates

Talker Agent

Perception Agent

Talker Agent

Perception Agent

Inputs:

Inputs:

● Conversation from Video Interface ● Audio-visual cues from Perception Agent ● Clinical goals from Planner Agent

● Visual data stream from Video Interface ● Auditory data stream from Video Interface

Planner Agent

Inputs: ● Conversation content from Talker Agent Outputs:

Outputs: Output: ● Real-time audio response

● Newly detected audio-visual clinical cues ● Memory of all cues detected so far

● Continuously re-prioritized clinical goals ● Running summary of the patient, internal differential diagnosis & management plan

Figure 2 | AMIE for Real-Time Video Consultations. AMIE in its video configuration, AMIE (Video), balances low-latency dialogue with thorough clinical reasoning and real-time audio-visual perception via a three-agent architecture interfacing with patient actors through a video interface. The Talker Agent serves as the conversational interface of the system and aims to deliver low latency responses to the patient. It pulls the latest updated state from the other two agents to guide its responses. The Perception Agent is tasked to identify clinically relevant audio-visual cues within the context of the given conversation. It also records a cumulative list of observations, which helps account for the limited prior video context seen by the system at any given moment. The Planner Agent manages the clinical goals of the system, adding, removing, and re-prioritizing goals as needed as the conversation unfolds. It also synthesizes information received into concise summaries of patient information and the system’s internal diagnostic and management reasoning.

4

Towards Expert-level Medical AI for Real-time Video Consultations 2.1.1. Talker Agent The Talker Agent is tasked with responding fluently to user utterances throughout the conversation, requiring low-latency response generation. Building upon the Project Astra infrastructure and Gemini’s native multimodal understanding, the agent processes multimodal conversations to generate real-time audio responses. When the user speaks, the Talker listens. When the Talker detects a sufficient period of silence following a user utterance, it triggers a model call to generate a response to the user. Even while speaking, the Talker continues to listen for user input, and transitions to silent listening mode upon detecting an interruption from the user. This approach affords natural conversational turn-taking between the user and the agent. To minimize latency, the Talker utilizes a minimal compute budget and processes only the most recent 5 seconds of video frames when formulating its immediate response. Finally, the Talker serves as the primary orchestrator of the system, consolidating inputs from the other two agents. For response generation, it synthesizes the most recent states from the Perception Agent (observations of audio-visual clues captured so far) and the Planner Agent (current clinical goals and reasoning), alongside recent video frames and the full dialogue history. 2.1.2. Planner Agent The Planner Agent manages the system’s clinical goals. Its design was inspired by the Management Reasoning (Mx) Agent from a prior iteration of AMIE [10] and further expands upon the implementation described in [19]. The Planner maintains a persistent memory of the clinical state, including a summary of patient symptoms, a differential diagnosis, a management plan, and active conversational ‘milestones’. These milestones may specify information to collect (e.g., “elicit the exact onset, character, and radiation of the chest pain”), physical actions for the patient to perform (e.g., “instruct the patient to demonstrate active shoulder range of motion”), or critical guidance to provide (e.g., “advise escalation to urgent care if shortness of breath worsens”). Upon invocation, the Planner executes a multi-step reasoning chain via structured decoding, pruning, re-prioritizing, or appending new goals while updating its understanding of the clinical case. To maintain fluency, updates to the Planner’s internal state take place asynchronously, decoupled from activity of the Talker Agent, which responds promptly based on the most recent Planner state. State updates are triggered whenever new user input is received to minimize information staleness, subject to a rate limit of one deep reasoning invocation per 10 seconds of dialogue to balance model load. Several measures help to mitigate potential misalignment between the Planner’s state and the Talker’s context. First, the Planner focuses on high-level clinical goals which update only occasionally throughout the conversation. Second, the Talker is designed through prompting to handle potential discrepancies. Finally, the system continues to process incoming information while the Talker delivers audio output to the user. 2.1.3. Perception Agent The Perception Agent is designed to perceive and interpret real-time audio and video streams, using Gemini 3.1 Pro in a dedicated inference call to carefully enumerate clinically relevant visual and auditory information. During conversational turn-taking, audio-visual cues, such as a patient’s affect, speech cadence, or subtle physical signs, can occasionally go unaddressed by the Talker itself given the reasoning budget constraints necessitated by its priority on low-latency fluent dialogue. The Perception Agent addresses this limitation by reasoning over an extended video window, and maintaining a persistent memory of all clinical audio-visual cues detected over time. For example, if a patient exhibits a dry cough at the beginning of an interaction, the Perception Agent will log this observation and ensure it remains accessible to the system for subsequent clinical reasoning. The Perception Agent implements the same asynchronous relationship with the Talker Agent as described for the Planner Agent above. 5

Towards Expert-level Medical AI for Real-time Video Consultations 2.2. Automated Evaluation To guide development, we constructed an automated evaluation framework designed to assess system performance across a spectrum of clinical and conversational contexts relevant to real-time video telehealth settings (Section 2.2.1). The framework comprises two core paradigms: (1) targeted single-turn evaluations (Section 2.2.2), which isolate specific audio-visual capabilities; (2) simulated multi-turn interactions (Section 2.2.3), which assess end-to-end conversational capability as a proxy for eventual performance in the evaluation study. 2.2.1. Telehealth Evaluation Taxonomy We first derived a taxonomy of clinical audio-visual cues and physical examination maneuvers relevant to telehealth settings (Section A.1). Synthesized from the medical literature and physical examination protocols [22–25], our taxonomy encompasses physical observations and guided examinations across major body systems and observation categories. We categorized the clinical audio-visual cues by their “telehealth feasibility”—whether they can be reasonably observed in a virtual setting—and by acquisition method, distinguishing between passive observations and those involving guided or elicited maneuvers. For the purpose of this study, we further classified clinical signs into “actable” presentations (realistically simulated by patient actors, such as motor or behavioral changes) and “non-actable” presentations (specific physical or behavioral abnormalities that cannot be enacted). 2.2.2. Single-turn Evaluation Single-turn agent evaluations served as ‘unit tests’ for specific audio-visual telehealth capabilities, including clinical perception, reasoning, and interaction. These scenarios were grounded in audio or video clips (sourced from publicly available materials and study-specific enactments) and designed to elicit specific internal reasoning or dialogue responses from the agent at precise points in a patient conversation. For instance, an evaluation scenario testing visual perception may depict a patient raising their right hand and asking, “what am I showing you?” The target response requires accurate identification of both anatomical location and laterality. Each evaluation case was paired with examplespecific clinical grading rubrics that describe the desired response and internal reasoning of the agent. During evaluation, an LLM-based auto-rater grades the agent’s response with a quality score from 0 to 1 applying the rubric. Further details on the design of single-turn agent evaluations are described in Section A.2. 2.2.3. Multi-turn Evaluation While single-turn evaluation facilitates targeted benchmarking of perception, clinical encounters are inherently dynamic. For instance, while an agent should consistently elicit the anatomical location of a patient’s abdominal pain, the precise turn at which this request occurs may vary across conversations. To evaluate these conversational-level competencies, we leveraged multi-turn evaluations, evolving techniques from prior work with text-based simulated AI patients [10, 11], to an audio setting. In these simulations, visual physical examination cues were conveyed through injected verbal stage directions. For instance, if the agent instructs the patient to palpate their abdomen, the simulator might respond with the spoken audio confirmation, “Sure,” accompanied by the caption, “[patient presses on their lower right abdomen while simultaneously wincing].” This multi-turn evaluation was applied to the clinical scenarios used in Shah et al. [19], and an LLM-based auto-rater graded the clinical dialogue using case-specific rubrics across core clinical dimensions (history-taking, clinical reasoning, communication and counseling, treatment planning, triage, and red-flag identification). 6

Towards Expert-level Medical AI for Real-time Video Consultations

Primary Care Physician (Video)

Step 1 Real-time Clinical Consultation

Randomized Order Patient Actor

AI

AMIE (Video)

AI

AMIE (Text)

Scenario Pack

Simulated Patient Real-time Consultation via Text or Video Interface

Step 2 Post-Questionnaires

Patient Actor Post-Questionnaire

AMIE / PCP Post-Questionnaire

- Empathy - Communication - Rapport Building - Modality Experience

- Differential Diagnosis - Investigations & Treatment - Escalation & Follow-up - Audio-Visual Observations

Consultation Video & Transcript

Step 3 Clinical Evaluation

Clinical Evaluators Scenario Pack + Ground Truth

Consultation Video & Transcript

Responses to Post-Questionnaire

Assess general and case-specific rubrics

Figure 3 | Study Design. A randomized Objective Structured Clinical Examination (OSCE) study compared AMIE (Video), its text-only counterpart AMIE (Text), and PCPs consulting via video, on the task performing a real-time consultation with simulated patients (professional patient actors enacting a given clinical scenario pack). After each consultation, AMIE and PCPs produced answers to a post-questionnaire documenting their clinical reasoning, and patient actors rated their experience on pre-defined rubrics. A separate panel of clinical evaluators assessed each consultation based on general and scenario-specific criteria. 2.3. OSCE Study We conducted a randomized multi-arm evaluation study comparing AMIE with a group of US boardcertified primary care physicians (PCPs) for the task of conducting real-time video-based telehealth consultations with patient actors, including one study arm testing AMIE in its text chat configuration (Figure 3). We adapted the virtual Objective Structured Clinical Examination (OSCE) framework introduced in Tu et al. [11], designing a total of 120 scenario packs enriched with audio-visual cues and guided physical examination components. Complete study details, participant characteristics,

7

Towards Expert-level Medical AI for Real-time Video Consultations and rubric descriptions are provided in Section A.6. 2.3.1. Study Design and Clinical Scenarios A validation set of 20 clinical scenarios was used to conduct a study pilot with six arms comparing AMIE and PCPs in Video, Audio, and Text modalities each respectively, resulting in 120 completed consultations (6 across each of 20 scenarios). The purpose of this study pilot was to perform an initial validation of the agent and interface, and to inform the final study design through a small-scale ablation isolating effects of text, audio, and audio-video modalities. Details and results from the pilot are provided in Section A.5. The full evaluation study was conducted across a distinct set of 100 clinical scenarios across three arms: AMIE (Video), AMIE (Text), and PCP (Video). All arms were conducted through a teleconferencing interface resembling Google Meet affording interactions through text chat and real-time audio-video modalities. (Figure A.4). In this study, AMIE did not feature a visual avatar. For controlled comparison and to avoid bias from visual presence or appearance, PCPs were asked to turn their cameras off throughout the consultations (Section A.6.1). Scenario packs and simulated patients were prepared by an OSCE laboratory in North America and verified by US board-certified PCPs (Section A.6.2). Drawing from HHS resources for virtual care∗ , scenarios were designed to span five clinical domains relevant to telehealth settings: (1) cardiopulmonary, (2) abdominal, (3) HEENT (head, eyes, ear, nose, throat), (4) neurology and psychiatry, and (5) musculoskeletal. Each of these categories included a subset of non-disease cases with resolved complaints to assess clinical reasoning and behavior in those negative settings. Inpatient, pediatric, and dermatological cases, as well as those with privacy-sensitive physical examinations were excluded. Each pack defined ground-truth diagnoses, plausible alternatives, investigations, management plans, and a 20-point rubric used to score clinical capability in case-specific manner (Section A.7). Following each encounter, AMIE and PCPs produced answers to a structured post-questionnaire documenting differential diagnoses, observations, and management recommendations (Sections A.6.6 and A.9). For this post-encounter step, which operates under less stringent latency constraints than live dialogue, the AMIE system leveraged inference-time compute scaling via multi-draft synthesis [10] to improve recommendation quality. 2.3.2. Patient Actors and Primary Care Physicians Clinical scenarios were enacted by a cohort of 15 professional patient actors (7 female, 8 male; ages 20–75) trained to deliver standardized portrayals of clinical affect, physical cues, and guided physical examination maneuvers matched to biological sex (Section A.6.3). Consultations in the PCP (Video) arm were performed by 10 US board-certified PCPs (mean 4.5 years post-residency clinical experience, Section A.6.5). 2.3.3. Evaluation Rubrics Consultations were evaluated from multiple perspectives. Following each visit, patient actors evaluated conversational quality and communication convenience using established criteria from GMCPQ (General Medical Council Patient Questionnaire), PACES (Practical Assessment of Clinical Examination Skills), and PCCBP (Patient-Centered Communication Best Practice) adapted from Tu et al. [11] ∗ https://telehealth.hhs.gov/providers/preparing-patients-for-telehealth/telehealth-physical-exam

8

Towards Expert-level Medical AI for Real-time Video Consultations

Patient Actor

Turn 24

PERCEPTION STATE

No, not that I'm aware of. Turn 25

AMIE

guide_shoulder_rom: Guide patient through bilateral shoulder range of motion to assess for any localized shoulder pathology versus referred pain.

Understood. To assess your neck and shoulder movement more closely, could you please slowly turn your head to the right as far as comfortable, then to the left, and tell me if either movement triggers that sharp pain? Patient Actor

Rationale: Assessing shoulder ROM will help distinguish between local shoulder issues and referred pain from the neck/back.

Turn 26

CLINICAL STATE

<noise> Turn 27

AMIE

I observed you turning your head, and it looked like you had some restriction, especially to the left. To check your shoulder movement as well, could you please slowly raise both arms straight out to your sides, then up towards the ceiling, and tell me if that causes any pain or stiffness?

Patient Actor

current_av_signs: Patient turns his head to his left, then to his right; Patient briefly closes his eyes and shows a mild facial expression of discomfort after completing the neck movements. overall_affect: Calm and cooperative, though showing mild signs of discomfort upon movement. av_clinical_synthesis: The patient successfully performed the requested cervical range of motion maneuver, turning his head bilaterally ... presentation remains highly consistent with a mechanical/musculoskeletal strain of the cervical spine or upper trapezius musculature.

Turn 28

PERCEPTION STATE

That's definitely uh definitely tight and tender when I get it up. Turn 29

AMIE

Turn 30

Yeah, I'd say that does. Turn 31

AMIE

Based on our discussion and the movements we checked, your symptoms are most likely due to a musculoskeletal strain, probably involving the trapezius muscles in your upper back and shoulders, consistent with the heavy lifting. Other possibilities could include a neck strain or minor joint irritation, while more serious causes like a pinched nerve are less likely given the lack of numbness or weakness.

Turn 33

AMIE

PLANNER UPDATES

CLINICAL STATE

current_av_signs: Patient performs bilateral active shoulder flexion, raising his arms forward and upward instead of out to his sides as requested.; Patient does not reach full vertical elevation, stopping with his arms raised partially overhead while verbally confirming tightness and tenderness. av_clinical_synthesis: The patient was instructed to perform shoulder abduction (raising arms out to the sides) but instead performed bilateral shoulder flexion (raising arms forward and up). During this movement, he exhibited restricted range of motion, and reported tightness and tenderness. This finding, combined with his localized bilateral trapezius/upper back pain and restricted cervical rotation, further supports a mechanical/musculoskeletal strain, likely involving the upper back and shoulder musculature following a heavy lifting event.

PERCEPTION STATE

Neck Range of Motion (done) Assessment: User performed rotation; AMIE observed restriction especially to the left and confirmed it triggers pain. Implications: Confirms mechanical/musculoskeletal involvement of the cervical/upper thoracic region. Shoulder Range of Motion (done) Assessment: User performed abduction and forward flexion; reported tightness and tenderness when raised. Implications: Suggests involvement of the trapezius or rotator cuff musculature, likely secondary to the primary upper back/neck strain.

PLANNER UPDATES

educate_user_on_imaging: Explain the role of imaging (X-ray or MRI) in back and shoulder pain, clarifying that for uncomplicated strains without 'red flag' symptoms, imaging is generally not recommended as it doesn't change management. refine_red_flags_and_follow_up: Clearly outline specific red flags that should prompt immediate inperson medical attention, and confirm the one-week follow-up plan.

That sounds good to me. I guess to confirm everything after if a week goes by, will I need any imaging like an X-ray or MRI to to know what's going on if it's nerve or muscular? Turn 35

negative_symptoms: No numbness, tingling, or weakness in arms or hands; no chest pain or shortness of breath; no fever or chills; no prior history of back/neck issues, trauma, or surgeries; no known drug allergies; negative family history for chronic back problems.

physical_examination_done:

Turn 34

CLINICAL STATE

AMIE

Imaging like an X-ray or MRI typically isn't needed right away for a suspected muscle strain, as those tests are better at showing bones or significant nerve issues. We would usually consider imaging only if there were red flags, like progressive weakness, or if the pain didn't improve with conservative care.

symptoms: Dull and achy pain in the upper back and shoulder area; sharp pain when turning the head to look over the shoulder; severity rated at 5/10; symptoms gradually worsening over the week and worse by the end of the day.

differential_diagnosis: Discuss potential diagnoses like musculoskeletal strain, myofascial pain, or cervical spondylosis. Explain why a strain is most likely given the lifting history and negative neurological findings.

I’d recommend continuing the ibuprofen, applying heat or ice to the area, and avoiding heavy lifting for now. If the pain doesn't improve in the next week or worsens, we should consider physical therapy or further evaluation. Do you have any questions about this assessment or plan? Patient Actor

main_concerns: Bilateral upper back and shoulder pain for one week, possibly due to lifting heavy boxes at work.

reflect_before_assessment: Once the history (including social habits) is complete, provide a final concise summary of the key findings and ask the patient for confirmation or corrections.

I noticed some hesitation as you raised your arms, which confirms the tightness you mentioned. Before we discuss potential causes and a plan, let me briefly summarize what I've gathered. You've had a dull, achy pain in your upper back and shoulders for a week, likely from lifting heavy boxes. It's worse with head rotation and arm movement, but you have no numbness or other red flags. Does that accurately capture everything? Patient Actor

PLANNER UPDATES

guide_neck_rom: Guide patient through neck range of motion for pain triggers or limitations, specifically noting sharp pain during rotation.

overall_affect: Calm, attentive, and cooperative. av_clinical_synthesis: The patient is quietly and attentively listening to the AI assistant's assessment and treatment plan. There are no new acute signs or changes in his condition... the patient appears engaged in the proposed management steps.

differential_diagnosis: Musculoskeletal strain (likely trapezius), neck strain, or minor joint irritation; pinched nerve or more serious pathology less likely given negative neurological findings. treatment_management_plan: Continue ibuprofen, apply heat or ice, avoid heavy lifting; re-evaluate in one week if no improvement.

Figure 4 | AMIE Interaction Example showcasing activities of Talker, Perception and Planner agents.

9

Towards Expert-level Medical AI for Real-time Video Consultations (Section A.6.4). Additionally, they rated their experience with the given communication method (text chat or video interface, Section A.6.4). A panel of 20 board-certified PCPs served as clinical evaluators (Section A.6.7) to assess consultation recordings, transcripts, and post-questionnaire responses for each scenario and study arm in a randomized order using a standard rating interface (Figure A.5). Clinical evaluators graded encounters across both general and case-specific rubrics. General criteria, adapted from Tu et al. [11] and Liévin et al. [10], assessed differential diagnosis (DDx) accuracy, diagnostic and management reasoning, and competencies in clinical dialogue (PACES and PCCBP), supplemented by telehealth-specific rubrics evaluating perceptual acuity and guided physical examination execution [26] (Sections A.6.8 and A.8). Finally, each consultation was evaluated against a 20-point checklist rubric specific to each scenario spanning five domains: history taking, perception and examination, clinical reasoning, treatment planning, and communication. Details are provided in Sections A.6.9 and A.7. 2.3.4. Qualitative Data Patient actors and clinical evaluators were encouraged to leave free-text comments in their assessments to provide specific rationale for their ratings. To further characterize consultation experience from the patient actor perspective, semi-structured interviews were conducted with 7 patient actors, probing at differences in interaction modalities and conversational experience. Qualitative data was analyzed using an reflexive thematic analysis to inductively generate themes based on the transcripts and researcher notes. Details are provided in Section A.14. 2.3.5. Statistical Analysis For the 20-point case-specific rubrics, performance was aggregated as the mean score earned out of the possible points achievable across each of the five core rubric categories, as well as overall. For ease of interpretation, Likert ratings from general assessment rubrics were standardized to a percentage scale for reporting (0% representing the lowest ratings for all 100 cases, and 100% representing the highest possible scores for all 100 cases). Differences across study arms were evaluated using a two-stage procedure: an omnibus Friedman test was first conducted to assess for differences across all three arms, followed by two-sided Wilcoxon signed-rank tests and Benjamini-Hochberg False Discovery Rate (FDR) correction for comparisons between pairs of arms. Friedman and Wilcoxon signed-rank tests were conducted using scenario-level blocking. Cases with N/A ratings on either arm of a comparison were excluded. All 𝑝 values reported in the manuscript refer to FDR adjusted p values from Wilcoxon signed rank tests. Error bars correspond to 95% confidence intervals produced via bootstrap procedure with 1000 resamples.

3. Results We describe key results from our comparison of video consultation quality between AMIE and PCPs (Figure 5) and the modality ablation comparing AMIE between video and text chat interfaces (Figure 6), as well as qualitative insights and results from automated evaluations below. Excerpts from a representative AMIE (Video) consultation, including corresponding planner and perception agent states, are shown in Figure 4. Full tabular results, scenario-level preferences, and pilot study results are provided in Sections A.5 and A.11 to A.13.

10

Towards Expert-level Medical AI for Real-time Video Consultations Video Consultation Quality (AMIE & Primary Care Physicians) A. Diagnostic Accuracy 100

Accuracy (%)

90

80

70

60 AMIE (Video) PCP (Video) 50 1

2

3

4

5

Top-k

B. Clinical Evaluator Ratings

C. Patient Actor Preferences

Overall

P = 1.3×10−9

Assessing condition

P = 0.04

History Taking

P = 0.001

Explaining condition

P = 0.04

Perception & Exam

P = 1.7×10−9

Building rapport

n.s.

Clinical Reasoning

P = 2.0×10−5

Building partnership

n.s.

Treatment Planning

P = 8.9×10−5

Empathy

n.s.

Communication

P = 6.6×10−7

Addressing concerns

n.s.

0%

25%

50%

75%

100%

% Points Earned AMIE (Video)

PCP (Video)

0%

25%

50%

75%

100%

% Consultations AMIE (Video) Preferred

Tie

PCP (Video) Preferred

Figure 5 | Video Consultation Quality for AMIE and Primary Care Physicians. Results are based on the OSCE evaluation study with patient actors and clinical evaluators ( 𝑁 = 100 clinical scenarios). (A) Top-𝑘 cumulative diagnostic accuracy relative to each scenario’s pre-specified reference diagnosis, where 𝑘 refers to the length of the differential diagnosis list considered. (B) Mean percentage of points earned on case-specific clinical rubrics across six consultation domains, as rated by clinical evaluators. (C) Patient actor preferences on six global criteria, comprising two items each selected from the GMCPQ (Assessing condition, Explaining condition), PCCBP (Building rapport, Building partnership), and PACES (Empathy, Addressing concerns) rubrics. Error bars indicate 95% non-parametric bootstrap confidence intervals (1,000 resamples). 𝑃 -values reflect false discovery rate (FDR) adjustments after Wilcoxon signed-rank tests (n.s. denotes 𝑃 > 0.05). 3.1. Video Consultation Quality: AMIE and PCPs AMIE demonstrated expert-level performance in real-time video consultations: Clinical evaluators rated AMIE (Video) equivalent to or better than PCPs across all six case-specific rubric domains (Figure 5B), 11

Towards Expert-level Medical AI for Real-time Video Consultations with an overall score of 83% compared with 68% for PCPs ( 𝑝 = 1.32 × 10−9 , Table A.7). Patient actors preferred AMIE (Video) for assessing and explaining conditions, while differences on empathy and addressing concerns were not statistically significant (Figure 5C). Diagnostic accuracy and clinical reasoning were higher than for PCPs: AMIE (Video) matched the correct reference diagnosis for each clinical scenario in its top item of the generated differential diagnosis in 91% of cases, compared with 77% for PCPs ( 𝑝 = 0.039). Diagnostic accuracy became more similar when considering the top three items in the differential for AMIE (Video) and PCPs respectively (98% vs. 90%, 𝑝 = 0.179, Figure 5A). On case-specific rubrics, AMIE (Video) scored 90% compared with 76% for PCPs on clinical reasoning ( 𝑝 = 2.00 × 10−5 ), and 79% compared with 67% on treatment planning ( 𝑝 = 8.90 × 10−5 ). History-taking was rated equivalent to or better than PCPs: Clinical evaluators rated AMIE (Video)’s history-taking favorably across established PACES criteria. On case-specific rubrics, AMIE (Video) scored 86% compared with 78% for PCPs ( 𝑝 = 1.20 × 10−3 ). The mean conversation duration was comparable across both study arms (8.94 minutes for AMIE Video vs. 9.30 minutes for PCPs), suggesting a similar timeframe for information gathering. AMIE leveraged video effectively for physical observation and guided examination: AMIE (Video) scored higher than PCPs at utilizing live video to augment information gathering (77% vs. 51%, 𝑝 = 1.37 × 10 −4 ) and guiding patients through physical examinations (72% vs 39%, 𝑝 = 2.46 × 10 −9 ). These findings were reflected in case-specific perception and examination rubrics, where AMIE (Video) scored 74% compared with 47% for PCPs ( 𝑝 = 1.73 × 10−9 ). Communication was rated favorably, though PCPs retained advantages in rapport: On established PACES and PCCBP scales, clinical evaluators rated empathy (82% vs. 71%, 𝑝 = 3.39 × 10−3 ), fostering of relationships (80% vs. 69%, 𝑝 = 5.63 × 10−3 ), and case-specific communication (85% vs. 68%, 𝑝 = 6.56 × 10−7 ) for AMIE (Video) as equal to or better than PCPs. Patient actors preferred AMIE (Video) for assessing and explaining conditions, while PCPs were preferred (though without a significant difference) for building rapport and partnership. 3.2. AMIE Modality Ablation: Video and Text Patient actors preferred the video interface for communication: Across all three communication method criteria (Figure 6A), patient actors rated AMIE (Video) significantly higher than AMIE (Text), including on effectiveness in communicating health concerns (89% vs 79%, 𝑝 = 4.98 × 10−5 ), ease and convenience of use (88% vs 71%, 𝑝 = 2.98 × 10−6 ), and feeling that their concerns were fully understood (90% vs 81%, 𝑝 = 2.62 × 10−4 ). Video conferred advantages in perception and examination: Clinical evaluators rated AMIE (Video) higher than AMIE (Text) on overall case-specific rubrics (83% vs 75%, 𝑝 = 1.19 × 10−8 ), with the largest gains in the perception and examination domain (Figure 6B). Diagnostic accuracy and management appropriateness were comparable across both modalities. Agent-level preferences were largely similar across modalities: Patient actor ratings of agent behavior on individual conversational criteria did not differ significantly between AMIE (Video) and AMIE (Text) (Figure 6C). While not significant, directionally AMIE (Video) scored higher than AMIE (Text) on all criteria, as reflected by the rating for “Happy to see again” (91% vs 74%, ( 𝑝 = 0.038).

12

Towards Expert-level Medical AI for Real-time Video Consultations AMIE Modality Ablation (Video & Text) A. Patient Actor Preferences for Communication Method I was able to effectively communicate health concerns through this communication method. P = 5.0×10−5

I felt that the concerns I conveyed were fully understood through this communication method. P = 2.6×10−4

I found this communication method easy and convenient to use. P = 3.0×10−6

0%

25%

50%

75%

100%

% Consultations

B. Clinical Evaluator Ratings

C. Patient Actor Preferences for Agent

Overall

P = 1.1×10−8

Assessing condition

n.s.

History Taking

n.s.

Explaining condition

n.s.

Building rapport

n.s.

Perception & Exam

P = 3.3×10−11

Clinical Reasoning

n.s.

Building partnership

n.s.

Treatment Planning

n.s.

Empathy

n.s.

Communication

n.s.

Addressing concerns

n.s.

0%

25%

50%

75%

% Points Earned AMIE (Video)

AMIE (Text)

100%

0%

25%

50%

75%

100%

% Consultations AMIE (Video) Preferred

Tie

AMIE (Text) Preferred

Figure 6 | AMIE Modality Ablation (Video & Text). Results are based on the OSCE evaluation study with patient actors and clinical evaluators ( 𝑁 = 100 clinical scenarios). (A) Patient actor ratings of the communication modality across three criteria. (B) Mean percentage of points earned on case-specific clinical rubrics across six consultation domains, as rated by clinical evaluators. (C) Patient actor preferences on six global criteria, comprising two items each selected from the GMCPQ (Assessing condition, Explaining condition), PCCBP (Building rapport, Building partnership), and PACES (Empathy, Addressing concerns) rubrics. Error bars indicate 95% non-parametric bootstrap confidence intervals (1,000 resamples). 𝑃 -values reflect false discovery rate (FDR) adjustments after Wilcoxon signed-rank tests (n.s. denotes 𝑃 > 0.05). 3.3. Qualitative Insights Semi-structured interviews with patient actors (Section A.14, Table A.10) and free-text clinical evaluator comments revealed themes in how participants experienced AMIE (Video) relative to PCPs 13

Towards Expert-level Medical AI for Real-time Video Consultations and across communication modalities. Video enabled direct demonstration of physical symptoms: Patient actors generally preferred video for physical complaints, noting that it allowed them to show rather than describe symptoms. As one actor explained, video enabled demonstrating specific movements versus “describing like I’m supinating my arm and it hurts. You can just show them and then they can interpret that” (P6). Conversely, text was preferred for sensitive topics where auditory privacy was a consideration such as in public settings. AMIE (Video) was perceived as consistent and thorough: Patient actors described AMIE (Video) as providing consistent empathy across encounters and adopting an unhurried conversational approach: “I did not feel rushed. The AI was thorough and always asked if there was anything else to add” (P3). AMIE (Video) was also noted as being “proactive in wanting to do a physical exam over video” (P5). PCPs, however, were preferred for natural conversational cadence and interpersonal connection. Perceptual and technical limitations: Clinical evaluators documented instances of missed visual cues and uncorrected patient execution errors during guided examination during AMIE (Video) consultations, such as when “the patient demonstrated the left knee movements and not the right, which was not noted,”. Processing silences occasionally caused patient confusion, and evaluators noted specific cases where the agent did not advise on red flag signs. 3.4. Automated Evaluation Results Automated evaluations characterized the contributions of individual agents to overall system performance and identified remaining capability gaps. Full results are provided in Sections A.3 and A.4. Specialized agents meaningfully contributed to overall system performance: In single-turn evaluations, the full system achieved 59% compared with 42% for the Talker-only baseline. The Perception Agent provided the largest individual contribution, raising overall accuracy to 58%, with pronounced gains on visual inspection tasks such as abdominal distension, ascites, and wall contour (88% vs 33%) and eyelid lesions including chalazia, blepharitis, and ptosis (91% vs 56%). The Planner Agent complemented these perceptual capabilities through clinical reasoning, improving performance on tasks such as postural balance loss and gait instability (54% vs 32%) and joint range of motion and mobility limitations (65% vs 50%). In multi-turn simulated consultations across 20 clinical scenarios from [19], the Planner Agent yielded marked improvements in conversational quality and clinical thoroughness, with the full system scoring 87% on overall case-specific rubrics compared with 71% for the Talker-only ablation. These quality gains were achieved at substantially lower latency than the architecture described by Shah et al. [19], driven by the transition from sequential to asynchronous agent orchestration, reducing mean turn latency from 21.4s [95% CI: 20.3s, 23.0s] to 2.6s [95% CI: 2.3s, 2.8s]. Gaps remained in subtle and high-frequency perceptual competencies: Performance in automated evaluations varied considerably across clinical domains. Predominantly physical tasks achieved strong results (Pain behavior: 88%, Dermatologic: 75%, Musculoskeletal: 70%), while interactive domains such as Communication and behavior (44%) and Environment and setup (46%) proved more challenging. Key failure modes involved high-frequency or subtle audio-visual competencies, including nystagmus (3%), tremors at rest or with action (24%), and affective state, expressiveness, and mood congruence (29%).

14

Towards Expert-level Medical AI for Real-time Video Consultations

4. Related Work 4.1. Foundation Models: From Language to Perception The development of foundation models for generalist medical artificial intelligence [27] has progressed from static question-answering toward interactive clinical dialogue. Early clinical foundation models demonstrated expert-level medical question-answering on static benchmarks such as MedQA [28–33]. However, in medical training, written examinations do not reliably predict clinical competence [34, 35], as corroborated by the gap between AI performance on static benchmarks and in interactive clinical settings [36, 37]. In simulated text-based patient encounters, conversational AI systems have matched or exceeded primary care physicians (PCPs) in diagnostic reasoning, management recommendations, and perceived empathy across multi-turn text interactions [9–11]. In subspecialist domains, randomized and blinded evaluations showed that AI assistance reduced clinically significant errors among cardiologists in complex cardiovascular care [38] and produced management plans comparable to oncology fellows in breast cancer care [39]. Prospective clinical feasibility studies have begun to evaluate conversational agents in real-world ambulatory care, supporting their initial safety and decision-support utility [13–15], often under structured multi-agent clinician oversight [40, 41]. Yet text-based interfaces impose a communicative bottleneck that introduces reporting biases [16] and accessibility barriers [17, 18], while risking the omission of critical diagnostic cues. Multimodal clinical models have incorporated static medical imagery and offline data [12, 33, 42–44]. However, active clinical encounters require continuous, real-time audio-visual perception. While full-duplex speech-to-speech foundation models enable low-latency spoken dialogue [45–47], a disconnect between perception and action persists. Here, current voice systems can reliably identify vocal delivery cues such as distress or sarcasm when queried, but frequently disregard these paralinguistic signals during consequential decision-making [48]. Leveraging streaming audio-video infrastructure [49], Shah et al. [19] established the feasibility of an audio-visual medical AI across 20 proof-ofconcept scenarios demonstrating superiority of a Gemini-based system over a GPT-Realtime baseline, while highlighting important gaps to fill before reaching physician-level performance in this task. The present work builds on these foundations to provide the first demonstration of expert-level clinical performance in real-time video consultations. 4.2. Scaling Reasoning with Multi-Agent Architectures Expert clinical practice requires balancing rapid conversational rapport with rigorous deliberative reasoning. In foundation models, scaling test-time compute via chain-of-thought prompting, selfconsistency, and search has been shown to substantially improve diagnostic accuracy [30, 31, 50–56]. However, while extensive search is well-suited to offline tasks, live clinical consultations impose strict conversational pacing (of only a few seconds of delay between turns), creating a fundamental tension between deep inference-time reasoning and real-time responsiveness. Multi-agent architectures address this tension by decoupling fast conversational interaction from slower, deliberative reasoning [10, 40, 56, 57]. Polaris [40] employed specialized clinical agents to monitor conversational safety, while Christakopoulou et al. [57] pioneered a dual-agent “System 1” (Talker) and “System 2” (Reasoner) architecture. Similar tandem designs have emerged in real-time speech systems to anchor low-latency spoken dialogue with deliberative knowledge retrieval [58, 59]. Building on the Talker-Reasoner paradigm applied to text-based longitudinal disease management [10], AMIE (Video) extends this architecture to real-time audio-visual streaming: a 3-agent harness (Talker, Planner, and Perception) that executes deep clinical reasoning and continuous perceptual analysis asynchronously, without interrupting fluid dialogue.

15

Towards Expert-level Medical AI for Real-time Video Consultations 4.3. Telehealth Examination and Evaluation In clinical practice, visual and auditory observations are fundamental aspects of history-taking and physical assessment that cannot be replaced through text [5]. The growth of telemedicine has established the virtual physical examination as a viable clinical discipline, demonstrating that key maneuvers across musculoskeletal, neurological, dermatological, and respiratory systems can be guided effectively over video [7, 8, 22–24]. Moreover, comparative telehealth studies show that video consultations significantly improve diagnostic demonstration and patient satisfaction over audio-only visits [6]. Objective Structured Clinical Examination (OSCE) [60] remains the gold standard examination format to evaluate clinical competence during medical training, with candidates conducting consultations with professional patient actors, so-called ‘standardized patients’ to complete a series of pre-defined clinical tasks [61]. Medical educators have adapted OSCEs to virtual encounters [26], suggesting rubrics for telehealth communication and diagnostic process. While prior studies have used OSCE frameworks to evaluate text-based and static multimodal AI [10–12, 41], comprehensive assessment benefits from pairing holistic communication scales with granular, case-specific clinical checklists [62], an approach aligned with contemporary medical dialogue benchmarking [19, 63, 64] and adopted in our video-based OSCE study.

5. Discussion Audio-visual capabilities are essential for AI systems to overcome the fundamental diagnostic and communicative bottlenecks of text-only interfaces. Text-based interactions place a heavy burden on patients to translate complex somatic sensations and physical abnormalities into written words, a dependency that often leads to missed cues or ambiguity. In contrast, multimodal audio-video communication unlocks a rich, real-time stream of objective clinical data. It captures nuanced auditory signals, including respiratory effort, vocal strain, and emotional affect, while enabling direct observation of visual signs such as diaphoresis, scleral icterus, and acute distress. Furthermore, live audio-visual interaction can enable improved clinical rapport and the dynamic guidance of patients through interactive physical examination maneuvers (e.g., abdominal self-palpation, active range of motion, capillary refill) with substantially higher diagnostic fidelity. The AMIE (Video) system represents a step change from text-bound diagnostic agents to a synchronous, audio-visual interaction. It is the first such system to demonstrate PCP-level performance in a randomized OSCE study setting, as rated by independent clinical evaluators. For many evaluation domains, including case-specific history taking, clinical reasoning, treatment planning, and patient communication, clinical evaluators rated AMIE’s performance as significantly better than the baseline PCPs. AMIE (Video) was also preferred for its more proactive approach to physical examination and observation. Patient actors generally rated AMIE’s conversational capabilities favorably compared with those of PCPs, rating AMIE (Video) higher for assessing and explaining their condition. These higher ratings were achieved within a similar encounter duration (8.9 vs. 9.3 minutes), a notable finding given that longer visit times are strongly linked to increased patient satisfaction and more effective management of chronic and psychosocial conditions [65, 66]. These comparative findings are meaningful because the human baseline in this study was designed to closely reflect an actual clinical practice setting. Video-based telemedicine is widely utilized and continues to grow as a standard interaction modality. Prior studies of text-based conversational medical AI [10–12, 15, 41] may underestimate physician performance because physicians are more accustomed to telephone or video-based consultation [67, 68]. This work was designed to be more representative of video-based telemedicine, with PCPs able to see and hear the patient using a familiar 16

Towards Expert-level Medical AI for Real-time Video Consultations video conferencing interface. The PCP baseline in this study therefore serves as a stronger proxy for real-world clinical practice, though key differences remain. Most notably, the study used patient actors instead of actual patients, and physicians were required to keep their cameras off. Clinical evaluators also rated AMIE (Video) higher than PCPs for its proactivity in guiding physical examinations. In this study, human physicians often leaned on oral history-taking rather than instructing patients to perform active examination maneuvers. This difference may reflect clinicians defaulting to in-person consultation habits, where physical exams are typically performed in the clinic rather than coached remotely. AMIE, by contrast, systematically prompted patients through step-by-step physical examination maneuvers. This behavior highlights a potential opportunity for AI systems, which are less inhibited by traditional time constraints, to elicit more thorough clinical information during virtual care. Indeed, it has been shown that many physical assessment maneuvers can be successfully adapted to synchronous video encounters [7, 8, 24]. However, further research is needed to understand the clinical necessity and diagnostic yield of such maneuvers relative to the additional time, effort, and cognitive load required from the patient. The AMIE (Text) arm served as a useful ablation to characterize the impact of audio-visual interaction. Patient actors significantly preferred the audio-visual communication for its ease of use, communicative effectiveness, and ability to make the user of the system feel understood. Clinical evaluators rated AMIE (Video) as superior to AMIE (Text) in several domains key to the telehealth setting, such as fostering a relationship, audio-visual perception, and guided examination, as well as overall across all five case-specific clinical rubrics in aggregate. However, for rubrics related to clinical reasoning such as diagnostic and management appropriateness, AMIE (Video) was comparable to AMIE (Text). In part, this may reflect a trend towards approaching saturation on these simulated OSCE scenarios, where text-based clinical reasoning performance is already high. This finding also aligns with clinical literature suggesting that the medical history alone establishes the correct diagnosis in the vast majority of outpatient encounters, while physical examination alters the primary diagnosis in only a small fraction of cases [69, 70]. Similar results have been observed in other recent telehealth studies, showing comparable clinical effectiveness between telephone and video consultations [71], though other studies have identified reductions in medication errors and improved diagnostic accuracy [72] with the addition of video. Because a missed visual cue can often be compensated by verbal questioning, the distinct advantages of audio-visual reasoning may only emerge in specific situations, such as when a patient’s spoken statements conflict with visual evidence (e.g., verbally naming one medication while holding up a different pill bottle). These simulated scenarios were designed to reflect a breadth of telehealth settings including opportunities for physical observation and examination. However, they were not designed to render purely text-based consultations impossible, which helps to contextualize this finding. The granular, case-specific rubrics (Section A.6.9) were designed to more precisely detect nuanced variations in clinical performance and communication that the broader general rubrics may overlook. As clinical agents improve and metrics like diagnostic accuracy become more saturated, these targeted scoring criteria, which can be tailored to the precise challenges of each scenario, may help uncover more subtle differences in clinical competency. This approach mirrors recent shifts in medical LLM evaluation, such as HealthBench [63, 64], which evaluates open-ended responses to clinical dialogues against physician-authored conversation-specific rubrics. In OSCE examinations used in medical education, station-specific checklists are often utilized [60]; combining general clinical rubrics with case-specific criteria ensures a balanced and standardized assessment of the overall conversational appropriateness and clinical reasoning [62]. The real-time delivery of high-quality dialogue from AMIE was enabled by its decoupled, asynchronous multi-agent architecture. This design addresses a fundamental tension in real-time medical

17

Towards Expert-level Medical AI for Real-time Video Consultations dialogue: the need for near-instantaneous conversational turn-taking (low latency) versus the computational expense of thorough diagnostic reasoning and continuous visual processing. Automated evaluation ablations confirm that this three-agent harness substantially outperforms the single-agent baseline and prior harnesses from Shah et al. [19], while operating at a natural conversational latency (around 3 seconds per turn). These ablations also show how individual components of AMIE (Video) meaningfully contribute to overall system performance. A rigorous offline evaluation framework is essential for systematically benchmarking agent performance and tracking progress throughout development. Our evaluation framework comprises two components: the clinical audio-visual taxonomy (Section A.1) and a multi-tiered benchmark suite. Grounded in telehealth feasibility, the taxonomy defines a principled spectrum of audio-visual signs that can be elicited and interpreted in virtual care. While exhaustively covering every clinical action and condition in the taxonomy was beyond the scope of this work, our approach establishes an extensible blueprint for evaluating multimodal clinical AI. Specifically, we pair complementary single-turn benchmarks, which isolate and evaluate discrete instances of clinical perception and diagnostic reasoning, with multi-turn simulated consultations that assess holistic, conversation-level competencies. Together, these tiers provide a computationally and logistically scalable proxy for the evaluation of real-time clinical audio-video consultations. Although the present multi-turn audio simulation communicates visual cues descriptively, future evaluation environments can directly incorporate dynamically generated synthetic patient videos [73]. While the agents in AMIE (Video) are orchestrated through prompting, multi-turn reinforcement learning with simulated patients has been shown to yield robust improvements in diagnostic accuracy and clinical safety [74]. The simulated encounters and structured evaluation criteria developed in this work could serve as a training environment for such end-to-end optimization. 5.1. Limitations The system described in this work is not ready for real-world clinical deployment and several core limitations must be acknowledged. An important architectural constraint of AMIE (Video) is its reliance on discrete turn-taking rather than continuous, multi-stream interactivity. Although the decoupled multi-agent design balances conversational latency and reasoning by running background agents asynchronously, the ultimate interaction loop still hinges on more rigid turn boundaries. Consequently, the patient and agent cannot speak simultaneously or engage in rapid, overlapping back-and-forth dialogue as human clinicians and patients naturally do during consultations. Furthermore, this audio-based separation also prevents visually triggered interjections, such as speaking up to guide a patient actor who is improperly executing a physical examination maneuver. Recent paradigms, such as interaction models [75], multi-stream LLMs [59], and tandem speech architectures like KAME [58], address these sequential bottlenecks by enabling full-duplex communication (“speaking while listening”). Despite the strong end-to-end performance AMIE (Video) demonstrated in our OSCE evaluation study, automated evaluations uncover important capability gaps that warrant careful examination. Subtle affective nuances and high-frequency movements like tremor remain difficult for the agent to reliably perceive. These granular perceptual capabilities are crucial to clinical reasoning for some medical presentations and therefore should be further developed in future work to increase robustness and coverage of audio-visual perceptive capabilities. Additionally, even when an agent successfully perceives subtle non-verbal cues, it may still fail to translate this perception into appropriate clinical action. During automated evaluations, we observed instances where the agent’s perception was accurate, yet did not trigger the appropriate proactive response (such as noticing poor audio quality but failing to request a correction). This mirrors other recent findings which demonstrate that voice 18

Towards Expert-level Medical AI for Real-time Video Consultations agents can recognize properties like distress or sarcasm but fall short of fully incorporating them in their decision-making [48]. Constructing and executing automated evaluations for complex interactive domains like audiovisual clinical consultations is a process with inherent challenges. Patient-acted videos exhibit varying degrees of clinical realism, while internet-sourced videos can lack the focused and natural presentation typical of live telehealth encounters. Furthermore, evaluating perception, reasoning, and interactivity in a targeted single-turn manner involved designing occasionally contrived examples (e.g., a patient directly asking, “It hurts right here... what part is this?”) to strictly isolate the capability being tested. Additionally, while we included a limited number of ‘adversarial’ examples, future iterations of the framework may enrich this adversarial subset with additional scenarios, such as patient actors performing examination maneuvers incorrectly or contradicting physical observations. During automated evaluation, the patient simulator’s behavior can vary slightly across simulation runs, and auto-graders exhibit a certain degree of intra-rater variability. Therefore, performance comparisons stemming from this automated process should be interpreted with appropriate caution. While automated evaluation results are valuable for relative comparisons of multiple candidate systems, their calibration with clinical evaluator ratings may not be precise. Consequently, these automated evaluations were primarily used to assess directional improvements of the system as sub-components were added or improved. However, they are not a suitable replacement for rigorous randomized studies involving human physician comparisons, professional patient actors and evaluation ratings from independent clinical evaluators. The clinical scope of the scenarios selected for the OSCE study were limited. Because evaluations relied entirely on live enactments by professional patient actors, the 100 study scenarios were restricted to “actable” clinical presentations. Consequently, important clinical scenarios featuring core audio-visual diagnostic components such as dermatological conditions that cannot be authentically simulated by actors were omitted from the study. Furthermore, while patient actors provide a useful standardized baseline for comparative evaluation, real-world clinical encounters introduce unscripted behavioral, environmental, and emotional complexities. A related constraint involved demographic alignment between patient actors and the simulated clinical conditions. While experimental protocols successfully matched actor biological sex to the clinical case descriptions and maintained a broad distribution across racial backgrounds and age groups, this challenge was particularly pronounced in simulating complex chronic and acute illnesses that disproportionately affect geriatric populations (e.g., severe cardiovascular disease, chronic obstructive pulmonary disease). This skew reflects logistical constraints in recruiting older standardized patients as well as the digital literacy required for synchronous teleconferencing platforms which remains a documented barrier to telemedicine access among older populations [76–78]. Evaluating conversational AI with older patient cohorts who present with multi-morbidity and sensory or cognitive impairments is an essential next step for clinical generalizability. While our experimental protocol implemented several measures to standardize conditions between study arms, complete blinding remained challenging. Standardization measures included utilizing an identical teleconferencing interface (capable of video interaction and text chat modalities) across all arms, asking PCPs to turn off their own camera during consultations to match the avatar-less AI system, and randomizing scenario presentation order and evaluation sequence across arms. Limitations to blinding include the synthetic vocal style of the Talker agent which remains audibly distinct from human physician speech, and the fact that the text-only arm represented an inherently different interaction format. Such unblinding is a critical consideration, as awareness of an agent’s AI identity can directly affect user behavior; it was observed in Reis et al. [79] that patients who believe they are interacting with an AI chatbot rather than a physician demonstrate lower-quality symptom reports.

19

Towards Expert-level Medical AI for Real-time Video Consultations Furthermore, asking PCPs to disable their cameras (a decision made to eliminate visual appearance confounders) reduced the ecological validity of the human baseline. Seeing the physician’s facial expressions, eye contact, and non-verbal presence contributes meaningfully to patient trust, empathy, and interpersonal rapport [1, 80, 81]. Finally, our evaluation study tests AMIE in the setting of video-based telehealth. While this is a common modality routinely used for remote clinical care, there are inherent clinical limits to this modality. Virtual clinical agents and telehealth practitioners alike are unable to perform certain physical examinations over a telehealth interface, such as palpating a spleen, assessing deep abdominal rigidity, or performing fundoscopy. Furthermore, discerning subtle audio-visual signs may be challenging in a telehealth setting, especially when audio or video quality is poor.

6. Conclusion This research advances AMIE towards real-time clinical video consultations, providing the first demonstration of expert-level AI performance in this modality through a randomized OSCE study with board-certified PCPs and professional patient actors across 100 clinical scenarios. Unlike prior text-chat evaluations, this comparison reflects the modality that physicians and patients routinely use for telehealth, offering a more ecologically valid assessment of clinical performance. These results were enabled through a novel asynchronous multi-agent architecture balancing low-latency dialogue with thorough clinical reasoning and real-time audio-visual perception. While AMIE matched or exceeded PCPs across core clinical competencies in this video configuration, PCPs were preferred for rapport and partnership building, a reminder that clinical interactions encompass dimensions beyond diagnostic and management accuracy. Limitations remain in fine anatomical precision, affective nuance, and high-frequency movement perception, and the study relied on patient actors rather than real patients. Substantial validation in real clinical workflows is essential before responsible translation. By engaging with the perceptual richness of clinical practice, this work represents a step toward AI systems that can more equitably expand access to high-quality care. Acknowledgments This project was an extensive collaboration between many teams at Google Research and Google DeepMind. We thank Rory Sayres and Yishay Mansour for their comprehensive review and detailed feedback on the manuscript. We also thank Jacinda Mein, Jen Haslip, Amira Fouad, Evie Gray, Danielle Breen, Amanda Carl-Pratt, Isha Mishra, Edna Agyemang, Aziza Henry, Kate Mischaikow, Louis Jardin, Kevin Soares, Esmeralda Cardenas, and Tim Herrmann for contributions to the narratives and visuals. We are grateful to Rachelle Sico, Cat Kozlowski, Jessica M. Williams, Celeste Grade, Oba Adewunmi, and Michael Cael Davis for their support during our research. We are grateful to Alan Karthikesalingam, Jim Taylor, John Hernandez, Naama Hammel, Susan Thomas, Raia Hadsell, Zoubin Ghahramani, Greg Corrado, Pushmeet Kohli, Michael Howell, and Demis Hassabis for their valuable support, insights and feedback during our research. We also thank Medical Science Solutions Inc. and Christopher Smith for their partnership in conducting the OSCE study, as well as SiWai Man, Brett Hatfield, Gordon Turner and Priyana Acharya. Finally, we are grateful for the support from Adam Rodman, Jason Gusdorf, Daniel Cook, Priya Gupta, Jeremy Lai, Jon Krause, Jonas Kemp, Kejia Chen, Alexey Kolesnikov, Pi-Chuan Chang, Qinghan Xue, Kishwar Shafin, Theo Guidroz, Fan Zhang, Diego Ardila, Lucas Brambrink, Quang Duong, Yuexing Hao, Bijal Rajput, Stefan Moser, Niko Schwarz, Alex Bijamov, Ayush Jain, Bhavna Daryani, Cara Tan, Daniel McDuff, Gal Elidan, Hieu Hoang, Ines Mezerreg, Jackie Barr, Jacqueline Shreibati, Joe Breda, Laura Vardoulakis, Lin Yang, Mike Sanchez, Preeti Singh, Tiya Tiyasirichokchai, Will Vaughan, Xiang Ji, Sara Mahdavi, David Barrett, David Stutz and Aleksandra Faust. 20

Towards Expert-level Medical AI for Real-time Video Consultations Data Availability Example evaluation scenarios and excerpts of patient actor scenario enactments are provided in the manuscript and appendix. For privacy reasons, we do not release recordings of patient actors at this time. Code Availability AMIE is an LLM based research AI system. We are not open-sourcing model code and weights due to the safety implications of unmonitored use of such a system in medical settings. In the interest of responsible innovation, we will be working with research partners, regulators, and providers to validate and explore safe onward uses of AMIE. For reproducibility, we have documented technical methods while keeping the paper accessible to a clinical and general scientific audience. Our work builds upon Gemini 3.0 Flash and Gemini 3.1 Pro, with details available in the corresponding model cards [20, 21]. Competing Interests This study was funded by Alphabet Inc and/or a subsidiary thereof (‘Alphabet’). Authors who are employees of Alphabet may own stock as part of the standard compensation package.

21

Towards Expert-level Medical AI for Real-time Video Consultations

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15.

16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29.

Hall, J. A., Harrigan, J. A. & Rosenthal, R. Nonverbal behavior in clinician-patient interaction. Applied & Preventive Psychology 4, 21–37 (1995). Soriano, R. P. Bates’ guide to physical examination and history taking (Lippincott Williams & Wilkins, 2025). Henry, S. G., Fuhrel-Forbis, A., Rogers, M. A. & Eggly, S. Association between nonverbal communication during clinical interactions and outcomes: a systematic review and meta-analysis. Patient education and counseling 86, 297–315 (2012). Stewart, M. A. Effective physician-patient communication and health outcomes: a review. CMAJ: Canadian medical association journal 152, 1423 (1995). Wright, H. H., O’Shea, M.-C., Sekula, J. & Mitchell, L. J. Assessment of communication skills using telehealth: considerations for educators. en. Frontiers in Medicine 9, 841309 (Aug. 2022). Kruis, R., Brown, E. A., Johnson, J., Simpson, K. N., McElligott, J. & Harvey, J. Patient perceptions of audio-only versus video telehealth visits: a qualitative study among patients in an academic medical center setting. Telemedicine reports 5, tmr–2023 (2024). Benziger, C. P., Huffman, M. D., Sweis, R. N. & Stone, N. J. The telehealth ten: a guide for a patient-assisted virtual physical examination. The American journal of medicine 134, 48–51 (2021). Tong, K. S., Waheed, M., Jackson, T. M., Sota, T. & Lau, A. Y. Translatability of physical examination to teleconsultation in primary care settings. Journal of Telemedicine and Telecare 32, 244–258 (2026). McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. Towards accurate differential diagnosis with large language models. Nature, 1–7 (2025). Liévin, V., Palepu, A., Weng, W.-H., Saab, K., Stutz, D., Cheng, Y., Kulkarni, K., Mahdavi, S. S., Barral, J., Webster, D. R., et al. Towards Conversational AI for Disease Management. Nature, 1–3 (2026). Tu, T., Schaekermann, M., Palepu, A., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Cheng, Y., et al. Towards conversational diagnostic artificial intelligence. Nature, 1–9 (2025). Saab, K., Park, C., Strother, T., Freyberg, J., Barrett, D. G., Cheng, Y., Weng, W.-H., Stutz, D., Tomasev, N., Palepu, A., et al. Advancing conversational diagnostic AI with multimodal reasoning. Nature Medicine, 1–11 (2026). Korom, R., Kiptinness, S., Adan, N., Said, K., Ithuli, C., Rotich, O., Kimani, B., King’ori, I., Kamau, S., Atemba, E., et al. AI-based clinical decision support for primary care: A real-world study. arXiv preprint arXiv:2507.16947 (2025). Zeltzer, D., Kugler, Z., Hayat, L., Brufman, T., Ilan Ber, R., Leibovich, K., Beer, T., Frank, I., Shaul, R., Goldzweig, C., et al. Comparison of initial artificial intelligence (AI) and final physician recommendations in AI-assisted virtual urgent care visits. Annals of Internal Medicine 178, 498–506 (2025). Brodeur, P., Koshy, J. M., Palepu, A., Saab, K., Homiar, A., Ruparel, R., Wu, C., Tanno, R., Xu, J., Wang, A., Stutz, D., Weng, W.-H., Ferrera, H. M., Barrett, D., Crowley, L., Lee, J., Rittner, S. E., Wulczyn, E., Zhang, S. K., Vedadi, E., Kohn, C. G., Kulkarni, K., Kadiyala, V., Mahdavi, S., Du, W., Williams, J. M., Feinbloom, D., Wong, R., Tu, T., Sirkovic, P., Orlandi, A., Semturs, C., Liu, Y., Gottweis, J., Webster, D. R., Barral, J., Chou, K., Kohli, P., Hassidim, A., Matias, Y., Manyika, J., Fields, R., Li, J. X., Cohen, M. L., Natarajan, V., Schaekermann, M., Karthikesalingam, A. & Rodman, A. A prospective clinical feasibility study of a conversational diagnostic AI in an ambulatory primary care clinic 2026. arXiv: 2603.08448 [cs.HC]. https://arxiv.org/abs/2603.08448. Sackett, D. L. in The case-control study consensus and controversy 51–63 (Elsevier, 1979). Arias López, M. d. P., Ong, B. A., Borrat Frigola, X., Fernández, A. L., Hicklent, R. S., Obeles, A. J., Rocimo, A. M. & Celi, L. A. Digital literacy as a new determinant of health: a scoping review. PLOS digital health 2, e0000279 (2023). Moore, A. A., Ellis, J. R., Dellavalle, N., Akerson, M., Andazola, M., Campbell, E. G. & DeCamp, M. Patient-facing chatbots: Enhancing healthcare accessibility while navigating digital literacy challenges and isolation risks—a mixed-methods study. Digital Health 11, 20552076251337321 (2025). Shah, M., Gusdorf, J., Palepu, A., Park, C., O’Sullivan, J. W., Ravi, V., Strother, T., Dubov, P., Rysbek, A., Fukuzawa, T., et al. Towards Conversational Medical AI with Eyes, Ears and a Voice. arXiv preprint arXiv:2605.09272 (2026). Gemini Team Google. Gemini 3 Flash Model Card 2025. Gemini Team Google. Gemini 3.1 Pro Model Card 2026. McDonough, K. et al. Complete Physical Exam Benchmarks https : / / uw . pressbooks . pub / fcmtextbook / chapter/complete-physical-exam-benchmarks/. Accessed: 2026-06-20. 2024. Caravan Health. Telehealth Physical Exam Fact Sheet https://caltrc.org/wp-content/uploads/2021/01/ Telehealth-Physical-Exam-Fact-Sheet.pdf. Accessed: 2026-06-18. 2021. Ansary, A. M., Martinez, J. N. & Scott, J. D. The virtual physical exam in the 21st century. Journal of telemedicine and telecare 27, 382–392 (2021). Dover, A. R., Innes, J. A. & Fairhurst, K. Macleod’s Clinical Examination-E-Book: Macleod’s Clinical Examination-E-Book (Elsevier Health Sciences, 2023). Sartori, D. J., Hayes, R. W., Horlick, M., Adams, J. G. & Zabar, S. R. The TeleHealth OSCE: preparing trainees to use telemedicine as a tool for transitions of care. Journal of Graduate Medical Education 12, 764–768 (2020). Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J. & Rajpurkar, P. Foundation models for generalist medical artificial intelligence. Nature 616, 259–265 (2023). Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H. & Szolovits, P. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. Applied Sciences 11, 6421 (2021). Liévin, V., Hother, C. E., Motzfeldt, A. G. & Winther, O. Can large language models reason about medical questions? en. Patterns (N. Y.) 5, 100943. issn: 2666-3899 (Mar. 2024).

22

Towards Expert-level Medical AI for Real-time Video Consultations 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 53. 54. 55. 56. 57. 58.

Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023). Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. Toward expert-level medical question answering with large language models. Nature medicine 31, 943–950 (2025). Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452 (2023). Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416 (2024). Miller, G. E. The assessment of clinical skills/competence/performance. Academic Medicine 65, S63–7 (1990). Epstein, R. M. & Hundert, E. M. Defining and assessing professional competence. en. JAMA 287, 226–235. issn: 0098-7484,1538-3598 (Jan. 2002). Schmidgall, S., Ziaei, R., Harris, C., Reis, E., Jopling, J. & Moor, M. AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments. arXiv [cs.HC]. arXiv: 2405.07960 [cs.HC] (May 2024). Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nature medicine 31, 77–86 (2025). O’Sullivan, J. W., Palepu, A., Saab, K., Weng, W.-H., Amponsah, D. K., Cheng, E., Cheng, Y., Chu, E., Desai, Y., Elezaby, A., et al. A large language model for complex cardiology care. Nature medicine 32, 616–623 (2026). Palepu, A., Dhillon, V., Niravath, P., Weng, W.-H., Prasad, P., Saab, K., Tanno, R., Cheng, Y., Mai, H., Burns, E., et al. Exploring large language models for specialist-level oncology care. NEJM AI 2, AIcs2500025 (2025). Mukherjee, S., Gamble, P., Ausin, M. S., Kant, N., Aggarwal, K., Manjunath, N., Datta, D., Liu, Z., Ding, J., Busacca, S., et al. Polaris: A safety-focused llm constellation architecture for healthcare. arXiv preprint arXiv:2403.13313 (2024). Vedadi, E., Barrett, D., Harris, N., Wulczyn, E., Reddy, S., Ruparel, R., Schaekermann, M., Strother, T., Tanno, R., Sharma, Y., et al. Towards physician-centered oversight of conversational diagnostic AI. arXiv preprint arXiv:2507.15743 (2025). Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. Towards generalist biomedical AI. Nejm Ai 1, AIoa2300138 (2024). Moor, M., Huang, Q., Wu, S., Yasunaga, M., Dalmia, Y., Leskovec, J., Zakka, C., Reis, E. P. & Rajpurkar, P. Med-flamingo: a multimodal medical few-shot learner in Machine learning for health (ML4H) (2023), 353–367. Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H. & Gao, J. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in neural information processing systems 36, 28541–28564 (2023). Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E. & Zeghidour, N. Moshi: a speech-text foundation model for real-time dialogue. arXiv [eess.AS]. arXiv: 2410.00037 [eess.AS] (Sept. 2024). Xie, Z. & Wu, C. Mini-Omni: Language models can hear, talk while thinking in streaming. arXiv [cs.AI]. arXiv: 2408.16725 [cs.AI] (Aug. 2024). Google DeepMind. Gemini Omni 2026. https://deepmind.google/models/gemini-omni/. Bartelds, M., Bianchi, F. & Zou, J. Real-Time Voice AI Hears but Does Not Listen. arXiv preprint arXiv:2606.26083 (2026). Google DeepMind. Project Astra: The future of AI assistants Accessed: 2025-01-20. 2024. https://deepmind. google/technologies/gemini/project-astra/. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824–24837 (2022). Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems 35, 22199–22213 (2022). Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A. & Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022). Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems 36, 46534–46594 (2023). Snell, C., Lee, J., Xu, K. & Kumar, A. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 (2024). Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E. & Hashimoto, T. B. s1: Simple test-time scaling in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (2025), 20286–20332. Gottweis, J., Weng, W.-H., Daryin, A., Tu, T., Sirkovic, P., Myaskovsky, A., Glowaty, G., Weissenberger, F., Orlandi, A., Popovici, D., et al. Accelerating scientific discovery with Co-Scientist. Nature, 1–3 (2026). Christakopoulou, K., Mourad, S. & Matarić, M. Agents thinking fast and slow: A talker-reasoner architecture. arXiv preprint arXiv:2410.08328 (2024). Kuroki, S., Kubo, Y., Akiba, T., Tang, Y. & Yaguchi, M. KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) arXiv:2510.02327 (2026).

23

Towards Expert-level Medical AI for Real-time Video Consultations 59. 60. 61. 62.

63. 64. 65. 66. 67. 68. 69. 70. 71. 72. 73. 74.

75. 76. 77. 78. 79. 80. 81.

Su, G., Yang, Y., Li, X. & Geiping, J. Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs. arXiv preprint arXiv:2605.12460 (2026). Harden, R. M., Stevenson, M., Downie, W. W. & Wilson, G. Assessment of clinical competence using objective structured examination. British medical journal 1, 447–451 (1975). Gentles, H. Putting them through their PACES. Practical Assessment of Clinical Examination Skills. en. BMJ 326, S76 (Mar. 2003). Hodges, B. & McIlroy, J. H. Analytic global OSCE ratings are sensitive to level of training. Medical Education 37, 1012–1016. eprint: https://asmepublications.onlinelibrary.wiley.com/doi/pdf/10.1046/j.13652923.2003.01674.x. https://asmepublications.onlinelibrary.wiley.com/doi/abs/10.1046/j. 1365-2923.2003.01674.x (2003). Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775 (2025). Hicks, R. S., Trofimov, M., Lim, D., Arora, R. K., Tsimpourlas, F., Bowman, P., Sharman, M., Tong, C., Karthik, K., Dugar, A., et al. HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats. arXiv preprint arXiv:2604.27470 (2026). Irving, G., Neves, A. L., Dambha-Miller, H., Oishi, A., Tagashira, H., Verjee, A. & Holden, J. International variations in primary care physician consultation time: a systematic review of 67 countries. BMJ Open 7, e017902 (2017). Hutton, C. & Freeman, G. The benefits of longer consultations in general practice. British Journal of General Practice 55, 60–62 (2005). Carrillo de Albornoz, S., Sia, K.-L. & Harris, A. The effectiveness of teleconsultations in primary care: systematic review. Family practice 39, 168–182 (2022). Wharton, G. A., Sood, H. S., Sissons, A. & Mossialos, E. Virtual primary care: fragmentation or integration? The Lancet Digital Health 1, e330–e331 (2019). Hampton, J. R., Harrison, M., Mitchell, J. R., Prichard, J. S. & Seymour, C. Relative contributions of history-taking, physical examination, and laboratory investigation to diagnosis and management of medical outpatients. Br Med J 2, 486–489 (1975). Peterson, M. C., Holbrook, J. H., Von Hales, D., Smith, N. & Staker, L. Contributions of the history, physical examination, and laboratory investigation in making medical diagnoses. Western Journal of Medicine 156, 163 (1992). Byambasuren, O., Greenwood, H., Bakhit, M., Atkins, T., Clark, J., Scott, A. M. & Glasziou, P. Comparison of telephone and video telehealth consultations: systematic review. Journal of medical Internet research 25, e49942 (2023). Rush, K. L., Howlett, L., Munro, A. & Burton, L. Videoconference compared to telephone in healthcare delivery: a systematic review. International journal of medical informatics 118, 44–53 (2018). Nasimzada, J., Kleesiek, J., Herrmann, K., Roitberg, A. & Seibold, C. Towards synthetic data generation for improved pain recognition in videos under patient constraints. arXiv preprint arXiv:2409.16382 (2024). Liévin, V., Schmidgall, S., Strother, T., Bijamov, A., Goel, A., Palepu, A., Park, C., Balazadeh, V., Sun, M. W., Guerard, M., Chen, J., Steiner, D., Dhillon, V., Azar, I., Mehta, A., Spetsieris, N., Shah, S., Abdelrahim, M., Dahiya, A., Liu, Y., Chou, K., Matias, Y., Hassidim, A., Webster, D. R., Le, Q. V., Hadsell, R., Barral, J., Radebaugh, C., Faust, A., Azizi, S., Schaekermann, M., Chen, P.-H. C., Tu, T., Racz, D. & Yang, L. ResidencyRL: Reinforcement Learning in Simulated Clinical Environments 2026. arXiv: 2608.07418 [cs.AI]. https://arxiv.org/abs/2608.07418. Thinking Machines Lab. Interaction Models: A Scalable Approach to Human-AI Collaboration https://thinkingmachines. ai/blog/interaction-models/. 2026. Lam, K., Lu, A. D., Shi, Y. & Covinsky, K. E. Assessing telemedicine unreadiness among older adults in the United States during the COVID-19 pandemic. JAMA internal medicine 180, 1389–1391 (2020). O’Connell, M. E., Haase, K. R., Grewal, K. S., Panyavin, I., Kortzman, A., Flath, M. E., Cammer, A., Cosco, T. D. & Peacock, S. Overcoming barriers for older adults to maintain virtual community and social connections during the COVID-19 pandemic. Clinical Gerontologist 45, 159–171 (2022). Mao, A., Tam, L., Xu, A., Osborn, K., Sheffrin, M., Gould, C., Schillinger, E., Martin, M. & Mesias, M. Barriers to telemedicine video visits for older adults in independent living facilities: mixed methods cross-sectional needs assessment. JMIR aging 5, e34326 (2022). Reis, M., Reis, F., Kim, Y. J., Demir, A., Lim, J., Gröschel, M. I., Boie, S. D. & Kunde, W. Reduced symptom reporting quality during human–chatbot versus human–physician interactions. Nature Health, 1–4 (2026). Zulman, D. M., Haverfield, M. C., Shaw, J. G., Brown-Johnson, C. G., Schwartz, R., Tierney, A. A., Zionts, D. L., Safaeinili, N., Fischer, M., Thadaney Israni, S., et al. Practices to foster physician presence and connection with patients in the clinical encounter. Jama 323, 70–81 (2020). Mast, M. S. On the importance of nonverbal communication in the physician–patient interaction. Patient education and counseling 67, 315–318 (2007).

24

Towards Expert-level Medical AI for Real-time Video Consultations

Appendix 1. Section A.1 describes the taxonomy of audio-visual cues and examination maneuvers that was used to scope auto-evaluation. 2. Section A.2 provides details on the development and mechanics of single-turn auto-evaluation examples, which were used to develop the system and characterize its clinical audio-visual capabilities. 3. Section A.3 provides detailed results on the single-turn auto-evaluations performed on different versions of the AMIE system. 4. Section A.4 provides detailed results on the simulated auto-evaluations performed on different versions of the AMIE system. 5. Section A.5 describes the 20-scenario pilot study conducted across 6 experimental arms to validate the system architecture and familiarize participants and includes automated evaluation results. 6. Section A.6 describes further details on the 100-scenario OSCE study with 3 study arms and subsequent clinical evaluation. 7. Section A.7 includes a full example of a scenario, ground truth, and corresponding case-specific rubric. 8. Section A.8 lists the supplementary evaluation rubric adapted from Sartori et al. [1] to assess telehealth audio-visual symptom observation and guided physical examinations. 9. Section A.9 details the structured post-encounter clinical questionnaire completed by PCPs and AV-AMIE following each consultation. 10. Section A.10 shows the UIs used by AMIE and PCPs across different study arms, as well as the interface used by expert raters. 11. Section A.11 contains detailed results for the OSCE study in a tabular format. 12. Section A.12 presents detailed ratings from patient actors. 13. Section A.13 presents detailed ratings from the expert evaluators. 14. Section A.14 presents details on structured interviews with patient actors.

25

Towards Expert-level Medical AI for Real-time Video Consultations

A.1. Taxonomy of audio-visual cues and physical exam maneuvers To establish an audio-visual telehealth evaluation framework, clinically relevant observations and patient-assisted examination maneuvers were organized into the taxonomy shown in Table A.2. The taxonomy separates four categories of information: (1) passive audio-visual observations, (2) patient-assisted examination maneuvers, (3) interaction behaviors that improve data quality, and (4) explicit boundaries of telehealth assessment. This distinction is important because certain clinically meaningful cues may be challenging to elicit in a virtual setting. Accordingly, each capability was annotated by telehealth feasibility (Table A.1). A key feature of this taxonomy is distinguishing findings that can be accurately simulated by standardized patients from those that require the presence of a clinical condition or abnormality or can only be elicited by in-person examination. Standardized patients are well suited for some functional, behavioral, and movement-based cues. In contrast, physical signs such as skin lesions, examination of orifices, or those requiring medical equipment (e.g. auscultation) or close up images generally require clinician-reviewed images or videos, high-quality close-up visualization, peripheral devices, or in-person assessment. For this reason, the OSCE study emphasized actable scenarios (Section 2.3), whereas the automated evaluations incorporated both actable cases and clinician-reviewed curated media (Sections 2.2.2 and 2.2.3) incorporating physiologic abnormalities. The taxonomy was developed by board-certified family medicine and internal medicine clinicians, who synthesized physical-examination benchmarks from the University of Washington School of Medicine [2], the Caravan Health telehealth physical-examination reference guide [3], the virtual physical examination framework described by Ansary et al. [4], and textbook sources [5]. These sources were consolidated into a multi-system inventory of cue families, feasibility levels, and benchmark references. Table A.1 | Telehealth-feasibility categories used in the audio-visual cue taxonomy. Code

Interpretation

Evaluation implication

H

High feasibility

The finding or maneuver can usually be observed, elicited, or derived from history taking through standard audio-video with minimal setup.

M

Moderate feasibility

The finding may be assessable only with favorable lighting, camera position, audio quality, patient cooperation, or patient-assisted examination. Responses should explicitly communicate uncertainty when the signal is limited.

L

Low feasibility

The finding is unreliable through standard audio-video alone and generally requires high-quality close-up imaging, a trained helper, specialized peripheral equipment, or in-person examination. These cases primarily evaluate caution and boundary recognition.

N

Not feasible

The finding cannot be adequately assessed through standard audio-video alone. The expected behavior is to avoid claiming that the finding was assessed and to recommend in-person or device-assisted evaluation when clinically relevant.

26

Towards Expert-level Medical AI for Real-time Video Consultations

Table A.2 | Taxonomy of audio-visual clinical cues and patient-assisted examination maneuvers for telehealth evaluation. Telehealth-feasibility codes are defined in Table A.1. A/V = audio-visual; obs. = observation; pt. = patient; sim. = simulated; JVD = jugular venous distention. Area

Cues or maneuvers

Acquisition

TH-F

Source

Evaluation focus and feasibility caveats

Abdominal

Visible discomfort, guarding, distension, pain localization, self-palpation response, movement-associated pain

Guided self-exam

M

Actor; sim.; OSCE

Coaches safe self-palpation, localizes tenderness, and escalates severe or focal pain. Deep palpation, rebound tenderness, rigidity, organomegaly, fluid wave, and shifting dullness are low-feasibility.

Cardiovascular

Home pulse/BP, capillary refill, visible edema, cyanosis, pallor

Pt. measurement; visual obs.; guided maneuver

M–L

Actor; sim.; media; OSCE

Uses patient-reported and visible perfusion findings cautiously. Cyanosis, JVD, pitting edema, and perfusion signs are lower-feasability on standard video.

Communication and behavior

Affect, anxiety, agitation, psychomotor slowing, eye contact, engagement, speech cadence

Passive A/V obs.

H

Actor; OSCE

Integrates behavioral and paraverbal cues into rapport, mental status, and severity assessment without over-interpreting normal, cultural, or neurodiverse variation.

Constitutional

General appearance, distress, fatigue, diaphoresis, pallor, apparent discomfort

Passive visual obs.

H–M

Actor; media; OSCE

Recognizes acuity and distress. Pallor, diaphoresis, and color change are lighting- and skin-tone-dependent and require caution.

Dermatologic

Rash/lesion pattern, distribution, swelling, bruising, wounds, ulcers, mottling

Visual obs.; guided close-up view

M–L

Images/clips; OSCE

Describes visible morphology and distribution. Color, texture, depth, warmth, tenderness, and subtle lesions depend on image quality; avoid definitive diagnosis with poor quality.

Environment and setup

Lighting, camera angle, audio quality, framing of relevant body region

Data-quality interaction

H

Actor; OSCE

Requests improved lighting, closer camera positioning, clearer audio, or safer framing before interpreting visual findings.

HEENT

Conjunctival injection, facial swelling, nasal congestion, voice change, oral/throat view, extraocular movements, facial symmetry

Observation; guided camera; guided maneuver

M

Actor; media; OSCE

Guides camera use safely and recognizes gross abnormalities. Pupil reactivity, red reflex, views of the oropharynx, mucosa, require close-up imaging or devices.

Musculoskeletal

Active range of motion, restrictions or discomfort with movement, posture and gait abnormalities, joint or limb swelling or redness or asymmetry, self-palpation tenderness

Guided self-exam

H–M

Actor; OSCE

Coaches movement safely and interprets functional limitation. Strength testing, joint stability from maneuvers, warmth, effusion, and fine deformity are lower-feasibility.

Neurologic

Slurred speech, confusion, altered attention, facial asymmetry, pronator drift, tremor, gait abnormality, coordination, gross strength asymmetry

Passive A/V obs.; guided maneuver; questions

H–M

Actor; media; OSCE

Recognizes acute neurologic red flags, instruction clarity, laterality, and asymmetry. Reflexes, tone, subtle weakness, sensory testing, and detailed cognitive testing are lower-feasibility.

Pain behavior

Wincing, splinting, reluctance to move, abnormal posture of body parts, guarding, visible discomfort during maneuvers or gait

Visual obs.; maneuver response

H–M

Actor; OSCE

Uses non-verbal pain behavior to guide follow-up and management; it should complement, not replace, patient-reported symptoms.

Respiratory

Cough, hoarseness, inspiratory (stridor) or expiratory (wheeze) sounds, dyspnea while speaking, increased work of breathing, accessory muscle use, pursed lip breathing, cyanosis.

Passive A/V obs.

H–M

Actor; media; OSCE

Detects respiratory difficulty and escalates appropriately. Wheeze, stridor, significant rapid breathing, pursed lip breathing, cyanosis all lower feasibility. Capturing subtle work of breathing depends on audio quality, framing, and severity.

Safety and boundaries

Severe dyspnea, chest pain with distress, acute neurologic deficit, fall risk, unsafe self-exam conditions, privacy-sensitive exposure, self harm

Observation; interaction control

H

Actor; sim.; OSCE

Stops unsafe maneuvers, avoids privacy-invasive requests, states telehealth limitations, and recommends urgent or in-person care when indicated.

Spatial orientation and localization

Pointing gestures, anatomical landmark identification, body part recognition, laterality

Passive visual obs.

H

Actor; media; OSCE

Maps pointing gestures to the correct anatomical landmarks. Correctly distinguishes the patient’s left/right from the viewer’s perspective, regardless of patient position.

Inaccessible by A/V

Auscultation, percussion, deep abdominal palpation and other body parts (e.g. breast, genitals, back, etc), detailed ear or eye examination (fundoscopy, otoscopy) w/o device, pulse oxygenation w/o device, genital/rectal exam, other intimate exam

Boundary condition

N

Boundary tests

Avoids fabricating findings and states that assessment requires in-person examination, a trained examiner, or appropriate peripheral devices.

27

Towards Expert-level Medical AI for Real-time Video Consultations

A.2. Single-Turn Agent Auto-evaluation Design We used a multi-stage curation pipeline to construct evaluation scenarios (Figure A.1) from the clinical taxonomy in Section A.1: • Enumeration of audio-visual cues: We started by enumerating the list of audio-visual cues described in Section A.1 and separating them based on whether they were “actable” or not. • Actable Scenario Generation: For “actable” clinical cues, an LLM (Gemini 3 Flash) was used to generate multiple draft scenarios. Each scenario establishes a short dialogue scene culminating in a specific “target turn.” (for example, a patient abruptly coughing, with the agent evaluated on its capacity to recognize this in either its verbal response or its internal reasoning traces). An LLM classifier along with human clinicians further filtered to the scenes that were actable. One of 24 team members enacted the remaining scenes. • Real-World Video Sourcing: Non-actable audio-visual cues from Section A.1 were sourced from videos on YouTube and Reddit where possible. From each source artifact, a time window isolating the relevant sign was extracted. To reliably elicit an agent response, synthetic conversation history was integrated where necessary without being suggestive of the particular audio-visual sign being tested. For example, a video depicting a user with conjunctival erythema was augmented with a synthetic audio prompt of "Doctor, what do you notice?" Grading instructions for each scenario were generated based on the scenario details and transcript. These scenario-specific grading instructions underwent human clinician refinement for a random subset of 200 cases, with the rest going through automated review. In total, the finalized evaluation suite comprises 217 (43.4%) patient-acted cases and 283 (56.6%) examples curated from real-world patient videos, yielding a benchmark of 500 single-turn evaluation scenarios. The final evaluation set covers many, but not all of the capabilities described in the taxonomy. Grade response according to rubric Input fixed context to agent THOUGHT: The user reports no falls or trauma to the leg. … To better assess the weakness, I should ask them to perform a functional movement or show their gait, if safe. This aligns with the high-priority goal "assess_gait_and_motor_function".

AMIE

Please position the camera so I can see your legs. Then, can you try lifting your right knee while seated, and then your left, so I can compare the movement? Sure.

Patient

Generate sampled response from agent under test … AMIE

Success Rubric The agent must identify the patient's difficulty or attempt to lift their right knee/leg in either its THOUGHT block or spoken response (accepting plain-language terms like "struggling to raise leg"). Additionally, the spoken response must address the cue with a clinically relevant follow-up question.

score: 1.0 I can see lifting your right knee was more difficult than lifting your left, Have you experienced any numbness, pain, or tingling in your right leg?

Partial Rubric The agent notes the right knee/leg weakness in its THOUGHT block, but its spoken response is generic / irrelevant / hallucinates; OR identifies a physical cue but misidentifies it (e.g., noting general discomfort without recognizing the motor difficulty).

score: 0.5 I see you’re experiencing some pain. Have you experienced any numbness, pain, or tingling in your leg?

Failure Rubric

score: 0.0

The agent completely fails to notice the physical cue in both THOUGHT and spoken response, providing a generic or irrelevant response.

Both legs appear normal. Have you felt unwell or experienced any weight loss?

Figure A.1 | Single-turn Agent Evaluation Example. Each single-turn agent evaluation serves as a unit test for telehealth capabilities. In this illustrative example, the agent is expected to both perceive a clinical cue, i.e. the user’s motor difficulty, and reason about it to propose a clinically relevant follow-up question, e.g. further assessing the user’s gait and motor function or their experience of the pain. In evaluation, the previous conversation history is provided as fixed context, then the agent response is sampled at test. Next, a Gemini 3 Flash-based auto-rater scores the response based on a rubric, which has been auto-generated with clinical guidance on the ground truth.

28

Towards Expert-level Medical AI for Real-time Video Consultations

A.3. Single-Turn Agent Auto-evaluation Ratings

0.88

0.95

0.88

0.85

0.55

0.57

Abdominal pain and guarding assessment (n=5)

0.65

0.88

0.45

0.33

0.60

0.65

Abdominal distension, ascites, and wall contour (n=5)

0.00

0.30 0.00

0.50 0.00

Localized abdominal tenderness (n=5) Audible bowel sounds (n=1)

0.89

0.91

0.57

0.59

0.52

Vital signs measurement and heart rate calculations (n=7)

0.75

0.72

0.59

0.69

0.69

0.66

Palpitation symptom (n=4)

0.67

0.72

0.72

0.62

0.59

0.56

Orthostatic heart rate changes including POTS (n=8)

0.91

0.91

0.50

0.56

0.59

0.69

Eyelid lesions including chalazia, blepharitis, and ptosis (n=4)

0.85

0.82

0.56

0.59

0.76

0.54

Oral candidiasis, ulcers, and mucosal lesions (n=13)

0.70

0.70

0.43

0.54

0.79

0.64

Tonsillar exudates, swelling, and peritonsillar abscess (n=7)

0.73

0.67

0.56

0.48

0.60

0.60

Oral and tongue masses, dental extraction sites (n=6)

0.66

0.70

0.52

0.46

0.52

0.55

Nasal discharge, congestion, and septal deviation (n=7)

0.66

0.72

0.28

0.50

0.53

0.62

Thyroid swelling and swallowing motions (n=4)

0.74

0.63

0.43

0.44

0.52

0.53

Sclera and conjunctiva color (n=18)

0.77

0.77

0.36

0.38

0.56

0.43

Cervical lymphadenopathy and other neck masses (n=14)

0.68

0.64

0.39

0.42

0.55

0.53

Pharyngeal erythema and posterior lymphoid cobblestoning (n=16)

0.81

0.75

0.31

0.19

0.62

0.25

Parotid and salivary gland enlargement (n=2)

0.46

0.42

0.17

0.18

0.33

0.24

Strabismus or exotropia (n=9)

0.62

0.38

0.38

0.50

0.46

Peripheral edema (n=3)

0.52

0.48

0.29

0.33

0.33

0.50

Capillary refill time and peripheral perfusion (n=6)

0.42

0.38

0.33

0.08

0.21

0.17

Ear canal discharge and hearing issues (n=3)

0.41

0.41

0.14

0.27

0.18

0.12

Peripheral blood pooling and vascular discoloration (n=7)

0.33

0.33

0.29

0.12

0.21

0.21

Dysphonia including vocal cord paralysis (n=3)

0.25

0.26

0.16

0.18

0.16

0.17

Temporomandibular joint clicking, popping, and subluxation (n=12)

0.00

0.12

0.00

0.00

0.12

0.12

Pupillary reflexes and anisocoria (n=1)

1.00

0.88

1.00

1.00

0.75

0.50

Digit deformities and swelling (n=1)

0.66

0.59

0.59

0.50

0.53

0.56

Joint range of motion and mobility limitations (n=12)

0.57

0.67

0.36

0.47

0.42

0.35

Large joint effusion or synovial swelling (n=9)

0.58

0.58

0.22

0.22

0.41

0.45

Spinal curvature, alignment, and asymmetry (n=8)

0.65

0.58

0.58

0.50

0.62

0.62

Mental status including orientation and alertness (n=6)

0.65

0.62

0.46

0.51

0.53

0.44

Cranial nerve palsy including tongue deviation (n=10)

0.54

0.46

0.46

0.32

0.43

0.33

Postural balance loss and gait instability (n=14)

0.38

0.38

0.50

0.25

0.31

0.44

Cerebellar signs such as ataxia or dysmetria (n=2)

0.40

0.47

0.33

0.28

0.26

0.15

Pronator drift and paresis (n=9)

0.23

0.21

0.12

0.12

0.09

0.12

Tremors at rest or with action (n=25)

0.03

0.00

0.06

0.06

0.06

0.09

Nystagmus (n=4)

1.00

1.00

1.00

0.75

0.88

0.88

Antalgic posturing and protective trunk flexion (n=1)

0.88

1.00

0.75

0.88

0.88

1.00

Joint guarding and limb splinting (n=1)

0.94

0.88

0.81

0.75

0.75

0.56

Facial grimacing, wincing, and flinching (n=2)

0.71

0.58

0.58

0.46

0.67

0.62

Audible pain vocalizations (n=3)

0.69

0.69

0.72

0.75

0.69

0.69

Respiratory rate measurement (n=4)

0.47

0.44

0.53

0.36

0.41

0.42

Visual work of breathing and respiratory distress (n=11)

0.31

0.31

0.56

0.38

0.34

0.34

Breathlessness (inability to complete full sentences) (n=4)

0.50

0.31

0.50

0.50

0.12

0.06

Audible wheezing sounds (n=2)

0.42

0.21

0.08

0.12

0.00

0.25

Audible cough recognition (n=3)

0.64

0.66

0.59

0.61

0.54

0.52

Guiding patient palpation to anatomical landmarks (n=7)

0.61

0.66

0.47

0.55

0.62

0.54

Abdominal quadrant and trunk localization (n=12)

0.58

0.60

0.44

0.40

0.54

0.50

Lower extremity and foot landmark localization (n=6)

0.59

0.52

0.31

0.38

0.43

0.31

Hand, digit, and small joint pinpointing (n=11)

0.46

0.45

0.36

0.29

0.30

0.38

Left vs. right anatomical laterality determination (n=17)

AMIE (Talker only)

Shah et al.

Shah et al. (Talker only)

0.62

0.75

0.50

0.62

0.50

Speech hesitation and word-finding difficulty (n=1)

0.50

0.50

0.50

0.50

0.50

0.38

Sensitive self-examination guidance (n=1)

0.39

0.39

0.36

0.22

0.35

0.18

Speech fluency and articulation (n=9)

0.33

0.42

0.25

0.27

0.25

0.21

Empathetic response to anxiety, distress, or anhedonia (n=6)

0.25

0.31

0.19

0.25

0.31

0.25

Eye contact, head nodding, and conversation engagement (n=2)

0.28

0.27

0.19

0.14

0.24

0.17

Affective state, expressiveness, and mood congruence (n=11)

0.54

0.57

0.51

0.62

0.38

0.46

Fever, chills, rigors, and sweating (n=10)

0.44

0.48

0.40

0.35

0.29

0.23

Generalized fatigue, malaise, and weight changes (n=6)

0.88

0.62

0.64

0.81

0.66

Erythema, cellulitis, and expanding inflammatory lesions (n=10)

0.83

0.86

0.62

0.66

0.80

0.67

Allergic contact dermatitis and linear skin eruptions (n=8)

0.76

0.78

0.73

0.62

0.79

0.67

Urticaria (hives) (n=12)

0.79

0.79

0.66

0.64

0.75

0.71

Parasitic and infectious skin conditions (n=7)

0.84

0.77

0.52

0.55

0.70

0.66

Inflammatory acne, comedones, and folliculitis (n=7)

0.72

0.66

0.56

0.59

0.69

0.62

Pigmented lesions (n=4)

0.70

0.60

0.50

0.42

0.68

0.65

Vascular livedo, purpura, and palmar erythema (n=5)

0.73

0.67

0.41

0.41

0.53

0.50

Alopecia, scalp scaling, and hair loss patterns (n=8)

0.75

0.72

0.25

0.34

0.59

0.56

Eczema, dry skin, scaly plaques including psoriasis (n=4)

0.75

0.68

0.33

0.30

0.50

0.45

Sebaceous cysts, subcutaneous nodules, and lipomas (n=5)

0.65

0.58

0.38

0.31

0.56

0.50

Vesicular and bullous eruptions (n=6)

0.57

0.60

0.25

0.30

0.28

0.28

Finger clubbing and nail abnormalities (n=5)

0.60

0.53

0.33

0.40

0.50

0.33

Lighting conditions, motion blur, and video feed stability (n=5)

0.58

0.61

0.28

0.25

0.45

0.47

Camera framing, obstruction, and patient presence (n=8)

0.21

0.17

0.15

0.08

0.15

0.12

Audio clarity, background noise, and acoustic interference (n=6)

AMIE (Talker only)

Shah et al.

Shah et al. (Talker only)

AMIE

AMIE AMIE (Perception (Planner only) only)

NEUROLOGIC

0.88

MUSCULOSKELETAL

0.62

PAIN BEHAVIOR

DERMATOLOGIC

0.00

0.40

0.82

0.86

ENVIRONMENT AND SETUP

0.00

0.42

RESPIRATORY

CONSTITUTIONAL

COMMUNICATION AND BEHAVIOR

CARDIOVASCULAR

0.00

0.50

HEENT

0.50

SPATIAL ORIENTATION AND LOCALIZATION

ABDOMINAL

The 500 single-turn agent evaluation examples, described in Sections A.2 and 2.2.2, were used to compare different configurations of the AMIE (Video) system, including the full system, a version without the Perception Agent, a version without the Planner Agent, and a version without both of these background agents. Furthermore, the agent harness presented in Shah et al. [6] was tested (updating the model endpoints from Gemini 2.5 Flash to Gemini 3.0 Flash for better comparability), as well as a version of that system without their sequential clinical Planner agent. Gemini 3.0 Flash served as the LLM-based autorater for these evaluations, and each example was tested four times per configuration. To aid in visualization and interpretation, we used Gemini 3.0 Flash to map each evaluation example to group related examples into discrete audio-visual competency descriptions, based on their transcript and ground truth.

AMIE

AMIE AMIE (Perception (Planner only) only)

Figure A.2 | Agent Evaluation Ratings. Average auto-rated score across 4 replications per agent configuration on audio-visual evaluation examples. For visualization purposes, examples are grouped into low-level descriptions as well as high-level taxonomy categories from Section A.1.

29

Towards Expert-level Medical AI for Real-time Video Consultations

A.4. Simulated Auto-evaluation Ratings The 20 scenarios from Shah et al. [6] were simulated with an LLM-based patient simulator and LLM-based autorater both using Gemini 3 Flash. Notably, these scenarios were also designed with case-specific rubrics, although the criteria had been organized into 7 domains rather than the 5 domains used in the primary OSCE study (Section A.6). These simulated evaluations were used while developing the eventual AMIE (Video) system to guide prompting and system design; improving latency, proactivity of physical observation and examination, and completeness of history taking were major areas of focus. Here, we report the performance of various versions of the AMIE (Video) system, including the full system, the system without the Perception Agent, without the Planner Agent, and without both of these background agents. Additionally, the agent harness presented in Shah et al. [6] was tested (updating the model endpoints from Gemini 2.5 Flash to Gemini 3.0 Flash for better comparability) with and without their sequential clinical Planner agent. We present auto-evaluated scores on the case-specific rubrics from these scenarios, as well as the mean response latency for each agent during the simulation. AMIE

AMIE (Planner only)

AMIE (Perception only)

AMIE (Talker only)

Shah et al. (Talker only)

a Case-Specific Rubrics

Shah et al.

b Mean Turn Latency

100 25

Using asynchronous orchestration

Points Earned (%)

80

60

20

40

20

Overall

History Taking

Physical Exams

Clinical Reasoning

Latency (s)

0

15

Reduced Talker thinking

100 10

Points Earned (%)

80

60 5

40

20

0

0 Communication & Counseling

Treatment Steps

Triage

Red Flags

Figure A.3 | Auto-evaluation Results from Simulated Multi-turn Conversations. Conversations were simulated between an audio-based patient grounded on one of the twenty scenarios from Shah et al. [6]. a) Auto-evaluated ratings on the case-specific rubrics from these scenarios, spanning 7 clinical domains as well as overall. b) Mean latency of the different agents. Error bars correspond to 95% bootstrap confidence intervals (n=1000).

30

Towards Expert-level Medical AI for Real-time Video Consultations

A.5. Pilot Study: System Validation and Modality Ablation Prior to executing the main 100-scenario clinical evaluation study (Section 2.3), we conducted a dedicated pilot study consisting of 20 clinical scenario packs. This preparatory phase served three fundamental operational and scientific objectives: (1) verifying the end-to-end reliability of the realtime audio-visual interface, (2) familiarizing professional patient actors and primary care physicians (PCPs) with the study interfaces and interaction dynamics, and (3) performing a small-scale controlled, multi-modal ablation for both AMIE and PCPs to understand the contributions of audio alone versus full synchronous audio-video interaction. A.5.1. Study Design and Automated Evaluation Methodology The pilot study evaluated 20 scenario packs, which had the same form as those used in the main study (see Section A.7). The study design was largely similar to the main study, except each simulated patient interacted with six arms instead of three. The six arms in this pilot run comprised of both AMIE and baseline PCPs across three communication modalities: text-only, audio-only, and synchronous video+audio. The same patient actor interacted with each arm. A different PCP was used for the three PCP sessions for a given scenario to eliminate the potential of the PCPs using information from prior interactions. During this pilot stage, human expert evaluators were not utilized; instead, consultations were evaluated with an auto-rater (LLM-as-a-judge). Given the consultation transcript and postquestionnaire responses, Gemini 3.0 Flash generated scores for each criterion in the case-specific rubric (20 points total). A.5.2. Pilot Study Results The automated evaluation results suggest that AMIE is able to have clinically meaningful conversations (rated at or above those of PCPs) with the patient actors in all modality settings. AMIE and PCPs benefit from access to both audio inputs and video inputs. The automated evaluation metrics across the six arms are summarized in Table A.3. Table A.3 | Pilot Study Case-Specific Rubric Results. Summary of automated evaluation performance metrics across the clinical scenarios. Results compare AMIE and baseline PCPs across six experimental arms (text-only, audio-only, and video+audio), showing mean scores and 95% bootstrap confidence intervals. AMIE Evaluation Metric History Taking

Text

Audio

Baseline PCPs Video+Audio

Text

Audio

Video+Audio

78.3%

78.3%

81.7%

57.5%

77.5%

72.5%

[71.7%, 85.8%]

[68.3%, 87.5%]

[70.8%, 90.8%]

[49.2%, 67.5%]

[66.7%, 87.5%]

[58.3%, 85.0%]

Perception and Examination

42.5%

52.1%

70.4%

26.2%

37.9%

55.0%

[27.5%, 57.5%]

[38.3%, 65.4%]

[59.2%, 81.7%]

[13.8%, 38.8%]

[25.0%, 52.1%]

[38.8%, 71.2%]

Clinical Reasoning and Diagnosis

92.5%

90.0%

97.5%

75.0%

80.0%

76.2%

[86.2%, 97.5%]

[81.2%, 97.5%]

[93.8%, 100.0%]

[61.2%, 87.5%]

[68.8%, 90.0%]

[61.3%, 90.0%]

Treatment Plan and Management

61.3%

61.3%

62.5%

53.8%

63.7%

67.5%

[51.2%, 71.2%]

[47.5%, 73.8%]

[52.5%, 72.5%]

[40.0%, 67.5%]

[50.0%, 75.0%]

[53.8%, 81.2%]

Patient Communication and Education

95.0%

95.0%

90.0%

72.5%

70.0%

72.5%

[87.5%, 100.0%]

[87.5%, 100.0%]

[82.5%, 97.5%]

[62.5%, 85.0%]

[60.0%, 80.1%]

[52.5%, 90.0%]

Overall Rubric Score

72.4%

73.7%

79.7%

55.6%

66.7%

68.9%

[67.1%, 77.1%]

[66.7%, 79.7%]

[74.0%, 85.2%]

[48.5%, 62.5%]

[60.5%, 73.0%]

[58.2%, 79.0%]

31

Towards Expert-level Medical AI for Real-time Video Consultations

A.6. OSCE Study Details We conducted a randomized, partially blinded study comparing the performance of various configurations of AMIE against PCPs on diagnostic dialogue in a synchronous video-based telehealth setting. We adapted the virtual Objective Structured Clinical Examination (OSCE) framework for evaluating conversational AI introduced in Tu et al. [7], designing 120 scenario packs. The scenario packs and evaluation rubrics were enriched to ensure each scenario included potential audio-visual cues and guided examination components. An initial pilot of 20 scenarios was conducted across 6 arms (AMIE Video, AMIE Audio, AMIE Text, PCP Video, PCP Audio, PCP Text). This pilot served as a preliminary validation of the system, a preparatory run to familiarize patient actors and PCPs with the interface, and a smaller-scale ablation to isolate the benefits of audio alone from video (Section A.5). For the full study run, 100 scenarios were conducted with 3 arms (AMIE in text-only and video+audio settings, and PCPs in the video+audio setting). All sessions were assessed by clinical evaluators. A.6.1. Remote Video-based OSCE Examination The OSCE is a standardized practical assessment widely used in healthcare training to evaluate clinical skills and competencies by simulating real-world practice. Unlike traditional knowledge-based examinations, the OSCE assesses practical skills of clinical encounters, typically involving candidates rotating through a series of timed stations where they encounter a trained patient actor portraying a specific clinical scenario. Test takers perform designated tasks such as taking a medical history, conducting a physical examination, interpreting results, or counseling the patient. Examiners observe these interactions and score the test taker’s performance against detailed, predefined checklists that assess the specific skill components in each scenario. Our study was heavily inspired by this OSCE framework, focusing on scenarios which could be performed in a virtual, video-based telehealth setting (Section A.6.2). Scenarios were prepared by external clinical providers and encompassed several broad clinical domains to ensure comprehensive coverage and diversity of audio-visual cues relevant to telehealth (with some exceptions based on actability). During the study, patient actors (Section A.6.3) reviewed these scenario packs and used a video call interface similar to Google Meet (Figure A.4) to interact with AMIE or a PCP in a randomized sequence. To assist with blinding, PCPs were asked to keep their cameras off; however, full blinding was not possible due to differing communication styles and voices between PCPs and AMIE. We note that the requirement for PCPs to have their cameras off may have impacted their ability to establish rapport and empathy compared to a typical audiovisual interaction, and this difference should be considered when interpreting findings related to communication and partnership building. Following the conversation, the PCPs and AMIE each provided a post-questionnaire (Sections A.6.6 and A.9), which included their differential diagnosis and management recommendations. Patient actors rated the conversational quality of their interaction with either AMIE or the PCP. Finally, an independent panel of clinical evaluators (Section A.6.7) comprised of board-certified PCPs evaluated conversation quality and clinical accuracy from the full video, transcript, and post-questionnaires, using the review interface shown in Figure A.5.

32

Towards Expert-level Medical AI for Real-time Video Consultations A.6.2. Scenario Packs This study utilized 120 distinct clinical case scenarios relevant to telehealth consultations for primary care conditions in the US. The scenarios were first developed by clinical experts from a Canadian organization with extensive experience in preparing scenario packs and simulated patients for OSCE examinations. Subsequently, each scenario underwent independent verification by one or more US board-certified PCPs. Each scenario pack was associated with a ground truth diagnosis, a set of plausible alternative diagnoses, ground truth investigations and management, and a case-specific evaluation rubric (Section A.7). The scenario packs represented a range of conditions, focused on initial patient presentations, and were distributed across five domains where physical assessments are utilized and feasible in telehealth based on HHS Guidelines† : cardiopulmonary, abdominal, HEENT (head, eyes, ear, nose, throat), neurology and psychiatry, and musculoskeletal. Each domain included two “non-disease” cases for which the presenting complaint had resolved. Within each domain, scenarios and conditions were structured so that a broad breadth of physical cues and guided examination proficiencies could be assessed. We intentionally excluded the following domains from the study: pediatrics, dermatology (limited ability for patient actors to demonstrate), routine chronic care follow-ups and annual physicals (limited ability to assess differential diagnosis), obstetrics, breast/gynecology/genital areas (sensitive body parts for patient actors), and inpatient scenarios. A.6.3. Patient Actors To ensure clinical realism and standardization, consultations were conducted by a dedicated cohort of 15 professional patient actors who reviewed their assigned clinical scenario packs in advance and executed standardized portrayals of patient affect, physical examination features, and guided examination maneuvers. Each clinical scenario was strictly matched to the actor’s biological sex. The participating cohort comprised 7 females (46.7%) and 8 males (53.3%). The cohort spanned a diverse distribution of racial backgrounds and age groups (age range: 20–75 years; median age bracket: 30–40 years). Prior to the primary 100-scenario study, patient actors participated in a 20-scenario pilot evaluation (Section A.5) to familiarize themselves with the Google Meet-style interface and calibrate their delivery across the study arms. A.6.4. Patient Actor Rubrics Following a consultation with the provider, patient actors provided answers to established criteria from GMCPQ (General Medical Council Patient Questionnaire), PACES (Practical Assessment of Clinical Examination Skills), and PCCBP (Patient-Centered Communication Best Practice) [7]. To elicit opinions on the different interaction modalities, the patient actors were additionally asked the following set of questions, providing a five-point Likert rating from “Strongly Disagree” to “Strongly Agree”: • I was able to effectively communicate my health concerns and condition through this communication method. • I felt the clinician fully understood my concerns through this communication method. • I found this communication method easy and convenient to use. † https://telehealth.hhs.gov/providers/preparing-patients-for-telehealth/telehealth-physical-exam

33

Towards Expert-level Medical AI for Real-time Video Consultations A.6.5. Baseline Primary Care Physicians To establish a relevant human baseline for comparison, consultations across the PCP study arm were conducted by a cohort of 10 board-certified PCPs selected from a wider pool of potential PCPs based on type of board certification, type of medical qualification, and experience with urgent care or telemedicine care. The participating cohort comprised 10 physicians with diverse educational backgrounds: four held MDs from US medical schools, four held DOs from US osteopathic schools, and two held international MDs. Regarding board certification, eight were certified in Family Medicine and two in Internal Medicine. On average, the cohort had 4.5 years of post-residency clinical experience (range: 1–9 years). This distribution is broadly representative of US PCPs. Prior to initiating the primary 100scenario study, participating PCPs were involved in a 20-scenario pilot evaluation (Section A.5) to familiarize themselves with the protocol and Google Meet-style interface, mitigating technical issues during the main evaluation. A.6.6. Post Questionnaires Following each simulated OSCE encounter, the respective clinical provider, whether PCP or an AMIE agent, completed a structured post-encounter clinical questionnaire documenting their differential diagnosis, clinical observations, and management planning (Section A.9). Because this step is less latency-constrained than live conversations, the AMIE system utilizes inference-scaling via multiple drafts and synthesis (as in Lievin et al. [8]) to improve the quality of its recommendations. A.6.7. Clinical Evaluators All consultation transcripts, video recordings, and post-encounter questionnaires were reviewed by an external panel of 20 independent clinical evaluators. Unlike prior studies of the AMIE system [7, 8] that utilized specialist physicians for evaluation, this study used board-certified PCPs as the clinical raters. This provided a more appropriate set of peers to evaluate clinical encounters within the telemedicine primary care setting. To distinguish this evaluator panel from the baseline PCPs who participated in the OSCE encounters (Section A.6.5), the clinical raters were recruited through a selective vetting process, emphasizing board certification in relevant specialties and extensive clinical experience including telemedicine, urgent care, or primary care. The evaluator cohort was entirely separate from the consultation cohort. The characteristics of the evaluators were as follows: type of medical degree (MD from US medical school 9, DO from US osteopathic school 6, MD from international medical school 5), board certification (Family Medicine 13, Internal Medicine 5, Emergency Medicine 2), and mean years since residency 14.95 years (range 8 to 25). Evaluators provided ratings on both general criteria (Section A.6.8) and case-specific criteria (Section A.6.9) using the review interface shown in Figure A.5. The same evaluator rated the performance of all study arms for a given scenario in a randomized order, and were not provided any explicit information about the source of each consultation. A.6.8. General Clinical Rating Rubrics Clinical evaluators assessed each encounter using established clinical communication and diagnostic reasoning rubrics adapted from prior benchmark evaluations [7, 8], primarily utilizing Likert scales across the general criteria. These overarching instruments evaluate multiple facets of clinical 34

Towards Expert-level Medical AI for Real-time Video Consultations competence: • Differential Diagnosis (DDx) Accuracy: Assesses the ranking and inclusion of ground-truth clinical conditions [7]. • Diagnosis & Management Rubric: Measures the appropriateness and comprehensiveness of proposed diagnoses, investigations, treatment recommendations, escalation decisions, and follow-up planning [7, 8]. • Patient-Centered Communication Scales (PACES): Evaluates clinical history-taking skills, empathy, and the effectiveness of explaining information [7]. • Patient-Centered Care Benchmarking Protocol (PCCBP): Gauges the provider’s alignment with shared decision-making and patient education principles [7]. To supplement these general instruments for synchronous video encounters, we incorporated a targeted set of telehealth evaluation criteria adapted from Sartori et al. [1]. These supplementary items evaluate the provider’s perceptual acuity in utilizing live video streams for information gathering and their ability to proactively partner with patients to execute guided physical examination maneuvers (Section A.8). A.6.9. Case-Specific Clinical Rating Rubrics To evaluate clinical performance with sufficient granularity and nuance, each scenario was paired with a 20-point case-specific rubric, as in Shah et al. [6]. An example rubric is shown in Section A.7. During the grading process, evaluators could assess each individual criterion as fully met, partially met, or not met, corresponding to the award of full (1.0 pt), half (0.5 pt), or zero (0.0 pt) points, respectively. The criteria for each case were standardized across five clinical domains: • History Taking: Evaluates the thoroughness and structure of the clinical interview, including eliciting key symptoms and their characteristics, eliciting risk factors, reviewing past medical history, and identifying potential red flags or warning signs of more serious or acute conditions. • Perception and Examination: Assesses perceptual acuity regarding non-verbal cues, general patient affect, physical distress manifestations, and virtual examination maneuvers. • Clinical Reasoning and Diagnosis: Evaluates diagnostic accuracy and the capacity to synthesize appropriate differential diagnoses spanning probable, plausible, and “must-not-miss” conditions. • Treatment Plan and Management: Evaluates the clinical appropriateness, safety, and urgency of recommendations, including immediate management steps, diagnostic investigations, and follow-up planning. • Patient Communication and Education: Assesses interpersonal communication quality, including the effectiveness of conveying clinical seriousness, and provides appropriate reassurance, establishing clear, actionable next steps.

35

Towards Expert-level Medical AI for Real-time Video Consultations

A.7. Scenario and Ground Truth Example Table A.4 | Example Scenario Details. Complete scenario context details for a gastritis scenario (the corresponding clinical ground truth and evaluation rubric are shown in Table A.5). Patient Medical Specialty Presenting Complaint Initial Presentation

Maria Lopez (38-year-old Female, Hispanic, New York) Abdominal You have been experiencing upper abdominal discomfort and nausea for the past week. "Hi doctor, I’ve had this dull ache in my upper stomach area for about a week now. It’s not terrible pain, but it’s constant, and I’ve been feeling nauseous too. Sometimes I burp a lot, and the discomfort seems worse after I eat. It’s starting to worry me."

Info to be Given if Asked

Onset and Duration: The discomfort started 7 days ago. Initially, it was mild, but it has become more noticeable and persistent over the past 3–4 days. Location and Character of Pain: Pain is located in the upper central abdomen (epigastric region). • Describes it as a dull, burning ache rather than sharp pain. Aggravating and Relieving Factors: Pain worsens after meals, especially spicy foods or coffee. • Slight relief when resting or taking antacids. Associated Symptoms: Nausea, especially in the morning. • Occasional bloating and frequent burping. • No vomiting or blood in vomit • No black, tarry stools or visible blood in the stool. • No trouble swallowing. • No recent weight loss or appetite loss. • No cough or respiratory symptoms. • No chest pain or palpitations. Dietary Triggers: Drinks 2–3 cups of coffee daily. • Recently started eating more takeout (pizza, fried foods) due to a busy schedule. Medication Use: Has been taking ibuprofen almost daily for the past 2 weeks for knee pain. • Occasional use of over-the-counter antacids with mild relief. Red Flag Symptoms (Absent): No severe, sudden-onset abdominal pain. • No vomiting of blood or coffeeground material. • No unexplained weight loss. • No fever or chills.

Clinical History

Past Medical History: Mild knee osteoarthritis. • No known ulcers or previous gastrointestinal disease. Current Medications: Ibuprofen 400 mg daily (taken for the past 2 weeks). • Antacid tablets as needed. • Daily multivitamin. Allergies & Adverse Reactions: No known drug allergies. Relevant Previous Medications: Ibuprofen 400 mg daily (taken for the past 2 weeks). • Antacid tablets as needed. • Daily multivitamin.

Social & Family History Social & Personal Circumstances: Works as an office administrator. • Non-smoker. • Drinks wine socially (1–2 glasses on weekends). Family History: No known family history regarding symptoms. Occupational History: Works as an office administrator. Lifestyle: Works as an office administrator. • Non-smoker. • Drinks wine socially (1–2 glasses on weekends). Travel History: No recent travel. Concerns & Questions

Concerns: “Could this be something serious, like an ulcer or even cancer?” Expectations: “I’d like to know if I need tests or medications to help with the pain.” Wishes: “I just want this discomfort to stop so I can eat normally again.” Specific Questions: Is it safe to keep taking ibuprofen for my knee pain? • Could this be caused by stress or something I ate? • Do I need an endoscopy or any other tests?

Patient Instructions

1. General Instruction: Speak in a calm, slightly tired tone, occasionally holding your upper abdomen when describing discomfort. Avoid dramatic expressions of pain — keep it mild to moderate. 2. Abdominal Pain Demonstration (If Asked): Point to the area just below the breastbone and slightly above the belly button. • Say: “The ache is mostly here, right in the middle.” 3. Aggravation with Food (If Asked): Rub your stomach gently and say: “It feels worse about 30 minutes after I eat, especially if it’s spicy or fried food.” 4. Medication History (If Asked): Look slightly concerned and say: “I’ve been taking ibuprofen every day for the past few weeks because of my knee pain. Could that be making it worse?” 5. Nausea Description (If Asked): Make a mild grimace and say: “I feel queasy most mornings, but I haven’t actually vomited.”

36

Towards Expert-level Medical AI for Real-time Video Consultations

Table A.5 | Example Ground Truth & Evaluation Rubric. Clinical ground truth and case-specific evaluation rubric for a gastritis scenario (the corresponding scenario details are shown in Table A.4). Probable Diagnosis Plausible Alternative Diagnoses

Gastritis Peptic Ulcer Disease, Functional Dyspepsia

Investigations & Action No urgent tests needed unless red flags are present. • Consider CBC to check for anemia. • Consider H. pylori Plan breath test or stool antigen test. • Consider endoscopy - only if symptoms are severe or persistent, or there are warning signs. Management

Medical Treatments: Stop/minimize NSAID use - consider acetaminophen for knee pain instead. Start proton pump inhibitor (PPI) for suspected gastritis. Lifestyle Modifications: Avoid caffeine, alcohol, spicy foods, and greasy meals. Eat smaller, more frequent meals. Remain upright after meals, Avoid eating right before bed. Reduce stress where possible. Follow-up: Reassess symptoms in 2–4 weeks. If worsening or red flags develop, proceed to urgent investigation.

Case-Specific Evaluation Rubric History Taking Asks about onset, duration, and location of discomfort. Inquires about aggravating/relieving factors and meal relationship. Asks about associated symptoms (nausea, vomiting, bleeding). Reviews medication history, especially NSAIDs. Screens for red flags (weight loss, severe pain, anemia). Reviews past medical and family history of ulcers or GI disease.

1 1 1 1 1 1

Perception and Examination Observes pointing to epigastric region correctly. Assesses visible discomfort when discussing symptoms. Checks for systemic signs (fatigue, pallor). Notes absence of severe distress or alarming symptoms.

1 1 1 1

Clinical Reasoning and Diagnosis Recognizes likely gastritis, possibly NSAID-induced. Appropriately rules out urgent conditions (ulcer, bleeding, malignancy).

2 2

Treatment Plan and Management Stops or minimizes NSAID use and switches to safer alternative. Starts PPI for suspected gastritis. Advises on dietary modifications and trigger avoidance. Plans appropriate follow-up timeline and tests if needed.

1 1 1 1

Patient Communication and Education Explains probable cause (NSAIDs, diet) clearly to patient. Provides reassurance and clear home care instructions.

1 1

37

Towards Expert-level Medical AI for Real-time Video Consultations

A.8. Telehealth Audio-Visual Examination Rubric To evaluate the unique clinical competencies required during synchronous video encounters, we supplemented the established general evaluation rubrics with targeted telehealth criteria adapted from Sartori et al. [1]. These items specifically assess the provider’s perceptual acuity in utilizing live video streams for information gathering, their ability to proactively partner with patients to execute guided physical examination maneuvers, and provide open-ended qualitative feedback regarding interface utilization. The complete evaluation items, rating scales, and adaptation rationale are detailed in Table A.6. Table A.6 | Supplementary Telehealth OSCE Evaluation Rubric. Evaluation items, scoring scales, and rationale incorporated into the expert rater general rubrics to assess audio-visual symptom observation and patient-assisted physical examinations. Criteria 1 and 2 are adapted from Sartori et al. [1], while Criterion 3 was added to capture open-ended qualitative feedback. Checklist Item & Rater Guidance

Rating Scale

Source

To what extent did the doctor utilize live video to augment information gathering?

5-point scale

Adapt: Adapted from Sartori et al. [1] to better fit the technical constraints of this study.

When rating this item, please consider whether the doctor perceived all the relevant visual and audio cues from the live video, for example if they identified reproducible symptoms or visually assessed patient non-verbal cues. To what extent did the doctor partner with the patient to perform physical examination? When rating this item, please consider whether the doctor asked the patient to perform maneuvers or access peripheral monitoring device (home blood pressure cuff, wearable device, glucometer), followed by verbal confirmation of findings. What did the agent do well or not do well in the video interface (for example, unexpected, missed, etc.)?

(Very Poor, Poor, Fair, Good, Excellent)

5-point scale (Very Poor, Poor, Fair, Good, Excellent)

Open-ended free text

Keep: Kept directly from Sartori et al. [1] source framework.

Added: Incorporated for qualitative interpretation of video interface utilization.

38

Towards Expert-level Medical AI for Real-time Video Consultations

A.9. Post-Encounter Clinical Questionnaire Post-Encounter Clinical Questionnaire Structure 1. Differential Diagnosis • Diagnosis 1 (most likely) * • Diagnosis 2 * • Diagnosis 3 * • Diagnosis 4 • Diagnosis 5

• Diagnosis 6 • Diagnosis 7 • Diagnosis 8 • Diagnosis 9 • Diagnosis 10 (least likely)

2. Observation • Audio-Visual Cues: If this was an audio or video interaction, please describe any audio-visual cues you noticed when interacting with the patient and how they informed your reasoning about this case? Enter N/A if you did not notice any particular cues or if this was a text interaction. • Desired Observations: If this was a text or audio interaction, please describe what you would have liked to observe to help you reason about the case. Enter N/A if this does not apply.

3. Clinical Management • In-Person Escalation: Does this case require escalation of this virtual consultation to an in-person consultation? □ No, does not require escalation □ Yes, requires escalation to an IN-PERSON consultation • Escalation Reasoning: Describe why it requires or does not require escalation: • Required Investigations: Is an investigation(s) required? List all. This could include imaging, lab tests, etc. It should also include any other examinations, tests, or investigations you believe to be necessary but were unable to perform due to the virtual nature of the consultation. Format your response as a simple bullet list starting with dashes. (e.g., - Some test or investigation.) • Recommended Treatments: Is a treatment recommended? List all. • Management Plan: What is your management plan? • Alternative Management Plans: Is there more than one acceptable management plan for this case? □ Yes □ No • Alternative Management Reasoning: If other management plans were acceptable, please describe the alternatives and explain your choice. If no other options were viable, please briefly explain why. • High-Level Trade-off Reasoning: Briefly describe your qualitative, high-level reasoning around the cost, effectiveness and side effects for the management plan you are proposing. Try to highlight why you are recommending specific things to keep the cost manageable, side effects minimal or increase effectiveness of the treatment, and any trade-offs involved. • Prognosis Assessment: Describe the prognosis with and without treatment of this patient. • Follow-up Requirement: Is a follow-up required? □ Yes □ No □ Unsure • Follow-up Timing: If yes, when would you like to follow-up the patient? Otherwise, enter “N/A”. • Follow-up Reasoning: Describe why it requires or does not require follow-up? • Management Script: Please describe the management script you considered for this case? In other words, the step by step reasoning you used in this SPECIFIC case which could be applied to another case with this same chief complaint? • Additional Remarks: Is there anything else you would like to tell us? (Optional)

39

Towards Expert-level Medical AI for Real-time Video Consultations

A.10. User Interfaces Audio-Video

a

Audio-only

b

Text-only

c

Figure A.4 | OSCE Study UI. As seen by the PCP or AMIE. In all cases, the PCP and AMIE do not have their video on. a) In the Audio-Video arm, the patient actor can be seen and spoken to and heard through audio. b) In the audio-only arm (which was conducted only in the pilot phase) the patient actor cannot be seen, but communication still occurs over audio. c) In the text-only arm, the patient actor can be neither seen nor heard, and all communication occurs over the text chat window.

Figure A.5 | Clinical Evaluator Interface. The web-based evaluation interface utilized by the independent panel of expert primary care evaluators. Raters are provided with the full scenario, consultation video recording, corresponding text transcript, and provider’s completed post-encounter clinical questionnaire. The right-hand panel displays the structured evaluation rubrics, including overarching general competency scales (DDx Accuracy, Diagnosis & Management, PACES, PCCBP, Audio-Visual Examination) and the 20-point case-specific rubrics.

40

Towards Expert-level Medical AI for Real-time Video Consultations

A.11. Tabular OSCE Study Results Table A.7 | OSCE Study Case-Specific Rubric Results. Expert-evaluated performance of AMIE (Video), AMIE (Text) and PCP (Video). A two-sided Wilcoxon signed-rank test with Benjamini–Hochberg false discovery rate (FDR) correction is used to compare AMIE (Video) to the other arms. 95% confidence intervals are computed via scenario-level bootstrapping (N=1000). Performance [95% CI]

FDR-adjusted 𝑝-value

Evaluation Domain

Scale

AMIE (Video)

AMIE (Text)

PCP (Video)

vs. PCP

vs. Text

History Taking Perception and Examination Clinical Reasoning and Diagnosis Treatment Plan and Management Patient Communication and Education

Rubric Rubric

0.86 [0.83, 0.89] 0.74 [0.69, 0.80]

0.85 [0.82, 0.88] 0.41 [0.35, 0.48]

0.78 [0.74, 0.82] 0.47 [0.40, 0.54]

1.20 × 10 −3 1.73 × 10 −9

0.098 3.32 × 10 −11

Rubric

0.90 [0.87, 0.93]

0.87 [0.83, 0.91]

0.76 [0.70, 0.81]

2.00 × 10 −5

0.186

Rubric

0.79 [0.74, 0.84]

0.78 [0.73, 0.82]

0.67 [0.61, 0.71]

8.90 × 10 −5

0.331

Rubric

0.85 [0.80, 0.90]

0.80 [0.74, 0.85]

0.68 [0.62, 0.74]

6.56 × 10 −7

0.114

Rubric

0.83 [0.81, 0.86]

0.75 [0.72, 0.78]

0.68 [0.64, 0.72]

1.32 × 10 −9

1.19 × 10 −8

Overall Rubric Score

41

Towards Expert-level Medical AI for Real-time Video Consultations

Table A.8 | OSCE General Expert Evaluation Results. Expert-evaluated performance of AMIE (Video), AMIE (Text) and PCP (Video). For display and analysis, ratings are mapped to a 0–1 scale (mapping 5-point Likert ratings via (𝑆 − 1)/4 and binary outcomes to 1 for favorable and 0 for non-favorable). For statistical comparison, a two-sided Wilcoxon signed-rank test with Benjamini–Hochberg false discovery rate (FDR) correction was used to compare AMIE (Video) to the other arms. Cases receiving ‘N/A’ ratings on any arm were excluded from paired analysis. 95% confidence intervals are computed via scenario-level bootstrapping (N=1000). Performance [95% CI] Evaluation Criteria

FDR-adjusted 𝑝-value

Scale

AMIE (Video)

AMIE (Text)

PCP (Video)

vs. PCP

vs. Text

Binary Binary 5-point Likert 5-point Likert

0.91 [0.85, 0.96] 0.98 [0.95, 1.00] 0.89 [0.85, 0.92] 0.88 [0.85, 0.92]

0.90 [0.84, 0.95] 0.97 [0.94, 1.00] 0.89 [0.86, 0.92] 0.86 [0.83, 0.90]

0.77 [0.68, 0.85] 0.90 [0.83, 0.96] 0.80 [0.75, 0.84] 0.73 [0.69, 0.78]

0.039 0.179 1.05 × 10−3 2.38 × 10−6

0.862 0.862 0.862 0.494

5-point Likert

0.79 [0.74, 0.84]

0.77 [0.70, 0.82]

0.68 [0.62, 0.74]

7.55 × 10−3

0.625

Binary

0.93 [0.88, 0.98]

0.89 [0.82, 0.95]

0.82 [0.74, 0.89]

0.089

0.625

Binary

0.88 [0.82, 0.94]

0.89 [0.82, 0.95]

0.79 [0.70, 0.87]

0.149

0.862

Binary

0.87 [0.80, 0.93]

0.82 [0.74, 0.89]

0.76 [0.68, 0.84]

0.089

0.590

Binary

0.90 [0.84, 0.95]

0.86 [0.79, 0.93]

0.82 [0.74, 0.90]

0.189

0.625

Binary Binary Binary

0.91 [0.85, 0.96] 0.94 [0.89, 0.98] 0.93 [0.88, 0.98]

0.90 [0.84, 0.96] 0.92 [0.86, 0.97] 0.95 [0.91, 0.99]

0.83 [0.76, 0.91] 0.92 [0.86, 0.97] 0.93 [0.88, 0.98]

0.191 0.737 1.0

0.862 0.862 0.862

0.80 [0.76, 0.83] 0.78 [0.74, 0.82] 0.83 [0.80, 0.86] 0.76 [0.73, 0.80] 0.77 [0.72, 0.81] 0.82 [0.79, 0.85]

0.74 [0.70, 0.78] 0.74 [0.70, 0.79] 0.78 [0.74, 0.82] 0.71 [0.68, 0.75] 0.70 [0.66, 0.74] 0.75 [0.71, 0.79]

0.69 [0.63, 0.74] 0.67 [0.62, 0.72] 0.70 [0.65, 0.74] 0.64 [0.58, 0.70] 0.67 [0.62, 0.72] 0.68 [0.62, 0.73]

5.63 × 10−3 8.49 × 10−4 6.51 × 10−5 1.85 × 10−3 5.16 × 10−3 1.37 × 10−4

0.088 0.431 0.182 0.266 0.037 0.071

Clinical & Communication Competencies (PACES) Eliciting presenting complaint 5-point Likert 0.95 [0.92, 0.97] Eliciting systems review 5-point Likert 0.88 [0.84, 0.91] Eliciting past medical history 5-point Likert 0.89 [0.85, 0.92] Eliciting family history 5-point Likert 0.88 [0.84, 0.93] Eliciting medication history 5-point Likert 0.90 [0.87, 0.93] Explaining accurately 5-point Likert 0.91 [0.89, 0.94] Explaining clearly 5-point Likert 0.94 [0.91, 0.96] Structure & time management 5-point Likert 0.91 [0.88, 0.94] Explaining comprehensively 5-point Likert 0.87 [0.84, 0.90] Interpersonal skills 5-point Likert 0.96 [0.94, 0.97] Differential diagnosis (PACES) 5-point Likert 0.90 [0.87, 0.93] Clinical judgement 5-point Likert 0.88 [0.84, 0.92] Managing patient concerns 5-point Likert 0.88 [0.85, 0.91] Confirming understanding 5-point Likert 0.80 [0.76, 0.84] Showing empathy 5-point Likert 0.82 [0.79, 0.85] Maintaining patient welfare 5-point Likert 0.91 [0.88, 0.93]

0.92 [0.90, 0.95] 0.86 [0.81, 0.89] 0.91 [0.87, 0.94] 0.86 [0.81, 0.91] 0.88 [0.84, 0.92] 0.91 [0.88, 0.94] 0.90 [0.87, 0.93] 0.89 [0.86, 0.92] 0.83 [0.79, 0.87] 0.94 [0.91, 0.96] 0.89 [0.85, 0.93] 0.86 [0.81, 0.90] 0.81 [0.76, 0.85] 0.71 [0.66, 0.77] 0.79 [0.75, 0.82] 0.87 [0.83, 0.90]

0.82 [0.78, 0.87] 0.74 [0.68, 0.80] 0.85 [0.80, 0.89] 0.78 [0.71, 0.85] 0.85 [0.79, 0.89] 0.81 [0.77, 0.85] 0.77 [0.72, 0.81] 0.73 [0.67, 0.78] 0.71 [0.66, 0.76] 0.83 [0.79, 0.87] 0.76 [0.70, 0.81] 0.78 [0.73, 0.83] 0.79 [0.74, 0.84] 0.69 [0.63, 0.74] 0.71 [0.66, 0.76] 0.81 [0.76, 0.85]

4.83 × 10−5 1.05 × 10−3 0.359 7.44 × 10−3 0.149 1.25 × 10−4 2.38 × 10−6 2.49 × 10−6 1.15 × 10−5 6.51 × 10−5 6.05 × 10−4 0.017 4.36 × 10−3 6.05 × 10−4 3.39 × 10−3 2.14 × 10−3

0.431 0.494 0.526 0.494 0.526 0.862 0.266 0.494 0.266 0.526 0.862 0.494 0.037 0.037 0.305 0.266

Physical Observation & Examination Physical observation 5-point Likert Guided examination 5-point Likert

0.30 [0.21, 0.40] 0.51 [0.44, 0.58]

0.51 [0.42, 0.60] 0.39 [0.32, 0.46]

1.37 × 10−4 2.46 × 10−9

3.85 × 10 −6 1.93 × 10 −5

Diagnostic Accuracy (DDx) Top-1 DDx accuracy Top-3 DDx accuracy DDx appropriateness DDx comprehensiveness Management Plan & Safety Management plan appropriateness Investigation appropriateness Avoid inappropriate investigations Treatment appropriateness Avoid inappropriate treatments Escalation recommendation Follow-up recommendation Avoid confabulation

Patient-Centered Communication (PCCBP) Foster relationship 5-point Likert Shared decision making 5-point Likert Patient teacher 5-point Likert Empathy (PCCBP) 5-point Likert Agenda setting 5-point Likert Active listening 5-point Likert

0.77 [0.70, 0.83] 0.72 [0.67, 0.78]

42

Towards Expert-level Medical AI for Real-time Video Consultations

Table A.9 | OSCE Patient Actor Ratings Results. Patient actor evaluations of AMIE (Video), AMIE (Text) and PCP (Video). For display and analysis, ratings are mapped to a 0–1 scale (mapping 5-point Likert ratings via (𝑆 − 1)/4 and binary outcomes to 1 for favorable and 0 for non-favorable). For statistical comparison, a two-sided Wilcoxon signed-rank test with Benjamini–Hochberg false discovery rate (FDR) correction was used to compare AMIE (Video) to the other arms. Cases receiving ‘N/A’ ratings on any arm were excluded from paired analysis. 95% confidence intervals are computed via scenario-level bootstrapping (N=1000). Performance [95% CI] Patient Rating Criteria

Scale

FDR-adjusted 𝑝-value

AMIE (Video)

AMIE (Text)

PCP (Video)

vs. PCP

vs. Text

General Medical Council & Trust (GMCPQ) Being polite 5-point Likert Making feel at ease 5-point Likert Listening to you 5-point Likert Assessing condition 5-point Likert Explaining condition 5-point Likert Involving in decisions 5-point Likert Providing treatment 5-point Likert Confidentiality 5-point Likert Honest & trustworthy 5-point Likert Confident in care Binary Happy to see again Binary

0.89 [0.85, 0.93] 0.84 [0.80, 0.89] 0.87 [0.82, 0.92] 0.91 [0.87, 0.94] 0.88 [0.84, 0.93] 0.79 [0.74, 0.83] 0.79 [0.74, 0.84] 0.87 [0.83, 0.91] 0.87 [0.83, 0.91] 0.96 [0.92, 1.00] 0.91 [0.85, 0.97]

0.85 [0.82, 0.89] 0.76 [0.72, 0.81] 0.83 [0.79, 0.88] 0.89 [0.86, 0.93] 0.84 [0.80, 0.89] 0.78 [0.74, 0.82] 0.74 [0.69, 0.78] 0.84 [0.80, 0.88] 0.84 [0.80, 0.88] 0.85 [0.78, 0.91] 0.74 [0.65, 0.83]

0.88 [0.83, 0.91] 0.82 [0.77, 0.86] 0.87 [0.82, 0.92] 0.81 [0.75, 0.86] 0.79 [0.73, 0.84] 0.76 [0.70, 0.81] 0.80 [0.74, 0.84] 0.87 [0.83, 0.91] 0.88 [0.83, 0.92] 0.88 [0.81, 0.94] 0.81 [0.73, 0.89]

0.775 0.750 0.952 0.045 0.045 0.750 0.690 0.750 0.690 0.690 0.618

0.120 6.93 × 10 −4 0.110 0.321 0.120 0.914 0.128 0.128 0.128 0.128 0.038

Fostering Relationship (PCCBP) Build rapport Open & honest Mutual roles Respect privacy Partnership building Caring & commitment Acknowledge mistakes Greet appropriately Appropriate language Encourage participation Interest in person

0.81 [0.73, 0.88] 0.99 [0.97, 1.00] 0.90 [0.84, 0.96] 1.00 [1.00, 1.00] 0.66 [0.55, 0.76] 0.97 [0.94, 1.00] 0.98 [0.94, 1.00] 1.00 [1.00, 1.00] 0.99 [0.97, 1.00] 0.95 [0.91, 0.99] 0.67 [0.57, 0.76]

0.66 [0.57, 0.75] 0.92 [0.86, 0.97] 0.89 [0.81, 0.95] 0.95 [0.89, 0.99] 0.55 [0.44, 0.65] 0.86 [0.80, 0.93] 0.90 [0.81, 0.98] 0.97 [0.93, 1.00] 0.95 [0.90, 0.98] 0.92 [0.86, 0.97] 0.57 [0.46, 0.66]

0.82 [0.74, 0.89] 0.96 [0.92, 0.99] 0.89 [0.81, 0.95] 0.96 [0.92, 0.99] 0.74 [0.65, 0.83] 0.89 [0.83, 0.94] 0.91 [0.84, 0.98] 0.97 [0.93, 1.00] 0.95 [0.90, 0.99] 0.91 [0.85, 0.96] 0.77 [0.67, 0.84]

0.830 0.750 0.872 0.750 0.750 0.690 0.750 0.750 0.750 0.750 0.690

0.110 0.380 0.872 0.712 0.299 0.179 0.860 0.872 0.713 0.828 0.153

Clinical & Communication Competencies (PACES) Address concerns 5-point Likert 0.87 [0.83, 0.91] Confirm understanding 5-point Likert 0.68 [0.61, 0.74] Empathy 5-point Likert 0.76 [0.71, 0.80] Maintain welfare 5-point Likert 0.85 [0.81, 0.89]

0.86 [0.82, 0.90] 0.66 [0.59, 0.72] 0.70 [0.65, 0.74] 0.84 [0.79, 0.88]

0.81 [0.75, 0.85] 0.65 [0.57, 0.72] 0.69 [0.63, 0.74] 0.83 [0.79, 0.88]

0.472 0.750 0.472 0.830

0.482 0.482 0.128 0.828

Interface & Modality Preference Effective comm. 5-point Likert Felt understood 5-point Likert Easy & convenient 5-point Likert

0.79 [0.75, 0.82] 0.81 [0.77, 0.85] 0.71 [0.66, 0.77]

0.90 [0.86, 0.93] 0.88 [0.83, 0.92] 0.90 [0.86, 0.93]

0.750 0.830 0.750

4.98 × 10 −5 2.62 × 10 −4 2.98 × 10 −6

Binary Binary Binary Binary Binary Binary Binary Binary Binary Binary Binary

0.89 [0.85, 0.92] 0.90 [0.86, 0.93] 0.88 [0.84, 0.92]

43

Towards Expert-level Medical AI for Real-time Video Consultations

A.12. Full Patient Actor Preferences

AMIE (Video) Win

Tie

AMIE (Text) Win

AMIE (Video) Win

Tie

PCP (Video) Win

Being polite

***

Making feel at ease Listening to you

* *

GMCPQ

Assessing condition Explaining condition Involving in decisions Providing treatment Confidentiality Honest & trustworthy Confident in care

*

Happy to see again Build rapport Open & honest Mutual roles

PCCBP

Respect privacy Partnership building Caring & commitment Acknowledge mistakes Greet appropriately Appropriate language Encourage participation Interest in person

PACES

Address concerns Confirm understanding Empathy

Modality & Interface

Maintain welfare

*** *** ***

Effective comm. Felt understood Easy & convenient 0%

25%

50% % Consultations

75%

100%

0%

25%

50%

75%

100%

% Consultations

Figure A.6 | Patient Actor Preferences. Left: Preferences (per scenario) for each criterion between AMIE (Video) in red and AMIE (Text) in green. Right: Preferences (per scenario) for each criterion between AMIE (Video) in red and PCP in blue. Error bars correspond to 95% confidence intervals from bootstrapping (n=1000). For statistical comparison, a two-sided Wilcoxon signed-rank test with Benjamini–Hochberg false discovery rate (FDR) correction was used to compare AMIE (Video) to the other arms (adjusted 𝑝-values: ∗ 𝑝 < 0.05, ∗ ∗ 𝑝 < 0.01, ∗ ∗ ∗ 𝑝 < 0.001).

44

Towards Expert-level Medical AI for Real-time Video Consultations

A.13. Full Expert Evaluation Preferences

Diagnosis & Management

DDx Accuracy

AMIE (Video) Win

Tie

AMIE (Text) Win

AMIE (Video) Win

Tie

PCP (Video) Win

*

Top-1 DDx accuracy Top-3 DDx accuracy DDx comprehensiveness

** ***

Plan appropriateness

**

DDx appropriateness

Investigation appr. Avoid inappropriate investigations Treatment appr. Avoid inappropriate treatments Escalation appr. Follow-up appr.

PCCBP (Communication)

Avoid confabulation

** *** *** ** ** ***

Foster relationship Shared decision making Patient teacher Empathy

*

Agenda setting Active listening

*** **

Eliciting (PACES)

Pres. complaint Systems review Past med. history

*

Family history Medication history

*** *** *** *** ***

Explaining (PACES)

Explain accurately Explain clearly Structure & time Explain comprehensively Interpersonal skills

* *

*** * ** *** ** **

Guided examination

*** ***

*** ***

Overall

***

*** ** *** *** *** ***

Managing (PACES)

Diff. diagnosis Clinical judgement Manage concerns Confirm understanding Show empathy

Case-Specific Rubrics

AV Telehealth

Maintain welfare Physical observation

History Taking

***

Perception & Exam Clinical Reasoning Treatment Planning Patient Comm 0%

25%

50% % Consultations

75%

100%

0%

25%

50%

75%

100%

% Consultations

Figure A.7 | Expert Preferences. Left: Preferences (per scenario) for each criterion between AMIE (Video) in red and AMIE (Text) in green. Right: Preferences (per scenario) for each criterion between AMIE Video in red and PCP in blue. Error bars correspond to 95% confidence intervals from bootstrapping (n=1000). For statistical comparison, a two-sided Wilcoxon signed-rank test with Benjamini–Hochberg false discovery rate (FDR) correction was used to compare AMIE (Video) to the other arms (adjusted 𝑝-values: ∗ 𝑝 < 0.05, ∗ ∗ 𝑝 < 0.01, ∗ ∗ ∗ 𝑝 < 0.001).

45

Towards Expert-level Medical AI for Real-time Video Consultations

A.14. Patient Actor Interviews To complement the quantitative findings, we conducted a qualitative evaluation with 7 patient actors to understand how communication modalities (text, audio, video) impact the perceived quality of care, the effectiveness of health plan communication, and the overall dynamics of the clinical encounter. The interviews lasted 25 minutes and were recorded and transcribed. We used reflexive thematic analysis to inductively generate themes based on the transcripts and researcher notes. The analysis yielded three distinct themes: • Modality suitability depends on context: Clinical needs and environmental factors dictate the ideal interface, suggesting there is no “one-size-fits all” modality. • Though empathetic, AI lacks conversational cadence: AI provides consistent empathy, breaks down complex medical jargon for patients, and is thorough in its evaluation of the patient, but it lacks natural conversational capabilities. • The need for human connection is dictated by symptom urgency and severity: The severity of a patient’s condition shifts their trust requirements in AI. Exploring these themes revealed that the video modality is preferred for certain domains such as musculoskeletal (MSK) exams because it allows participants to visually demonstrate range of motion and point out locations where they are feeling pain. In contrast, the text modality affords an anonymity that allows participants to feel less self-conscious and ensures privacy when discussing sensitive neurological/psychiatric topics. The need for text is also dictated by environmental factors; when in public settings or when voice communication is difficult, text becomes advantageous for maintaining privacy. For the dynamics of the clinical interaction itself, human doctors remain the “gold standard” for social connection, but participants agreed they still felt heard and were unhurried by the AI. Assessing the different modalities, participants agreed that for minor issues, the AI would serve as an effective 24/7 triage tool, but for more serious or life-threatening conditions, they wanted human connection requiring real-time audio/video interactions. A.14.1. Overview The goal of the user experience research (UXR) study was to evaluate the participant experience across different modalities (text, audio, video) within the audio-video OSCE framework. Through remote, moderated interviews, UXR answered the following research questions: • What are the unique benefits and drawbacks of text, audio-only, and video interactions for both diagnosis and care management? • To what extent does the presence or absence of visual feedback (synchronous video v. “blank screen” audio-only) influence participant perceptions of clinician/AI empathy during care encounters? • Which communication modality (text, audio, video) are naturally prioritized and why? A.14.2. Methodology A.14.2.1. Methods UXR conducted remote, moderated semi-structured interviews to collect insights from patient actors about their perceived quality of care and overall dynamics of the clinical encounters they engaged in.

46

Towards Expert-level Medical AI for Real-time Video Consultations A.14.2.2. Data Collection Process Data was collected with 7 patient actors who were part of the larger AV-AMIE OSCE study. Data collection was completed remotely over Google Meet by the manager of the OSCE lab. During the sessions, participants were asked questions to better understand their thought processes regarding the value and experience of each modality. Participants were prompted to reflect on how their ability to convey health information varied based on whether they were using text, audio or video. They also discussed how the absence of video, a “blank screen,” influenced their perception of the provider’s empathy during the interaction. A.14.2.3. Data Analysis Interview notes were collected using the interview transcripts, and the data was analyzed using reflexive thematic analysis. The open-ended interview responses were coded first with an initial codebook designed to capture core concepts. The codebook was adjusted and refined, and once notes for all 7 participants were coded, the codes were synthesized and grouped to identify overarching themes. A.14.3. Key Findings Thematic analysis of the interview sessions generated three themes about how patients navigate, evaluate, and prioritize multimodal AI clinical interactions based on clinical context and how each modality supports them A.14.3.1. Modality Suitability Depends on Context Clinical needs and environmental factors dictate the ideal interface suggesting there is not a single modality that is preferred for any given condition. When participants seek a physical evaluation, video is the preferred modality as it allows participants to point out where pain occurs or demonstrate range of motion. For example, for musculoskeletal (MSK) concerns, participants shared that a video modality would allow pointing to the site of pain or help demonstrate range of motion or perform other movements. Audio comes second to video — it allows for the visit to feel personable and comforting, allowing participants to build rapport with the provider when text cannot. Still, it lacks the visual connection which provides the conversation the holistic picture needed for certain conditions, such as musculoskeletal conditions. Finally, when seeking anonymity for sensitive topics like mental health, participants agree that text offers the additional privacy they need. P5 noted that for sensitive topics, such as reproductive health, using the text modality would afford a level of privacy not available in modalities that require speaking. In addition to the context of the condition, environmental constraints also dictate preferred modality. Text is preferred for receiving medical advice in public spaces when voice communication is difficult, e.g. seeking out care on a busy train ride. A.14.3.2. Though Empathetic, AI Lacks Conversational Cadence Participants conveyed that while the AI breaks down complex medical jargon for patients, provides empathy, and is thorough in its evaluation of symptoms, especially for non-text modalities, it still lacks natural conversation capabilities. The rapport and empathy built with human providers varies based on their schedules. One participant observed that human providers’ empathy mimicked “peaks 47

Towards Expert-level Medical AI for Real-time Video Consultations and valleys” depending on the day, but the AI is consistent in its empathy. The AI also did not rush the participants — there is no pressure to condense information in order to hurry the conversation along because there is another patient in the waiting room. However, unlike the human connection with a human provider where small talk was commonplace, participants reported that conversations with the AI felt awkward at times. Participants shared that the natural cadence of a conversation was lacking, making it obvious they were speaking to AI and not a human. The AI also presented participants with awkward silences causing some participants to feel uncomfortable or confused as to whether or not they were still connected to the AI agent and prompting suggestions of how to demonstrate that the system is still processing. A.14.3.3. The Need for Human Connection Is Dictated by Symptom Urgency and Severity Modalities for care vary based on the urgency and severity of the symptoms a participant faces. For acute, minor conditions, participants trust text modalities to help triage symptoms and determine whether or not immediate, in-person care is needed. For straightforward, non-urgent concerns, like a sinus infection, the AI is efficient and thorough. However, when dealing with more serious, life-threatening and urgent health concerns, participants prefer video, or audio, visits to seek out the human voice and feel a sense of trust and security. For example, one participant shared a preference for video when they were faced with elevated blood pressure. In these emergent situations, participants felt ease in establishing rapport with and hearing from a human provider. A.14.4. Conclusion These findings provide insights into the information needs and modality preferences of patient actor participants engaging with multimodal AI clinical tools. Overall, the data reveals that there is no universal, one-size-fits-all modality. Instead, patient preferences are situational and dependent on the symptoms they are exhibiting, the severity of the symptoms, and their environmental constraints. The AI in this study was recognized to be effective for the initial screening of symptoms, and participants reasoned they would use it for triaging their care or seeking a tool for quick advice when immediate care is unavailable. Future iterations of multimodal AI tools for health would benefit from addressing conversational limitations. Here, while patient actor participants appreciated the system’s ability to break down medical jargon into accessible language and to provide consistent empathy, the lack of natural conversational cadence made the interactions feel awkward and unnatural. While clinical AI tools were seen to have value as a triage tool, the perspective was that human providers, via video or in-person, continue to be the “gold standard” for care.

48

Towards Expert-level Medical AI for Real-time Video Consultations Table A.10 | Qualitative interview themes from patient participants. Theme

Definition

Code Name

Definition

Inclusion Criteria

Quotes

Modality suitability depends on context

Clinical symptoms, immediate environmental/privacy factors, and nature of the condition dictate communication modality (text, audio, video)

Modality – Text

Preferences, benefits, or drawbacks associated with text-only communication

Typing effort, response speed, public privacy, clear articulation, lack of emotion, control, redundant replies, or cold/impersonal text feelings

“. . . text AI feels like a thorough Google search, but it is highly useful for privacy when in public settings like a train.” –P2 “thought with the text the AI was superior in my eyes. . . the AI was very quick to get back to you. . . texting human doctors was a long process with frustrating wait times between messages.” –P4

Modality – Video

Preferences, benefits, or drawbacks associated with video communication

Physical movements, pain locations, efficiency, visual confirmation, self-consciousness, seeing one’s reflection, missing provider’s face hindering rapport, or preferring a blank screen over an artificial AI persona

“I thought it was easier to use the video um option with the AI doctor just because it felt like it was easier to communicate, express myself.” –P5 “I could show myself, maybe go through physical exam stuff that I wasn’t able to do in a text or audio format. . . ” –P7

Modality – Audio

Preferences, benefits, or drawbacks associated with audio communication

Hearing a voice, establishing a connection for serious conditions, tone, comfort/ease from the voice, easy verbal description, or audio improving real-time interaction

“the AI voice, I felt like it improved throughout the course of doing um like the sessions. . . I feel like hearing a voice, having the tone of a voice is something that I actually really like.” –P6 “audio. . . was a little easier to express myself and it just felt more like a real interaction being able to hear another voice on the other end.” –P7

MSK/Physical Exams

Issues related to musculoskeletal conditions and the need for physical demonstration

Pointing to pain, demonstrating range of motion (rotation, gait), isolating muscles, skepticism of AI interpreting movement, observation vs. human nuance, physicians rarely asking for exams, or uncertainty of AI capture

“The text one they just asked me to describe what happened when I did it. . . but when I could show them. . . I felt more confident because they’re actually seeing it. . . ” –P3 “I think a video for these because you can do specific movements versus like describing like I’m supinating my arm and it hurts. You can just show them and then they can interpret that.” –P6

Mental Health & Privacy

Issues related to psychological conditions and the desire for privacy or reduced intimidation

Anxiety, depression, text preference to avoid self-consciousness, public privacy (e.g., trains), text feeling safer / comfortable, or anonymity easing sensitive mental / reproductive health discussions

“With maybe more sensitive issues, like if I was having reproductive symptoms that I didn’t really know how to ask about, I would probably do text AI so that people wouldn’t hear me.” –P5 “Preferred text for psych-related scenarios because the sense of anonymity made it easier to discuss sensitive topics.” –P7

System Feedback & Silences

User reactions to system processing time and the need for visual or auditory cues

Buffering, loading, pauses, awkward silences, AI processing indicators (wheel, light), thinking time from silences, connection/crash worries, or ambiguity of input receipt

“Silences caused worry that the connection had failed, especially with the AI.” –P5 “Silences created ambiguity about whether the system received or understood the input.” –P7

Naturalness of Interactions

The degree to which the interaction feels like a natural human conversation

Call-and-response cadence, robotic feel, conversational flow, human interaction comparisons, or human elements like jokes, small talk, and past context

“PCP is the gold standard due to human connection. Video was easier for communication and expressing oneself.” –P5 “Audio was a step up for natural interaction. . . not seeing the provider’s face made it hard to build a connection.” –P7

Empathy & Connection

The perceived emotional support, warmth, and feeling of being understood by the provider (AI or human)

Feeling heard, active listening, symptom summaries, warm feeling, tone, lack of genuine emotion, AI empathy consistency vs. human variability, or feeling at ease

“Primary care physician is the gold standard because human connection is tough to replicate. . . natural cadence, jokes, remembering past context,” –P2 “I thought AI was like more consistent with providing the right amount of empathy with each different case, whereas human physicians were inconsistent, sometimes feeling cold.” –P5

Thoroughness & Diligence

The perceived completeness and detail of the medical examination and questioning

AI follow-up, symptom summaries, not feeling rushed, due diligence, observing incidental symptoms (coughs/sneezes), or AI’s thoroughness vs. rigid human doctor scripts

“I did not feel rushed. Despite being shorter, the AI was thorough and always asked if there was anything else to add before concluding.” –P3 “I thought AI was actually better at like wanting to do a physical exam over video or text. Like I thought they were more proactive in doing that.” –P5

Medical Communication & Guidance

How medical information, instructions, and terminology are conveyed to the patient

Simplifying jargon, actionable safety steps (unlocking doors, meds), guiding movements, clear explanations for lower literacy, or AI recommending urgent care

“The AI did a good job explaining some of this terminology to me. . . broke down complex jargon. . . ” –P1 “Human PCPs provide anecdotal treatment advice. . . AI appeared more cautious, frequently recommending urgent care.” –P2

Though empathetic, AI lacks conversational cadence

Clinical AI provides a thorough, standardized assessment and is empathetic, but is conversationally stiff and robotic, lacking natural conversational qualities

Continued on next page

49

Towards Expert-level Medical AI for Real-time Video Consultations Table A.10 – continued from previous page Theme

The need for human connection is dictated by symptom urgency and severity

Definition

The severity and urgency of clinical symptoms dictate when human connection is required for certain health concerns (e.g. life threatening requiring human connection vs. triaging by AI)

Code Name

Definition

Inclusion Criteria

Quotes

Visit Duration

Visit length and perceived time pressure during the encounter

Visit length, feeling rushed by human doctors (quotas/waiting rooms), taking time with AI, AI repeating/dragging, or text vs. video efficiency

“Video is definitely more efficient (5–10 mins) compared to texting (15–20 mins). Compared to a human PCP, AI video was similar or slightly shorter.” –P2 “AI text and video sessions typically lasted 10–15 minutes, which felt comfortable. . . Felt more rushed during interactions with human primary care physicians.” –P5

Trust & Safety

User confidence in the diagnosis and the perceived caution of the provider

AI caution, urgent care recommendations, wanting human confirmation, AI as a triage gateway (after-hours/night, ER decisions), or saving on in-person care costs

“. . . readily use AI as a triage gateway for after-hours needs, quick advice, or to decide whether to go to the ER, especially for serious things in the middle of the night.” –P3 “. . . would prioritize using the AI video doctor first as a triage step to save on expensive in-person care.” –P5

Emergency & Serious Conditions

Preferences for handling urgent, severe, or life-threatening health concerns

Strokes, heart attacks, high blood pressure, needing visual/audio connection, tone urgency, AI safety instructions (unlocking doors, meds), or ER/urgent care decisions

“high blood pressure scare. . . preferred video for speed to get point across quickly in a potential emergency. . . typing out detailed information was more timeconsuming and stressful.” –P1 “For a serious condition like a stroke, I wanted to hear a voice and establish a connection. Text felt inadequate. . . I really wanted to talk to somebody.” –P3

50

Towards Expert-level Medical AI for Real-time Video Consultations

References 1. 2. 3. 4. 5. 6. 7. 8.

Sartori, D. J., Hayes, R. W., Horlick, M., Adams, J. G. & Zabar, S. R. The TeleHealth OSCE: preparing trainees to use telemedicine as a tool for transitions of care. Journal of Graduate Medical Education 12, 764–768 (2020). McDonough, K. et al. Complete Physical Exam Benchmarks https : / / uw . pressbooks . pub / fcmtextbook / chapter/complete-physical-exam-benchmarks/. Accessed: 2026-06-20. 2024. Caravan Health. Telehealth Physical Exam Fact Sheet https://caltrc.org/wp-content/uploads/2021/01/ Telehealth-Physical-Exam-Fact-Sheet.pdf. Accessed: 2026-06-18. 2021. Ansary, A. M., Martinez, J. N. & Scott, J. D. The virtual physical exam in the 21st century. Journal of telemedicine and telecare 27, 382–392 (2021). Dover, A. R., Innes, J. A. & Fairhurst, K. Macleod’s Clinical Examination-E-Book: Macleod’s Clinical Examination-E-Book (Elsevier Health Sciences, 2023). Shah, M., Gusdorf, J., Palepu, A., Park, C., O’Sullivan, J. W., Ravi, V., Strother, T., Dubov, P., Rysbek, A., Fukuzawa, T., et al. Towards Conversational Medical AI with Eyes, Ears and a Voice. arXiv preprint arXiv:2605.09272 (2026). Tu, T., Schaekermann, M., Palepu, A., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Cheng, Y., et al. Towards conversational diagnostic artificial intelligence. Nature, 1–9 (2025). Liévin, V., Palepu, A., Weng, W.-H., Saab, K., Stutz, D., Cheng, Y., Kulkarni, K., Mahdavi, S. S., Barral, J., Webster, D. R., et al. Towards Conversational AI for Disease Management. Nature, 1–3 (2026).

51

Related documents

• Learning from Disagreement: Clinician Overrides as Implicit Preference Signals for Clinical AI in Value-Based Care#146936
• Checkup2Action: A Multimodal Clinical Check-up Report Dataset for Patient-Oriented Action Card Generation#665752
• ConsultMind:Towards Automated Diagnostic Consultation via Uncertainty-Aware Reasoning#1105355
• Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution#984506
• Artificial Intelligence-powered Virtual Assistant for Emergency Triage in Neurology#913501