Google Tests AMIE AI for Clinical Video Consultations

Google is taking another step toward AI-powered healthcare with AMIE (Articulate Medical Intelligence Explorer) Video, a research system designed to conduct real-time clinical consultations through video. In a controlled study involving professional patient actors, Google reported that AMIE received clinical evaluation scores comparable to primary care physicians across several important measures.

The research explores whether multimodal AI can go beyond text-based medical conversations by interpreting visual and auditory information during a consultation. However, Google stresses that these findings should not yet be interpreted as proof that the system is ready for routine clinical use. Studies involving real patients and their actual health conditions will be necessary before broader conclusions can be made.

How Google’s AMIE Video System Works

One of the most notable features of AMIE Video is its multi-agent architecture. Instead of asking a single AI process to manage conversation, medical reasoning, and continuous video and audio analysis at the same time, Google separates these responsibilities among three specialized agents.

The Talker Agent manages the patient-facing conversation. Its primary responsibility is to maintain a natural flow of dialogue while using information generated by the other agents.

The Planner Agent works in the background. It continuously updates possible diagnoses, considers management options, identifies missing information, and adjusts the clinical objectives as new information becomes available.

The Perception Agent continuously analyzes the audio and video streams. It looks for potentially relevant visual signs, physical findings, non-verbal behaviour, and auditory signals before placing those observations into the broader clinical context.

This division of responsibilities addresses one of the biggest challenges facing medical AI: latency. Detailed clinical reasoning can require significant processing time, while patients expect immediate responses during a conversation. By separating dialogue from slower reasoning and perception tasks, AMIE can continue communicating while background processes work on more complex analysis.

Video Consultation Adds a New Layer to Medical AI

Traditional AI medical assistants have largely relied on text-based conversations. Video introduces additional information that may be important during clinical assessment.

A video consultation can potentially provide clues through a patient’s facial expressions, movements, posture, breathing patterns, speech, and other observable characteristics. The system can also guide patients through certain virtual examination manoeuvres.

Google’s automated evaluations found improvements across areas including history-taking, clinical reasoning, treatment recommendations, communication, and response latency. The company also evaluated whether individual agents contributed to these clinical capabilities.

The researchers then moved from automated testing to a controlled human evaluation involving simulated clinical consultations.

Study Compared AMIE With Doctors and Text-Based AI

Google designed the human evaluation as a multi-arm randomised study. AMIE conducted synchronous video consultations, while another version of AMIE operated through text to provide a comparison with a text-only system.

The study also included 10 board-certified primary care physicians, who used the same video consultation interface.

A separate panel of 20 experienced primary care physicians reviewed the consultations. The evaluators used established clinical assessment frameworks, including general measures of clinical competence and criteria specifically designed for each medical scenario.

The research covered five broad areas of medicine: cardiopulmonary, abdominal, HEENT (head, eyes, ears, nose and throat), neurological or psychiatric, and musculoskeletal conditions.

Fifteen professionally trained actors portrayed patients with different clinical presentations. Each participant followed a standardised scenario designed to provide a consistent basis for comparing AMIE and physicians.

According to Google, evaluators rated AMIE similarly to the primary care physician group for several major measures, including the thoroughness of history-taking, diagnostic accuracy, appropriateness of management recommendations, and communication quality.

AMIE Video also reportedly performed as well as or better than the text-based AMIE system on these measures.

AMIE Showed Strength in Virtual Examination Guidance

One of the more interesting findings involved the system’s ability to identify physical signs and guide patients through virtual examination activities.

Evaluators rated the video version of AMIE higher for eliciting physical signs and proactively guiding patients through virtual examination manoeuvres compared with both the physician group and the text-based AMIE system in the reported study.

This result highlights a potential advantage of multimodal medical AI. Text-only systems have access primarily to what a patient says and what the system asks. A video-based system can potentially incorporate information from what it sees and hears while also instructing a patient to perform specific actions.

However, these results came from controlled scenarios involving trained actors. Real-world patients may respond differently, and real clinical environments introduce far greater variability.

Patients Preferred Video Over Text Chat

The study also examined how the simulated patients experienced the different consultation formats.

Google reported that patient actors preferred synchronous video consultations over text-based interactions. Participants considered video easier to use and more effective for explaining health concerns.

The actors also rated AMIE positively in areas such as empathy, rapport, and confidence in the care provided when compared with the study alternatives.

These findings suggest that the communication format could be an important part of AI-assisted healthcare. Patients may find a conversational video interface more natural than typing detailed descriptions of symptoms into a chatbot.

Nevertheless, satisfaction in a simulated environment does not necessarily predict patient trust or acceptance during real medical encounters.

Automated Testing Came Before Human Evaluation

Before conducting the actor-based study, Google developed an automated evaluation framework to test AMIE’s capabilities.

The framework was based on telehealth competencies described in medical literature and included visual cues, auditory information, and virtual physical examination tasks.

Some tests examined individual perception and reasoning capabilities. Examples included identifying anatomical laterality or recognising signs associated with respiratory distress.

Other evaluations involved longer simulated conversations designed to test the system’s ability to maintain clinical dialogue over multiple turns.

Google also used scenarios in which visual information was converted into text descriptions. For example, in a simulated Parkinson’s disease scenario, an AI patient could describe showing a piece of paper containing cramped, small handwriting to the camera.

This approach allowed researchers to evaluate the system’s response to visual information while testing conversational behaviour. However, it is not equivalent to an end-to-end live video consultation.

Important Limitations of the AMIE Study

Despite the promising results, the study has significant limitations.

The biggest limitation is the use of professional actors rather than real patients. Actors can follow scenarios consistently, but they cannot fully reproduce the unpredictability of real medical encounters. Real patients may provide incomplete information, have multiple health conditions, behave unexpectedly, experience emotional distress, or describe symptoms differently from a prepared script.

The scenarios also excluded certain conditions that could not be authentically represented by actors. Some medical problems may depend heavily on real-world audio-visual information, making them particularly important for future research.

Google also reported occasional errors in perception and clinical reasoning during targeted automated evaluations. Technical issues could also interrupt the natural flow of conversations.

These limitations mean the findings should be viewed as evidence of research capability rather than proof of clinical safety.

Real-Patient Research Will Be the Critical Next Step

Google says research involving real patients and their actual medical conditions will be required before AMIE Video can be evaluated for clinical deployment.

The company has already been exploring related work involving text-based AMIE in clinical settings. Google reports that a feasibility study with Beth Israel Deaconess Medical Center provided early evidence regarding safety and usefulness in clinical practice. An ongoing randomised study with Included Health is also examining AI in real-world virtual care.

These studies could provide a more realistic understanding of how medical AI performs when dealing with genuine patient histories, unpredictable conversations, diverse symptoms, and real healthcare environments.

What Google’s AMIE Video Research Means for Healthcare

The AMIE Video study demonstrates how medical AI is evolving from simple text-based question answering toward multimodal clinical interaction.

The system combines conversation, medical reasoning, and continuous audio-visual perception through a multi-agent design. In controlled testing, Google reports performance comparable to primary care physicians across several clinical measures, along with advantages in virtual examination guidance and patient interaction.

However, the distinction between a successful research demonstration and a clinically validated healthcare product remains important.

AMIE Video has not yet demonstrated that it can safely diagnose or manage real patients independently in routine medical practice. Future studies involving real patients, broader clinical conditions, different healthcare environments, and long-term safety monitoring will be essential.

For now, Google’s research represents an important development in AI-powered telemedicine and virtual healthcare. It shows how combining video perception with medical reasoning and natural conversation could expand the capabilities of clinical AI, while also highlighting the significant testing, validation, safety, and regulatory challenges that remain before such systems can become part of everyday patient care.