Google’s experimental medical chatbot has moved from test scenarios to real patients, and the first results are now in The Lancet. In a study at Beth Israel Deaconess Medical Center (BIDMC) in Boston, the AI system, called AMIE, interviewed patients before urgent primary care visits, and its list of possible diagnoses included the one doctors eventually confirmed 90% of the time.
The study was published Oct. 8. It is small, it was funded by Google’s parent company, and a doctor watched every conversation live. Still, it is one of the first looks at how a diagnostic chatbot behaves with real people instead of actors or textbook cases.
How the Study Worked
Patients with new, non-emergency problems who had booked an urgent care appointment were invited to chat with AMIE by text from home, up to five days before their visit. One hundred patients completed a chat, and 98 went on to see a clinician.
A board-certified internal medicine physician monitored each conversation in real time and could step in if any of four predefined safety rules were triggered. Final diagnoses were confirmed through chart review eight weeks after each visit.
What the Researchers Found
- Diagnosis: The final diagnosis appeared in AMIE’s top seven possibilities in 90% of cases, in its top three in 75% and as its single top pick in 56%.
- Safety: No conversation had to be stopped. Supervisors flagged one hallucination, an invented detail, and added clinical clarification in five cases.
- Doctors’ prep: Clinicians read AMIE’s summary before 44 visits and said it helped them prepare in 75% of those cases.
- Patients: Attitudes toward AI in health care became significantly more positive after the chats and stayed that way after the in-person visit.
In a blinded comparison, evaluators rated AMIE’s diagnosis lists and management plans as similar in quality and safety to those of the clinic’s providers. The human providers scored better on how practical and cost-effective their plans were. Google attributes that gap to AMIE having no access to medical records, no physical exam and text-only input.
The Catch
This was a feasibility study at a single clinic, with no control group, so it can’t show that AMIE improves care or health outcomes. Participants skewed younger than the clinic’s usual urgent care patients. Patients liked how AMIE listened and explained, but some raised concerns about confidentiality and the chatbot’s honesty. One provider judged an exchange somewhat harmful because a patient may have become anxious after AMIE listed lymphoma as a possibility.
Alphabet funded the work, and senior author Adam Rodman of BIDMC was a visiting researcher at Google during part of it. Google’s own summary says larger clinical trials are needed before patient-facing AI like this can be judged at scale, and it has announced no plans to release AMIE to the public.
If you use a general chatbot for health questions in the meantime, the same caution applies as always: these tools can make things up, so check anything important with a clinician.
More on Contoh
- Why AI Chatbots Make Things Up — and How to Catch Them When They Do
- AI Chatbots Give a Narrower Range of Information Than a Google Search, Study Finds
- Common Sense Media Rates ChatGPT for Teens an “Unacceptable Risk.” OpenAI Disputes the Findings