Nearly a hundred patients at Boston's Beth Israel Deaconess Medical Center first chatted with a chatbot last year before seeing a doctor—a computer program that people can message as if they were talking to another person. It was AMIE, an experimental Google system designed to detect diseases. The results have now been published in The Lancet, one of the world's most respected medical journals, which only accepts papers that have undergone rigorous review by other experts. Doctors did not have to interrupt any of the ninety-eight conversations for safety reasons, and in nine out of ten cases, the chatbot included the disease the patient actually had among its predictions. For Google, this is its first article in the journal's main edition. However, the authors acknowledge that the study has yet to prove many things.
A conversation from the living room, then a visit to the doctor
Researchers from Google and BIDMC tested the chatbot from April to November 2025 in a primary care practice. They approached English-speaking adult patients scheduled for an urgent visit, meaning a prompt appointment for cases in which a person is suddenly experiencing one specific problem, such as pain or fever. They chatted with the chatbot from home, no more than five days before their appointment.
AMIE, whose full name is Articulate Medical Intelligence Explorer, asked questions much like a doctor. It wanted to know what was bothering the patient, how long it had been going on, and what illnesses they had experienced in the past. It then offered several possible explanations for the symptoms that the patient could discuss with their doctor and prepared a concise summary of the conversation for the physician. According to internist Adam Rodman, an internal medicine physician who helped lead the study, the chatbot runs on a more powerful version of the Gemini artificial intelligence model than the one used by Google's search engine.
The study did not examine whether the chatbot helped people recover faster. The authors wanted to determine whether such a program could safely talk to real patients in the routine operation of a medical practice. Rodman explained that before considering time savings, artificial intelligence tools must first demonstrate that they will not harm anyone and that people will accept them.
Doctors monitored every conversation live
Seven experienced internists oversaw safety. They monitored each conversation via a secure video link as it took place. The patients could not see them, and the monitoring was never performed by their own primary care physician.
The team established four reasons in advance for ending a conversation early. These included the risk that the patient might harm themselves or others, significant distress caused by the chatbot, a potential health threat identified by the supervising physician, and the patient's own wish to stop. In the end, none of the conversations met any of these conditions.
However, it was not error-free. After each conversation, the supervising physician reviewed everything and corrected any mistakes. The researchers recorded one hallucination, which is a situation in which artificial intelligence makes something up and presents it as fact. In five conversations, doctors added clarification, and one case was classified as mildly harmful. In that case, the chatbot mentioned lymphoma, a cancer of the lymph nodes, which could have frightened the patient unnecessarily. Rodman points out that the chatbot works only with a short conversation, not the patient's complete medical record, and therefore fabricates information less often.
Predictions on par with a doctor
A doctor usually does not start with a single definitive diagnosis but draws up a list of diseases that could be causing the problem and gradually rules them out. The AMIE system did the same. Researchers compared its lists with what the patient actually had, as determined by doctors eight weeks after the visit using the results of all examinations. The correct disease appeared on the list in ninety percent of cases and among the top three predictions in three out of four cases.
In a so-called blinded evaluation, experts assessed the suggestions made by the chatbot and by real doctors without knowing who had written each one. The chatbot performed similarly to primary care physicians both in predicting the disease and in planning the next steps. However, the doctors in white coats prevailed when it came to practicality and cost—for example, which tests actually made sense and which would waste money. Physician Peter Brodeur of BIDMC said his colleagues compared it to a highly capable medical student or junior doctor.
Patients remained cautious
Doctors completed a questionnaire for sixty patients, and in forty-four cases they had read the conversation or the chatbot's summary beforehand. In thirty-three of those cases, or roughly three quarters, it helped them prepare. In more than half of the cases, they said the information could have influenced how the patient was treated.
Patients were more divided. Their opinion of artificial intelligence improved after the conversation, and this impression persisted even after their visit to the doctor; most said they would use AMIE again. However, fewer than half believed that the chatbot would keep their information confidential or deal with them honestly. Rodman noted that no patient trusted the chatbot until they understood that it was passing the information on to their doctor. Marc Cohen, who leads primary care at BIDMC and is one of the lead authors, envisions a future in which patients, doctors, and artificial intelligence work together.
What the study did not prove
This was a so-called feasibility study, intended to show whether something can be tested safely at all, not whether it works better than the existing approach. It lacked a control group of patients who did not use the chatbot, against whom the others could be compared. Without one, it is impossible to say whether AMIE actually improved care. Moreover, the research took place in a single practice and involved only volunteers. Of more than fourteen hundred urgent visits, only just over one hundred people signed up, and ninety-eight of them completed both parts.
The participants were also not typical patients at the practice. They were younger and technologically savvy, even though more than half of all urgent care patients were over sixty. Pregnant women, people with mental health problems, and anyone who needed emergency care were excluded from the study, so we do not know how the chatbot would cope with a life-threatening situation. In addition, AMIE could not access medical records and could not examine patients or measure their blood pressure. People who know they are being watched may also behave differently than usual.
Experts warn that such programs cannot yet replace doctors. AMIE remains an experimental project, is not available to the public, and has not been authorized for use in hospitals. Google itself acknowledges that the benefits for patients must be confirmed by much larger studies.
The research was funded by Alphabet, Google's parent company. During part of the study, Adam Rodman worked at Google as a visiting researcher, an expert who temporarily joins another institution and contributes to its research.
Sources: nytimes.com and thelancet.com
