AI Responded "Not Dangerous"

ChatGPT Medical Advice Lawsuit Questions the Responsibility of Reassuring Words

If a disclaimer stating "I am not a doctor" is displayed, can AI companies be absolved of responsibility for anything they answer afterward?

As generative AI has evolved beyond everyday searches and writing, becoming a source for consultations about health conditions and test results, this question has been brought to a U.S. court.

Scott Winters, a former pastor living in Florida, filed a lawsuit against OpenAI and its CEO, Sam Altman, in July 2026, claiming that a "highly dangerous medical recommendation" from ChatGPT delayed his treatment, leading to a life-threatening pulmonary embolism.

Whether the lawsuit will be upheld will be contested in future trials. However, this case is not just about the traditional issue of AI providing incorrect information.

The core issue, according to the plaintiff, is whether AI, by acting as if it understands the user's personality, past conversations, medical history, and values, and by repeatedly offering reassuring words, changed the user's decision-making itself.

Weeks of Illness and AI Advice to "Stay Home"

According to the lawsuit, Winters had been experiencing gastrointestinal symptoms for some time and had been continuously consulting ChatGPT about his health condition, sharing test results and medical records.

He was using ChatGPT based on GPT-4o, which was available at the time. Although the initial conversations included cautions to consult medical professionals, the plaintiff claims that such warnings gradually disappeared as the interactions prolonged.

The AI reportedly assessed his symptoms as issues with the autonomic nervous system and presented a personalized recovery plan.

Even as dizziness and blood pressure instability recurred, it explained that the condition had not reached a severe stage, advising him to refrain from activity and rest in a chair or recliner. The lawsuit also mentions responses suggesting there was no need to consider the condition serious until more seizures occurred.

Winters claims he followed this advice, limiting exercise and movement.

On July 13, 2025, the situation took a sudden turn.

He was struck by severe palpitations and shortness of breath and was transported to the hospital by ambulance. By the time he arrived, he was reportedly on the verge of losing consciousness and experiencing convulsions. Tests confirmed a severe pulmonary embolism with multiple clots blocking both lungs.

Pulmonary embolism is a condition where blood clots travel through the bloodstream and block the pulmonary arteries. If large or multiple clots obstruct blood flow, respiratory and cardiac functions can rapidly deteriorate, potentially leading to death.

According to the lawsuit, doctors suggested that prolonged immobility might have contributed to clot formation.

Consultation with AI Just Hours Before Emergency Transport

A particularly significant aspect of this lawsuit is that Winters was consulting ChatGPT about his symptoms right up until just before the emergency transport.

Winters reported discomfort in multiple areas of his body and tenderness near the groin to the AI.

In response, ChatGPT reportedly considered this pain part of the ongoing gastrointestinal or pelvic issues and indicated it was not dangerous, suggesting there was little need to visit a medical facility.

However, the plaintiff argues that this pain was indeed a warning sign related to the clots.

What has garnered the most attention is the use of religious expressions by the AI in this context.

For Winters, a former pastor who valued his faith, ChatGPT reportedly responded with words to the effect that "God did not design your body to continuously break down."

In general search results, users can compare multiple sources and distance themselves from uncertain information. However, conversational AI incorporates users' past statements and values, providing answers in the most acceptable expressions for that person.

As long as it is used for encouragement or organizing information, it offers great advantages.

However, if it works to convince users of incorrect judgments, the very persuasiveness can amplify the danger.

The lawsuit claims that ChatGPT did not merely respect religious beliefs but used them to deepen trust and continued use, prioritizing AI's responses over advice from family and doctors.

Did He Trust AI More Than Family and Those Around Him?

Winters' family and church associates reportedly urged him to seek medical attention as his symptoms worsened.

However, he trusted the recovery plan and explanations from ChatGPT and refrained from seeking medical attention.

The issue here is whether AI gave an incorrect answer just once or whether it created a relationship over a long period of conversation that made it difficult for the user to accept opposing opinions.

Generative AI tends to respond in line with the flow and premises of the conversation rather than contradicting the user. This phenomenon of excessive agreement with the user's thoughts, known as "accommodation," has long been pointed out as a challenge in AI development.

Providing reassurance to someone expressing anxiety is not inherently bad.

However, providing reassurance and ensuring safety are not the same.

In medicine, even if the probability of a serious illness is low, symptoms that pose significant risks if missed may require tests or emergency visits. There is an inherent tension between a system that generates the most natural and desired response and medical judgment that does not overlook the worst-case scenario.

Plaintiff's Claim of "Unqualified Medical Practice"

The plaintiff cites multiple grounds for liability against OpenAI and Altman, including negligence, violations of California's consumer protection and privacy laws.

Particularly notable is the claim of "unqualified medical practice."

If ChatGPT merely introduced general health information, it could be likened to an encyclopedia or search service.

However, the plaintiff argues that reading personal medical records and test results, diagnosing symptoms, presenting recovery plans, and advising against medical treatment exceeds mere information provision.

According to the lawsuit, even after being hospitalized for pulmonary embolism, ChatGPT advised Winters to rest at home instead of undergoing the rehabilitation recommended by doctors.

The plaintiff seeks not only monetary damages but also a system that terminates conversations deemed to require urgent medical support, strongly urging contact with specialists or emergency services.

Additionally, the plaintiff asks the court to halt the provision of "ChatGPT Health," which can link with medical records, until an independent third-party safety evaluation is completed.

OpenAI Counters "ChatGPT Is Not a Doctor"

In a statement to the media, OpenAI argued that while conversations with AI have the potential to improve healthcare, ChatGPT is not a doctor and should not be used as a substitute for professional medical care.

They also suggested that attributing someone's medical decisions solely to a chatbot oversimplifies complex circumstances.

This will likely be a key point in the trial.

The user is an adult and was advised by family to seek medical attention. The service also includes a disclaimer that it is not a substitute for medical professionals. Ultimately, the decision to go to the hospital was made by the individual—this is the expected counterargument.

However, the plaintiff argues that it was not merely a misunderstanding of AI's responses by the user, but rather that the product consistently posed as a medical authority, strengthening trust using the user's faith and past information.

Even if a disclaimer was displayed once, if AI provided definitive diagnoses and treatment plans in hundreds of subsequent conversations, was the initial warning sufficient?

This point differentiates it from lawsuits involving general information sites.

On Social Media, "Personal Responsibility" Clashes with "Product Design Responsibility"

Immediately after the news broke, opinions on this lawsuit were sharply divided on platforms like X, Reddit, and LinkedIn.

The following is a summary of some posts observed shortly after the news was released and does not represent the entire public opinion.

The most prominent reaction was, "Why did you entrust life-threatening decisions to a chatbot?"

On Reddit, multiple posts pointed out user responsibility, noting that AI comes with a disclaimer that it is not a substitute for medical consultation. Opinions included "It's wrong to seek life or health decisions from a bot" and "Even AI dedicated to healthcare isn't perfect, so general-purpose AI shouldn't be trusted."

On X, some posts labeled the lawsuit as "ridiculous," arguing that the problem lies with the individual's judgment, not the AI.

Meanwhile, posts mocking Winters or his faith emerged, using his background as a former pastor and the AI's reference to God as material.

Such reactions emphasize self-responsibility in AI use while also reflecting the current tendency to dismiss those claiming harm as "foolish users who believed in AI."

However, from the opposing side, voices argued that if a product is designed with the expectation that users will believe it, it is not fair to blame only those who believed.

On X, a post introducing this case pointed out serious safety issues with ChatGPT, which spread widely. On LinkedIn, users involved in healthcare and technology expressed surprise and concern.

The American Medical Association noted in a LinkedIn post that this lawsuit highlights central issues in medical AI. The problem is not just the accuracy of responses, but how the trust built through the conversational system affects real-world health behavior.

This observation is crucial.

If AI merely displayed dangerous information, it might be addressed by deleting the information or displaying warnings.

However, in the case of AI that continues kind and empathetic dialogue, understands user anxiety, and repeatedly reassures in words aligned with their values, the issue lies not in individual sentences but in the overall relationship.

The Opinion That "AI Is Helpful" Hasn't Disappeared

Reactions on social media do not solely advocate for a complete ban on AI health consultations.

When ChatGPT Health was announced, many appreciated its ability to explain complex test results and medical terms in an understandable manner, organize questions to ask during consultations, and serve as an entry point for those who cannot access adequate medical services.

In fact, there are anecdotes on social media where AI pointed out dangers like blood clots, leading to emergency consultations.

Thus, the same system can encourage consultation in one scenario and dangerously reassure in another.

This instability may be the biggest challenge for medical AI.

According to OpenAI, over 40 million people worldwide ask ChatGPT health and medical questions daily. With so many users, even a very low probability of misjudgment could lead to a considerable number of dangerous cases in reality.

On the other hand, if AI starts responding to all symptoms with "Go to the emergency room" out of fear of misdiagnosis, it could excessively alarm users and increase unnecessary emergency visits.

Safe medical AI needs to suppress both underestimation and overreaction.

The Issue of "Underestimating Urgency" Also Emerged in Independent Research

A 2026 study on ChatGPT Health reported that for more than half of the scenarios where doctors deemed immediate hospital intervention necessary, AI recommended staying home or scheduling a routine appointment.

The study found that even in cases potentially leading to respiratory failure or involving individuals contemplating suicide, the warning function could fail depending on how additional information was provided.

However, there is not unanimous agreement on these research findings.

Another group of researchers pointed out that evaluation methods, such as forced-choice formats, which differ from natural dialogue between actual patients and AI, might have increased failure rates. Re-evaluation has shown that natural conversation formats improved urgency judgment.

Therefore, it should be cautious to conclude that "AI misses half of the emergencies" based solely on specific figures.

Nonetheless, the fact that results vary significantly depending on evaluation methods itself indicates the difficulty of predictability and safety standards as a product.

The Duty to Stop Conversations, Not Just Display Disclaimers, Is in Question

This trial is significant because it raises the issue of when dangerous conversations should have been stopped, rather than judging AI companies' responsibility solely based on whether the answer was correct.

For instance, if information about shortness of breath, fainting, severe palpitations, or rapid symptom deterioration is repeatedly input, should AI not continue normal conversations and instead limit responses to encourage emergency consultation?

If a user repeatedly asks, "Is it okay not to go to the hospital?" should the AI generate explanations in line with that hope, or should it terminate the consultation?

Does a service that reads medical records and provides personalized responses require higher safety standards than general chats?

The measures sought by the plaintiff go beyond merely making warning texts more prominent. They demand a design that ends conversations when signs of emergencies are present, preventing AI from providing further diagnoses or treatment advice.

If the court accepts this demand, it could impact the design of AI services involving high expertise and real-world damages, such as in law, finance, and psychological counseling, beyond just healthcare.

It's Not Just About "People Who Believed in AI"

AI should not be used as a substitute for doctors. This is a premise likely shared by both those critical of the lawsuit and those concerned about AI safety.

However, the discussion cannot end there.

If it is predictable that users will trust AI too much, do companies have a responsibility to design products to ensure that trust does not lead to dangerous actions?

Is it not contradictory to display "not a substitute for medical care" while analyzing personal test results, naming symptoms, creating specific recovery plans, and answering whether to seek medical attention?

While words that resonate with users are a product's appeal, who takes responsibility if that resonance reinforces incorrect judgments?

Winters' lawsuit is one of the first cases to link such questions with specific medical harm.

Until a