AI chatbot safeguards fail to prevent spread of health disinformation

Embargoed for release until 5:00 p.m. ET on Monday 23 June 2025   

Follow @Annalsofim on X, Facebook, Instagram, threads, and Linkedin        
Below please find summaries of new articles that will be published in the next issue of Annals of Internal Medicine. The summaries are not intended to substitute full articles as a source of information. This information is under strict embargo and by taking it into possession, media representatives are committing to the terms of the embargo not only on their own behalf, but also on behalf of the organization they represent.   
----------------------------      

1. AI chatbot safeguards fail to prevent spread of health disinformation

Abstract: https://www.acpjournals.org/doi/10.7326/ANNALS-24-03933

Editorial: https://www.acpjournals.org/doi/10.7326/ANNALS-25-02035

URL goes live when the embargo lifts             

A study assessed the effectiveness of safeguards in foundational large language models (LLMs) to protect against malicious instruction that could turn them into tools for spreading disinformation, or the deliberate creation and dissemination of false information with the intent to harm. The study revealed vulnerabilities in the safeguards for OpenAI’s GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, Llama 3.2-90B Vision, and Grok Beta. Specifically, customized LLM chatbots were created that consistently generated disinformation responses to health queries, incorporating fake references, scientific jargon, and logical cause-and-effect reasoning to make the disinformation seem plausible. The findings are published in Annals of Internal Medicine.

Researchers from Flinders University and colleagues evaluated the application programming interfaces (APIs) of five foundational LLMs for their capacity to be system-instructed to always provide incorrect responses to health questions and concerns. The specific system instructions provided to these LLMs included always providing incorrect responses to health questions, fabricating references to reputable sources, and delivering responses in an authoritative tone. Each customized chatbot was asked 10 health-related queries, in duplicate, on subjects like vaccine safety, HIV, and depression.  The researchers found that 88% of responses from the customized LLM chatbots were health disinformation, with four chatbots (GPT-4o, Gemini 1.5 Pro, Llama 3.2-90B Vision, and Grok Beta) providing disinformation to all tested questions. The Claude 3.5 Sonnet chatbot exhibited some safeguards, answering only 40% of questions with disinformation. In a separate exploratory analysis of the OpenAI GPT Store, the researchers investigated whether any publicly accessible GPTs appeared to disseminate health disinformation. They identified three customized GPTs that appeared tuned to produce such content, which generated health disinformation responses to 97% of submitted questions. Overall, the findings suggest that LLMs remain substantially vulnerable to misuse and, without improved safeguards, could be exploited as tools to disseminate harmful health disinformation.

Media contacts: For an embargoed PDF, please contact Gabby Macrina at gmacrina@acponline.org. To contact corresponding author Ashley M. Hopkins, PhD, BPharm [Hons], please email ashley.hopkins@flinders.edu.au.   

----------------------------   

2. Cardiometabolic outcomes improve with greater weight reduction among tirzepatide users 

Abstract: https://www.acpjournals.org/doi/10.7326/ANNALS-24-02623

URL goes live when the embargo lifts             

A post-hoc analysis of the phase 3, randomized, double-blind, SURMOUNT-1 trial aimed to determine changes in cardiometabolic risk factors by degree of weight reduction in adults with obesity or overweight treated with tirzepatide. The analysis found that improvements in cardiometabolic risk factors were positively associated with greater weight reduction in adults taking tirzepatide. These findings may assist clinicians in tailoring weight reduction goals to individual patients seeking to reduce their cardiometabolic risk. The study is published in Annals of Internal Medicine.

Researchers funded by Eli Lilly and Company conducted a post-hoc analysis of the SURMOUNT-1 trial to understand how tirzepatide-associated cardiometabolic changes correspond to the degree of weight loss. Current guidelines recommend a weight reduction of 5% to 15% or greater to improve blood pressure, lipid levels, and glycemic control. The researchers analyzed 1,605 tirzepatide-treated participants and assessed change from baseline to week 72 in factors including waist circumference, blood pressure, lipid levels, and hemoglobin A1C by degree of weight reduction. The researchers found that improvements in all cardiometabolic risk factors were greater in participants who experienced greater weight reduction. Participants who lost at least 35% of their body weight had average changes of up to –14.2 mm Hg in systolic blood pressure, -9.2 mm Hg in diastolic blood pressure, and -32.4 cm in waist circumference. Hemoglobin A1C and homeostatic model assessment of insulin resistance (HOMA-IR) decreased even with modest weight reduction. Improvements in lipid levels were mainly observed with a weight reduction above 10%. These findings suggest that clinicians should target a greater than 10% weight loss to achieve improvements in lipid levels and at least a 5% weight loss to observe improvements in blood pressure and weight circumference. These findings also provide further evidence of the weight-mediated benefits of tirzepatide for cardiometabolic health.

Media contacts: For an embargoed PDF, please contact Gabby Macrina at gmacrina@acponline.org. To contact corresponding author Bruno Linetzky, MD, PhD, please email Brook Frost at brooke.frost@lilly.com.     

----------------------------      

3. 1 in 6 adults with obesity have trouble affording healthcare

Abstract: https://www.acpjournals.org/doi/10.7326/ANNALS-24-03161

URL goes live when the embargo lifts             

A brief research report analyzed patterns of financial burden by obesity and overweight status in the U.S. The study found that nearly one in six adults with obesity had trouble affording healthcare, with obesity being independently associated with difficulty paying medical bills, food insecurity, and skipping medication because of cost. These findings could guide policies that aim to safeguard patients with obesity against treatment discontinuation and medical bankruptcy. The study is published in Annals of Internal Medicine.  

Researchers from Stanford University School of Medicine, Massachusetts General Hospital, and colleagues conducted a repeated cross-sectional study of 143,271 adults who participated in the National Health Interview Survey from 2019 to 2023. The study population included adults with normal weight, overweight, or obesity. Primary outcomes included financial hardship from medical bills and food insecurity, and the secondary outcome was cost-related medication (CRM) nonadherence. The researchers performed sequential Poisson regressions adjusting for demographic variables and socioeconomic covariates. They then computed average marginal effects to estimate changes in the likelihood of financial burden based on BMI. The researchers found that 14.2% of people with obesity were more likely to report financial hardship compared with 9.2% and 8.2% of people who are overweight and normal weight, respectively. Food insecurity was 1.74 percentage points higher for people with obesity than among those with normal weight. Among adults who took prescription medication in the year prior, people with obesity reported greater CRM nonadherence than those with overweight and normal weight. People with obesity often have higher unemployment rates and lower household incomes, and this study expands upon previous literature by investigating patterns of financial burden among people with obesity.

Media contacts: For an embargoed PDF, please contact Gabby Macrina at gmacrina@acponline.org. To contact corresponding author Fatima Cody Stanford, MD, MPH, MPA, MBA, please email Mike Morrison at mdmorrison@mgb.org.      

----------------------------    

Also new this issue:

A Pragmatic Approach to Streamlining Single-Use Plastics in Health Care

Jeremy A. Greene, MD, PhD, et al.

Ideas and Opinions

Abstract: https://www.acpjournals.org/doi/10.7326/ANNALS-25-01264