Brief conversations with a large language model chatbot increased parents' short-term intentions to vaccinate their children against human papillomavirus, but the effects did not persist as long as those observed with government public health materials, according to a randomized clinical trial published in JAMA Network Open.
The online trial included 1,297 parents in the US, Canada, and the United Kingdom who had at least 1 human papillomavirus (HPV) vaccine–eligible child who was unvaccinated or whose vaccination status was unknown. Mean baseline likelihood of vaccinating the index child within 12 months was 27.4 on a 0- to 100-point scale, which the researchers described as reflecting substantial baseline hesitancy.
Participants were randomly assigned to receive no message, country-matched government public health materials, a GPT-4o chatbot using a default response style, or a GPT-4o chatbot using a shorter conversational style. The public health materials, which included official content from the CDC, NHS, or Public Health Agency of Canada, were displayed for a minimum of 3 minutes. Chatbot participants completed a 3-minute minimum multiturn interaction, and chatbot prompts incorporated each participant's top-rated reason for HPV nonvaccination.
The primary outcome was parents' self-reported likelihood of vaccinating their child against HPV within 12 months, measured immediately following the intervention. Follow-up assessments occurred at 15 and 45 days.
Compared with no message, all 3 active interventions increased immediate HPV vaccination intent. Public health materials produced the largest effect, followed by the default chatbot and the conversational chatbot. The default chatbot did not statistically significantly differ from public health materials, whereas the conversational chatbot produced a smaller immediate effect than public health materials.
Durability differed across arms. At 15 days, public health materials and the conversational chatbot remained associated with modest increases in intent vs no message, while the default chatbot did not. By 45 days, only public health materials remained statistically significant. The researchers noted that lower-than-planned enrollment may have reduced statistical power to detect follow-up effects and rarer binary outcomes.
None of the interventions increased self-reported HPV vaccination uptake at 15 or 45 days. The interventions also showed no spillover effects on influenza or COVID-19 vaccination intentions, general parental vaccine hesitancy, charitable donation willingness, or use of a vaccine scheduling link among US participants.
The researchers also reported differential attrition. Both chatbot arms had lower odds of providing primary outcome data than the no-message group, largely because some participants entered the intervention survey but did not complete the chatbot interaction and subsequent intent item. The primary analysis followed an intention-to-treat framework, adjusted for baseline HPV vaccination intent, and used multiple imputation for missing postbaseline outcomes.
In exploratory analyses, researchers did not detect statistically significant differences in treatment effects across demographic groups or baseline beliefs. Participants rated the default chatbot as more empathetic than public health materials and spent more time interacting with it, but greater engagement did not translate into larger or more durable effects on intent. The conversational chatbot was rated as less effective than public health materials on a message-effectiveness scale.
The researchers noted that their findings differed from two recent HPV-specific chatbot trials. In one US trial, a GPT-4–based chatbot increased parental HPV vaccination intentions more than a CDC brochure, although parents spent more time with the chatbot and outcomes were assessed within the same survey session. In a school-based cluster trial in China, parents given 2 weeks of on-demand access to a chatbot were more likely to schedule or obtain HPV vaccination for their children compared with no intervention. The researchers wrote that design differences may help explain the smaller and less persistent chatbot effects in the current trial, including equalized minimum exposure, use of public health materials as a strong comparator, a delay between baseline and intervention assessment, and follow-up through 45 days.
Several limitations temper interpretation. Vaccination intent and uptake were self-reported, and follow-up was limited to 45 days. The sample was recruited through online survey platforms, was predominantly White and female, and skewed toward moderate-to-high artificial intelligence familiarity, which may limit generalizability to parents in clinical settings, those with limited digital access, and those with lower digital literacy. The study also enrolled fewer participants than planned, which may have reduced power to detect effects on follow-up intent and rarer outcomes such as vaccination uptake.
The findings suggest that brief, one-time chatbot interactions may raise short-term HPV vaccination intentions but offer limited advantage over existing public health communication materials. The authors suggested that future research should examine whether integrating AI-based conversations with broader structural interventions could produce more durable effects.
"High-quality, evidence-based public health materials performed at least as well as brief chatbot interactions and demonstrated greater durability," wrote lead study author Neil K. R. Sehgal, MS, of the University of Pennsylvania, and colleagues. "Chatbots may still serve as a supplementary resource, particularly for addressing questions or reinforcing existing guidance, but should not blindly be used in place of established communication strategies."
Disclosures: The researchers reported no conflicts of interest. The study was funded in part by the University of Pennsylvania's Penn Medicine Communication Research Institute, the Penn Global Research and Engagement Fund, and grants from the National Institute on Minority Health and Health Disparities. The funders had no role in the design or conduct of the study; collection, management, analysis, or interpretation of the data; manuscript preparation, review, or approval; or the decision to submit the manuscript for publication.
Source: JAMA Network Open