Ethics approval
The designs of our megastudy and our forecasting studies were reviewed and approved by the Institutional Review Board of the University of Pennsylvania. A waiver of informed consent was granted for our megastudy because of the following reasons: (1) it was deemed to pose minimal risk to patients; (2) it could not be practically carried out otherwise; and (3) only CVS Pharmacy patients who had already consented to receive SMS communications were included in the study.
Megastudy participants
Megastudy participants were CVS Pharmacy patients who (1) were 18 years or older, (2) resided in one of 65 US metropolitan areas selected for study inclusion (see Supplementary Information, section 4 for a complete list), (3) had previously received at least their primary COVID-19 vaccine series but not the bivalent booster according to CVS Pharmacy records (only patients who had completed their primary COVID-19 vaccination series were eligible for a bivalent booster according to the US Food and Drug Administration), and (4) had consented in writing to receive text messages from CVS Pharmacy (this requirement was imposed to comply with the FCC’s Telephone Consumer Protection Act, which outlaws sending communications by text without an individual’s consent).
The average age of CVS Pharmacy patients in our megastudy was 47.30 years (s.d. = 17.15), and 40.43% of patients were male. Information on the race of a patient was not available. CVS Pharmacy used SMS short codes to contact all patients in our study, and for roughly 60% of patients, the SMS short code used to contact them was familiar—meaning it had been used to send that same patient one or more pharmacy-related messages (for example, about prescription refills) in the previous 22 months.
Megastudy conditions and randomization procedures
Our megastudy included nine different conditions: eight intervention conditions and a holdout control condition. A patient’s condition determined which (if any) text messages they received from CVS Pharmacy reminding them to obtain a COVID-19 bivalent booster vaccine as part of this megastudy.
All intervention messages consisted of an initial set of reminder texts sent on 3 November (hereafter called day 1), 5 November (hereafter called day 2) or 8 November (hereafter called day 3) with a follow-up set of reminder texts sent 7 days later. All text reminders conveyed to patients that a vaccine was ‘recommended’ and ‘waiting for you’, building on past research19,20,21. Our key intervention—which was designed to test the value of free round-trip rides to vaccination sites—included this standard reminder language but also provided patients with one free round-trip ride to and from a CVS Pharmacy in the month ahead. The free ride was provided by Lyft (Extended Data Fig. 2), a popular ride-sharing company supporting over one quarter of all rideshare rides in the United States68. The Lyft codes provided were geofenced so that patients could only take the round-trip ride to and from the CVS Pharmacy locations in their metropolitan area (or subregion). CVS Pharmacy did not cover the cost of the Lyft rides or provide any incentive to patients in this study. All costs of the free Lyft rides to and from CVS Pharmacies were funded by the Social Science Research Council’s Mercury Project.
Our free-ride offering was designed to emulate the 2021 White House programme that offered people free rides by Lyft and Uber to and from vaccination sites for a limited time. Specifically, the free rides offered by the White House were available from 24 May to 4 July 2021 and required customers of Lyft (or Uber) to enter a claim code into their app to receive a free round trip ride (worth up to US$15 per ride at Lyft or up to $25 per ride at Uber29,51,52). Our programme arguably made claiming free rides slightly easier than the White House programme because it simply required one click on a link in our text message to accept our offer code (the code was also supplied directly for manual entry if preferred), and our offer was then automatically applied to the next qualifying ride taken to or from a CVS Pharmacy in the patient’s metropolitan area.
Patients encountered a price cap of $25 per ride only if they attempted to book a ride that exceeded this price limit, at which point they would be billed for spending in excess of $25. We estimated that in the zip codes where our test was conducted, the median resident lived 1.70 miles (2.7 km) from a CVS Pharmacy, such that the cost of a Lyft to or from the pharmacy would typically be under $10. Even patients living at the estimated 99th percentile distance from a CVS Pharmacy were only 9.40 miles (15.1 km) from a CVS Pharmacy, such that the cost of a Lyft to or from the pharmacy would typically be under $25 (see Supplementary Information, section 5 for calculation details and a complete distribution of distance and ride cost estimates).
The interventions tested that did not offer free round-trip Lyft rides to CVS Pharmacy in a standard reminder message instead layered a range of different strategies for encouraging immunization on top of the standard reminder, from conveying current (high) rates of infection in a patient’s county to providing resources to combat misinformation (see Table 1 for a summary of our eight interventions).
Randomization of each eligible participant to one of our nine megastudy conditions was conducted using data obtained from CVS Pharmacy on 18 October 2022 with the splitsample routine in Stata (v.17.0)69. Patients were assigned with equal probability to one of nine megastudy conditions except the intervention offering free round-trip Lyft rides to CVS Pharmacy—this intervention was capped at 50,000 people to ensure study costs would not exceed our budget. Owing to a technical error, reminder messages in two megastudy conditions (interventions 7 and 8 in Table 1) were not successfully sent on 3 November (day 1), and thus no follow-up reminder messages were sent to these intended study patients 1 week later either. These intended participants simply were not messaged or included in the megastudy. As a result, an average of 328,556 patients were included in two megastudy conditions (interventions 7 and 8), whereas an average of 492,573 patients were included in the megastudy’s remaining six conditions. See Fig. 1 for a CONSORT flow diagram depicting randomization.
Megastudy data
All megastudy data supplied by CVS Pharmacy were de-identified through the Safe Harbor method pursuant to 45 Code of Federal Regulations 164.514(b)(2). Supplied data for each patient included sex, age, dates of all previous COVID-19 and flu vaccinations at CVS Pharmacy since 2020, primary insurance type and the zip codes of the CVS Pharmacy locations that were closest to the patient’s home, the most frequently visited and the site of the patient’s last COVID-19 vaccination. We merged in several additional variables that describe the composition of residents of the zip code or county of the CVS Pharmacy closest to the patient’s home address (see Supplementary Information sections 9 and 10 for details).
Calculation of Bayes factors in support of null results
Throughout this article, we support null results by reporting approximate Bayes factors. For all nulls derived from linear probability models estimated using OLS regression, we first estimated the corresponding generalized linear model for binary data with the identity link function70 using Maximum Likelihood, then obtained the Bayesian information criterion (BIC) from the likelihood for both the null (that is, restricted) and the non-null (that is, full) model, and then tightly approximated the Bayes factor in support of the null hypothesis as Bayes factor ≈ exp([BICFull – BICRestricted]/2)71,72,73. For a null result involving a continuous dependent variable, we tightly approximated the Bayes factor directly from the sums of squared errors of the null and non-null models73.
Megastudy data analysis
We evaluated the impact of the eight interventions tested in our megastudy using a pre-registered OLS regression to predict vaccination within 30 days of the start of a patient’s intervention period (or control period). The start of a patient’s intervention period was defined as the (randomly assigned) date when they received their first reminder message (day 1, day 2 or day 3). The start of a patient’s control period was defined as the (randomly assigned) control period start date selected for purposes of comparison with the intervention conditions (day 1, day 2 or day 3). The key predictors in our regression were eight indicator variables for assignment to each intervention condition with an indicator for assignment to the holdout control condition omitted. We also included indicators for the date on which patients were assigned to receive their first reminder text (day 1 or day 2; day 3 was omitted). We estimated this regression with HC1 robust standard errors and adjusted all P values for multiple comparisons using the BH procedure74.
As noted above, a technical error prevented interventions 7 and 8 from deploying on day 1 of our study (so no patients were actually assigned to these interventions on day 1). To assess the ability of our pre-registered OLS regression to produce unbiased results despite the absence of patients in interventions 7 and 8 on launch day 1, we followed a method laid out in our second pre-registration (which was posted after this launch error became apparent but before any outcome data had been received by our research team). Specifically, we ran our standard OLS regression to predict vaccination within 30 days and added interaction terms between indicators for interventions 1–6 and launch day 1. We then conducted an undirected F-test assessing whether these interaction terms were jointly equal to zero. We failed to reject the null hypothesis (P = 0.140), which indicated a lack of heterogeneous treatment effects between launch day 1 and launch days 2 and 3 pooled (Extended Data Table 5). Furthermore, we found strong support for the null hypothesis that the effects of interventions 1–6 were identical on day 1 and days 2 and 3 pooled (Bayes factor ≈ 4.051 × 1017). Following our second pre-registration, we therefore proceeded with analysing data from all 25 available intervention-by-launch day combinations jointly and including indicators for the eight intervention conditions and launch days but no interaction terms.
In addition, after conducting our pre-registered OLS regression (outlined above) to evaluate the impact of our eight interventions on COVID-19 vaccinations, we ran a robustness check that included the following pre-registered additional controls: (1) age (as of October 2022); (2) an indicator for being 50 years or older; (3) an indicator for being male; (4) indicators for insurance type as of December 2022 (Medicare, Medicaid or unknown; commercial insurance omitted); (5) total number of previous COVID-19 vaccinations received at any CVS Pharmacy (as measured in December 2022); and (6) total number of previous COVID-19 boosters received at any CVS Pharmacy (as measured in December 2022). However, these robustness tests have the limitation that variables extracted in December 2022 were likely to be influenced patients’ condition assignment (that is, patients in megastudy conditions that produced more CVS Pharmacy visits for bivalent booster vaccinations apparently had more previous vaccinations ‘updated’, making it appear that they received more vaccinations pre-treatment than other patients; see discussion in the section ‘Megastudy to promote COVID-19 vaccination’).
In further robustness checks presented in Extended Data Table 2 and Supplementary Fig. 1, we also re-ran both regression specifications excluding all data from patients assigned to launch day 1 (because interventions 7 and 8 were not deployed on launch day 1).
Forecasting experiment with laypeople
We recruited 216 US residents who were 18 years and older from Prolific (48.15% male; average age = 35.69 years, s.d. = 13.39 years) and paid them $1.40 to complete a 7-min forecasting survey. All participants were required to take our survey on a desktop computer or tablet rather than a mobile device to ensure images would display properly. All participants were told: “We’ll ask you to review nine different sets of text messages that encouraged pharmacy patients to get their bivalent COVID-19 booster in November 2022. We’ll ask you to predict the impact each message set had on bivalent booster vaccination uptake.” They then all learned about the inclusion criteria for patients in our vaccination megastudy and were told what fraction of patients in our holdout control condition received a vaccine within 30 days of our megastudy’s launch. Because we had not yet determined that data on patients’ vaccinations before 3 November 2022 were unreliable (because it was probably differentially influenced by patients’ condition assignment) at the time of these studies, we told survey forecasters that 5.31% of patients in our control condition had been vaccinated. At this point, the forecasters were required to pass a comprehension check before proceeding—17 laypeople did not do so, leaving 199 forecasters who completed our survey and are therefore included in all analyses (52.26% male; average age = 35.69, s.d. = 13.39).
Next, the forecasters were separately shown each of the different text messaging interventions that patients in our study could have received from their pharmacy. These messages were displayed overlaid on a mobile phone screen (as they would have appeared to recipients). After viewing each set of messages and being reminded what fraction of patients in our holdout control group received a booster vaccine within 30 days of our study’s launch, the forecasters were asked: “For patients who did receive the above text messages from their pharmacy—what percentage of them do you think got the bivalent COVID-19 booster at their pharmacy within 30 days of receiving the first message above? Please enter your response to the hundredth decimal place (for example, X.XX% or XX.XX%)”. For complete study stimuli, which were closely modelled on those used in past forecasting studies21,46,75, see Supplementary Information, section 12.
Although there were only eight intervention conditions in our megastudy, one of our interventions (intervention 4: infection rates) displayed a different message to patients who lived in US counties with above median infection rates in late October 2022. To simplify the way this was communicated to survey respondents, we showed forecasters each of these two message separately and then created a weighted average of their two forecasts (weighted proportionally to the number of megastudy patients who saw each version of intervention 4) to estimate the forecasts of the impact of intervention 4 on vaccination rates.
When depicting the free-ride intervention, we did not show forecasters the Lyft app screens they would have seen had they been in the megastudy and clicked the link in their intervention message to claim a free ride to CVS Pharmacy (see Supplementary Information, section 12, screen 7). Because so few individuals in our megastudy clicked the link to claim a free ride (see Supplementary Information, section 4), giving forecasters the information shown to this small subpopulation would have provided them with a nonrepresentative experience of our stimuli.
These forecasting procedures followed standard practices in the literature21,46,75. Although incentives are sometimes provided for forecasting accuracy, they often are not21,46,75,76,77.
To analyse the accuracy of the estimates of the forecasters, we calculated the absolute change in vaccination rates they forecasted. All results we describe are robust to instead analysing the percentage change in vaccination rates that were forecasted (Supplementary Information, section 11).
Extended Data Table 6 presents the median, mean and standard deviation of the predicted effectiveness of each intervention provided by lay forecasters. Extended Data Table 7 presents the average rank order of intervention performance based on laypeople’s forecasts of intervention efficacy as well as the fraction of laypeople who forecasted each intervention would be the top performer.
Forecasting experiment with experts
We recruited 215 volunteer participants who held a PhD in psychology, economics, business or a related field in the social sciences (37.21% male; average age = 41.86 years, s.d. = 10.73 years) to complete our second forecasting survey. Participants were recruited by posting invitations on the Society for Judgement and Decision Making and the Economic Science Association listservs to anyone with the aforementioned qualifications to make predictions about “a study testing the efficacy of eight different sets of text messages encouraging people to get bivalent COVID-19 boosters”. Invitations to participate in the forecasting study were also posted on social media (Twitter and LinkedIn) in early 2023 by the study’s principal investigators with the message: “Can you predict what text messages worked best to increase bivalent COVID-19 booster vax rates this past fall? Do you have a PhD in #psych, #econ, #business, or a related field?”
The first question in our survey asked respondents to confirm that they held the requisite PhD. The remainder of the study procedures were identical to those described above for lay forecasters. Fifty-two individuals failed our attention check or dropped out of our survey before reaching it, leaving 163 participants who completed our survey and are therefore included in all analyses (49.07% male; average age = 41.86 years, s.d. = 10.73 years). For complete study stimuli, see Supplementary Information, section 12.
Extended Data Table 6 presents the median, mean and standard deviation of the predicted effectiveness of each intervention provided by expert forecasters. Extended Data Table 8 presents the average rank order of intervention performance based on the forecasts by experts of intervention efficacy as well as the fraction of experts who forecasted each intervention would be the top performer.
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.