Strategy
Public messaging case studies: how US cities test behavioral insights
US cities use randomized trials to test public messaging. Case studies show effect sizes and costs. Measurement tools often leave out behavior change.
What to take away
- US cities have run randomized tests of public messages, with reported effect sizes such as a 26 percent drop in missed court dates in Philadelphia.
- Test designs usually compare message variants against a control group.
- Costs per message and staff time are rarely published, which makes full evaluation hard.
- Disclosure rules from the FTC apply when cities pay influencers or place sponsored content.
Documented US city tests and effect sizes
Philadelphia and New York City have published results from randomized controlled trials of public messaging. These tests show how behavioral insights public messaging works in practice.
Documented US city tests
| City | Message tested | Test design | Effect size |
|---|---|---|---|
| Philadelphia | Text reminders for court dates | Randomized controlled trial | 26 percent reduction in failure to appear; more than $3 million saved |
| New York City | Text reminders for flu shots | Randomized controlled trial | 5 percentage point increase in vaccination |
The Philadelphia court reminder trial is one of the most cited US city public messaging examples. It sent text messages to people with upcoming court dates. The messages reduced failure to appear by 26 percent, according to published results. The city also saved more than $3 million in costs tied to missed hearings.
The New York City flu shot trial tested text reminders against a control group. It found a 5 percentage point increase in vaccination among recipients. That result is smaller but still meaningful for public health.
How cities design and run these tests
A typical test follows a sequence. City staff first define the behavior they want to change. Then they write two or more message variants. They randomize residents to receive one variant or no message. Finally they compare outcomes.
- Define the target behavior and the population.
- Write two or more message variants.
- Randomize residents to receive one variant or a control message.
- Measure the outcome against the control group.
- Compare costs and decide whether to scale the winning variant.
Many cities use text messaging platforms and simple A/B testing software. For a full walkthrough of writing variants and running randomized trials, see the design and test campaigns process. Some cities partner with the Behavioral Insights Team, a provider that has run trials in Philadelphia and New York City. Others build internal behavioral insight units.
To see how municipal nudge units test small changes in wording to nudge residents, read the guide on public messaging and behavioral insights.
For a detailed process on sourcing and evaluating pilots, see how US cities use behavioral insights.
Most published case studies describe the test design but omit the cost per message and the staff hours required to run the trial.
Costs and what gets left out
Cost data is the weakest part of most public messaging case studies. Philadelphia reported savings, but not the cost per text message. New York City did not publish a full budget for its flu shot trial.
Staff time for writing variants, managing randomization, and analyzing results is almost never itemized. This gap makes it hard for other cities to estimate what a similar test would cost.
When a city pays a local influencer to share a public message, the content must be identifiable as advertising under the FTC native advertising guide. The FTC disclosures 101 guidance requires influencers to clearly and conspicuously disclose material connections to brands.
Public relations messaging aims to build understanding and trust, as the Wikipedia article on public relations explains. But measurement often stops at impressions and clicks.
To check what the dashboards leave out, compare PR measurement tools by what they capture and what strategy needs. That comparison includes disclosure rules and a tool checklist.
Tracking results after the campaign ends
Most US city public messaging examples track immediate outputs like message opens or website visits. Fewer track behavioral outcomes like court appearances or vaccination rates over time. The Philadelphia and New York City trials are exceptions because they used control groups.
Public messaging evaluation requires baseline data and a clear outcome measure. Without a control group, a city cannot know if its message caused the change. This is why randomized tests are the standard for behavioral insights public messaging.
