What statistics tells us about the efficacy of the Pfizer, Moderna, and Oxford vaccines

BY JAMES CAPRETTA AND SCOTT GANZ

The news that the Pfizer-BioNTech and Moderna vaccines are 95 percent effective at preventing COVID-19 disease, and the AstraZeneca-Oxford vaccine might be 90 percent effective (with the right dosage), has given the world a much-needed glimmer of hope. These observed efficacy rates of the vaccines from their Phase 3 clinical trials puts them in the range of the most valuable vaccines in use today and imply that the pandemic would be tamed rapidly when sufficient numbers of people received them. They also substantially exceed what was expected by the trial sponsors and public health officials.

While this is certainly welcome news, it requires context, based on the statistical methods used to evaluate efficacy in clinical trials. In the case of the Pfizer-BioNTech vaccine, for example, the estimated efficacy is based on 170 observed cases of COVID-19 among trial participants, 162 of whom received the placebo. Thus, only 8 trial volunteers who received the vaccine developed COVID-19, which translates into an observed efficacy rate of about 95 percent.

A reasonable question for those trying to understand the implications of the trial, however, is whether this impressive result could have occurred by chance, given that the sample size of 170 cases is relatively small. One way to gauge the robustness of the finding is to calculate the range of likely true efficacy rates — i.e., the efficacy rate of the vaccine when given to millions or billions of people — based on the results of the Phase 3 trial.

Here, there is more good news. Statistical modeling of vaccine efficacy in phase 3 trials reveals that, based on the Pfizer-BioNTech test, there is around a 99 in 100 chance that the true vaccine efficacy rate for this vaccine is above 90 percent. In short, in addition to having a high reported efficacy rate from the trial, the results should make us confident that the vaccine will be very effective when widely distributed.

The Phase 3 trial for the Moderna vaccine also reports approximately 95 percent efficacy. In the trial, 185 out of the 196 total observed cases of the disease among trial participants received the placebo. These results imply that the vaccine has a 99 in 100 chance of being at least 89 percent effective.

Many observers have noted that we should not be surprised that the Pfizer-BioNTech and Moderna vaccines have similar efficacy rates because they are built on the same mRNA technologies. The downside of the mRNA vaccines is that they must be stored at very low temperatures, which complicate widespread distribution efforts. Further, relative to traditional vaccines, the price tag for two doses of the mRNA vaccines — $39 for Pfizer-BioNTech and $74 for Moderna for the first round of U.S. purchases — are many times higher than for traditional vaccines.

For this reason, reports of the efficacy of the AstraZeneca-Oxford Phase 3 trial brings more good news, but also more uncertainty. Unlike the other two vaccines, the AstraZeneca-Oxford vaccine is based on a vaccine technology that only requires refrigeration storage and is priced at around $5-$10 for two doses. However, the results from the trial are less straightforward to interpret.

The average efficacy based on 131 cases of the disease was 70 percent, implying a 99 in 100 likelihood of true efficacy exceeding 53 percent. The results of the AstraZeneca-Oxford trial indicate it may be less effective than the mRNA vaccines. And there is also more uncertainty in the estimate of its true efficacy. Still, 70 percent efficacy exceeds the regulatory criteria for approval and compares favorably to pre-trial expectations.

The trial also suggests that the AstraZeneca-Oxford vaccine could be most effective when administered as a half dose followed by a full dose at least one month apart — instead of as two full doses at least one month apart; the observed efficacy for volunteers receiving the half-dose/full dose combination was 90 percent.

However, this result is based on a much smaller sample of participants and thus should be interpreted with some caution. Fewer than one in four of the participants in the trial who received the vaccine had this dosing regimen. More data from the trial is required before it can be known whether the true efficacy rate under this regimen is indeed in line with the mRNA vaccines.

The results of vaccine trials reported in the past few weeks indicate the world is nearing a turning point in the long-term fight against the pandemic. However, there are still many open questions. Early indications are that vaccines are equally effective at preventing the disease among high-risk populations as among lower-risk groups. But, these conclusions are drawn from limited subsamples of the trials. More data is necessary to determine average efficacy for the elderly and individuals with comorbidities and to see whether efficacy differs across the vaccine technologies.

Still, there is no question the recent news is promising. If the successes from the early trials persist as more data is collected, then the world should soon see an easing of the heavy health and economic burdens that the pandemic has imposed.

James C. Capretta is a resident fellow and the Milton Friedman Chair at the American Enterprise Institute, and Scott Ganz a research fellow at the American Enterprise Institute.