An experimental Covid-19 vaccine from Pfizer and German drug developer BioNTech (BNT) seems to be working very well, according to early trial data. The announcement, made on Monday, has enormous implications. If we end up with a few vaccines that are as effective as this one appears to be, then very widespread vaccination could be a steamroller for public health, slowly crushing the pandemic.
The details, though, are complicated; and the press coverage to this point has often been off target. Some stories underplay the good news with spurious caveats, such as the claim that the numbers are drawn from “a very small number” of cases. Others have overplayed it—wrongly claiming, for example, that the vaccine has been shown to be more than 90 percent effective at preventing infections, or that it has “displayed no side effects.” Here’s what we do know: Pfizer and BNT have just hiked up the bar for Covid-19 vaccines; and while it’s not yet clear exactly how high that bar now sits, there’s every reason to think it’s pretty high.
First, was the number of infections really that small? Long before any results came in, the protocol for this trial calculated what sort of data—including how many people ended up getting sick—would need to be tallied in order to know that the vaccine had cleared the minimum requirement of 50-percent effectiveness set by the Food and Drug Administration. (A vaccine that prevents about half of illnesses would be more like a flu shot than a measles immunization—not enough to squash the pandemic on its own, but still much better than nothing.) It’s tricky, because there was no telling where the virus might surge in the first few months of the trial. To get an answer fast, you would need to have a lot of people in both the vaccinated and control groups who were sitting in the virus’ possible path—far more people than you would need if you were running your trial on, say, a giant cruise ship about to have an out-of-control outbreak. That’s why it was necessary to have tens of thousands of wonderful people as volunteers, but the required number of actual “events”—in this case, volunteers getting symptomatic Covid-19—would be small, if the vaccine were to end up making a really big difference.
An independent data monitoring group was scheduled to look at the data at several points along the way, to check whether it was having a dramatically good or bad impact. That’s a process with a lot of planning and safeguards. BNT and Pfizer originally planned to have this group check in with four interim analyses, once certain numbers of events had been recorded. The companies were criticized, though, for allowing themselves too many opportunities to stop the trial early, and for planning to have the first analysis when there were just 32 events. “Take the time, the extra weeks,” one expert told The New York Times. “No shortcuts. Nobody will regret it.”
Monday’s press release reports there had been negotiations with the FDA to drop the earliest analysis, but by the time the new protocol was signed off, there were already enough events to skip ahead to what would have been the third analysis. Now, with the virus surging in the US, the full number of events required for this trial—a total of 164 volunteers’ getting sick—may even be reached in the coming weeks. For the moment, the companies are only reporting on the first 94 events.
So there really isn’t a reason to be concerned about the number of events included in this analysis. What about the results—have BNT and Pfizer really shown that the vaccine is “more than 90 percent effective at preventing infections”? No. For one thing, we have no information on how many people in the study were infected with the virus—only a number who actually ended up getting sick and diagnosed with Covid-19. The “90 percent” figure is also up in the air, for the time being. Because there’s more uncertainty in the smaller numbers that emerge early in the trial, the hurdle for declaring success must be set higher than 50 percent. The companies had calculated ahead of time that if they’d found the vaccine to be 62.7 percent effective at around this point, they could be confident the vaccine would turn out to be at least 50 percent effective overall. On the other hand, if that number were lower than 38.6 percent, it would mean the trial was never going to be able to clamber over the bar. When the data monitoring board made its examination, the effectiveness rate was more than 90 percent for people who had definitely been Covid-free before the trial started. (Wow.) That rate could drop when the next 70 events join this unlucky group, but it should still be well clear of the FDA’s minimum requirement. On the other hand, it could end up considerably higher than 90 percent, too.
