Our Unstatistic of the Month for November addressed the question of how to correctly interpret statements about vaccine effectiveness. That contribution continues to make big waves to this day.
Immediately after its publication, BR Faktenfuchs very differentiatedly addressed the topic of absolute and relative risk reduction, clarifying why the statement that the active ingredient protects 95% of vaccinated individuals from getting sick or even infected is not correct. However, some websites, at least partly belonging to the conspiracy theory milieu, conclude from the numbers that the BioNTech/Pfizer COVID-19 vaccine would practically protect no one, with only 0.84% of vaccinated individuals.
Recently, Bunte.de referred to the Unstatistic to explain why a vaccine efficacy of 70% does not mean that 30% of those vaccinated with AstraZeneca will get sick. Just now, Correctiv.org cited the Unstatistic in a fact-check of six claims about the BioNTech vaccine.
"Life-threatening side effects" and supposedly no protection against Covid-19? Six claims about the BioNTech vaccine fact-checked
In blog articles, false and misleading claims about the side effects and effectiveness of the Covid-19 vaccine are being spread.
A core issue is the (in everyday German language) imprecise use of the term "effectiveness." In a clinical study like the one Pfizer submitted for the approval of the vaccine, what is examined is the so-called Efficacy. Efficacy is measured in terms of the relative risk reduction in the vaccinated group over a predefined period.
Thus, the question concerns how much the relative frequency of becoming sick with Covid-19 over the observation period is reduced in the vaccinated group compared to the unvaccinated. The median observation time was two months, and it was not the frequency of infections that was studied, but the frequency of illnesses.
Efficacy is different from effectiveness, which refers to the efficacy of a vaccine or more generally an intervention in 'real life'.
Effectiveness depends on many factors that cannot be observed in a clinical study. There are often good reasons why effectiveness could at least match efficacy over a comparable period if the conditions (such as receiving two doses at the recommended interval, similar age structure among the vaccinated, etc.) are roughly similar. For this reason, confidence intervals or credible intervals (depending on the statistical analysis approach used) are provided for efficacy in such studies, allowing conclusions to be drawn about a larger population. In the case of the BioNTech vaccine, it can be assumed that there is a 95% probability that the efficacy is at least 90.3%.
Firstly, the study showed that the observed relative frequency of COVID-19 disease within the observation period, defined as the occurrence of at least one typical symptom, was about one percent in unvaccinated individuals, i.e., 20 out of 2,000 people. Furthermore, the study demonstrated that only about one out of 2,000 vaccinated individuals became ill in the same period. It is justified to conclude that the vaccine prevented 19 out of 20 expected illnesses, i.e., 95%.
However, this does not mean that the vaccine protects 95% of those vaccinated from disease. Why? Because in absolute terms, 19 out of 2,000 vaccinated individuals did not get sick due to the vaccination – this is 0.95 percentage points of absolute risk reduction, from one percent to 0.05 percent. Using the actual numbers from the study, the 0.84 percentage points, which are often cited by vaccine opponents, are derived.
Confusion arises when trying to infer effectiveness from efficacy if the observation period is overlooked.
One could argue with some justification that over a longer period, more people would become ill and accordingly more would be protected. It has now been demonstrated that antibodies can still be detected in vaccinated individuals even after four months.
Let's consider an extreme scenario. Suppose 2,000 unvaccinated individuals, who have shown no signs of a past or current COVID-19 infection, are locked in a closed room with the SARS-CoV-2 virus and held there for two months. It is quite plausible to assume that all of them would get infected.
Studies show, however, that only 2/3 to 3/4 of those infected actually develop symptoms, i.e., at least one symptom. Therefore, up to 1,500 of the unvaccinated individuals might get sick. Among 2,000 otherwise comparable vaccinated individuals who also spend two months with the virus in a closed room, only 75 would get sick with a 95% efficacy rate, which is 5% of 1,500.
In fact, the vaccine then protects 1,425 individuals from getting sick. If the vaccine were to immunize for life, a maximum of 71.2% of the vaccinated individuals would be causally protected from illness by the vaccine, since the others would not have become ill even under the most extreme conditions, or they still did. Accordingly, the absolute risk reduction is 71.2 percentage points.
If the vaccine's immunity lasts for a shorter period, its protective effect relative to those vaccinated decreases. For example, if the vaccine-induced immunity lasts for one year and 12% of people become infected within that year, up to 75% of the unvaccinated (i.e., 9% of the unvaccinated) would get sick and 95% of these expected illnesses would be prevented by the vaccine in the vaccinated. Therefore, causally, 8.55% of the vaccinated are protected from illness by the vaccine, and the absolute risk reduction is correspondingly about 8.6 percentage points.
To date, we, the Unstatistik team, continue to receive numerous inquiries on this topic. Some of these inquiries have a noticeably irritated undertone, suggesting that we might at least be negligently providing ammunition to vaccine critics or even openly siding with them. Therefore, it should be clarified:
Neither Unstatistik nor this article argues against vaccination.
I personally would get vaccinated if I could—which, due to my previous COVID-19 illness, will probably not be possible in the foreseeable future. Of course, that's beside the point here.
However, I strongly advocate for using terminology accurately because only those who truly understand what is being discussed can make informed decisions based on data and statistics.