Study shows 95% of all omicron cases in Ontario were in the vaccinated

Another failed test-negative design.

For those of you that have not read my article on the test-negative case-control design, I consider it essential reading in the COVID Era. But if you are non-technical, a one-line explanation is simply this: the study design assumes equal levels of health-seeking behavior between vaccinated and non-vaccinated test-takers. The study design is ludicrous on its face, and good test-negative designs recognize that this assumption can only be true under rigorous conditions, or as Vandenbroucke and Pearce note,

“Test-negative studies recruit cases who attend a healthcare facility and test positive for a particular disease; controls are patients undergoing the same tests for the same reasons at the same healthcare facility and who test negative."

In any case, a study out of Ontario, Canada got a mention in the mainstream news recently. The Canadian Broadcasting Corporation article titled “Canadian COVID-19 vaccine study seized on by anti-vaxxers — highlighting dangers of early research in pandemic” claims that the study vastly underestimated the effectiveness of the vaccines. Of course, CBC’s senior health writer Adam Miller, whose expertise in data stems from his degree in journalism, made sure to mention that this study is not peer-reviewed and is being revised. He claims that the negative effectiveness only appears because of behavioral issues and the fact that the authors only had data until December 19.

Well, Adam, there were behavioral issues. The entire study is a behavioral issue. It’s a test-negative design. That does not seem to stop the vast majority of these flawed designs from getting peer-reviewed and published, and in fact, the only thing preventing it from being peer-reviewed and published is that the findings do not match with vaccinated expectations. And since we’re talking about the timing of studies, why did you have no problem with the timing of any other studies or the fact that we give the vaccinated a two-week bonus for nearly every accounting of the statistics? Really, "adding two more weeks of data and it looks like there's no more negative VE” is the argument here? Yikes.

The fact of the matter is the study design did not overestimate vaccine effectiveness. Anyone stupid enough to think so did not even glance at the numbers the authors were working with. Let’s dive in.

The author’s biased findings that still show negative VE in omicron.

First of all, we do not actually have the population characteristics in this study. The study uses individuals 18+ that received an mRNA vaccine. Ontario does not, as far as I am aware, publish what the breakdown of vaccines is in the province. The number that have not received this vaccine may be in excess of 500,000, so understand that the fully vaccinated cohort is almost definitely overestimated. The study also excludes the partially vaccinated.

In total, 88.3% of Ontario’s population over the age of 18 was fully vaccinated as of the end of the study period. Ontario data actually did not aggregate the data on those with a booster until January 5th, at which point about a third of adults had received a booster. During this period, around 2.4 million people received a dose, and only around 200,000 people received a first or second dose, so let’s say the boosted population since that study period was around 2.2 million people. In other words, around 17.1% of adults were boosted at the end of the study period. Additionally, around 9.16% of individuals were not vaccinated.

For ease of calculation (ie., I’m feeling lazy), let’s assume vaccination status did not change much over the study period (ie., the final numbers are the real numbers). In reality, around 10.72% of the adult population had not received any vaccinations at the start of the study period, and around 86.37% of the population was fully vaccinated. That is pretty much a wash¹. The true implication is the boosted population is drawing a lot of vaccine effectiveness from the fully vaccinated population. In fact, since the majority of people got boosted during this period, it may mean that the vaccine effectiveness and in the boosted population and the fully vaccinated population is essentially the same (ie., boosting does nothing).

Here are the characteristics of those infected. Note, I took all the junk we don’t care about of this screen capture.

So, 94.9% of cases were in the vaccinated for omicron. Because we eliminated the partially vaccinated (essentially 300,000 individuals) from the numbers to match the study, normalizing the population of the vaccinated and not vaccinated gives us 9.4% of the normalized population that do not have any vaccinations, 75.1% of the population with two doses, and 15.5% of population with a booster. Reminder: the boosted population is overestimated and the two doses population is underestimated.

In other words, the relative risk of catching omicron for the three groups was as follows (rounded):

• no vaccination: ~0.02%

• two doses: ~0.05%

• booster: ~0.01%

• all vaccinations: ~0.03%

The vaccine effectiveness for each group would be:

• two doses: -120.6%

• booster: 43.5%

• all vaccinations: -92.5%

I wish I had better data on boosters by day (and better data on everything else, really, so I could adjust for variables such as comorbidites and did not have to make conjectures) as it would be interesting to see the full extent of the booster effectiveness leech here, but in any case, as the authors noted in their original paper, this has huge implications for vaccine passes. You cannot justify a vaccine pass for even 43.5% effectiveness in the dominant strain, let alone -92.5%. Maybe we should demand the vaccinated be excommunicated from society with numbers like these?

1. On second reading, no, it’s extraordinarily generous to the vaccinated as proportionally that is a very large percentage of the not vaccinated population while being only a small change in the vaccinated population. I will leave it as-is to take the sting off the highly negative effectiveness already present in the vaccinated cohort.