The conundrum of the test-negative case-control design

Assumptions are where art meets science, and, these days, the scientific literature is beginning to look like a Jackson Pollock drip-painting.

Those that keep up-to-date on the pandemic literature have undoubtedly encountered the test-negative case-control design (TND). Famously, the Center for Disease Control used a TND to declare that vaccination in 5.5x superior to natural immunity and researchers out of South Africa used a TND to declare that the vaccination remained somewhat effective against omicron.

It is easy to understand why researchers love this design. The design does not require any excess data collection. Thus, the funding and effort requirements to conduct a “study” using a TND lie solely in administrative costs. Furthermore, as the data is already being collected and is available, more or less, in real-time, researchers can use this method to derive vaccine effectiveness “instantaneously”. But the third argument for TNDs is really the cherry on top that provides legitimacy to the whole ordeal: researchers claim that the design reduces bias in estimating the true-VE.

A design that is quick and cheap while reducing bias? Who can’t get behind that? But as most things in life that sound too good to be true, it is not that simple.

So, what is a TND?

Let’s say we want to compare distinct groups of individuals, like the vaccinated and unvaccinated. One of the problems with a straight up comparison of these groups is there is a lot of unobserved error that will bias any estimates we arrive at. Perhaps the largest unobserved error is the perceived difference in health-seeking behaviors between the two groups. TNDs try to reduce this error by using testing behavior as an instrumental variable that will, in theory, equalize the differences in health-seeking behaviors.

The logic is that any two people taking a PCR or rapid test have the same health-seeking behavior (and, on top of that, the same access to care) as one another. This is the main assumption that TNDs make. Instead of looking at populations proportionally, then, we can look at test-taking behavior proportionally. Let’s use the South African example.

In the Discovery health network for Gauteng province between November 15th and December 7th, there were 78,173 tests administered with 19,070 positive results. The unvaccinated took 26,331 tests with 7,889 positive results; whereas, the vaccinated took 51,842 tests with 11,181 positive results.

Using a TND, the underlying population of vaccinated and unvaccinated individuals in the province does not matter. Only the proportion of positive results from the tests matter. The unvaccinated group had a test-positivity rate of 29.96% and the vaccinated group had a test-positivity rate of 21.57%. VE, then, would be estimated to be (29.96-21.57)/29.96 or 28%. That is a far-cry from the less than -200% VE that we would arrive at by looking at the proportion of positive results in vaccinated individuals in Gauteng province over that period of time.

Other adjustments can be made to account for differences in underlying factors. In fact, many TNDs match individuals up to those with similar attributes (ie., propensity score matching) and discarding individuals that cannot be matched, then it becomes easy to look at the proportion of positive tests in matched individuals. And generally, only the first positive test an individual takes is considered part of the data (and all subsequent test results discarded).

But those adjustments are standard and non-controversial. What I do find controversial is the initial assumption of test-taking being an instrument for health-seeking behavior. Though, I seem to be one of the few that takes issue with this assumption as the controversy in the literature is nearly non-existent. The hidden assumption in the initial assumption is that it relies on people only testing for the virus when they are symptomatic and under similar conditions.

A large subset of society has, through fear and manipulation, been turned into fervent hypochondriacs. The very assumption of similar health seeking behavior that makes TNDs theoretically useful is nullified by the very fact that individuals who are vaccinated and unvaccinated have very different health-seeking behaviors. That difference does not end simply at vaccination, and thus, we cannot use TNDs to reduce that bias. Vaccinated test-takers and unvaccinated test-takers are two different beasts altogether.

Let’s look at the acute forms of bias that arise from being the test-taking differences in the two groups. First of all, what are the exogenous factors here? Government incentives and disincentives would be one example. Many unvaccinated, in many countries, are forced to test for work. These tests will generally be counted as a positive case. Many vaccinated, on the other hand, are told by government not to test even if symptomatic, and just to self-isolate. We do not know how this affects testing behavior among the two groups, and which way the estimates are biased. Frankly, negative externalities created by government messaging cannot be quantified and there is no use in trying.

Secondly, who, in a vacuum, is more likely to be tested? The vaccinated is the obvious answer. In fact, one of the arguments for TNDs is that the unvaccinated would be less likely to test if positive and may be underrepresented in a more naive estimate. But that effect is likely small compared to the massive difference in asymptomatic testing that the vaccinated are doing. Whether that is to do things like flying, something the unvaccinated cannot even do test or not in my country, or to see their parents in long-term-care homes. Or, more often, just “to be safe” as that homeless guy they passed on the street looked like a little tired. I personally know many vaccinated people that get tested if they have a headache — even if it’s from drinking the night before.

The differences in testing behavior is enormous and this is seen by large lineups of vaccinated individuals who, other than mental distress, are otherwise health individuals. Unvaccinated individuals, on the other hand, may be less likely to test in general, but more likely to test with symptoms proportional to the vaccinated testing with symptoms. Which obliterates the entire point of the test-negative design.

A perfect example of this distinctly different testing behavior is the woman who quarantined herself in a plane washroom after testing herself positive mid-air because she had a sore throat. This was after taking two PCR tests and five rapid tests directly before the flight. That kind of testing behavior is not normal (and neither physically nor mentally beneficial as these tests do contain, often carcinogenic, chemicals).

The health-seeking behavior of vaccinated and unvaccinated behaviors are not similar at all, which begs the question, what do TNDs give us? A completely unknown estimate of VE that is biased in different ways than simply calculating VE from the proportion of the vaccinated and unvaccinated populations. We will be seeing this design used a lot more because they have, throughout the pandemic, tended to overestimate VE, and overestimate it by a lot, with regards to COVID. TNDs will be the new justification for why vaccines work despite them not working, be ready.