Just how reliable are those at-home Corona tests?
TL;DR The publicly available information makes it difficult to know with any certainty how good at-home Corona tests are, but I’m personally skeptical of their accuracy for individuals with low viral loads.
Testing, testing, …
It’s early September 2021 and the latest tidal wave of Corona cases continues to bring a seemingly inexorable rise in the number of infections.
As in each of the previous waves, I perceive a multi-week lag between the reported number of cases and the apparent recognition by the broader population of the risk of infection. It reminds me of when I’m forced to merge onto the Autobahn with an Opel. I floor the gas pedal and… one… two… three… acceleration (or something like it).
Apart from the general disappointment/frustration/sadness/despair I feel when considering the number of unnecessary lives this communication-shortcoming-induced lag costs, it also impacts me personally through my partner. Due to her work situation, as long as the broader population perceives a comparatively low risk, she feels immense pressure to maintain a busy travel/meeting schedule.
To give us (okay, me) a bit less uncertainty, I took advantage of the current easy availability of cheap at-home Corona tests, and bought a bunch of them online. But do they deliver reliable results?
Enter the Confusion Matrix
The particular test I bought (pictured above) includes instructions accompanied by some impressive-seeming statistics on its reliability (as compared with a PCR test):
This is a classic “confusion matrix”[1][2]. It seems to show that my at-home test (“neuartigen Coronavirus (2019-CoV)-Antigentest”) agrees with the PCR test (“PCR-Testergebnisse”) the overwhelming majority of the time (>95%). So I think: awesome, done. Not quite.
I mentioned this to a friend and former colleague of mine who was highly skeptical as this contradicted the experience of a number of folks in his network who had repeatedly tested negative on at-home tests only to test positive on a PCR test.
Cycle what?
After overcoming my initial smug thoughts of “well my at-home test must just be better than theirs”, I started digging. It wasn’t long before I came across this EU document, which outlines the standards for evaluating antigen tests[3] and includes the following statement:
In independent evaluations of unselected participants, assays should have a sensitivity of 90% or greater for subjects with a Ct < 25, in symptomatic people (positive samples from early infection within the first 7 days after symptom onset).
The phrase that caught my eye is “with a Ct<25”. What does that even mean? It turns out that “Ct” is an abbreviation for “cycle threshold” and it is a rough measure of how much viral material is in a sample. This document does a great job of explaining the nuances of cycle threshold values within the context of Sars-Cov-2 testing and sums it up as follows:
A low Ct indicates a high concentration of viral genetic material, which is typically associated with high risk of infectivity.
A high Ct indicates a low concentration of viral genetic material which is typically associated with a lower risk of infectivity. [emphasis mine]
A helpful mnenonic here this is the imaginary concept of “How much do I need to ‘zoom in’ to see virus in the sample?”
Low Ct = Not much zoom needed = Lots of virus present
High Ct = Lots of zoom needed = Not a lot of virus present
The unconditional distribution of cycle threshold values in samples with positive PCR test results seems to look something like this:
Or this:
of the virus circulating in communities”
My rough-eye-squint-guesstimate is that the median appears to be somewhere in the very rough vicinity of a cycle threshold of about 28 to 30.
Conditional Confusion
At this point I start to smell a rat. The issue is that rather than simply looking at the confusion matrix shown above, which is “unconditional”, one would ideally want to see additional confusion matrices for sub-groups of the population with various viral loads (i.e., conditional confusion matrices).
The Paul Ehrlich Institut’s protocol for its ongoing comparative testing of various antigen tests implicitly recognizes this:
18 samples each were analysed with CT<25,
23 samples with CT between 25 and 30, and
9 samples with CT>30.
Recall the quote that kicked off our journey into cycle thresholds?
In independent evaluations of unselected participants, assays should have a sensitivity of 90% or greater for subjects with a Ct < 25, in symptomatic people (positive samples from early infection within the first 7 days after symptom onset). [emphasis mine]
The evaluation criteria for antigen tests appears primarily focused on the performance of the tests conditional on high viral loads. This may be appropriate from a public policy perspective, as these are likely those who are most contagious, but it doesn’t help me as much as I would like in identifying whether my partner has a non-obvious/asymptomatic (likely low viral load) infection.
Test Result: Negative
As of this writing, my understanding is that test providers are not required to provide any information on how well their tests perform conditional on high/low cycle thresholds. Given the first rule of economics (“People respond to incentives”) I personally infer that because test manufacturers face strong incentives to report the most favorable statistics possible, they have a correspondingly strong incentive to skew their samples towards those with higher viral loads (i.e., lower cycle thresholds), and I am therefore skeptical of how good at-home Corona tests are for samples with lower viral loads (i.e., higher cycle thresholds). At the same time, it’s still prohibitively expensive (in both time and money) to get a PCR test a few times a week. Lesson for me: the at-home tests are definitely better than nothing and still probably the only practical choice, but I think I’ll trust them about as much as I do weather forecasts (and bring both a mask and an umbrella).
Note: I have also determined that, no surprise, I am not the first to realize this issue. I came across this article (sorry, German only), which has some additional details and a slightly different perspective.
Notes
[1] This name always makes me smile as I am always very confused by the forest of (often redundant) names associated with various simple calculations based on these counts (e.g., sensitivity, recall, specificity, selectivity, precision, accuracy).
[2] Interesting anecdote: my sleuthing revealed some slight disagreements/revisions as some online documentations versions show a slightly alternate confusion matrix with one of the cases reclassified:
[3]Technically “at-home” tests are specifically excluded, but in practice it seems like the same kits are being marketed to both professionals and at-home users (e.g., my kit is in the table in the document).
Disclaimer
All views are my own and not of my employer. I am not a doctor. I am not licensed to practice medicine. You should not make any medical decisions based on my writings. Or, really, any meaningful decisions at all.