Improve screening practices: Understanding false positive and false negative rates of COVID-19 tests

With news articles about COVID-19 testing in Iceland, you can find statements like “half those who tested positive have no symptoms of Covid-19” [1] or " ~50% of People with COVID-19 Not Showing Symptoms, ~50% Have Very Moderate Cold Symptoms" [2] in the media. The latter was flagged with an update that high false-positive rates [3] in COVID-19 testing make this statement not reliable.

However, as a recent article showed high concentrations of viral RNA in a patient without clinical symptoms when tested with “nasal (cycle threshold [ct] values, 22 to 28) and throat swabs (Ct values, 30 to 32) tested positive on days 7, 10, and 11 after contact” [4].

This proposed milestones of this mini project are to

  • gather data regarding testing sample rates
  • visualize them for health care professionals

The proposed aim would be understanding the relationship of COVID-19 tests and screening or isolation practices.

Suggestions and contributions are very welcome.

[1] https://www.bloomberg.com/news/articles/2020-03-22/one-third-of-coronavirus-cases-may-show-no-symptom-scmp-reports, 24.03.2020

[2] https://cleantechnica.com/2020/03/21/iceland-is-doing-science-50-of-people-with-covid-19-not-showing-symptoms-50-have-very-moderate-cold-symptoms/ 24.03.2020

[3] Zhuang, G.H., Shen, M.W., Zeng, L.X., Mi, B.B., Chen, F.Y., Liu, W.J., Pei, L.L., Qi, X. and Li, C., 2020. Potential false-positive rate among the’asymptomatic infected individuals’ in close contacts of COVID-19 patients. Zhonghua liu xing bing xue za zhi= Zhonghua liuxingbingxue zazhi , 41 (4), p.485. doi:10.3760/cma.j.cn112338-20200221-00144

[4] Zou, L., Ruan, F., Huang, M., Liang, L., Huang, H., Hong, Z., Yu, J., Kang, M., Song, Y., Xia, J. and Guo, Q., 2020. SARS-CoV-2 viral load in upper respiratory specimens of infected patients. New England Journal of Medicine .

4 Likes

Hi Tom,

This is a very crucial effort, wishing you good luck! I really can’t offer any help but would like to pick your brain as a complete outsider.

What’s your take on the “there can’t be any false positives from PCR tests as SARS-COV-2 has a very distinctive genome”? That seems to be the consensus of epi twitter.

Also, my understanding is that paper [3] was based on infection rates of asymptomatic patients (or the lack there of), and hence their definition of false positive does not really state that these people did not have COVID-19. I also couldn’t find the manuscript, let alone an English translation.

Regarding false negatives, a relevant reference is the Tao et al paper that compares PCR test results with CT scans and finds that PCR might have a 30-40% false negative rate.

Hi EmbreB,

thanks for opening up the discussion!

The statement “there can’t be any false positives from PCR tests as SARS-COV-2 has a very distinctive genome” shows that those people are completely inexperienced working with data coming from real lab settings.

Let us dissect it a bit: the amplicons used are very specific, that is true, however, PCR is an ultra-sensitive method. With enough amplification cycles, you can amplify from the tiniest amounts - therefore working with a lot of positive samples - the workbench has to be extremely clean (!) and otherwise single molecules can interfere. There are plenty of reasons, contaminated reagents, problems with samples that were prepared in close proximity in well plates.

source:wikimedia

Evaporations/condensations in other wells (which we see in drug screens a lot) - there are guidelines saying that the no template control should be as far away from the positive samples! Aerosol contaminations in a hood they are working in are another possibility. Never underestimate problems with human processing of the samples, if the pipetting was not clean, reagents were mislabeled or even people taking the samples were in contact with COVID-19 cases and then took samples from healthy individuals which could be enough to get some functional or non-functional virus molecules in there. Also if there are other problems with PCR primers, oligorimizations or other molecular dynamics this can also screw things up.
If the strategy is to multiplex lot of samples into a next-generation sequencing run, problems with indices or sequencing errors (~1/1000 in most common machines) can lead to cross-contamination.

In other words, there are many many reasons why there would be false positives.

Regarding false negatives, there are also many reasons, like problems with reagents, PCR conditions, machines etc. Or e.g.
(credits to Aindrilla Chatterjee, PostDoc EMBL) “if the swabs used are not collecting enough cells, degradation when the time between sample collection and RNA extraction is too long. Viral load could be another problem, basically how many copies of viral RNA are there in the sample? Based on the Korean screening strategy, they found that 1 copy of viral RNA will give a rough Ct of ~ 35 which is really pushing the limit of detection, especially when you consider that they are often considered the border of qPCR itself. So, you are easily going to miss a person if screened too early or if not swabbed properly (its stochastic). Primers of course. CDC has published a list of primers and I guess many are using that. In Korean screen they actually found some of them behaving erratically, so there is definitely some scope of improvement there. They actually made their own ones too and used them.
but common there are point mutations recorded in the course of this outbreak itself ! Considering that we have NGS data of only a small sub fraction of the infected the nextrain data collection ( https://nextstrain.org/ncov?branchLabel=aa&l=clock&m=div) shows already that there can and HAVE been nucleotide mutations in the viral genome. So, coming back to original qPCR Primer problem…if they are not using the right set or multiple sets (which can of course be a blind screening) they can easily have false negatives or even positives, if they accidentally end up looking at other SARSstrains”.

Overall with tests, we always assume that there will be false positives and false negatives. With clinical tests, you usually want to keep the false negatives low, in our case, we do not want to release a COVID-19 positive patient who was wrongly tested negative into the wild.

On the other hand, if somebody was wrongly tested positive, those people are looked together with infected people and therefore might catch the disease. This is unethical and dangerous for the individuals.

Therefore I would like to estimate for certain tests the false positive and false negative rate to develop guidelines: for your example, if a test has a 35% false-negative rate, people at risk have to be tested more often (e.g. 1 - 0.35 * 0.35) to decrease the likelihood of missing infected people.

If the false-positive rate is very high, then the positive tested people should not be mixed with other COVID-19 patients until a second or third test would confirm the infection.

One might say that multiple tests are probably not feasible, I would say you could probably save a lot of tests if this is understood better.

In terms of deCODE iceland, I would be interested if the positive cases without symptoms have been tested again. This would be invaluable information, but we need to talk to them!

Thanks for the paper and the discussion!

All the best.

1 Like

Thank you very much for the thorough reply!

Since there are so many factors in play that could determine FP and FN, how does one actually estimate these? What are some seminal papers in the area of FP/FN estimation for PCR?

My pleasure! Actually, one does not need to ‘understand’ everything about the test. It would be enough to get hands on data (1) how often was a patient re-tested (and in what time) and (2) did the patient show clinical symptoms or is there any proof of viral activity despite no clinical symptoms (like in the patient in Zou, L., Ruan, F., Huang, M., Liang, L., Huang, H., Hong, Z., Yu, J., Kang, M., Song, Y., Xia, J. and Guo, Q., 2020. SARS-CoV-2 viral load in upper respiratory specimens of infected patients. New England Journal of Medicine ).

We would assume that the probability that the test is wrong does not change, so each test has the same probability to fail - then you could estimate easily how often the test result could be reproduced.

Of course, there are some simplifications - but things we could also estimate would be if the second test is more likely to “stay” positive (because they are in quarantine with infected people).

I was trying to write somebody at deCode if they have data on the 50% without symptoms, but no response - is there anybody out there with some data on that or who has contact with institutes?

1 Like