Validation of a molecular assay to detect SARS-CoV-2 in saliva | OPEN ACCESS
View Article PDF
Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is the cause of the current coronavirus disease 2019 (COVID-19) pandemic. Nucleic acid amplification testing using reverse transcription quantitative polymerase chain reaction (RT-qPCR) for SARS-CoV-2 genes is the diagnostic test of choice because it detects extremely low levels of virus within biological fluids. However, the sensitivity and accuracy of such tests are dependent upon the type of specimen, its method of collection and the RT-qPCR test itself.
In the haste to develop a detection test during the onset of the COVID-19 pandemic, specimens from the nasopharyngeal (NP) site were used. Although this specimen has become regarded as the gold standard test to which all others are compared, it has its own limitations. The NP site can be difficult to reach, impacting on NP specimen quality,[[1]] which in turn accounts for significant differences in NP assay sensitivity and increases the risk of false negatives.[[2]] Moreover, the invasiveness of sample collection often causes discomfort and is unpopular with those requiring frequent testing.
The use of saliva as a specimen for detecting SARS-CoV-2 is increasing around the world. Saliva samples perform well compared to NP in detecting respiratory viruses including coronaviruses,[[3–6]] with both SARS-CoV[[7]] and SARS-CoV-2[[1,5,8,9]] being present in high titers in saliva. The advantage of saliva is that it bypasses the need for invasive sample collection. Although it is an attractive option for frequent testing, there has been a cautious approach to saliva testing in Aotearoa New Zealand. The New Zealand Microbiology Network (NZMN) has recommended that any saliva test would need to be validated locally using well characterised samples that were positive for SARS-CoV-2 and that testing be performed in a diagnostic laboratory accredited by International Accreditation New Zealand (IANZ) and aligned with the ISO 15189 quality framework.[[10]] NZMN noted that the accuracy of saliva tests is reliant upon the methods used for saliva collection, the extraction steps employed for the viral RNA and the commercial kit utilised.[[10]]
In September 2020, we established an international collaboration with investigators at the University of Illinois Urbana-Champaign (UIUC), USA, who had developed a simple and rapid direct saliva-to RT-qPCR assay for the detection of SARS-CoV-2.[[11]] The assay omits the common RNA extraction step and utilises a modified method of a commercially available RT-qPCR kit, thus avoiding reagent competition and supply-chain issues. It includes an initial heat step that inactivates the saliva sample, reducing viral transmission risk, as well as enabling accessibility of the viral genome to the PCR reagents. Finally, the subsequent addition of a detergent-buffer mix overcomes the drawback of saliva viscosity. This test has been approved for surveillance testing in a US FDA Emergency Use Authorization (EUA).[[12]]
The aim of the study was to validate the UIUC RT-qPCR protocol for detecting SARS-CoV-2 in saliva in two independent New Zealand laboratories by comparing our results to those measured at the UIUC laboratory (USA). This study used real-world saliva samples that were paired with nasal samples, of mainly NP origin, from participants that were positive and negative for SARS-CoV-2.
Methods
Test details
The validation was performed at Victoria University of Wellington (VUW) in Wellington and IGENZ Ltd in Auckland using both commercially available heat-inactivated SARS-CoV-2 (ATCC® VR-1986HK™) and saliva samples from consenting COVID-19 positive and negative participants located in Chicago and Wisconsin, USA. These samples, obtained by UIUC, were shipped to Aotearoa New Zealand for the studies described in this paper.
The RT-qPCR test used the ThermoFisher Scientific TaqPath[[TM]] COVID-19 Combo kit (Cat. No. A47814; ThermoFisher) and omits the RNA extraction step. The kit includes primers and probes specific to the SARS-CoV-2 nucleocapsid (N), spike (S) and replication (ORF1ab) genes and a spike-in bacteriophage (MS2) gene, with modifications to the manufacturer’s instructions, as previously published.[[11,13]] Controls included a positive control (TaqPath™ COVID19 Control; ThermoFisher Scientific TaqPath[[TM]] COVID-19 Combo kit) at 25 copies per µL, negative control (UltraPure dH2O) and no-template control (heat-inactivated saliva; collected in-house).
Saliva collection
All but five participants had saliva and nasopharyngeal specimens collected contemporaneously. The five non-contemporaneously collected saliva samples were taken following a positive low viral load result obtained from a mid-turbinate swab. These later samples were included to deliberately include paired samples with low viral load as required for FDA EUA validation purposes, which were also included in this current study to ensure that it contained samples of low viral load.
Participants provided a saliva sample by following guidelines that instructed them to allow saliva to collect in the mouth before gently expelling saliva into the collection tube (passive drool method). They then capped their tube and handed it to the healthcare professional or collection-site staff, who placed it into a collection container. A healthcare professional collected the nasopharyngeal (N=142) or mid-turbinate (N=5) sample and transferred the swab into the collection media (per the comparator manufacturer’s instructions). All nasal swabs (defined as both nasopharyngeal and mid-turbinate) were processed in the clinical pathology laboratory at the University of Illinois Chicago Hospital using an FDA EUA reference method for detection of SARS-CoV-2, the Abbott RealTime SARS-CoV-2 assay performed on the Abbott m2000 System with a limit of detection of 2,700 NDU per mL.[[13]]
Saliva samples that had been heat-inactivated at 95°C for 30 min in UIUC were divided into aliquots. Sample aliquots were stored for up to four months at -80oC before transportation from UIUC on dry ice to the two New Zealand laboratories. Even though the status of each aliquot was unknown upon arrival, samples were immediately placed in a random sequence and re-coded. Before testing, the heat-inactivated saliva aliquot was diluted 1:1 (v/v) in a Tris-Borate-EDTA/Tween 20 mix.
A total of 147 paired saliva samples were received. Of these, 33 saliva samples were from people that tested positive for SARS-CoV-2 based on a nasal sample test result (N=28 NP specimens and N=5 mid-turbinate specimens). The remaining 114 saliva samples were from people that tested negative with SARS-CoV-2 based on a nasal sample test result (N=73 NP specimens and N=41 mid-turbinate specimens). One of these NP samples that tested negative for SARS-CoV-2 did test positive for SARS-CoV-2 in its accompanying saliva sample. The 34 paired samples that tested as positive either in both the nasal (NP or mid-turbinate) and saliva specimen (N=32), or in the nasal (N=1) or saliva (N=1) specimen only, are listed in detail in the results. Of the participants that tested positive (in either test), 26 were symptomatic and eight were asymptomatic.
Analytical validation
The limit of detection (LOD) was determined by performing three separate experiments in triplicate, whereby the ATCC® SARS-CoV-2 control was serially diluted in heat-inactivated saliva from 192 to 0.75 viral copies per µL. The LOD was defined as the lowest concentration (and highest quantification cycle (Cq) value) at which at least two out of the three genes (ORF1ab, N and S) in all replicates were detected. The repeatability of the LOD was tested with 60 replicates at the LOD concentration to determine whether all 60 replicates would test positive. As a final measure of LOD, 20 heat-inactivated saliva samples spiked with 1x, 2x, 5x and 10x LOD concentrations of ATCC® SARS-CoV-2 control, and 20 heat-inactivated saliva samples negative for SARS-CoV-2 were tested blindly.
The stability of ATCC® SARS-CoV-2 control in heat-inactivated saliva was determined by preparing 12 replicates of saliva spiked with 1x, 2x, 4x, 8x and 16x LOD concentrations. Each of the replicates were divided into four groups and used either fresh or after one, two or three freeze-thaw cycles.
The cross-reactivity of Middle East respiratory syndrome coronavirus (MERS-CoV; ATCC® VR-3248SD™) was tested by preparing five replicates at 100 and 1,000 viral copies per µL of heat-inactivated saliva. These results were compared to heat-inactivated saliva spiked with ATCC® SARS-CoV-2 control at 10 viral copies per µL.
Diagnostic validation
A positive result was defined as any sample with a Cq value of <39 for two or more of the SARS-CoV-2 gene targets and the spike-in control gene (MS2). An indeterminate test was defined as any sample with a Cq value of <39 for only one of the SARS-CoV-2 gene targets and the spike-in control gene (MS2). Samples that gave an indeterminate result were re-tested when sufficient sample allowed. A negative result was defined as a sample with a Cq value >39 for all three of the SARS-CoV-2 gene targets and a spike-in control gene (MS2) Cq value of <39. An invalid result was defined as any situation where either the spike-in control gene (MS2) had a Cq value of >39, the positive control had a Cq value of >39 or the negative control had a Cq value of <39.
The reproducibility of the assay was determined by processing, in three different laboratories (VUW, IGENZ and UIUC), 147 saliva samples in blinded experiments from individuals that were positive (N=33) and negative (N=119) for SARS-CoV-2. The Cq values for each gene in every sample were compared. The VUW results were used to compare with UIUC results for diagnostic validation.
The diagnostic sensitivity of the test was determined by comparing saliva assay results with those of contemporaneously collected nasal swabs. Thirty-three positive SARS-CoV-2 samples, each paired with either an NP (N=28, 85%) or mid-turbinate (N=5) sample, and 114 negative samples, each paired with an NP sample, were tested. Concordance of sample test results between all three laboratories was calculated. Statistical analysis for a qualitative test with calculated diagnostic sensitivity, specificity and accuracy with confidence intervals was performed. Test accuracy was calculated using a disease prevalence value (estimated for managed isolation and quarantine (MIQ) facilities) of 0.74% (1,201 confirmed cases out of 162,733 individuals through MIQ facilities).[[14]] Finally, hypothesis tests were carried out to determine whether there was a statistically significant difference between Cq values obtained in different laboratories.
Ethics
Clinical trials comparing results from contemporaneously collected NP or mid-turbinate nasal swabs analysed by an FDA EUA reference method for detection of SARS-CoV-2, and saliva samples analysed by the UIUC assay (covidSHIELD), were approved by the Western Institutional Review Board, USA (WIRB 20202884). All participants gave written and informed consent both to the collection of all saliva samples and their use at external organisations for the purpose of test validation.
In-use experience of the assay
The assay was accredited for surveillance use in December 2020. The ease of sample handling and rate of inhibited samples was assessed by failure of detection of the spike-in control, MS2.
Results
Analytical validation
The LOD of SARS-CoV-2-spiked heat-inactivated saliva samples in three replicate experiments was <0.75 viral copies per μL (or 1.875 viral copies per reaction) (Figure 1). A positive result was given to all replicates at 0.75 viral copies per μL using the diagnostic criteria of two or more genes being detectable.
The LOD of <0.75 viral copies per μL was confirmed by all 60 replicates being detected as positive of SARS-CoV-2 (data not shown) and by a blinded experiment whereby all SARS-CoV-2-spiked samples were detected as positive at concentrations ranging from 1 to 10x LOD, and all non-spiked samples were detected as negative (Figure 2). Freeze-thaw cycles did not change assay results, with only one replicate of the ORF1ab gene at 1x LOD concentration at Cq of >39 (data not shown).
The assay showed no cross-reactivity with a similar virus, MERS-CoV. All heat-inactivated saliva samples spiked with MERS-CoV had Cq values of >39, even at high concentrations of 1,000 viral copies per μL (Figure 3).
Figure 1: Three replicate experiments of SARS-CoV-2 spiked heat-inactivated saliva serially diluted from 192 to 0.75 viral copies per μL. The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike-in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included. Dashed line indicates the limit of detection (LOD) for each replicate experiment.
Figure 2: A blinded experiment of heat-inactivated saliva samples that were either spiked with SARS-CoV-2 at 1 to 10x LOD concentrations (n=5 replicates per concentration; blinded positives) or not spiked (n=20 replicates; blinded negatives). The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike-in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included.
Figure 3: Cross-reactivity experiment where MERS-CoV spiked heat-inactivated saliva was not detected at high concentrations (n=5 replicates at 100 and 1,000 viral copies per μL), while SARS-CoV-2 spiked heat-inactivated saliva was detected in every sample (n=50 replicates at 10 viral copies per μL). The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included.
Diagnostic validation
Positive and negative SARS-CoV-2 saliva samples assayed are displayed in Table 1. The Cq values for the 113 concordant negative samples were >39 (data not shown). Thirty-two of 33 saliva samples that were paired with a positive nasal sample (NP and mid-turbinate) were detected as positive in the saliva test, resulting in a sensitivity of 97.0% (95% CI 84.2–99.9%). One saliva sample paired with a negative NP sample tested as positive, resulting in a specificity of 99.1% (95% CI 95.2–100%). Thus, the overall test concordance between saliva and nasal specimens was 98.6%, and the saliva test accuracy was 99.1% (95% CI 95.9–100) (Table 2).
Of those paired with NP specimens only, 27 out of the 28 samples that were positive in the NP test were detected as positive in the saliva test, resulting in a sensitivity of 96.4% (95% CI 81.7–99.9). One saliva sample that was paired with a negative NP sample tested as positive, resulting in a specificity of 98.6% (95% CI 92.6–100%). Thus, the overall test concordance between saliva and nasal specimens was 98.0%, and the saliva test accuracy was 98.6% (95% CI 94.1–99.9) (data not shown).
Table 1: Nasal Swab PCR as the reference test versus the saliva test.
Table 2: Diagnostic statistics for nasal swab PCR as the reference test versus the saliva test. The accuracy was calculated using a population prevalence of 0.74%, as estimated in MIQ.[[14]]
The Cq values for all 33 positive SARS-CoV-2 samples are displayed in Table 3. The one saliva sample (#34) taken from a symptomatic participant that tested positive and was accompanied by a contemporaneously collected negative NP specimen is also included in Table 3. The variability of the 2^(-△Cq) values between the two New Zealand laboratories was 0.24, 0.41 and 0.29 for the N, ORF1ab and S genes, respectively. A t-Test performed on the Cq values between the laboratories demonstrated no statistically significant differences. The low sample variability of 2^(-△Cq) values (particularly those stored in different aliquots; data not shown) between the laboratories confirmed high reproducibility.
Two samples with very low viral load, #6 and #33, were received for testing in the New Zealand laboratories. The samples were tested immediately in the VUW laboratory but there was a delay of one week in sample processing at the IGENZ laboratory, which resulted in Cq values of >39. Given the positive results obtained from these specimens at VUW and the similarity in Cq values between the two New Zealand laboratories in all samples that were processed promptly, samples #6 and #33 were assumed to be degraded and IGENZ data were omitted from Table 3. Sample #33 had a very low viral load and was called as indeterminate by all three laboratories. Repeat testing (#33 re-test) by UIUC on the original sample showed all three genes as being detected and was subsequently recorded as a positive result. There was insufficient sample for repeat analyses at VUW. However, when the degraded sample was repeated by IGENZ, no genes were detected. Again, given the close concordance of all other results, the absence of detectable genes was attributed to deterioration of the sample over time of storage and during transportation (>4 months).
An additional 20 non-paired saliva samples (N=10 positive and N=10 negative) were also tested and exhibited 100% concordance between the three laboratories (Supplementary Figure 1).
Table 3: Concordance of Cq values of 34 positive SARS-CoV-2 saliva samples analysed independently in two New Zealand laboratories (Victoria University of Wellington (VUW) and IGENZ) and one US laboratory (UIUC) in blind experiments. Saliva samples were processed immediately after collection at UIUC but 4–6 months later in the New Zealand laboratories. A dash indicates data omission due to assumed sample degradation. A blank entry denotes that the sample was not re-tested in that laboratory due to lack of sample volume. All samples had paired nasal swabs (nasopharyngeal NP, or mid-turbinate MT) as listed, and all but one nasal sample were called positive. (Pos = Positive, Neg = Negative, Indet = Indeterminate). Asterisk indicates a discordant result between the nasal and saliva sample. View Table 3.
In-use experience
To 1 September 2021, we have tested 13,304 saliva samples. Saliva viscosity that required manual pipetting only affected two specimens, both from the same participant. No sample exhibited assay inhibition.
Discussion
This study is the first to diagnostically validate a saliva test for SARS-CoV-2 in Aotearoa New Zealand. It used real-world paired saliva and NP samples from COVID-19 infected individuals, some of whom were exhibiting symptoms of COVID-19 infection (others were asymptomatic). Previous reports of sensitivities of SARS-CoV-2 detection in saliva are shown to be variable while specificity has been more consistent.[[15]] This emphasises the need for strict collection protocols and well-validated tests. The sensitivity (97.0%), specificity (99.1%) and accuracy (99.1%) of this UIUC saliva RT-qPCR test are very high, with the 98.6% concordance with NP and mid-turbinate specimens in all three laboratories. False-negative or false-positive results were not a test performance issue. The one positive saliva sample that was associated with a negative NP specimen suggests a difference in the timings of viral infection at the two anatomical sites during early or late disease. Overall, these results revealed the UIUC saliva RT-qPCR test to be a highly reproducible method for SARS-CoV-2 detection in both New Zealand laboratories. Moreover, the analytical validation using spiked saliva also exhibited a high sensitivity and specificity, as well as a lower LOD than that of another local NP qPCR assay reported recently (<2 versus ~10 copies per reaction, respectively).[[16]] The correlation of the Cq values for the three SARS-CoV-2 genes tested in each sample independently between the three laboratories was extremely high. This is particularly encouraging given the time between sample collection in the USA and processing in the New Zealand laboratories.
The recent Simpson–Roche report on the country’s COVID-19 testing strategy called for broadening the range of testing methods and recommended introducing saliva testing to increase the acceptability of testing across workforces in the community.[[17]] The availability of such an accurate saliva assay for selected testing situations, such as workplace testing, would significantly compliment the current NP testing used for New Zealand’s public health response. The low LOD, as well as the inclusion of samples from asymptomatic individuals, confirms that this assay can detect infection in asymptomatic and symptomatic people. Early detection enables individuals to be isolated quickly, reducing the risk of transmission.
It should be emphasised that, in addition to the specific RT-qPCR test being used, appropriate saliva collection and preparation procedures are essential.[[11]] This is important due to varying sample viscosity, which if not mitigated can make aliquoting difficult, particularly when using robotic equipment. Moreover, those undertaking saliva testing need instructions on adequate hydration and the need to abstain from food or drink, other than water, for an hour before sample collection. Additionally, there have been concerns about exogenous substances causing assay inhibition. However, the testing of several candidate substances, including nasal spray, different mouth lozenges, nicotine and mouthwash, revealed that only toothpaste (and in only one of three samples) was associated with saliva assay inhibition.[[12]] This is supported by our experience, as after providing detailed collection instructions and compliance, we have yet to encounter saliva samples where the PCR reaction has been inhibited.
This study does have limitations. At the time that this study was performed, there was no community transmission of SARS-CoV-2 in Aotearoa New Zealand. Therefore, it was impossible to obtain locally collected samples. Diagnostic validation using paired contemporaneously collected samples was essential to enable diagnostic validation of this test. This was required as part of IGENZ accreditation to ISO 15189 standards by International Accreditation New Zealand. Collaboration with UIUC enabled this validation study to take place. Although the number of positive pairs was limited, it was similar to many other studies.[[15]] We acknowledge that an increased number of positive pairs would enable greater understanding of variation between the two sample sites. The high analytical sensitivity and concordance, particularly with the NP samples, are specific to this assay and its sample preparation methods and cannot be taken as an indication of other assays’ performance. Saliva samples were obtained after comprehensive advice on hydration and avoiding food and drink. Assay performance may not be the same if collection advice is not followed.
Saliva is likely to participate in SARS-CoV-2 transmission due to the virus replicating in oral epithelial and salivary gland cells.[[18]] As saliva contains large numbers of oral epithelial cells, the detection of SARS-CoV-2 in this specimen is indicative of local virus production and does not rely on the virus passing through the oropharynx from the nasopharynx. Different replication rates at either site may result in a sample from one being positive when the other site is negative.[[1,19,20]] Moreover, there is emerging evidence that breakthrough infections in some people vaccinated with the Janssen Ad26.COV2.S COVID-19 and Pfizer/BioNTech vaccines elicit tissue compartmentalisation, whereby SARS-CoV-2 is detectable only in saliva and not in the nasal passages.[[21]] Our results support this disparity in viral load between tissue sites, but prospective studies are required to understand how frequently this occurs and how it impacts on diagnostic test performance.
Supply-chain issues, in particular the reagents and consumables required for RNA extraction, have hampered the testing of NP samples during the pandemic. The UIUC protocol bypasses the RNA extraction step and, in doing so, removes the supply-chain issues associated with this step. Furthermore, self-collection of saliva samples reduces the need for health professionals at collection sites and the heat-inactivation step reduces the risk of exposure to medical laboratory workers.
The country has now completed more than three million NP tests,[[22]] which is similar to the number of saliva tests conducted at UIUC.[[23]] SARS-CoV-2 testing used for our public health response needs to be scalable overnight, from the baseline testing of ~3,000–5,000 per day to more than 30,000 per day during possible community outbreaks.[[22]] This responsiveness has been achieved using the NP test. The NP swab remains the choice of the Ministry of Health for routine public health testing. However, a role for saliva testing in situations where high-frequency testing is required is now accepted.[[24]] This saliva test is also highly scalable and over 10,000 samples could be processed in one diagnostic laboratory in a single day.
Conclusions
The UIUC RT-qPCR has been tested locally and has been found to be an assay with high analytical and diagnostic sensitivity. It showed 99.1% accuracy and 98.1% concordance to that of nasal swabs in all three independent laboratories. In-use experience to date has not encountered either aliquoting problems or inhibited reactions. As a non-invasive test, it has significant appeal where high-frequency testing is required.
Supplementary Material
Supplementary Figure 1: Twenty saliva samples (10 positive and 10 negative) processed blind in triplicate by both New Zealand laboratories (data for Victoria University of Wellington shown) demonstrated 100% concordance with UIUC saliva results. Positive (PC), negative (NC) and no template (NTC) controls were included.
Summary
This paper presents the validation results of a qPCR test that was developed at University of Illinois Urbana-Champaign (UIUC) for non-invasively detecting the SARS-CoV-2 virus in saliva and tested in Aotearoa New Zealand laboratory. We used saliva samples that were collected from individuals that had also had nasal swabs taken at the same time. The nasal swabs for just over a third of these people were positive for SARS-CoV-2. Our results showed that the UIUC qPCR test is highly accurate (99.1%) for detecting SARS-CoV-2 in saliva and can detect very low copy numbers of SARS-CoV-2 in saliva. This UIUC qPCR for SARS-CoV-2 is as accurate as the qPCR tests used for detecting SARS-CoV-2 in nasopharyngeal samples in New Zealand. These results confirmed that this reliable option for SARS-CoV-2 testing, including for diagnostic testing for asymptomatic people requiring regular screening.
Abstract
Aim
To validate a reverse transcriptase-quantitative polymerase chain reaction (RT-qPCR) assay to detect SARS-CoV-2 in saliva in two independent Aotearoa New Zealand laboratories.
Method
An RT-qPCR assay developed at University of Illinois Urbana-Champaign, USA, was validated in two New Zealand laboratories. Analytical measures, such as limit of detection (LOD) and cross-reactivity, were performed. One hundred and forty-seven saliva samples, each paired with a contemporaneously collected nasal swab, mainly of nasopharyngeal origin, were received. Positive (N=33) and negative (N=114) samples were tested blindly in each laboratory. Diagnostic sensitivity and specificity were then calculated.
Results
The LOD was <0.75 copy per µL and no cross-reactivity with MERS-CoV was detected. There was complete concordance between laboratories for all saliva samples with the quantification cycle values for all three genes in close agreement. Saliva had 98.7% concordance with paired nasal samples: and a sensitivity, specificity and accuracy of 97.0%, 99.1% and 99.1%, respectively.
Conclusion
This saliva RT-qPCR assay produces reproducible results with a low LOD. High sensitivity and specificity make it a reliable option for SARS-CoV-2 testing, including for asymptomatic people requiring regular screening.
Author Information
Acknowledgements
Correspondence
Correspondence Email
Competing Interests
1) Wyllie AL, Fournier J, Casanovas-Massana A, et al. Saliva or Nasopharyngeal swab specimens for detection of SARS-CoV-2. N Engl J Med. 2020;383(13):1283-6.
2) Richard-Greenblatt MJ, Ziegler M, Bromberg V, et al. Impact of nasopharyngeal specimen quality on SARS-CoV-2 test sensitivity. medRxiv. 2020 Dec 11. doi: 10.1101/2020.12.09.20246520
3) To KKW, Yip CCY, Lai CYW, et al. Saliva as a diagnostic specimen for testing respiratory virus by a point-of-care molecular assay: a diagnostic validity study. Clin Microbiol Infect. 2019;25(3):372-8.
4) To KKW, Chan KH, Ho J, et al. Respiratory virus infection among hospitalized adult patients with or without clinically apparent respiratory infection: a prospective cohort study. Clin Microbiol Infect. 2019;25(12):1539-45.
5) To KK, Tsang OT, Yip CC, et al. Consistent detection of 2019 novel coronavirus in saliva. Clin Infect Dis. 2020;71(15):841-3.
6) Rao M, Rashid FA, Sabri FSAH, et al. Comparing nasopharyngeal swab and early morning saliva for the identification of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Clin Infect Dis. 2021;72(9):352-6.
7) Wang WK, Chen SY, Liu IJ, et al. SARS Research Group of the National Taiwan University/National Taiwan University Hospital. Detection of SARS-associated coronavirus in throat wash and saliva in early diagnosis. Emerg Infect Dis. 2004;10(7):1213-9.
8) Harikrishnan P. Saliva as a potential diagnostic specimen for COVID-19 testing. J Craniofac Surg. 2020;31(6):653-5.
9) Tee M, Ubial P, Ranoa DR, et al. Direct saliva versus conventional nasopharyngeal swab qRT-PCR to diagnose SARS-CoV-2: validity study. Asian J Res Infect Dis. 2021;6(2):37-46.
10) New Zealand Microbiology Network [Internet]. Statement on saliva testing for surveillance of SARS-CoV-2. Summary recommendations (2021) [cited 2021 Aug 13]. Available from: https://www.nzmn.org.nz/assets/NZMN/NZMN-statement-on-saliva-testing-for-surveillance-of-SARS-CoV-2-March-2021-v3.pdf
11) Ranoa DRE, Holland RL, Alnaji FG et al. Saliva-based molecular testing for SARS-CoV-2 that bypasses RNA extraction. bioRxiv. doi.org/10.1101/2020.06.18.159434
12) FDA [Internet]. Emergency Use Authorization for covidShield [cited 2021 Aug 13]. Available from: https://www.fda.gov/media/146314/download
13) Ranoa DRE, Holland RL, Alnaji FG, et al. Mitigation of SARS-CoV-2 Transmission at a Large Public University. medRxiv preprint: doi.org/10.1101/2021.08.03.21261548
14) Ministry of Business, Innovation & Employment [Internet]. Managed isolation and quarantine data [cited 2021 Aug 13]. Available from: https://www.mbie.govt.nz/business-and-employment/economic-development/covid-19-data-resources/managed-isolation-and-quarantine-data/
15) Butler-Laporte G, Lawandi A, Schiller I et al. Comparison of saliva and nasopharyngeal swab nucleic acid amplification testing for detection of SARS-CoV-2. A systematic review and meta-analysis. JAMA Int Med 2021;181(3):353-60.
16) Fox-Lewis S, Muttaiyah S, Rahnama F, et al. An understanding of discordant SARS-CoV-2 test results: an examination of the data from a central Auckland laboratory. NZ Med J. 2020;133(1519):81-8.
17) Simpson H, Roche B, Report of the Advisory Committee to oversee the implementation of the New Zealand COVID-19 Surveillance Plan and Testing Strategy [cited 2021 Aug 13]. NZ Govt Wellington 2020. Available from: https://covid19.govt.nz/assets/Review-of-Surveillance-Plan-and-Testing-Strategy/Final_Report-of-Advisory-Committee-to-Oversee-the-Implementation-of-the-....pdf
18) Huang N, Perez P, Kato T, SARS-CoV-2 infection of the oral cavity and saliva, Nat Med. 2021;27(5):892-903.
19) Miller M, Jansen M, Bisignano A, et al. Validation of a self-administrable, saliva-based RT-qPCR test detecting SARS-CoV-2. Preprint. Posted online June 9, 2020. medRxiv. doi:10.1101/2020.06.05.20122721
20) Rao M, Rashid FA, Sabri FSAH, et al. Comparing nasopharyngeal swab and early morning saliva for the identification of SARS-CoV-2. Clin Infect Dis. Published online August 6, 2020. doi:10.1093/cid/ciaa1156
21) Ke R, Martinez PP, Smith R, et al. Longitudinal analysis of SARS-CoV-2 vaccine breakthrough infections reveal limited infectious virus shedding and restricted tissue distribution. medRxiv preprint: doi.org/10.1101/2021.08.30.21262701
22) Ministry of Health [Internet]. “Testing for COVID-19,” Ministry of Health, 03 May 2021. Available: https://www.health.govt.nz/our-work/diseases-and-conditions/covid-19-novel-coronavirus/covid-19-data-and-statistics/testing-covid-19#testing
23) U. C. University of Illinois [Internet]. “Shield Testing Data,” University of Illinois, Urbana Champaign, 03 May 2021 [cited 2021 Aug 13].. Available: https://covid19.illinois.edu/on-campus-covid-19-testing-data-dashboard/
24) NZ Herald [Internet]. “Covid 19 coronavirus: Daily saliva testing for border, quarantine workers will be optional,” NZ Herald, 22 Jan 2021 [cited 2021 Aug 13]. Available: https://www.nzherald.co.nz/nz/covid-19-coronavirus-daily-saliva-testing-for-border-quarantine-workers-will-be-optional/FFODUHMKQLRC2CXFA3GV2PLO4Q/
For the PDF of this article,
contact nzmj@nzma.org.nz
Validation of a molecular assay to detect SARS-CoV-2 in saliva | OPEN ACCESS
View Article PDF
Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is the cause of the current coronavirus disease 2019 (COVID-19) pandemic. Nucleic acid amplification testing using reverse transcription quantitative polymerase chain reaction (RT-qPCR) for SARS-CoV-2 genes is the diagnostic test of choice because it detects extremely low levels of virus within biological fluids. However, the sensitivity and accuracy of such tests are dependent upon the type of specimen, its method of collection and the RT-qPCR test itself.
In the haste to develop a detection test during the onset of the COVID-19 pandemic, specimens from the nasopharyngeal (NP) site were used. Although this specimen has become regarded as the gold standard test to which all others are compared, it has its own limitations. The NP site can be difficult to reach, impacting on NP specimen quality,[[1]] which in turn accounts for significant differences in NP assay sensitivity and increases the risk of false negatives.[[2]] Moreover, the invasiveness of sample collection often causes discomfort and is unpopular with those requiring frequent testing.
The use of saliva as a specimen for detecting SARS-CoV-2 is increasing around the world. Saliva samples perform well compared to NP in detecting respiratory viruses including coronaviruses,[[3–6]] with both SARS-CoV[[7]] and SARS-CoV-2[[1,5,8,9]] being present in high titers in saliva. The advantage of saliva is that it bypasses the need for invasive sample collection. Although it is an attractive option for frequent testing, there has been a cautious approach to saliva testing in Aotearoa New Zealand. The New Zealand Microbiology Network (NZMN) has recommended that any saliva test would need to be validated locally using well characterised samples that were positive for SARS-CoV-2 and that testing be performed in a diagnostic laboratory accredited by International Accreditation New Zealand (IANZ) and aligned with the ISO 15189 quality framework.[[10]] NZMN noted that the accuracy of saliva tests is reliant upon the methods used for saliva collection, the extraction steps employed for the viral RNA and the commercial kit utilised.[[10]]
In September 2020, we established an international collaboration with investigators at the University of Illinois Urbana-Champaign (UIUC), USA, who had developed a simple and rapid direct saliva-to RT-qPCR assay for the detection of SARS-CoV-2.[[11]] The assay omits the common RNA extraction step and utilises a modified method of a commercially available RT-qPCR kit, thus avoiding reagent competition and supply-chain issues. It includes an initial heat step that inactivates the saliva sample, reducing viral transmission risk, as well as enabling accessibility of the viral genome to the PCR reagents. Finally, the subsequent addition of a detergent-buffer mix overcomes the drawback of saliva viscosity. This test has been approved for surveillance testing in a US FDA Emergency Use Authorization (EUA).[[12]]
The aim of the study was to validate the UIUC RT-qPCR protocol for detecting SARS-CoV-2 in saliva in two independent New Zealand laboratories by comparing our results to those measured at the UIUC laboratory (USA). This study used real-world saliva samples that were paired with nasal samples, of mainly NP origin, from participants that were positive and negative for SARS-CoV-2.
Methods
Test details
The validation was performed at Victoria University of Wellington (VUW) in Wellington and IGENZ Ltd in Auckland using both commercially available heat-inactivated SARS-CoV-2 (ATCC® VR-1986HK™) and saliva samples from consenting COVID-19 positive and negative participants located in Chicago and Wisconsin, USA. These samples, obtained by UIUC, were shipped to Aotearoa New Zealand for the studies described in this paper.
The RT-qPCR test used the ThermoFisher Scientific TaqPath[[TM]] COVID-19 Combo kit (Cat. No. A47814; ThermoFisher) and omits the RNA extraction step. The kit includes primers and probes specific to the SARS-CoV-2 nucleocapsid (N), spike (S) and replication (ORF1ab) genes and a spike-in bacteriophage (MS2) gene, with modifications to the manufacturer’s instructions, as previously published.[[11,13]] Controls included a positive control (TaqPath™ COVID19 Control; ThermoFisher Scientific TaqPath[[TM]] COVID-19 Combo kit) at 25 copies per µL, negative control (UltraPure dH2O) and no-template control (heat-inactivated saliva; collected in-house).
Saliva collection
All but five participants had saliva and nasopharyngeal specimens collected contemporaneously. The five non-contemporaneously collected saliva samples were taken following a positive low viral load result obtained from a mid-turbinate swab. These later samples were included to deliberately include paired samples with low viral load as required for FDA EUA validation purposes, which were also included in this current study to ensure that it contained samples of low viral load.
Participants provided a saliva sample by following guidelines that instructed them to allow saliva to collect in the mouth before gently expelling saliva into the collection tube (passive drool method). They then capped their tube and handed it to the healthcare professional or collection-site staff, who placed it into a collection container. A healthcare professional collected the nasopharyngeal (N=142) or mid-turbinate (N=5) sample and transferred the swab into the collection media (per the comparator manufacturer’s instructions). All nasal swabs (defined as both nasopharyngeal and mid-turbinate) were processed in the clinical pathology laboratory at the University of Illinois Chicago Hospital using an FDA EUA reference method for detection of SARS-CoV-2, the Abbott RealTime SARS-CoV-2 assay performed on the Abbott m2000 System with a limit of detection of 2,700 NDU per mL.[[13]]
Saliva samples that had been heat-inactivated at 95°C for 30 min in UIUC were divided into aliquots. Sample aliquots were stored for up to four months at -80oC before transportation from UIUC on dry ice to the two New Zealand laboratories. Even though the status of each aliquot was unknown upon arrival, samples were immediately placed in a random sequence and re-coded. Before testing, the heat-inactivated saliva aliquot was diluted 1:1 (v/v) in a Tris-Borate-EDTA/Tween 20 mix.
A total of 147 paired saliva samples were received. Of these, 33 saliva samples were from people that tested positive for SARS-CoV-2 based on a nasal sample test result (N=28 NP specimens and N=5 mid-turbinate specimens). The remaining 114 saliva samples were from people that tested negative with SARS-CoV-2 based on a nasal sample test result (N=73 NP specimens and N=41 mid-turbinate specimens). One of these NP samples that tested negative for SARS-CoV-2 did test positive for SARS-CoV-2 in its accompanying saliva sample. The 34 paired samples that tested as positive either in both the nasal (NP or mid-turbinate) and saliva specimen (N=32), or in the nasal (N=1) or saliva (N=1) specimen only, are listed in detail in the results. Of the participants that tested positive (in either test), 26 were symptomatic and eight were asymptomatic.
Analytical validation
The limit of detection (LOD) was determined by performing three separate experiments in triplicate, whereby the ATCC® SARS-CoV-2 control was serially diluted in heat-inactivated saliva from 192 to 0.75 viral copies per µL. The LOD was defined as the lowest concentration (and highest quantification cycle (Cq) value) at which at least two out of the three genes (ORF1ab, N and S) in all replicates were detected. The repeatability of the LOD was tested with 60 replicates at the LOD concentration to determine whether all 60 replicates would test positive. As a final measure of LOD, 20 heat-inactivated saliva samples spiked with 1x, 2x, 5x and 10x LOD concentrations of ATCC® SARS-CoV-2 control, and 20 heat-inactivated saliva samples negative for SARS-CoV-2 were tested blindly.
The stability of ATCC® SARS-CoV-2 control in heat-inactivated saliva was determined by preparing 12 replicates of saliva spiked with 1x, 2x, 4x, 8x and 16x LOD concentrations. Each of the replicates were divided into four groups and used either fresh or after one, two or three freeze-thaw cycles.
The cross-reactivity of Middle East respiratory syndrome coronavirus (MERS-CoV; ATCC® VR-3248SD™) was tested by preparing five replicates at 100 and 1,000 viral copies per µL of heat-inactivated saliva. These results were compared to heat-inactivated saliva spiked with ATCC® SARS-CoV-2 control at 10 viral copies per µL.
Diagnostic validation
A positive result was defined as any sample with a Cq value of <39 for two or more of the SARS-CoV-2 gene targets and the spike-in control gene (MS2). An indeterminate test was defined as any sample with a Cq value of <39 for only one of the SARS-CoV-2 gene targets and the spike-in control gene (MS2). Samples that gave an indeterminate result were re-tested when sufficient sample allowed. A negative result was defined as a sample with a Cq value >39 for all three of the SARS-CoV-2 gene targets and a spike-in control gene (MS2) Cq value of <39. An invalid result was defined as any situation where either the spike-in control gene (MS2) had a Cq value of >39, the positive control had a Cq value of >39 or the negative control had a Cq value of <39.
The reproducibility of the assay was determined by processing, in three different laboratories (VUW, IGENZ and UIUC), 147 saliva samples in blinded experiments from individuals that were positive (N=33) and negative (N=119) for SARS-CoV-2. The Cq values for each gene in every sample were compared. The VUW results were used to compare with UIUC results for diagnostic validation.
The diagnostic sensitivity of the test was determined by comparing saliva assay results with those of contemporaneously collected nasal swabs. Thirty-three positive SARS-CoV-2 samples, each paired with either an NP (N=28, 85%) or mid-turbinate (N=5) sample, and 114 negative samples, each paired with an NP sample, were tested. Concordance of sample test results between all three laboratories was calculated. Statistical analysis for a qualitative test with calculated diagnostic sensitivity, specificity and accuracy with confidence intervals was performed. Test accuracy was calculated using a disease prevalence value (estimated for managed isolation and quarantine (MIQ) facilities) of 0.74% (1,201 confirmed cases out of 162,733 individuals through MIQ facilities).[[14]] Finally, hypothesis tests were carried out to determine whether there was a statistically significant difference between Cq values obtained in different laboratories.
Ethics
Clinical trials comparing results from contemporaneously collected NP or mid-turbinate nasal swabs analysed by an FDA EUA reference method for detection of SARS-CoV-2, and saliva samples analysed by the UIUC assay (covidSHIELD), were approved by the Western Institutional Review Board, USA (WIRB 20202884). All participants gave written and informed consent both to the collection of all saliva samples and their use at external organisations for the purpose of test validation.
In-use experience of the assay
The assay was accredited for surveillance use in December 2020. The ease of sample handling and rate of inhibited samples was assessed by failure of detection of the spike-in control, MS2.
Results
Analytical validation
The LOD of SARS-CoV-2-spiked heat-inactivated saliva samples in three replicate experiments was <0.75 viral copies per μL (or 1.875 viral copies per reaction) (Figure 1). A positive result was given to all replicates at 0.75 viral copies per μL using the diagnostic criteria of two or more genes being detectable.
The LOD of <0.75 viral copies per μL was confirmed by all 60 replicates being detected as positive of SARS-CoV-2 (data not shown) and by a blinded experiment whereby all SARS-CoV-2-spiked samples were detected as positive at concentrations ranging from 1 to 10x LOD, and all non-spiked samples were detected as negative (Figure 2). Freeze-thaw cycles did not change assay results, with only one replicate of the ORF1ab gene at 1x LOD concentration at Cq of >39 (data not shown).
The assay showed no cross-reactivity with a similar virus, MERS-CoV. All heat-inactivated saliva samples spiked with MERS-CoV had Cq values of >39, even at high concentrations of 1,000 viral copies per μL (Figure 3).
Figure 1: Three replicate experiments of SARS-CoV-2 spiked heat-inactivated saliva serially diluted from 192 to 0.75 viral copies per μL. The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike-in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included. Dashed line indicates the limit of detection (LOD) for each replicate experiment.
Figure 2: A blinded experiment of heat-inactivated saliva samples that were either spiked with SARS-CoV-2 at 1 to 10x LOD concentrations (n=5 replicates per concentration; blinded positives) or not spiked (n=20 replicates; blinded negatives). The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike-in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included.
Figure 3: Cross-reactivity experiment where MERS-CoV spiked heat-inactivated saliva was not detected at high concentrations (n=5 replicates at 100 and 1,000 viral copies per μL), while SARS-CoV-2 spiked heat-inactivated saliva was detected in every sample (n=50 replicates at 10 viral copies per μL). The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included.
Diagnostic validation
Positive and negative SARS-CoV-2 saliva samples assayed are displayed in Table 1. The Cq values for the 113 concordant negative samples were >39 (data not shown). Thirty-two of 33 saliva samples that were paired with a positive nasal sample (NP and mid-turbinate) were detected as positive in the saliva test, resulting in a sensitivity of 97.0% (95% CI 84.2–99.9%). One saliva sample paired with a negative NP sample tested as positive, resulting in a specificity of 99.1% (95% CI 95.2–100%). Thus, the overall test concordance between saliva and nasal specimens was 98.6%, and the saliva test accuracy was 99.1% (95% CI 95.9–100) (Table 2).
Of those paired with NP specimens only, 27 out of the 28 samples that were positive in the NP test were detected as positive in the saliva test, resulting in a sensitivity of 96.4% (95% CI 81.7–99.9). One saliva sample that was paired with a negative NP sample tested as positive, resulting in a specificity of 98.6% (95% CI 92.6–100%). Thus, the overall test concordance between saliva and nasal specimens was 98.0%, and the saliva test accuracy was 98.6% (95% CI 94.1–99.9) (data not shown).
Table 1: Nasal Swab PCR as the reference test versus the saliva test.
Table 2: Diagnostic statistics for nasal swab PCR as the reference test versus the saliva test. The accuracy was calculated using a population prevalence of 0.74%, as estimated in MIQ.[[14]]
The Cq values for all 33 positive SARS-CoV-2 samples are displayed in Table 3. The one saliva sample (#34) taken from a symptomatic participant that tested positive and was accompanied by a contemporaneously collected negative NP specimen is also included in Table 3. The variability of the 2^(-△Cq) values between the two New Zealand laboratories was 0.24, 0.41 and 0.29 for the N, ORF1ab and S genes, respectively. A t-Test performed on the Cq values between the laboratories demonstrated no statistically significant differences. The low sample variability of 2^(-△Cq) values (particularly those stored in different aliquots; data not shown) between the laboratories confirmed high reproducibility.
Two samples with very low viral load, #6 and #33, were received for testing in the New Zealand laboratories. The samples were tested immediately in the VUW laboratory but there was a delay of one week in sample processing at the IGENZ laboratory, which resulted in Cq values of >39. Given the positive results obtained from these specimens at VUW and the similarity in Cq values between the two New Zealand laboratories in all samples that were processed promptly, samples #6 and #33 were assumed to be degraded and IGENZ data were omitted from Table 3. Sample #33 had a very low viral load and was called as indeterminate by all three laboratories. Repeat testing (#33 re-test) by UIUC on the original sample showed all three genes as being detected and was subsequently recorded as a positive result. There was insufficient sample for repeat analyses at VUW. However, when the degraded sample was repeated by IGENZ, no genes were detected. Again, given the close concordance of all other results, the absence of detectable genes was attributed to deterioration of the sample over time of storage and during transportation (>4 months).
An additional 20 non-paired saliva samples (N=10 positive and N=10 negative) were also tested and exhibited 100% concordance between the three laboratories (Supplementary Figure 1).
Table 3: Concordance of Cq values of 34 positive SARS-CoV-2 saliva samples analysed independently in two New Zealand laboratories (Victoria University of Wellington (VUW) and IGENZ) and one US laboratory (UIUC) in blind experiments. Saliva samples were processed immediately after collection at UIUC but 4–6 months later in the New Zealand laboratories. A dash indicates data omission due to assumed sample degradation. A blank entry denotes that the sample was not re-tested in that laboratory due to lack of sample volume. All samples had paired nasal swabs (nasopharyngeal NP, or mid-turbinate MT) as listed, and all but one nasal sample were called positive. (Pos = Positive, Neg = Negative, Indet = Indeterminate). Asterisk indicates a discordant result between the nasal and saliva sample. View Table 3.
In-use experience
To 1 September 2021, we have tested 13,304 saliva samples. Saliva viscosity that required manual pipetting only affected two specimens, both from the same participant. No sample exhibited assay inhibition.
Discussion
This study is the first to diagnostically validate a saliva test for SARS-CoV-2 in Aotearoa New Zealand. It used real-world paired saliva and NP samples from COVID-19 infected individuals, some of whom were exhibiting symptoms of COVID-19 infection (others were asymptomatic). Previous reports of sensitivities of SARS-CoV-2 detection in saliva are shown to be variable while specificity has been more consistent.[[15]] This emphasises the need for strict collection protocols and well-validated tests. The sensitivity (97.0%), specificity (99.1%) and accuracy (99.1%) of this UIUC saliva RT-qPCR test are very high, with the 98.6% concordance with NP and mid-turbinate specimens in all three laboratories. False-negative or false-positive results were not a test performance issue. The one positive saliva sample that was associated with a negative NP specimen suggests a difference in the timings of viral infection at the two anatomical sites during early or late disease. Overall, these results revealed the UIUC saliva RT-qPCR test to be a highly reproducible method for SARS-CoV-2 detection in both New Zealand laboratories. Moreover, the analytical validation using spiked saliva also exhibited a high sensitivity and specificity, as well as a lower LOD than that of another local NP qPCR assay reported recently (<2 versus ~10 copies per reaction, respectively).[[16]] The correlation of the Cq values for the three SARS-CoV-2 genes tested in each sample independently between the three laboratories was extremely high. This is particularly encouraging given the time between sample collection in the USA and processing in the New Zealand laboratories.
The recent Simpson–Roche report on the country’s COVID-19 testing strategy called for broadening the range of testing methods and recommended introducing saliva testing to increase the acceptability of testing across workforces in the community.[[17]] The availability of such an accurate saliva assay for selected testing situations, such as workplace testing, would significantly compliment the current NP testing used for New Zealand’s public health response. The low LOD, as well as the inclusion of samples from asymptomatic individuals, confirms that this assay can detect infection in asymptomatic and symptomatic people. Early detection enables individuals to be isolated quickly, reducing the risk of transmission.
It should be emphasised that, in addition to the specific RT-qPCR test being used, appropriate saliva collection and preparation procedures are essential.[[11]] This is important due to varying sample viscosity, which if not mitigated can make aliquoting difficult, particularly when using robotic equipment. Moreover, those undertaking saliva testing need instructions on adequate hydration and the need to abstain from food or drink, other than water, for an hour before sample collection. Additionally, there have been concerns about exogenous substances causing assay inhibition. However, the testing of several candidate substances, including nasal spray, different mouth lozenges, nicotine and mouthwash, revealed that only toothpaste (and in only one of three samples) was associated with saliva assay inhibition.[[12]] This is supported by our experience, as after providing detailed collection instructions and compliance, we have yet to encounter saliva samples where the PCR reaction has been inhibited.
This study does have limitations. At the time that this study was performed, there was no community transmission of SARS-CoV-2 in Aotearoa New Zealand. Therefore, it was impossible to obtain locally collected samples. Diagnostic validation using paired contemporaneously collected samples was essential to enable diagnostic validation of this test. This was required as part of IGENZ accreditation to ISO 15189 standards by International Accreditation New Zealand. Collaboration with UIUC enabled this validation study to take place. Although the number of positive pairs was limited, it was similar to many other studies.[[15]] We acknowledge that an increased number of positive pairs would enable greater understanding of variation between the two sample sites. The high analytical sensitivity and concordance, particularly with the NP samples, are specific to this assay and its sample preparation methods and cannot be taken as an indication of other assays’ performance. Saliva samples were obtained after comprehensive advice on hydration and avoiding food and drink. Assay performance may not be the same if collection advice is not followed.
Saliva is likely to participate in SARS-CoV-2 transmission due to the virus replicating in oral epithelial and salivary gland cells.[[18]] As saliva contains large numbers of oral epithelial cells, the detection of SARS-CoV-2 in this specimen is indicative of local virus production and does not rely on the virus passing through the oropharynx from the nasopharynx. Different replication rates at either site may result in a sample from one being positive when the other site is negative.[[1,19,20]] Moreover, there is emerging evidence that breakthrough infections in some people vaccinated with the Janssen Ad26.COV2.S COVID-19 and Pfizer/BioNTech vaccines elicit tissue compartmentalisation, whereby SARS-CoV-2 is detectable only in saliva and not in the nasal passages.[[21]] Our results support this disparity in viral load between tissue sites, but prospective studies are required to understand how frequently this occurs and how it impacts on diagnostic test performance.
Supply-chain issues, in particular the reagents and consumables required for RNA extraction, have hampered the testing of NP samples during the pandemic. The UIUC protocol bypasses the RNA extraction step and, in doing so, removes the supply-chain issues associated with this step. Furthermore, self-collection of saliva samples reduces the need for health professionals at collection sites and the heat-inactivation step reduces the risk of exposure to medical laboratory workers.
The country has now completed more than three million NP tests,[[22]] which is similar to the number of saliva tests conducted at UIUC.[[23]] SARS-CoV-2 testing used for our public health response needs to be scalable overnight, from the baseline testing of ~3,000–5,000 per day to more than 30,000 per day during possible community outbreaks.[[22]] This responsiveness has been achieved using the NP test. The NP swab remains the choice of the Ministry of Health for routine public health testing. However, a role for saliva testing in situations where high-frequency testing is required is now accepted.[[24]] This saliva test is also highly scalable and over 10,000 samples could be processed in one diagnostic laboratory in a single day.
Conclusions
The UIUC RT-qPCR has been tested locally and has been found to be an assay with high analytical and diagnostic sensitivity. It showed 99.1% accuracy and 98.1% concordance to that of nasal swabs in all three independent laboratories. In-use experience to date has not encountered either aliquoting problems or inhibited reactions. As a non-invasive test, it has significant appeal where high-frequency testing is required.
Supplementary Material
Supplementary Figure 1: Twenty saliva samples (10 positive and 10 negative) processed blind in triplicate by both New Zealand laboratories (data for Victoria University of Wellington shown) demonstrated 100% concordance with UIUC saliva results. Positive (PC), negative (NC) and no template (NTC) controls were included.
Summary
This paper presents the validation results of a qPCR test that was developed at University of Illinois Urbana-Champaign (UIUC) for non-invasively detecting the SARS-CoV-2 virus in saliva and tested in Aotearoa New Zealand laboratory. We used saliva samples that were collected from individuals that had also had nasal swabs taken at the same time. The nasal swabs for just over a third of these people were positive for SARS-CoV-2. Our results showed that the UIUC qPCR test is highly accurate (99.1%) for detecting SARS-CoV-2 in saliva and can detect very low copy numbers of SARS-CoV-2 in saliva. This UIUC qPCR for SARS-CoV-2 is as accurate as the qPCR tests used for detecting SARS-CoV-2 in nasopharyngeal samples in New Zealand. These results confirmed that this reliable option for SARS-CoV-2 testing, including for diagnostic testing for asymptomatic people requiring regular screening.
Abstract
Aim
To validate a reverse transcriptase-quantitative polymerase chain reaction (RT-qPCR) assay to detect SARS-CoV-2 in saliva in two independent Aotearoa New Zealand laboratories.
Method
An RT-qPCR assay developed at University of Illinois Urbana-Champaign, USA, was validated in two New Zealand laboratories. Analytical measures, such as limit of detection (LOD) and cross-reactivity, were performed. One hundred and forty-seven saliva samples, each paired with a contemporaneously collected nasal swab, mainly of nasopharyngeal origin, were received. Positive (N=33) and negative (N=114) samples were tested blindly in each laboratory. Diagnostic sensitivity and specificity were then calculated.
Results
The LOD was <0.75 copy per µL and no cross-reactivity with MERS-CoV was detected. There was complete concordance between laboratories for all saliva samples with the quantification cycle values for all three genes in close agreement. Saliva had 98.7% concordance with paired nasal samples: and a sensitivity, specificity and accuracy of 97.0%, 99.1% and 99.1%, respectively.
Conclusion
This saliva RT-qPCR assay produces reproducible results with a low LOD. High sensitivity and specificity make it a reliable option for SARS-CoV-2 testing, including for asymptomatic people requiring regular screening.
Author Information
Acknowledgements
Correspondence
Correspondence Email
Competing Interests
1) Wyllie AL, Fournier J, Casanovas-Massana A, et al. Saliva or Nasopharyngeal swab specimens for detection of SARS-CoV-2. N Engl J Med. 2020;383(13):1283-6.
2) Richard-Greenblatt MJ, Ziegler M, Bromberg V, et al. Impact of nasopharyngeal specimen quality on SARS-CoV-2 test sensitivity. medRxiv. 2020 Dec 11. doi: 10.1101/2020.12.09.20246520
3) To KKW, Yip CCY, Lai CYW, et al. Saliva as a diagnostic specimen for testing respiratory virus by a point-of-care molecular assay: a diagnostic validity study. Clin Microbiol Infect. 2019;25(3):372-8.
4) To KKW, Chan KH, Ho J, et al. Respiratory virus infection among hospitalized adult patients with or without clinically apparent respiratory infection: a prospective cohort study. Clin Microbiol Infect. 2019;25(12):1539-45.
5) To KK, Tsang OT, Yip CC, et al. Consistent detection of 2019 novel coronavirus in saliva. Clin Infect Dis. 2020;71(15):841-3.
6) Rao M, Rashid FA, Sabri FSAH, et al. Comparing nasopharyngeal swab and early morning saliva for the identification of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Clin Infect Dis. 2021;72(9):352-6.
7) Wang WK, Chen SY, Liu IJ, et al. SARS Research Group of the National Taiwan University/National Taiwan University Hospital. Detection of SARS-associated coronavirus in throat wash and saliva in early diagnosis. Emerg Infect Dis. 2004;10(7):1213-9.
8) Harikrishnan P. Saliva as a potential diagnostic specimen for COVID-19 testing. J Craniofac Surg. 2020;31(6):653-5.
9) Tee M, Ubial P, Ranoa DR, et al. Direct saliva versus conventional nasopharyngeal swab qRT-PCR to diagnose SARS-CoV-2: validity study. Asian J Res Infect Dis. 2021;6(2):37-46.
10) New Zealand Microbiology Network [Internet]. Statement on saliva testing for surveillance of SARS-CoV-2. Summary recommendations (2021) [cited 2021 Aug 13]. Available from: https://www.nzmn.org.nz/assets/NZMN/NZMN-statement-on-saliva-testing-for-surveillance-of-SARS-CoV-2-March-2021-v3.pdf
11) Ranoa DRE, Holland RL, Alnaji FG et al. Saliva-based molecular testing for SARS-CoV-2 that bypasses RNA extraction. bioRxiv. doi.org/10.1101/2020.06.18.159434
12) FDA [Internet]. Emergency Use Authorization for covidShield [cited 2021 Aug 13]. Available from: https://www.fda.gov/media/146314/download
13) Ranoa DRE, Holland RL, Alnaji FG, et al. Mitigation of SARS-CoV-2 Transmission at a Large Public University. medRxiv preprint: doi.org/10.1101/2021.08.03.21261548
14) Ministry of Business, Innovation & Employment [Internet]. Managed isolation and quarantine data [cited 2021 Aug 13]. Available from: https://www.mbie.govt.nz/business-and-employment/economic-development/covid-19-data-resources/managed-isolation-and-quarantine-data/
15) Butler-Laporte G, Lawandi A, Schiller I et al. Comparison of saliva and nasopharyngeal swab nucleic acid amplification testing for detection of SARS-CoV-2. A systematic review and meta-analysis. JAMA Int Med 2021;181(3):353-60.
16) Fox-Lewis S, Muttaiyah S, Rahnama F, et al. An understanding of discordant SARS-CoV-2 test results: an examination of the data from a central Auckland laboratory. NZ Med J. 2020;133(1519):81-8.
17) Simpson H, Roche B, Report of the Advisory Committee to oversee the implementation of the New Zealand COVID-19 Surveillance Plan and Testing Strategy [cited 2021 Aug 13]. NZ Govt Wellington 2020. Available from: https://covid19.govt.nz/assets/Review-of-Surveillance-Plan-and-Testing-Strategy/Final_Report-of-Advisory-Committee-to-Oversee-the-Implementation-of-the-....pdf
18) Huang N, Perez P, Kato T, SARS-CoV-2 infection of the oral cavity and saliva, Nat Med. 2021;27(5):892-903.
19) Miller M, Jansen M, Bisignano A, et al. Validation of a self-administrable, saliva-based RT-qPCR test detecting SARS-CoV-2. Preprint. Posted online June 9, 2020. medRxiv. doi:10.1101/2020.06.05.20122721
20) Rao M, Rashid FA, Sabri FSAH, et al. Comparing nasopharyngeal swab and early morning saliva for the identification of SARS-CoV-2. Clin Infect Dis. Published online August 6, 2020. doi:10.1093/cid/ciaa1156
21) Ke R, Martinez PP, Smith R, et al. Longitudinal analysis of SARS-CoV-2 vaccine breakthrough infections reveal limited infectious virus shedding and restricted tissue distribution. medRxiv preprint: doi.org/10.1101/2021.08.30.21262701
22) Ministry of Health [Internet]. “Testing for COVID-19,” Ministry of Health, 03 May 2021. Available: https://www.health.govt.nz/our-work/diseases-and-conditions/covid-19-novel-coronavirus/covid-19-data-and-statistics/testing-covid-19#testing
23) U. C. University of Illinois [Internet]. “Shield Testing Data,” University of Illinois, Urbana Champaign, 03 May 2021 [cited 2021 Aug 13].. Available: https://covid19.illinois.edu/on-campus-covid-19-testing-data-dashboard/
24) NZ Herald [Internet]. “Covid 19 coronavirus: Daily saliva testing for border, quarantine workers will be optional,” NZ Herald, 22 Jan 2021 [cited 2021 Aug 13]. Available: https://www.nzherald.co.nz/nz/covid-19-coronavirus-daily-saliva-testing-for-border-quarantine-workers-will-be-optional/FFODUHMKQLRC2CXFA3GV2PLO4Q/
For the PDF of this article,
contact nzmj@nzma.org.nz
Validation of a molecular assay to detect SARS-CoV-2 in saliva | OPEN ACCESS
View Article PDF
Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is the cause of the current coronavirus disease 2019 (COVID-19) pandemic. Nucleic acid amplification testing using reverse transcription quantitative polymerase chain reaction (RT-qPCR) for SARS-CoV-2 genes is the diagnostic test of choice because it detects extremely low levels of virus within biological fluids. However, the sensitivity and accuracy of such tests are dependent upon the type of specimen, its method of collection and the RT-qPCR test itself.
In the haste to develop a detection test during the onset of the COVID-19 pandemic, specimens from the nasopharyngeal (NP) site were used. Although this specimen has become regarded as the gold standard test to which all others are compared, it has its own limitations. The NP site can be difficult to reach, impacting on NP specimen quality,[[1]] which in turn accounts for significant differences in NP assay sensitivity and increases the risk of false negatives.[[2]] Moreover, the invasiveness of sample collection often causes discomfort and is unpopular with those requiring frequent testing.
The use of saliva as a specimen for detecting SARS-CoV-2 is increasing around the world. Saliva samples perform well compared to NP in detecting respiratory viruses including coronaviruses,[[3–6]] with both SARS-CoV[[7]] and SARS-CoV-2[[1,5,8,9]] being present in high titers in saliva. The advantage of saliva is that it bypasses the need for invasive sample collection. Although it is an attractive option for frequent testing, there has been a cautious approach to saliva testing in Aotearoa New Zealand. The New Zealand Microbiology Network (NZMN) has recommended that any saliva test would need to be validated locally using well characterised samples that were positive for SARS-CoV-2 and that testing be performed in a diagnostic laboratory accredited by International Accreditation New Zealand (IANZ) and aligned with the ISO 15189 quality framework.[[10]] NZMN noted that the accuracy of saliva tests is reliant upon the methods used for saliva collection, the extraction steps employed for the viral RNA and the commercial kit utilised.[[10]]
In September 2020, we established an international collaboration with investigators at the University of Illinois Urbana-Champaign (UIUC), USA, who had developed a simple and rapid direct saliva-to RT-qPCR assay for the detection of SARS-CoV-2.[[11]] The assay omits the common RNA extraction step and utilises a modified method of a commercially available RT-qPCR kit, thus avoiding reagent competition and supply-chain issues. It includes an initial heat step that inactivates the saliva sample, reducing viral transmission risk, as well as enabling accessibility of the viral genome to the PCR reagents. Finally, the subsequent addition of a detergent-buffer mix overcomes the drawback of saliva viscosity. This test has been approved for surveillance testing in a US FDA Emergency Use Authorization (EUA).[[12]]
The aim of the study was to validate the UIUC RT-qPCR protocol for detecting SARS-CoV-2 in saliva in two independent New Zealand laboratories by comparing our results to those measured at the UIUC laboratory (USA). This study used real-world saliva samples that were paired with nasal samples, of mainly NP origin, from participants that were positive and negative for SARS-CoV-2.
Methods
Test details
The validation was performed at Victoria University of Wellington (VUW) in Wellington and IGENZ Ltd in Auckland using both commercially available heat-inactivated SARS-CoV-2 (ATCC® VR-1986HK™) and saliva samples from consenting COVID-19 positive and negative participants located in Chicago and Wisconsin, USA. These samples, obtained by UIUC, were shipped to Aotearoa New Zealand for the studies described in this paper.
The RT-qPCR test used the ThermoFisher Scientific TaqPath[[TM]] COVID-19 Combo kit (Cat. No. A47814; ThermoFisher) and omits the RNA extraction step. The kit includes primers and probes specific to the SARS-CoV-2 nucleocapsid (N), spike (S) and replication (ORF1ab) genes and a spike-in bacteriophage (MS2) gene, with modifications to the manufacturer’s instructions, as previously published.[[11,13]] Controls included a positive control (TaqPath™ COVID19 Control; ThermoFisher Scientific TaqPath[[TM]] COVID-19 Combo kit) at 25 copies per µL, negative control (UltraPure dH2O) and no-template control (heat-inactivated saliva; collected in-house).
Saliva collection
All but five participants had saliva and nasopharyngeal specimens collected contemporaneously. The five non-contemporaneously collected saliva samples were taken following a positive low viral load result obtained from a mid-turbinate swab. These later samples were included to deliberately include paired samples with low viral load as required for FDA EUA validation purposes, which were also included in this current study to ensure that it contained samples of low viral load.
Participants provided a saliva sample by following guidelines that instructed them to allow saliva to collect in the mouth before gently expelling saliva into the collection tube (passive drool method). They then capped their tube and handed it to the healthcare professional or collection-site staff, who placed it into a collection container. A healthcare professional collected the nasopharyngeal (N=142) or mid-turbinate (N=5) sample and transferred the swab into the collection media (per the comparator manufacturer’s instructions). All nasal swabs (defined as both nasopharyngeal and mid-turbinate) were processed in the clinical pathology laboratory at the University of Illinois Chicago Hospital using an FDA EUA reference method for detection of SARS-CoV-2, the Abbott RealTime SARS-CoV-2 assay performed on the Abbott m2000 System with a limit of detection of 2,700 NDU per mL.[[13]]
Saliva samples that had been heat-inactivated at 95°C for 30 min in UIUC were divided into aliquots. Sample aliquots were stored for up to four months at -80oC before transportation from UIUC on dry ice to the two New Zealand laboratories. Even though the status of each aliquot was unknown upon arrival, samples were immediately placed in a random sequence and re-coded. Before testing, the heat-inactivated saliva aliquot was diluted 1:1 (v/v) in a Tris-Borate-EDTA/Tween 20 mix.
A total of 147 paired saliva samples were received. Of these, 33 saliva samples were from people that tested positive for SARS-CoV-2 based on a nasal sample test result (N=28 NP specimens and N=5 mid-turbinate specimens). The remaining 114 saliva samples were from people that tested negative with SARS-CoV-2 based on a nasal sample test result (N=73 NP specimens and N=41 mid-turbinate specimens). One of these NP samples that tested negative for SARS-CoV-2 did test positive for SARS-CoV-2 in its accompanying saliva sample. The 34 paired samples that tested as positive either in both the nasal (NP or mid-turbinate) and saliva specimen (N=32), or in the nasal (N=1) or saliva (N=1) specimen only, are listed in detail in the results. Of the participants that tested positive (in either test), 26 were symptomatic and eight were asymptomatic.
Analytical validation
The limit of detection (LOD) was determined by performing three separate experiments in triplicate, whereby the ATCC® SARS-CoV-2 control was serially diluted in heat-inactivated saliva from 192 to 0.75 viral copies per µL. The LOD was defined as the lowest concentration (and highest quantification cycle (Cq) value) at which at least two out of the three genes (ORF1ab, N and S) in all replicates were detected. The repeatability of the LOD was tested with 60 replicates at the LOD concentration to determine whether all 60 replicates would test positive. As a final measure of LOD, 20 heat-inactivated saliva samples spiked with 1x, 2x, 5x and 10x LOD concentrations of ATCC® SARS-CoV-2 control, and 20 heat-inactivated saliva samples negative for SARS-CoV-2 were tested blindly.
The stability of ATCC® SARS-CoV-2 control in heat-inactivated saliva was determined by preparing 12 replicates of saliva spiked with 1x, 2x, 4x, 8x and 16x LOD concentrations. Each of the replicates were divided into four groups and used either fresh or after one, two or three freeze-thaw cycles.
The cross-reactivity of Middle East respiratory syndrome coronavirus (MERS-CoV; ATCC® VR-3248SD™) was tested by preparing five replicates at 100 and 1,000 viral copies per µL of heat-inactivated saliva. These results were compared to heat-inactivated saliva spiked with ATCC® SARS-CoV-2 control at 10 viral copies per µL.
Diagnostic validation
A positive result was defined as any sample with a Cq value of <39 for two or more of the SARS-CoV-2 gene targets and the spike-in control gene (MS2). An indeterminate test was defined as any sample with a Cq value of <39 for only one of the SARS-CoV-2 gene targets and the spike-in control gene (MS2). Samples that gave an indeterminate result were re-tested when sufficient sample allowed. A negative result was defined as a sample with a Cq value >39 for all three of the SARS-CoV-2 gene targets and a spike-in control gene (MS2) Cq value of <39. An invalid result was defined as any situation where either the spike-in control gene (MS2) had a Cq value of >39, the positive control had a Cq value of >39 or the negative control had a Cq value of <39.
The reproducibility of the assay was determined by processing, in three different laboratories (VUW, IGENZ and UIUC), 147 saliva samples in blinded experiments from individuals that were positive (N=33) and negative (N=119) for SARS-CoV-2. The Cq values for each gene in every sample were compared. The VUW results were used to compare with UIUC results for diagnostic validation.
The diagnostic sensitivity of the test was determined by comparing saliva assay results with those of contemporaneously collected nasal swabs. Thirty-three positive SARS-CoV-2 samples, each paired with either an NP (N=28, 85%) or mid-turbinate (N=5) sample, and 114 negative samples, each paired with an NP sample, were tested. Concordance of sample test results between all three laboratories was calculated. Statistical analysis for a qualitative test with calculated diagnostic sensitivity, specificity and accuracy with confidence intervals was performed. Test accuracy was calculated using a disease prevalence value (estimated for managed isolation and quarantine (MIQ) facilities) of 0.74% (1,201 confirmed cases out of 162,733 individuals through MIQ facilities).[[14]] Finally, hypothesis tests were carried out to determine whether there was a statistically significant difference between Cq values obtained in different laboratories.
Ethics
Clinical trials comparing results from contemporaneously collected NP or mid-turbinate nasal swabs analysed by an FDA EUA reference method for detection of SARS-CoV-2, and saliva samples analysed by the UIUC assay (covidSHIELD), were approved by the Western Institutional Review Board, USA (WIRB 20202884). All participants gave written and informed consent both to the collection of all saliva samples and their use at external organisations for the purpose of test validation.
In-use experience of the assay
The assay was accredited for surveillance use in December 2020. The ease of sample handling and rate of inhibited samples was assessed by failure of detection of the spike-in control, MS2.
Results
Analytical validation
The LOD of SARS-CoV-2-spiked heat-inactivated saliva samples in three replicate experiments was <0.75 viral copies per μL (or 1.875 viral copies per reaction) (Figure 1). A positive result was given to all replicates at 0.75 viral copies per μL using the diagnostic criteria of two or more genes being detectable.
The LOD of <0.75 viral copies per μL was confirmed by all 60 replicates being detected as positive of SARS-CoV-2 (data not shown) and by a blinded experiment whereby all SARS-CoV-2-spiked samples were detected as positive at concentrations ranging from 1 to 10x LOD, and all non-spiked samples were detected as negative (Figure 2). Freeze-thaw cycles did not change assay results, with only one replicate of the ORF1ab gene at 1x LOD concentration at Cq of >39 (data not shown).
The assay showed no cross-reactivity with a similar virus, MERS-CoV. All heat-inactivated saliva samples spiked with MERS-CoV had Cq values of >39, even at high concentrations of 1,000 viral copies per μL (Figure 3).
Figure 1: Three replicate experiments of SARS-CoV-2 spiked heat-inactivated saliva serially diluted from 192 to 0.75 viral copies per μL. The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike-in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included. Dashed line indicates the limit of detection (LOD) for each replicate experiment.
Figure 2: A blinded experiment of heat-inactivated saliva samples that were either spiked with SARS-CoV-2 at 1 to 10x LOD concentrations (n=5 replicates per concentration; blinded positives) or not spiked (n=20 replicates; blinded negatives). The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike-in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included.
Figure 3: Cross-reactivity experiment where MERS-CoV spiked heat-inactivated saliva was not detected at high concentrations (n=5 replicates at 100 and 1,000 viral copies per μL), while SARS-CoV-2 spiked heat-inactivated saliva was detected in every sample (n=50 replicates at 10 viral copies per μL). The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included.
Diagnostic validation
Positive and negative SARS-CoV-2 saliva samples assayed are displayed in Table 1. The Cq values for the 113 concordant negative samples were >39 (data not shown). Thirty-two of 33 saliva samples that were paired with a positive nasal sample (NP and mid-turbinate) were detected as positive in the saliva test, resulting in a sensitivity of 97.0% (95% CI 84.2–99.9%). One saliva sample paired with a negative NP sample tested as positive, resulting in a specificity of 99.1% (95% CI 95.2–100%). Thus, the overall test concordance between saliva and nasal specimens was 98.6%, and the saliva test accuracy was 99.1% (95% CI 95.9–100) (Table 2).
Of those paired with NP specimens only, 27 out of the 28 samples that were positive in the NP test were detected as positive in the saliva test, resulting in a sensitivity of 96.4% (95% CI 81.7–99.9). One saliva sample that was paired with a negative NP sample tested as positive, resulting in a specificity of 98.6% (95% CI 92.6–100%). Thus, the overall test concordance between saliva and nasal specimens was 98.0%, and the saliva test accuracy was 98.6% (95% CI 94.1–99.9) (data not shown).
Table 1: Nasal Swab PCR as the reference test versus the saliva test.
Table 2: Diagnostic statistics for nasal swab PCR as the reference test versus the saliva test. The accuracy was calculated using a population prevalence of 0.74%, as estimated in MIQ.[[14]]
The Cq values for all 33 positive SARS-CoV-2 samples are displayed in Table 3. The one saliva sample (#34) taken from a symptomatic participant that tested positive and was accompanied by a contemporaneously collected negative NP specimen is also included in Table 3. The variability of the 2^(-△Cq) values between the two New Zealand laboratories was 0.24, 0.41 and 0.29 for the N, ORF1ab and S genes, respectively. A t-Test performed on the Cq values between the laboratories demonstrated no statistically significant differences. The low sample variability of 2^(-△Cq) values (particularly those stored in different aliquots; data not shown) between the laboratories confirmed high reproducibility.
Two samples with very low viral load, #6 and #33, were received for testing in the New Zealand laboratories. The samples were tested immediately in the VUW laboratory but there was a delay of one week in sample processing at the IGENZ laboratory, which resulted in Cq values of >39. Given the positive results obtained from these specimens at VUW and the similarity in Cq values between the two New Zealand laboratories in all samples that were processed promptly, samples #6 and #33 were assumed to be degraded and IGENZ data were omitted from Table 3. Sample #33 had a very low viral load and was called as indeterminate by all three laboratories. Repeat testing (#33 re-test) by UIUC on the original sample showed all three genes as being detected and was subsequently recorded as a positive result. There was insufficient sample for repeat analyses at VUW. However, when the degraded sample was repeated by IGENZ, no genes were detected. Again, given the close concordance of all other results, the absence of detectable genes was attributed to deterioration of the sample over time of storage and during transportation (>4 months).
An additional 20 non-paired saliva samples (N=10 positive and N=10 negative) were also tested and exhibited 100% concordance between the three laboratories (Supplementary Figure 1).
Table 3: Concordance of Cq values of 34 positive SARS-CoV-2 saliva samples analysed independently in two New Zealand laboratories (Victoria University of Wellington (VUW) and IGENZ) and one US laboratory (UIUC) in blind experiments. Saliva samples were processed immediately after collection at UIUC but 4–6 months later in the New Zealand laboratories. A dash indicates data omission due to assumed sample degradation. A blank entry denotes that the sample was not re-tested in that laboratory due to lack of sample volume. All samples had paired nasal swabs (nasopharyngeal NP, or mid-turbinate MT) as listed, and all but one nasal sample were called positive. (Pos = Positive, Neg = Negative, Indet = Indeterminate). Asterisk indicates a discordant result between the nasal and saliva sample. View Table 3.
In-use experience
To 1 September 2021, we have tested 13,304 saliva samples. Saliva viscosity that required manual pipetting only affected two specimens, both from the same participant. No sample exhibited assay inhibition.
Discussion
This study is the first to diagnostically validate a saliva test for SARS-CoV-2 in Aotearoa New Zealand. It used real-world paired saliva and NP samples from COVID-19 infected individuals, some of whom were exhibiting symptoms of COVID-19 infection (others were asymptomatic). Previous reports of sensitivities of SARS-CoV-2 detection in saliva are shown to be variable while specificity has been more consistent.[[15]] This emphasises the need for strict collection protocols and well-validated tests. The sensitivity (97.0%), specificity (99.1%) and accuracy (99.1%) of this UIUC saliva RT-qPCR test are very high, with the 98.6% concordance with NP and mid-turbinate specimens in all three laboratories. False-negative or false-positive results were not a test performance issue. The one positive saliva sample that was associated with a negative NP specimen suggests a difference in the timings of viral infection at the two anatomical sites during early or late disease. Overall, these results revealed the UIUC saliva RT-qPCR test to be a highly reproducible method for SARS-CoV-2 detection in both New Zealand laboratories. Moreover, the analytical validation using spiked saliva also exhibited a high sensitivity and specificity, as well as a lower LOD than that of another local NP qPCR assay reported recently (<2 versus ~10 copies per reaction, respectively).[[16]] The correlation of the Cq values for the three SARS-CoV-2 genes tested in each sample independently between the three laboratories was extremely high. This is particularly encouraging given the time between sample collection in the USA and processing in the New Zealand laboratories.
The recent Simpson–Roche report on the country’s COVID-19 testing strategy called for broadening the range of testing methods and recommended introducing saliva testing to increase the acceptability of testing across workforces in the community.[[17]] The availability of such an accurate saliva assay for selected testing situations, such as workplace testing, would significantly compliment the current NP testing used for New Zealand’s public health response. The low LOD, as well as the inclusion of samples from asymptomatic individuals, confirms that this assay can detect infection in asymptomatic and symptomatic people. Early detection enables individuals to be isolated quickly, reducing the risk of transmission.
It should be emphasised that, in addition to the specific RT-qPCR test being used, appropriate saliva collection and preparation procedures are essential.[[11]] This is important due to varying sample viscosity, which if not mitigated can make aliquoting difficult, particularly when using robotic equipment. Moreover, those undertaking saliva testing need instructions on adequate hydration and the need to abstain from food or drink, other than water, for an hour before sample collection. Additionally, there have been concerns about exogenous substances causing assay inhibition. However, the testing of several candidate substances, including nasal spray, different mouth lozenges, nicotine and mouthwash, revealed that only toothpaste (and in only one of three samples) was associated with saliva assay inhibition.[[12]] This is supported by our experience, as after providing detailed collection instructions and compliance, we have yet to encounter saliva samples where the PCR reaction has been inhibited.
This study does have limitations. At the time that this study was performed, there was no community transmission of SARS-CoV-2 in Aotearoa New Zealand. Therefore, it was impossible to obtain locally collected samples. Diagnostic validation using paired contemporaneously collected samples was essential to enable diagnostic validation of this test. This was required as part of IGENZ accreditation to ISO 15189 standards by International Accreditation New Zealand. Collaboration with UIUC enabled this validation study to take place. Although the number of positive pairs was limited, it was similar to many other studies.[[15]] We acknowledge that an increased number of positive pairs would enable greater understanding of variation between the two sample sites. The high analytical sensitivity and concordance, particularly with the NP samples, are specific to this assay and its sample preparation methods and cannot be taken as an indication of other assays’ performance. Saliva samples were obtained after comprehensive advice on hydration and avoiding food and drink. Assay performance may not be the same if collection advice is not followed.
Saliva is likely to participate in SARS-CoV-2 transmission due to the virus replicating in oral epithelial and salivary gland cells.[[18]] As saliva contains large numbers of oral epithelial cells, the detection of SARS-CoV-2 in this specimen is indicative of local virus production and does not rely on the virus passing through the oropharynx from the nasopharynx. Different replication rates at either site may result in a sample from one being positive when the other site is negative.[[1,19,20]] Moreover, there is emerging evidence that breakthrough infections in some people vaccinated with the Janssen Ad26.COV2.S COVID-19 and Pfizer/BioNTech vaccines elicit tissue compartmentalisation, whereby SARS-CoV-2 is detectable only in saliva and not in the nasal passages.[[21]] Our results support this disparity in viral load between tissue sites, but prospective studies are required to understand how frequently this occurs and how it impacts on diagnostic test performance.
Supply-chain issues, in particular the reagents and consumables required for RNA extraction, have hampered the testing of NP samples during the pandemic. The UIUC protocol bypasses the RNA extraction step and, in doing so, removes the supply-chain issues associated with this step. Furthermore, self-collection of saliva samples reduces the need for health professionals at collection sites and the heat-inactivation step reduces the risk of exposure to medical laboratory workers.
The country has now completed more than three million NP tests,[[22]] which is similar to the number of saliva tests conducted at UIUC.[[23]] SARS-CoV-2 testing used for our public health response needs to be scalable overnight, from the baseline testing of ~3,000–5,000 per day to more than 30,000 per day during possible community outbreaks.[[22]] This responsiveness has been achieved using the NP test. The NP swab remains the choice of the Ministry of Health for routine public health testing. However, a role for saliva testing in situations where high-frequency testing is required is now accepted.[[24]] This saliva test is also highly scalable and over 10,000 samples could be processed in one diagnostic laboratory in a single day.
Conclusions
The UIUC RT-qPCR has been tested locally and has been found to be an assay with high analytical and diagnostic sensitivity. It showed 99.1% accuracy and 98.1% concordance to that of nasal swabs in all three independent laboratories. In-use experience to date has not encountered either aliquoting problems or inhibited reactions. As a non-invasive test, it has significant appeal where high-frequency testing is required.
Supplementary Material
Supplementary Figure 1: Twenty saliva samples (10 positive and 10 negative) processed blind in triplicate by both New Zealand laboratories (data for Victoria University of Wellington shown) demonstrated 100% concordance with UIUC saliva results. Positive (PC), negative (NC) and no template (NTC) controls were included.
Summary
This paper presents the validation results of a qPCR test that was developed at University of Illinois Urbana-Champaign (UIUC) for non-invasively detecting the SARS-CoV-2 virus in saliva and tested in Aotearoa New Zealand laboratory. We used saliva samples that were collected from individuals that had also had nasal swabs taken at the same time. The nasal swabs for just over a third of these people were positive for SARS-CoV-2. Our results showed that the UIUC qPCR test is highly accurate (99.1%) for detecting SARS-CoV-2 in saliva and can detect very low copy numbers of SARS-CoV-2 in saliva. This UIUC qPCR for SARS-CoV-2 is as accurate as the qPCR tests used for detecting SARS-CoV-2 in nasopharyngeal samples in New Zealand. These results confirmed that this reliable option for SARS-CoV-2 testing, including for diagnostic testing for asymptomatic people requiring regular screening.
Abstract
Aim
To validate a reverse transcriptase-quantitative polymerase chain reaction (RT-qPCR) assay to detect SARS-CoV-2 in saliva in two independent Aotearoa New Zealand laboratories.
Method
An RT-qPCR assay developed at University of Illinois Urbana-Champaign, USA, was validated in two New Zealand laboratories. Analytical measures, such as limit of detection (LOD) and cross-reactivity, were performed. One hundred and forty-seven saliva samples, each paired with a contemporaneously collected nasal swab, mainly of nasopharyngeal origin, were received. Positive (N=33) and negative (N=114) samples were tested blindly in each laboratory. Diagnostic sensitivity and specificity were then calculated.
Results
The LOD was <0.75 copy per µL and no cross-reactivity with MERS-CoV was detected. There was complete concordance between laboratories for all saliva samples with the quantification cycle values for all three genes in close agreement. Saliva had 98.7% concordance with paired nasal samples: and a sensitivity, specificity and accuracy of 97.0%, 99.1% and 99.1%, respectively.
Conclusion
This saliva RT-qPCR assay produces reproducible results with a low LOD. High sensitivity and specificity make it a reliable option for SARS-CoV-2 testing, including for asymptomatic people requiring regular screening.
Author Information
Acknowledgements
Correspondence
Correspondence Email
Competing Interests
1) Wyllie AL, Fournier J, Casanovas-Massana A, et al. Saliva or Nasopharyngeal swab specimens for detection of SARS-CoV-2. N Engl J Med. 2020;383(13):1283-6.
2) Richard-Greenblatt MJ, Ziegler M, Bromberg V, et al. Impact of nasopharyngeal specimen quality on SARS-CoV-2 test sensitivity. medRxiv. 2020 Dec 11. doi: 10.1101/2020.12.09.20246520
3) To KKW, Yip CCY, Lai CYW, et al. Saliva as a diagnostic specimen for testing respiratory virus by a point-of-care molecular assay: a diagnostic validity study. Clin Microbiol Infect. 2019;25(3):372-8.
4) To KKW, Chan KH, Ho J, et al. Respiratory virus infection among hospitalized adult patients with or without clinically apparent respiratory infection: a prospective cohort study. Clin Microbiol Infect. 2019;25(12):1539-45.
5) To KK, Tsang OT, Yip CC, et al. Consistent detection of 2019 novel coronavirus in saliva. Clin Infect Dis. 2020;71(15):841-3.
6) Rao M, Rashid FA, Sabri FSAH, et al. Comparing nasopharyngeal swab and early morning saliva for the identification of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Clin Infect Dis. 2021;72(9):352-6.
7) Wang WK, Chen SY, Liu IJ, et al. SARS Research Group of the National Taiwan University/National Taiwan University Hospital. Detection of SARS-associated coronavirus in throat wash and saliva in early diagnosis. Emerg Infect Dis. 2004;10(7):1213-9.
8) Harikrishnan P. Saliva as a potential diagnostic specimen for COVID-19 testing. J Craniofac Surg. 2020;31(6):653-5.
9) Tee M, Ubial P, Ranoa DR, et al. Direct saliva versus conventional nasopharyngeal swab qRT-PCR to diagnose SARS-CoV-2: validity study. Asian J Res Infect Dis. 2021;6(2):37-46.
10) New Zealand Microbiology Network [Internet]. Statement on saliva testing for surveillance of SARS-CoV-2. Summary recommendations (2021) [cited 2021 Aug 13]. Available from: https://www.nzmn.org.nz/assets/NZMN/NZMN-statement-on-saliva-testing-for-surveillance-of-SARS-CoV-2-March-2021-v3.pdf
11) Ranoa DRE, Holland RL, Alnaji FG et al. Saliva-based molecular testing for SARS-CoV-2 that bypasses RNA extraction. bioRxiv. doi.org/10.1101/2020.06.18.159434
12) FDA [Internet]. Emergency Use Authorization for covidShield [cited 2021 Aug 13]. Available from: https://www.fda.gov/media/146314/download
13) Ranoa DRE, Holland RL, Alnaji FG, et al. Mitigation of SARS-CoV-2 Transmission at a Large Public University. medRxiv preprint: doi.org/10.1101/2021.08.03.21261548
14) Ministry of Business, Innovation & Employment [Internet]. Managed isolation and quarantine data [cited 2021 Aug 13]. Available from: https://www.mbie.govt.nz/business-and-employment/economic-development/covid-19-data-resources/managed-isolation-and-quarantine-data/
15) Butler-Laporte G, Lawandi A, Schiller I et al. Comparison of saliva and nasopharyngeal swab nucleic acid amplification testing for detection of SARS-CoV-2. A systematic review and meta-analysis. JAMA Int Med 2021;181(3):353-60.
16) Fox-Lewis S, Muttaiyah S, Rahnama F, et al. An understanding of discordant SARS-CoV-2 test results: an examination of the data from a central Auckland laboratory. NZ Med J. 2020;133(1519):81-8.
17) Simpson H, Roche B, Report of the Advisory Committee to oversee the implementation of the New Zealand COVID-19 Surveillance Plan and Testing Strategy [cited 2021 Aug 13]. NZ Govt Wellington 2020. Available from: https://covid19.govt.nz/assets/Review-of-Surveillance-Plan-and-Testing-Strategy/Final_Report-of-Advisory-Committee-to-Oversee-the-Implementation-of-the-....pdf
18) Huang N, Perez P, Kato T, SARS-CoV-2 infection of the oral cavity and saliva, Nat Med. 2021;27(5):892-903.
19) Miller M, Jansen M, Bisignano A, et al. Validation of a self-administrable, saliva-based RT-qPCR test detecting SARS-CoV-2. Preprint. Posted online June 9, 2020. medRxiv. doi:10.1101/2020.06.05.20122721
20) Rao M, Rashid FA, Sabri FSAH, et al. Comparing nasopharyngeal swab and early morning saliva for the identification of SARS-CoV-2. Clin Infect Dis. Published online August 6, 2020. doi:10.1093/cid/ciaa1156
21) Ke R, Martinez PP, Smith R, et al. Longitudinal analysis of SARS-CoV-2 vaccine breakthrough infections reveal limited infectious virus shedding and restricted tissue distribution. medRxiv preprint: doi.org/10.1101/2021.08.30.21262701
22) Ministry of Health [Internet]. “Testing for COVID-19,” Ministry of Health, 03 May 2021. Available: https://www.health.govt.nz/our-work/diseases-and-conditions/covid-19-novel-coronavirus/covid-19-data-and-statistics/testing-covid-19#testing
23) U. C. University of Illinois [Internet]. “Shield Testing Data,” University of Illinois, Urbana Champaign, 03 May 2021 [cited 2021 Aug 13].. Available: https://covid19.illinois.edu/on-campus-covid-19-testing-data-dashboard/
24) NZ Herald [Internet]. “Covid 19 coronavirus: Daily saliva testing for border, quarantine workers will be optional,” NZ Herald, 22 Jan 2021 [cited 2021 Aug 13]. Available: https://www.nzherald.co.nz/nz/covid-19-coronavirus-daily-saliva-testing-for-border-quarantine-workers-will-be-optional/FFODUHMKQLRC2CXFA3GV2PLO4Q/
Contact diana@nzma.org.nz
for the PDF of this article
Validation of a molecular assay to detect SARS-CoV-2 in saliva | OPEN ACCESS
View Article PDF
Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is the cause of the current coronavirus disease 2019 (COVID-19) pandemic. Nucleic acid amplification testing using reverse transcription quantitative polymerase chain reaction (RT-qPCR) for SARS-CoV-2 genes is the diagnostic test of choice because it detects extremely low levels of virus within biological fluids. However, the sensitivity and accuracy of such tests are dependent upon the type of specimen, its method of collection and the RT-qPCR test itself.
In the haste to develop a detection test during the onset of the COVID-19 pandemic, specimens from the nasopharyngeal (NP) site were used. Although this specimen has become regarded as the gold standard test to which all others are compared, it has its own limitations. The NP site can be difficult to reach, impacting on NP specimen quality,[[1]] which in turn accounts for significant differences in NP assay sensitivity and increases the risk of false negatives.[[2]] Moreover, the invasiveness of sample collection often causes discomfort and is unpopular with those requiring frequent testing.
The use of saliva as a specimen for detecting SARS-CoV-2 is increasing around the world. Saliva samples perform well compared to NP in detecting respiratory viruses including coronaviruses,[[3–6]] with both SARS-CoV[[7]] and SARS-CoV-2[[1,5,8,9]] being present in high titers in saliva. The advantage of saliva is that it bypasses the need for invasive sample collection. Although it is an attractive option for frequent testing, there has been a cautious approach to saliva testing in Aotearoa New Zealand. The New Zealand Microbiology Network (NZMN) has recommended that any saliva test would need to be validated locally using well characterised samples that were positive for SARS-CoV-2 and that testing be performed in a diagnostic laboratory accredited by International Accreditation New Zealand (IANZ) and aligned with the ISO 15189 quality framework.[[10]] NZMN noted that the accuracy of saliva tests is reliant upon the methods used for saliva collection, the extraction steps employed for the viral RNA and the commercial kit utilised.[[10]]
In September 2020, we established an international collaboration with investigators at the University of Illinois Urbana-Champaign (UIUC), USA, who had developed a simple and rapid direct saliva-to RT-qPCR assay for the detection of SARS-CoV-2.[[11]] The assay omits the common RNA extraction step and utilises a modified method of a commercially available RT-qPCR kit, thus avoiding reagent competition and supply-chain issues. It includes an initial heat step that inactivates the saliva sample, reducing viral transmission risk, as well as enabling accessibility of the viral genome to the PCR reagents. Finally, the subsequent addition of a detergent-buffer mix overcomes the drawback of saliva viscosity. This test has been approved for surveillance testing in a US FDA Emergency Use Authorization (EUA).[[12]]
The aim of the study was to validate the UIUC RT-qPCR protocol for detecting SARS-CoV-2 in saliva in two independent New Zealand laboratories by comparing our results to those measured at the UIUC laboratory (USA). This study used real-world saliva samples that were paired with nasal samples, of mainly NP origin, from participants that were positive and negative for SARS-CoV-2.
Methods
Test details
The validation was performed at Victoria University of Wellington (VUW) in Wellington and IGENZ Ltd in Auckland using both commercially available heat-inactivated SARS-CoV-2 (ATCC® VR-1986HK™) and saliva samples from consenting COVID-19 positive and negative participants located in Chicago and Wisconsin, USA. These samples, obtained by UIUC, were shipped to Aotearoa New Zealand for the studies described in this paper.
The RT-qPCR test used the ThermoFisher Scientific TaqPath[[TM]] COVID-19 Combo kit (Cat. No. A47814; ThermoFisher) and omits the RNA extraction step. The kit includes primers and probes specific to the SARS-CoV-2 nucleocapsid (N), spike (S) and replication (ORF1ab) genes and a spike-in bacteriophage (MS2) gene, with modifications to the manufacturer’s instructions, as previously published.[[11,13]] Controls included a positive control (TaqPath™ COVID19 Control; ThermoFisher Scientific TaqPath[[TM]] COVID-19 Combo kit) at 25 copies per µL, negative control (UltraPure dH2O) and no-template control (heat-inactivated saliva; collected in-house).
Saliva collection
All but five participants had saliva and nasopharyngeal specimens collected contemporaneously. The five non-contemporaneously collected saliva samples were taken following a positive low viral load result obtained from a mid-turbinate swab. These later samples were included to deliberately include paired samples with low viral load as required for FDA EUA validation purposes, which were also included in this current study to ensure that it contained samples of low viral load.
Participants provided a saliva sample by following guidelines that instructed them to allow saliva to collect in the mouth before gently expelling saliva into the collection tube (passive drool method). They then capped their tube and handed it to the healthcare professional or collection-site staff, who placed it into a collection container. A healthcare professional collected the nasopharyngeal (N=142) or mid-turbinate (N=5) sample and transferred the swab into the collection media (per the comparator manufacturer’s instructions). All nasal swabs (defined as both nasopharyngeal and mid-turbinate) were processed in the clinical pathology laboratory at the University of Illinois Chicago Hospital using an FDA EUA reference method for detection of SARS-CoV-2, the Abbott RealTime SARS-CoV-2 assay performed on the Abbott m2000 System with a limit of detection of 2,700 NDU per mL.[[13]]
Saliva samples that had been heat-inactivated at 95°C for 30 min in UIUC were divided into aliquots. Sample aliquots were stored for up to four months at -80oC before transportation from UIUC on dry ice to the two New Zealand laboratories. Even though the status of each aliquot was unknown upon arrival, samples were immediately placed in a random sequence and re-coded. Before testing, the heat-inactivated saliva aliquot was diluted 1:1 (v/v) in a Tris-Borate-EDTA/Tween 20 mix.
A total of 147 paired saliva samples were received. Of these, 33 saliva samples were from people that tested positive for SARS-CoV-2 based on a nasal sample test result (N=28 NP specimens and N=5 mid-turbinate specimens). The remaining 114 saliva samples were from people that tested negative with SARS-CoV-2 based on a nasal sample test result (N=73 NP specimens and N=41 mid-turbinate specimens). One of these NP samples that tested negative for SARS-CoV-2 did test positive for SARS-CoV-2 in its accompanying saliva sample. The 34 paired samples that tested as positive either in both the nasal (NP or mid-turbinate) and saliva specimen (N=32), or in the nasal (N=1) or saliva (N=1) specimen only, are listed in detail in the results. Of the participants that tested positive (in either test), 26 were symptomatic and eight were asymptomatic.
Analytical validation
The limit of detection (LOD) was determined by performing three separate experiments in triplicate, whereby the ATCC® SARS-CoV-2 control was serially diluted in heat-inactivated saliva from 192 to 0.75 viral copies per µL. The LOD was defined as the lowest concentration (and highest quantification cycle (Cq) value) at which at least two out of the three genes (ORF1ab, N and S) in all replicates were detected. The repeatability of the LOD was tested with 60 replicates at the LOD concentration to determine whether all 60 replicates would test positive. As a final measure of LOD, 20 heat-inactivated saliva samples spiked with 1x, 2x, 5x and 10x LOD concentrations of ATCC® SARS-CoV-2 control, and 20 heat-inactivated saliva samples negative for SARS-CoV-2 were tested blindly.
The stability of ATCC® SARS-CoV-2 control in heat-inactivated saliva was determined by preparing 12 replicates of saliva spiked with 1x, 2x, 4x, 8x and 16x LOD concentrations. Each of the replicates were divided into four groups and used either fresh or after one, two or three freeze-thaw cycles.
The cross-reactivity of Middle East respiratory syndrome coronavirus (MERS-CoV; ATCC® VR-3248SD™) was tested by preparing five replicates at 100 and 1,000 viral copies per µL of heat-inactivated saliva. These results were compared to heat-inactivated saliva spiked with ATCC® SARS-CoV-2 control at 10 viral copies per µL.
Diagnostic validation
A positive result was defined as any sample with a Cq value of <39 for two or more of the SARS-CoV-2 gene targets and the spike-in control gene (MS2). An indeterminate test was defined as any sample with a Cq value of <39 for only one of the SARS-CoV-2 gene targets and the spike-in control gene (MS2). Samples that gave an indeterminate result were re-tested when sufficient sample allowed. A negative result was defined as a sample with a Cq value >39 for all three of the SARS-CoV-2 gene targets and a spike-in control gene (MS2) Cq value of <39. An invalid result was defined as any situation where either the spike-in control gene (MS2) had a Cq value of >39, the positive control had a Cq value of >39 or the negative control had a Cq value of <39.
The reproducibility of the assay was determined by processing, in three different laboratories (VUW, IGENZ and UIUC), 147 saliva samples in blinded experiments from individuals that were positive (N=33) and negative (N=119) for SARS-CoV-2. The Cq values for each gene in every sample were compared. The VUW results were used to compare with UIUC results for diagnostic validation.
The diagnostic sensitivity of the test was determined by comparing saliva assay results with those of contemporaneously collected nasal swabs. Thirty-three positive SARS-CoV-2 samples, each paired with either an NP (N=28, 85%) or mid-turbinate (N=5) sample, and 114 negative samples, each paired with an NP sample, were tested. Concordance of sample test results between all three laboratories was calculated. Statistical analysis for a qualitative test with calculated diagnostic sensitivity, specificity and accuracy with confidence intervals was performed. Test accuracy was calculated using a disease prevalence value (estimated for managed isolation and quarantine (MIQ) facilities) of 0.74% (1,201 confirmed cases out of 162,733 individuals through MIQ facilities).[[14]] Finally, hypothesis tests were carried out to determine whether there was a statistically significant difference between Cq values obtained in different laboratories.
Ethics
Clinical trials comparing results from contemporaneously collected NP or mid-turbinate nasal swabs analysed by an FDA EUA reference method for detection of SARS-CoV-2, and saliva samples analysed by the UIUC assay (covidSHIELD), were approved by the Western Institutional Review Board, USA (WIRB 20202884). All participants gave written and informed consent both to the collection of all saliva samples and their use at external organisations for the purpose of test validation.
In-use experience of the assay
The assay was accredited for surveillance use in December 2020. The ease of sample handling and rate of inhibited samples was assessed by failure of detection of the spike-in control, MS2.
Results
Analytical validation
The LOD of SARS-CoV-2-spiked heat-inactivated saliva samples in three replicate experiments was <0.75 viral copies per μL (or 1.875 viral copies per reaction) (Figure 1). A positive result was given to all replicates at 0.75 viral copies per μL using the diagnostic criteria of two or more genes being detectable.
The LOD of <0.75 viral copies per μL was confirmed by all 60 replicates being detected as positive of SARS-CoV-2 (data not shown) and by a blinded experiment whereby all SARS-CoV-2-spiked samples were detected as positive at concentrations ranging from 1 to 10x LOD, and all non-spiked samples were detected as negative (Figure 2). Freeze-thaw cycles did not change assay results, with only one replicate of the ORF1ab gene at 1x LOD concentration at Cq of >39 (data not shown).
The assay showed no cross-reactivity with a similar virus, MERS-CoV. All heat-inactivated saliva samples spiked with MERS-CoV had Cq values of >39, even at high concentrations of 1,000 viral copies per μL (Figure 3).
Figure 1: Three replicate experiments of SARS-CoV-2 spiked heat-inactivated saliva serially diluted from 192 to 0.75 viral copies per μL. The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike-in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included. Dashed line indicates the limit of detection (LOD) for each replicate experiment.
Figure 2: A blinded experiment of heat-inactivated saliva samples that were either spiked with SARS-CoV-2 at 1 to 10x LOD concentrations (n=5 replicates per concentration; blinded positives) or not spiked (n=20 replicates; blinded negatives). The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike-in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included.
Figure 3: Cross-reactivity experiment where MERS-CoV spiked heat-inactivated saliva was not detected at high concentrations (n=5 replicates at 100 and 1,000 viral copies per μL), while SARS-CoV-2 spiked heat-inactivated saliva was detected in every sample (n=50 replicates at 10 viral copies per μL). The Cq values for the three SARS-CoV-2 specific genes (N, ORF1ab and S), as well as for the spike in control gene (MS2), are shown. A Cq value of 40 was given to undetermined (absence of gene) results. This was the cycle number in which the reaction was stopped. Positive (PC), negative (NC) and no template (NTC) controls were included.
Diagnostic validation
Positive and negative SARS-CoV-2 saliva samples assayed are displayed in Table 1. The Cq values for the 113 concordant negative samples were >39 (data not shown). Thirty-two of 33 saliva samples that were paired with a positive nasal sample (NP and mid-turbinate) were detected as positive in the saliva test, resulting in a sensitivity of 97.0% (95% CI 84.2–99.9%). One saliva sample paired with a negative NP sample tested as positive, resulting in a specificity of 99.1% (95% CI 95.2–100%). Thus, the overall test concordance between saliva and nasal specimens was 98.6%, and the saliva test accuracy was 99.1% (95% CI 95.9–100) (Table 2).
Of those paired with NP specimens only, 27 out of the 28 samples that were positive in the NP test were detected as positive in the saliva test, resulting in a sensitivity of 96.4% (95% CI 81.7–99.9). One saliva sample that was paired with a negative NP sample tested as positive, resulting in a specificity of 98.6% (95% CI 92.6–100%). Thus, the overall test concordance between saliva and nasal specimens was 98.0%, and the saliva test accuracy was 98.6% (95% CI 94.1–99.9) (data not shown).
Table 1: Nasal Swab PCR as the reference test versus the saliva test.
Table 2: Diagnostic statistics for nasal swab PCR as the reference test versus the saliva test. The accuracy was calculated using a population prevalence of 0.74%, as estimated in MIQ.[[14]]
The Cq values for all 33 positive SARS-CoV-2 samples are displayed in Table 3. The one saliva sample (#34) taken from a symptomatic participant that tested positive and was accompanied by a contemporaneously collected negative NP specimen is also included in Table 3. The variability of the 2^(-△Cq) values between the two New Zealand laboratories was 0.24, 0.41 and 0.29 for the N, ORF1ab and S genes, respectively. A t-Test performed on the Cq values between the laboratories demonstrated no statistically significant differences. The low sample variability of 2^(-△Cq) values (particularly those stored in different aliquots; data not shown) between the laboratories confirmed high reproducibility.
Two samples with very low viral load, #6 and #33, were received for testing in the New Zealand laboratories. The samples were tested immediately in the VUW laboratory but there was a delay of one week in sample processing at the IGENZ laboratory, which resulted in Cq values of >39. Given the positive results obtained from these specimens at VUW and the similarity in Cq values between the two New Zealand laboratories in all samples that were processed promptly, samples #6 and #33 were assumed to be degraded and IGENZ data were omitted from Table 3. Sample #33 had a very low viral load and was called as indeterminate by all three laboratories. Repeat testing (#33 re-test) by UIUC on the original sample showed all three genes as being detected and was subsequently recorded as a positive result. There was insufficient sample for repeat analyses at VUW. However, when the degraded sample was repeated by IGENZ, no genes were detected. Again, given the close concordance of all other results, the absence of detectable genes was attributed to deterioration of the sample over time of storage and during transportation (>4 months).
An additional 20 non-paired saliva samples (N=10 positive and N=10 negative) were also tested and exhibited 100% concordance between the three laboratories (Supplementary Figure 1).
Table 3: Concordance of Cq values of 34 positive SARS-CoV-2 saliva samples analysed independently in two New Zealand laboratories (Victoria University of Wellington (VUW) and IGENZ) and one US laboratory (UIUC) in blind experiments. Saliva samples were processed immediately after collection at UIUC but 4–6 months later in the New Zealand laboratories. A dash indicates data omission due to assumed sample degradation. A blank entry denotes that the sample was not re-tested in that laboratory due to lack of sample volume. All samples had paired nasal swabs (nasopharyngeal NP, or mid-turbinate MT) as listed, and all but one nasal sample were called positive. (Pos = Positive, Neg = Negative, Indet = Indeterminate). Asterisk indicates a discordant result between the nasal and saliva sample. View Table 3.
In-use experience
To 1 September 2021, we have tested 13,304 saliva samples. Saliva viscosity that required manual pipetting only affected two specimens, both from the same participant. No sample exhibited assay inhibition.
Discussion
This study is the first to diagnostically validate a saliva test for SARS-CoV-2 in Aotearoa New Zealand. It used real-world paired saliva and NP samples from COVID-19 infected individuals, some of whom were exhibiting symptoms of COVID-19 infection (others were asymptomatic). Previous reports of sensitivities of SARS-CoV-2 detection in saliva are shown to be variable while specificity has been more consistent.[[15]] This emphasises the need for strict collection protocols and well-validated tests. The sensitivity (97.0%), specificity (99.1%) and accuracy (99.1%) of this UIUC saliva RT-qPCR test are very high, with the 98.6% concordance with NP and mid-turbinate specimens in all three laboratories. False-negative or false-positive results were not a test performance issue. The one positive saliva sample that was associated with a negative NP specimen suggests a difference in the timings of viral infection at the two anatomical sites during early or late disease. Overall, these results revealed the UIUC saliva RT-qPCR test to be a highly reproducible method for SARS-CoV-2 detection in both New Zealand laboratories. Moreover, the analytical validation using spiked saliva also exhibited a high sensitivity and specificity, as well as a lower LOD than that of another local NP qPCR assay reported recently (<2 versus ~10 copies per reaction, respectively).[[16]] The correlation of the Cq values for the three SARS-CoV-2 genes tested in each sample independently between the three laboratories was extremely high. This is particularly encouraging given the time between sample collection in the USA and processing in the New Zealand laboratories.
The recent Simpson–Roche report on the country’s COVID-19 testing strategy called for broadening the range of testing methods and recommended introducing saliva testing to increase the acceptability of testing across workforces in the community.[[17]] The availability of such an accurate saliva assay for selected testing situations, such as workplace testing, would significantly compliment the current NP testing used for New Zealand’s public health response. The low LOD, as well as the inclusion of samples from asymptomatic individuals, confirms that this assay can detect infection in asymptomatic and symptomatic people. Early detection enables individuals to be isolated quickly, reducing the risk of transmission.
It should be emphasised that, in addition to the specific RT-qPCR test being used, appropriate saliva collection and preparation procedures are essential.[[11]] This is important due to varying sample viscosity, which if not mitigated can make aliquoting difficult, particularly when using robotic equipment. Moreover, those undertaking saliva testing need instructions on adequate hydration and the need to abstain from food or drink, other than water, for an hour before sample collection. Additionally, there have been concerns about exogenous substances causing assay inhibition. However, the testing of several candidate substances, including nasal spray, different mouth lozenges, nicotine and mouthwash, revealed that only toothpaste (and in only one of three samples) was associated with saliva assay inhibition.[[12]] This is supported by our experience, as after providing detailed collection instructions and compliance, we have yet to encounter saliva samples where the PCR reaction has been inhibited.
This study does have limitations. At the time that this study was performed, there was no community transmission of SARS-CoV-2 in Aotearoa New Zealand. Therefore, it was impossible to obtain locally collected samples. Diagnostic validation using paired contemporaneously collected samples was essential to enable diagnostic validation of this test. This was required as part of IGENZ accreditation to ISO 15189 standards by International Accreditation New Zealand. Collaboration with UIUC enabled this validation study to take place. Although the number of positive pairs was limited, it was similar to many other studies.[[15]] We acknowledge that an increased number of positive pairs would enable greater understanding of variation between the two sample sites. The high analytical sensitivity and concordance, particularly with the NP samples, are specific to this assay and its sample preparation methods and cannot be taken as an indication of other assays’ performance. Saliva samples were obtained after comprehensive advice on hydration and avoiding food and drink. Assay performance may not be the same if collection advice is not followed.
Saliva is likely to participate in SARS-CoV-2 transmission due to the virus replicating in oral epithelial and salivary gland cells.[[18]] As saliva contains large numbers of oral epithelial cells, the detection of SARS-CoV-2 in this specimen is indicative of local virus production and does not rely on the virus passing through the oropharynx from the nasopharynx. Different replication rates at either site may result in a sample from one being positive when the other site is negative.[[1,19,20]] Moreover, there is emerging evidence that breakthrough infections in some people vaccinated with the Janssen Ad26.COV2.S COVID-19 and Pfizer/BioNTech vaccines elicit tissue compartmentalisation, whereby SARS-CoV-2 is detectable only in saliva and not in the nasal passages.[[21]] Our results support this disparity in viral load between tissue sites, but prospective studies are required to understand how frequently this occurs and how it impacts on diagnostic test performance.
Supply-chain issues, in particular the reagents and consumables required for RNA extraction, have hampered the testing of NP samples during the pandemic. The UIUC protocol bypasses the RNA extraction step and, in doing so, removes the supply-chain issues associated with this step. Furthermore, self-collection of saliva samples reduces the need for health professionals at collection sites and the heat-inactivation step reduces the risk of exposure to medical laboratory workers.
The country has now completed more than three million NP tests,[[22]] which is similar to the number of saliva tests conducted at UIUC.[[23]] SARS-CoV-2 testing used for our public health response needs to be scalable overnight, from the baseline testing of ~3,000–5,000 per day to more than 30,000 per day during possible community outbreaks.[[22]] This responsiveness has been achieved using the NP test. The NP swab remains the choice of the Ministry of Health for routine public health testing. However, a role for saliva testing in situations where high-frequency testing is required is now accepted.[[24]] This saliva test is also highly scalable and over 10,000 samples could be processed in one diagnostic laboratory in a single day.
Conclusions
The UIUC RT-qPCR has been tested locally and has been found to be an assay with high analytical and diagnostic sensitivity. It showed 99.1% accuracy and 98.1% concordance to that of nasal swabs in all three independent laboratories. In-use experience to date has not encountered either aliquoting problems or inhibited reactions. As a non-invasive test, it has significant appeal where high-frequency testing is required.
Supplementary Material
Supplementary Figure 1: Twenty saliva samples (10 positive and 10 negative) processed blind in triplicate by both New Zealand laboratories (data for Victoria University of Wellington shown) demonstrated 100% concordance with UIUC saliva results. Positive (PC), negative (NC) and no template (NTC) controls were included.
Summary
This paper presents the validation results of a qPCR test that was developed at University of Illinois Urbana-Champaign (UIUC) for non-invasively detecting the SARS-CoV-2 virus in saliva and tested in Aotearoa New Zealand laboratory. We used saliva samples that were collected from individuals that had also had nasal swabs taken at the same time. The nasal swabs for just over a third of these people were positive for SARS-CoV-2. Our results showed that the UIUC qPCR test is highly accurate (99.1%) for detecting SARS-CoV-2 in saliva and can detect very low copy numbers of SARS-CoV-2 in saliva. This UIUC qPCR for SARS-CoV-2 is as accurate as the qPCR tests used for detecting SARS-CoV-2 in nasopharyngeal samples in New Zealand. These results confirmed that this reliable option for SARS-CoV-2 testing, including for diagnostic testing for asymptomatic people requiring regular screening.
Abstract
Aim
To validate a reverse transcriptase-quantitative polymerase chain reaction (RT-qPCR) assay to detect SARS-CoV-2 in saliva in two independent Aotearoa New Zealand laboratories.
Method
An RT-qPCR assay developed at University of Illinois Urbana-Champaign, USA, was validated in two New Zealand laboratories. Analytical measures, such as limit of detection (LOD) and cross-reactivity, were performed. One hundred and forty-seven saliva samples, each paired with a contemporaneously collected nasal swab, mainly of nasopharyngeal origin, were received. Positive (N=33) and negative (N=114) samples were tested blindly in each laboratory. Diagnostic sensitivity and specificity were then calculated.
Results
The LOD was <0.75 copy per µL and no cross-reactivity with MERS-CoV was detected. There was complete concordance between laboratories for all saliva samples with the quantification cycle values for all three genes in close agreement. Saliva had 98.7% concordance with paired nasal samples: and a sensitivity, specificity and accuracy of 97.0%, 99.1% and 99.1%, respectively.
Conclusion
This saliva RT-qPCR assay produces reproducible results with a low LOD. High sensitivity and specificity make it a reliable option for SARS-CoV-2 testing, including for asymptomatic people requiring regular screening.
Author Information
Acknowledgements
Correspondence
Correspondence Email
Competing Interests
1) Wyllie AL, Fournier J, Casanovas-Massana A, et al. Saliva or Nasopharyngeal swab specimens for detection of SARS-CoV-2. N Engl J Med. 2020;383(13):1283-6.
2) Richard-Greenblatt MJ, Ziegler M, Bromberg V, et al. Impact of nasopharyngeal specimen quality on SARS-CoV-2 test sensitivity. medRxiv. 2020 Dec 11. doi: 10.1101/2020.12.09.20246520
3) To KKW, Yip CCY, Lai CYW, et al. Saliva as a diagnostic specimen for testing respiratory virus by a point-of-care molecular assay: a diagnostic validity study. Clin Microbiol Infect. 2019;25(3):372-8.
4) To KKW, Chan KH, Ho J, et al. Respiratory virus infection among hospitalized adult patients with or without clinically apparent respiratory infection: a prospective cohort study. Clin Microbiol Infect. 2019;25(12):1539-45.
5) To KK, Tsang OT, Yip CC, et al. Consistent detection of 2019 novel coronavirus in saliva. Clin Infect Dis. 2020;71(15):841-3.
6) Rao M, Rashid FA, Sabri FSAH, et al. Comparing nasopharyngeal swab and early morning saliva for the identification of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Clin Infect Dis. 2021;72(9):352-6.
7) Wang WK, Chen SY, Liu IJ, et al. SARS Research Group of the National Taiwan University/National Taiwan University Hospital. Detection of SARS-associated coronavirus in throat wash and saliva in early diagnosis. Emerg Infect Dis. 2004;10(7):1213-9.
8) Harikrishnan P. Saliva as a potential diagnostic specimen for COVID-19 testing. J Craniofac Surg. 2020;31(6):653-5.
9) Tee M, Ubial P, Ranoa DR, et al. Direct saliva versus conventional nasopharyngeal swab qRT-PCR to diagnose SARS-CoV-2: validity study. Asian J Res Infect Dis. 2021;6(2):37-46.
10) New Zealand Microbiology Network [Internet]. Statement on saliva testing for surveillance of SARS-CoV-2. Summary recommendations (2021) [cited 2021 Aug 13]. Available from: https://www.nzmn.org.nz/assets/NZMN/NZMN-statement-on-saliva-testing-for-surveillance-of-SARS-CoV-2-March-2021-v3.pdf
11) Ranoa DRE, Holland RL, Alnaji FG et al. Saliva-based molecular testing for SARS-CoV-2 that bypasses RNA extraction. bioRxiv. doi.org/10.1101/2020.06.18.159434
12) FDA [Internet]. Emergency Use Authorization for covidShield [cited 2021 Aug 13]. Available from: https://www.fda.gov/media/146314/download
13) Ranoa DRE, Holland RL, Alnaji FG, et al. Mitigation of SARS-CoV-2 Transmission at a Large Public University. medRxiv preprint: doi.org/10.1101/2021.08.03.21261548
14) Ministry of Business, Innovation & Employment [Internet]. Managed isolation and quarantine data [cited 2021 Aug 13]. Available from: https://www.mbie.govt.nz/business-and-employment/economic-development/covid-19-data-resources/managed-isolation-and-quarantine-data/
15) Butler-Laporte G, Lawandi A, Schiller I et al. Comparison of saliva and nasopharyngeal swab nucleic acid amplification testing for detection of SARS-CoV-2. A systematic review and meta-analysis. JAMA Int Med 2021;181(3):353-60.
16) Fox-Lewis S, Muttaiyah S, Rahnama F, et al. An understanding of discordant SARS-CoV-2 test results: an examination of the data from a central Auckland laboratory. NZ Med J. 2020;133(1519):81-8.
17) Simpson H, Roche B, Report of the Advisory Committee to oversee the implementation of the New Zealand COVID-19 Surveillance Plan and Testing Strategy [cited 2021 Aug 13]. NZ Govt Wellington 2020. Available from: https://covid19.govt.nz/assets/Review-of-Surveillance-Plan-and-Testing-Strategy/Final_Report-of-Advisory-Committee-to-Oversee-the-Implementation-of-the-....pdf
18) Huang N, Perez P, Kato T, SARS-CoV-2 infection of the oral cavity and saliva, Nat Med. 2021;27(5):892-903.
19) Miller M, Jansen M, Bisignano A, et al. Validation of a self-administrable, saliva-based RT-qPCR test detecting SARS-CoV-2. Preprint. Posted online June 9, 2020. medRxiv. doi:10.1101/2020.06.05.20122721
20) Rao M, Rashid FA, Sabri FSAH, et al. Comparing nasopharyngeal swab and early morning saliva for the identification of SARS-CoV-2. Clin Infect Dis. Published online August 6, 2020. doi:10.1093/cid/ciaa1156
21) Ke R, Martinez PP, Smith R, et al. Longitudinal analysis of SARS-CoV-2 vaccine breakthrough infections reveal limited infectious virus shedding and restricted tissue distribution. medRxiv preprint: doi.org/10.1101/2021.08.30.21262701
22) Ministry of Health [Internet]. “Testing for COVID-19,” Ministry of Health, 03 May 2021. Available: https://www.health.govt.nz/our-work/diseases-and-conditions/covid-19-novel-coronavirus/covid-19-data-and-statistics/testing-covid-19#testing
23) U. C. University of Illinois [Internet]. “Shield Testing Data,” University of Illinois, Urbana Champaign, 03 May 2021 [cited 2021 Aug 13].. Available: https://covid19.illinois.edu/on-campus-covid-19-testing-data-dashboard/
24) NZ Herald [Internet]. “Covid 19 coronavirus: Daily saliva testing for border, quarantine workers will be optional,” NZ Herald, 22 Jan 2021 [cited 2021 Aug 13]. Available: https://www.nzherald.co.nz/nz/covid-19-coronavirus-daily-saliva-testing-for-border-quarantine-workers-will-be-optional/FFODUHMKQLRC2CXFA3GV2PLO4Q/
Contact diana@nzma.org.nz
for the PDF of this article
Validation of a molecular assay to detect SARS-CoV-2 in saliva | OPEN ACCESS
The aim of the study was to validate the UIUC RT-qPCR protocol for detecting SARS-CoV-2 in saliva in two independent New Zealand laboratories by comparing our results to those measured at the UIUC laboratory (USA).
Subscriber Content
The full contents of this pages only available to subscribers.
Login, subscribe or email nzmj@nzma.org.nz to purchase this article.
