In New Zealand, as in the rest of the world, COVID-19 is diagnosed by SARS-CoV-2-specific nucleic acid amplification tests (NAAT),[[1]] and past infection is determined by serological assays.[[2]] Typically, high-volume NAAT testing is performed in batches and results may take several hours. However, a rapid turnaround time of a test, from arrival in the laboratory to provision of results, is critical to the success of public health interventions, as well as for individual patient management.[[3]]

New Zealand currently has a very low prevalence of COVID-19,[[4]] and laboratories are generally operating well within existing capacity,[[5]] although, even with our current prevalence, increased levels of testing around suspected or actual community cases can strain local laboratory resources. Nevertheless, there is a need to consider what rapid-testing alternatives are available should current epidemiology change, and how these options could fit with SARS-CoV-2 diagnostic testing in the New Zealand setting.

Rapid SARS-CoV-2 antigen tests (RATs) were developed by many manufacturers by mid-2020. RATs promised low cost and scalable diagnostics that can be performed outside of a laboratory. Subsequently, these tests have been assessed and used in high-prevalence settings globally and are recommended in low income settings by the World Health Organization (WHO).[[6,7]] In New Zealand, due to concerns around the quality of manufacturing and accuracy, importation of RATs has been restricted since April 2020;[[8]] these restrictions have recently widened to include molecular point-of-care tests.[[9]]

Early in the pandemic, several commercial molecular assays utilising reverse-transcriptase PCR (RT-PCR) became available with time to results of <1 hour, such as the GeneXpert (Cepheid, USA) and FilmArray (Biomerieux, France), which were integrated into diagnostic workflows in New Zealand. These rapid molecular tests are generally regarded as point-of-care tests (POCTs), although in New Zealand use of the GeneXpert and FilmArray is currently confined to laboratory settings. Later in 2020, ID NOW (Abbott, US) and Cobas Liat (Roche, Germany), molecular assays able to provide results in 20 minutes, also became available to New Zealand laboratories. These platforms are generally able to process one sample at a time, whereas some NAAT tests, such as the Oxsed RaVid Direct (Oxford, UK) reverse-transcription loop-mediated isothermal amplification (RT-LAMP) assay, potentially offer rapid and scalable results.

The aim of this study was to assess the sensitivity and potential utility of five rapid antigen tests and three rapid NAAT tests in our population.

Methods

Rapid antigen tests

Five RAT kits were evaluated:

• Panbio COVID-19 Ag Rapid Test Device (Abbott, US)
• CareStart COVID-19 Ag Rapid Test (AccessBio, US)
• Clinitest Rapid COVID-19 Ag Test (Siemens, Germany)
• Innova SARS-CoV-2 Ag Rapid Test (Innova, US)
• Roche SARS-CoV-2 Rapid Ag Test (Roche, Germany)

These brands were selected because of their widespread use, post-marketing performance data and emergency use authorisations in either the USA or Australia. Specimen types recommended by the manufacturers for the Abbott, CareStart and Clinitest kits are nasopharyngeal swabs or nasal swabs The Roche kit is nasopharyngeal swabs only. The Innova kit is nasal swabs or throat swabs.

Thirty-three SARS-CoV-2 NAAT-positive, frozen, stored patient samples in viral transport media (VTM) were retrieved. Each sample was thawed at room temperature, vortexed for 10 seconds and then prepared as follows:

For rapid antigen testing, 200uL of the sample was aliquoted into the manufacturer’s extraction vessel and further testing proceeded as per the manufacturer’s instruction.

For routine RT-PCR comparison, 200uL of each sample was extracted using the MagNA Pure 96 (Roche) system followed by amplification using the A*STAR Fortitude Kit 2.1 (Mirxes, Singapore) on a Lightcycler 480 (Roche, Germany). The A*STAR Fortitude Kit 2.1 detects two different proprietary regions of the orf1ab gene.

One positive sample (with an initial Ct value of 16.1) was diluted tenfold out to 10[[-6]] in universal transport media (UTM). The original sample and resulting dilution series were tested by RT-PCR and by each of the five RAT kits.

Rapid NAAT testing

For the rapid NAAT tests, 50 additional SARS-CoV-2 NAAT-positive, frozen, stored nasopharyngeal samples were prepared as above.

Cobas Liat testing was performed using 200uL of sample, according to the manufacturer’s instructions. The same volume of sample was used for the ID NOW, as direct dry patient swabs were unavailable, although the manufacturer’s recommendation for this assay is that it be used for testing nasal, throat or nasopharyngeal swabs directly without elution in VTM.

For routine RT-PCR comparison, 200uL of each sample was extracted on the Kingfisher Flex system (Thermofisher, US), as per the manufacturer’s recommendation for extraction for the Oxsed RT-LAMP, using Machery Nagel (Machery-Nagel, Germany) kits followed by amplification using A*STAR Fortitude Kit 2.1 on a Lightcycler 480.

For Oxsed RT-LAMP, an aliquot of the above extracted RNA was prepared according to manufacturer’s instructions; reaction tubes were incubated in a heat block at 65oC for 30 minutes and colour changes determined in conjunction with “before and after” photographs.

Analysis

RAT (Figure 1) and rapid NAAT results (Figure 2) results were interpreted according to the manufacturers’ instructions by two independent operators. When these operators disagreed, a third operator determined the result.

Sensitivity of all the evaluated kits was compared with the qualitative result of the orf1ab region 1 target (FAM channel) of the A*STAR Fortitude Kit 2.1 RT-PCR using Chi square tests; Ct was used as a surrogate of viral load. Where samples tested negative using the A*STAR kit on retrieval from storage, they were excluded from analysis.

Results

Rapid antigen tests

For the 39 samples tested using RAT, the overall average sensitivity compared with RT-PCR was 39.0% (95% CI 30.3–43.9%, p=0.01), with individual results as shown in Table 1. All RAT kits showed a sensitivity of 100% where the RT-PCR Ct value was <25 (n=9), but RAT sensitivity dropped for all kits where samples had a Ct value of ≥25.0 (p<0.001). Clinitest had the highest sensitivity overall. However, there was no significant difference between the sensitivity of the tests (p=0.178). None of the kits detected SARS-CoV-2 in any of the samples with a Ct value ≥35.0.

Figure 1: Clinitest rapid antigen test: dilution series.

Table 1: Performance of RATs compared with reference RT-PCR.

Rapid NAAT testing

For the 50 stored positive samples tested using rapid NAAT, six samples tested negative on repeat by the A*STAR Fortitude Kit 2.1 assay and were excluded from further analysis.

The overall sensitivity compared with RT-PCR was 92% (95% CI 86.6–95.8, p=0.07), with individual performance presented in Table 2.

All three tests showed a sensitivity of 100% where the RT-PCR Ct value was <29, (n=29), but sensitivity dropped for the ID NOW and Oxsed assays for samples with a Ct value of ≥30.0 (p=0.042). In contrast, the Cobas Liat detected all 44 (100%) samples.

Table 2: Comparison of Cobas Liat, ID NOW and Oxsed assays with reference RT-PCR.

Figure 2: Oxsed RT-LAMP results.

Discussion

We evaluated the sensitivity of rapid antigen tests and rapid molecular assays in order to determine potential utility in the New Zealand setting. Specificity was not assessed for this evaluation, as limited kits were available; internationally, this has been found to be high across a range of devices, though the positive predictive value would be highly variable dependent on prevalence of infection in the setting in which the tests were used in New Zealand.[[10]]

We found that the Cobas Liat, which provides results in 20 minutes and uses RT-PCR technology, had equivalent sensitivity to our standard high-throughput RT-PCR assay. This has been demonstrated elsewhere and, notably, several authors have demonstrated concordance with the GeneXpert, a RT-PCR system that provides results in 40 minutes and is already in use in our laboratory.[[11,12]] The major limitation of the Liat is that only one sample can be tested at a time, which considerably limits throughput, although the Liat also has significant advantages: the simplicity, speed and portability would suit remote settings (eg, rural hospitals without molecular laboratories) when rapid results are needed. However, at >$10K per machine, financial considerations could impede scalability.

Otherwise, all evaluated assays were less sensitive than the standard RT-PCR used in our laboratory, which is evident at lower viral loads. RATs demonstrated a higher limit of detection compared with the ID NOW and Oxsed RaVid assays. Overall our results are consistent with many other reports that demonstrate sensitivity is related to viral load proxies such as Ct value in a laboratory or a field setting for these assays and others,[[6,10,13–17]] though comparative data for the Oxsed RaViD RT-LAMP assay against RT-PCR are limited.[[18]] Therefore, although we tested a narrow range of assays, we believe our findings and conclusions are more generalisable.

One note of caution: there is a high degree of variability in performance across RAT brands and some will perform worse than those we have presented. [[7,10]]

There are technical limitations to our study. A limited range of kits were available for the evaluation, which resulted in small numbers and consequent statistical challenges. Furthermore, limited residual sample volume precluded direct comparison of RATs and rapid NAATs. Importantly, for the RATs and rapid NAAT assays, we were constrained to using samples in VTM at a standard volume of 200uL (the average volume of fluid absorbed by a flocculated swab), as all swabs received at our laboratories are inoculated into VTM to optimise detection of SARS-CoV-2 using existing platforms and dry swabs were unavailable. The RAT or ID NOW manufacturers do not recommend the use of VTM, as there is a dilution effect when additional kit buffer is added to this volume. Accordingly, in theory, the sensitivity of the RATs and ID NOW kits may differ from that obtained by the manufacturers’ recommended methods; however, in practice our findings are consistent with those reported elsewhere for the RATs and for ID NOW using either dry swabs (the manufacturer’s recommended method) or VTM.[[6,10,13–17]]

Conversely, we were unable to evaluate the manufacturer’s alternate “extraction-free” Oxsed method, as VTM affects assessment of the end point colour and our results obtained using RNA extraction likely demonstrate higher sensitivity than we would have obtained using the faster method; further evaluation without extraction would be needed if this technique were under further consideration.

Our results highlight the importance of assessing the performance of RATs or rapid NAATs in the intended setting because epidemiology and public health approaches differ globally. In New Zealand, which is currently pursuing an elimination strategy, detection of any COVID-19 case is of paramount public health importance as it may indicate the presence of unknown transmission links. After applying our findings into three New Zealand scenarios with data from our laboratory, we found that the average clinical sensitivity of the RATs/NAATs (with the exclusion of the Cobas Liat) for detecting SARS-CoV-2 infections identified by RT-PCR would be 74.1%/93.9% in the initial outbreak of March–June 2020, 58.3%/92% in the Auckland outbreak in August 2020 and 55.7%/88% in individuals testing positive in managed isolation facilities between December 2020–January 2021.

Since August 2021, the Abbott, CareStart and Roche RATs have been included in the Australian Register of Therapeutic Goods (ARTG) for legal supply in Australia,[[19]] while the CareStart RAT has been authorised for emergency use by the US Food and Drug Administration (FDA).[[20]] In New Zealand, all RATs have been prohibited for import since April 2020 due to concerns around their performance in our low-prevalence setting.[[8]] This has recently been widened to include molecular point-of-care devices (outside diagnostic laboratories).[[9]] Our local data support these restrictions for RATs; we found an overall sensitivity of 39% for the samples we tested; none of the kits would meet the WHO interim guidance for rapid antigen device (RAD) diagnosis of SARS-CoV-2 infection (≥80% sensitivity and >97% specificity),[[7]] and they are not suitable for use as a standalone test.

RATs do offer potential advantages as diagnostic tests for SARS-CoV-2 infection in both high-disease-prevalence and low-technology settings—they provide rapid results, are relatively inexpensive compared to nucleic acid testing and can be used outside of the laboratory,[[13]] and multiple RATs can be performed simultaneously.[[7]] Consideration could be given to their use as an adjunct to RT-PCR in the event of a widespread outbreak in New Zealand where laboratories are unable to provide RT-PCR results in a timely fashion.[[3,10]] However, their results will have to be confirmed with RT-PCR testing even in this setting if an elimination strategy is being pursued, given performance characteristics and likely prevalence.[[10,21]] In our current setting, RATs could have a role in parallel with confirmatory RT-PCR in specific scenarios, such as within a relatively isolated group of individuals with acute respiratory symptoms where rapid RT-PCR tests are not readily available (eg, on a fishing boat or in a remote community), in which case a positive SARS-CoV-2 result would allow rapid escalation of interventions while awaiting definitive results.[[22]] There has been recent interest in the potential use of RATs at points of entry to New Zealand (eg, in screening incoming passengers), and there may be value in testing individuals from very high-risk countries in parallel with confirmatory RT-PCR, to determine their disposition. However, testing all arrivals indiscriminately is not advisable due to the low positive and negative predictive value of this approach[[7,23]] and the associated operational and logistical challenges.

It is likely these technologies will have wider utility as the public health strategy changes and COVID-19 becomes more prevalent in our population; this is because sensitivity will be less critical for diagnostic tests, positive predictive value will improve and the advantages of near-patient care may outweigh the risks associated with their use. At that stage, there may be a role for the use of RATs for single tests for mass screening, such as at large public events, and as part of a diagnostic algorithm in healthcare settings. Under these circumstances, regular and frequent testing for healthcare workers or in residential care facilities may also be appropriate to reduce the risk of transmission to patients in hospitals or nursing homes,[[22]] as in higher-prevalence settings frequency of testing has been shown to increase sensitivity.[[3]] However, in New Zealand’s current low-prevalence state, trying to offset the decrease in sensitivity by increasing the testing frequency of the same individual would likely give a false sense of security.

The ID NOW and Oxsed, which are both rapid isothermal NAAT assays, performed better than RAT and could be considered if increased national testing capacity were to be required and limited reagents and instruments were available for RT-PCR. However, each test is suited to a different setting: The ID NOW provides results within 20 minutes but, similarly to the Liat, only allows single sample testing.[[24]] The Oxsed assay is scalable if performed directly without extraction and would take approximately 40 minutes to results, but this assay is not an integrated sample-to-answer machine like the ID NOW or Liat, so it requires considerable scientific skill and resource to run, similarly to a batched RT-PCR test.

In summary, we found that the Cobas Liat, a RT-PCR POC platform that delivers a result in 20 minutes, performed equivalently to our standard RT-PCR, whereas the RATs and rapid isothermal NAAT assays demonstrated reduced sensitivity, which limits their utility in New Zealand’s currently very low prevalence setting. RATs and rapid isothermal NAAT assays may be useful in the event of a widespread outbreak in New Zealand, when laboratories may be overwhelmed and unable to provide prompt results with existing resources. With time, changes in public health strategy and disease prevalence may also widen the utility of these tests.

Addendum

This study was written prior to New Zealand’s August 2021 COVID-19 outbreak. Since then, the prevalence of COVID-19 has risen in Auckland, and while other regions are still pursuing an elimination strategy, it is expected that outbreaks will occur across the other regions in short order.  It is apparent that rapid antigen testing will be an integral part of testing strategy in regions with a high prevalence of infection over the coming months to supplement PCR testing, particularly for asymptomatic surveillance testing in health care and other settings.