New Zealand’s 2020 response to the rise and spread of COVID-19 has rightly been widely praised.[[1–3]]

However, as in all countries, the unintended effects of measures to control the pandemic on the operation of our health system are potentially dramatic. There is evidence that lockdown policies reduce some demand for healthcare, notably road trauma[[4,5]] and influenza and respiratory viral infections,[[6]] or, as appears to have been the case with respiratory syncytial virus, deferred healthcare, leading to larger outbreaks.[[7]] However, the demand associated with long-term conditions does not go away during periods of lockdown. Instead access to care is simply delayed.

The knock-on effects of this delay for health system access are likely to be large but difficult to forecast. This uncertainty has many risks, not least that efforts to address the backlogs that grow during lockdown periods will potentially exacerbate the New Zealand health system’s long-standing inequities in access, experience and outcome, quite apart from the likely inequitable spread and outcomes from the delta variant. This raises numerous research questions: What precisely changed in provision of healthcare during the COVID-19 lockdown periods? What has happened in the periods since? Is this consistent around the country? Did some services adapt to the challenges more effectively? What particular services and population groups have been most affected? What has been the impact on Māori? Or on Pacific peoples?

We present a new tool and use it to examine at least some of these questions, including recent trends in emergency department presentations.

REACH: how it works, what it shows

In health quality measurement, Klein et al broadly divide measures into two categories: tin openers and dials. Tin-opener metrics tend to open a “can of worms”—that is, create opportunities for potentially fruitful investigations around particular issues—as opposed to dials, which provide a more or less explicit judgment around “good” or “bad” based on a given normative standard.[[8]]

The Health Quality and Safety Commission (the Commission) has developed a simple-to-use tool that operates as a tin opener in order to explore the questions above and assist local services to understand and respond to the likely effects of COVID-19 on New Zealand’s health system. The REACH tool (Rapid Effects Assessment of Measures to Contain COVID-19 on Healthcare) compares expected activity in our health system based on past data with observed activity in 2020 as a way of understanding the effects of COVID-19 and associated public health measures, and the likely directions that activity is going in.

The tool applies Prophet, an open-source forecasting tool,[[9,10]] to New Zealand’s national data collections for admitted and non-admitted patients in our public hospitals. The tool is written in R,[[11]] an open-source programming language and software environment for statistical computing and graphics, which has rapidly become an industry standard, and visualised using the Shiny visualisation platform.[[12]]

REACH was originally developed and trialled on one district health board’s (DHB’s) live data and has since been applied to national collections, the National Minimum Dataset (NMDS) and the National Non-Admitted Patient Collection (NNPAC). REACH can quickly estimate, at multiple levels of granularity, the expected level of activity during the lockdown period based upon historical trends and patterns from three years of national data. The forecast is compared with actual activity to calculate the patterns of difference resulting from the pandemic.

Expected levels of activity in our public hospitals are calculated by applying the Prophet model to historical activity data from 1 January 2017 to 29 February 2020. This approach identifies both seasonal effects (such as seasons, week of the year, day of the week and public holidays) and underlying trends, and applies both to estimated and expected activity, all else being equal.

The granular levels at which REACH works allow this calculation to be undertaken by DHB, age group, sex, ethnic group, specialty, admission type and diagnosis-related group (DRG), or any combination thereof.

REACH has a number of features (Figure 1) to allow quick exploration of data. In the top left of the presentation (Figure 1), activity (such as national acute inpatient admissions in this example) is presented, by day, in two lines: the green line is expected activity based on a forecast from historical data, and the red line is the observed activity for 2020. Together these lines quickly show how 2020 activity was affected by COVID-19 and associated public health measures, including lockdowns of different levels in different areas.

Figure 1: Screenshot of REACH for actual and forecast acute inpatient admissions presentations, Aotearoa New Zealand, July 2019 to February 2021.

On the bottom left (Figure 1) is presented the cumulative effect of any differences between the two. This cumulative effect is calculated as expected activity minus observed activity, so that a positive number indicates lower than expected activity, or a “shortfall.” A rising curve thus shows a shortfall in inpatient admissions growing more than would be expected. A flattening of this curve indicates the system returning to expected levels of activity, and a fall in the curve represents a period of “catch up” where demand exceeds the normally expected level of activity.

On the right-hand side of the presentation (Figure 1) are a number of filter buttons that allow this analysis to be run for specific specialties, admission types, DRGs and groups of DRGs, different ethnicities, sex and age groups. Selection of specific filters allows recalculation of the display—the Prophet model is rerun on the national data collections just for cases that fit the parameters selected.

The central area of the display can be set as either “quick explorer” or “shortfall.” In quick explorer mode, a “light” version of the model is run to allow the user to get a broad though less precise comparison of relative effects on different groupings, such as specialty, ethnic group and so forth, within the selected parameters. In shortfall mode, the full model is used to show effects by week for patients within the selected parameters. These are presented as a detailed table and as a graph (Figure 2).

Figure 2: Screenshot of REACH tool central panel in shortfall mode.

REACH and emergency department presentations

A potentially useful example of REACH’s findings is differences in emergency department (ED) activity in 2020.

Articles about increasing pressures on EDs were widespread in the general media in March 2021 and prior.[[13–15]] These have included concerns about “code blacks” (EDs reaching full capacity and hospitals asking patients to go elsewhere, such as after-hours services) and “ramping” (patients held in ambulances outside EDs). The REACH tool allows us to understand whether these increases were real, sustained and reflect a genuine increase in demand (as opposed to system delays caused by problems with intra-hospital flow).

Data through to February 2021 show a complex picture, highlighting the value of two features of REACH: a robust forecasting method and the ability to investigate activity at more granular levels (ie, “disaggregation”). Figure 3 shows that there was a pronounced reduction in ED attendances during the first lockdown period, some of which was a real reduction in demand (such as trauma, mentioned above). However, it is likely at least some of this was displaced activity—that is, people with long-term conditions having acute episodes that would lead them to seek care from EDs avoided doing so.[[16]] This being so, the likelihood was that these patients would “re-emerge” in the system at some point, potentially with more advanced disease requiring more invasive or intensive treatments, or destabilised disease requiring greater medical input and time to address.

The evidence points to this re-emergence occurring in the last three months before Christmas 2020, with the red line of observed activity showing a trend of unprecedentedly high ED presentations nationally during this period. Not only was the number of these presentations significantly higher than expected for the time of year, but it was higher than historical trends for mid-winter (when the number of presentations tend to be higher) and still increasing at the end of the period.

If we look at the red line alone, the period since January 2021 shows a decrease in ED presentations. However, this apparently good news needs to be considered against what we might have expected to happen. Based upon historical trends, we would have expected a notable decline in ED presentations following the holiday period (the green line in Figure 3), and despite the absolute reduction in actual presentations in more recent months, these remain above both forecast and historical norms.

Figure 3: Actual and forecast ED presentations, Aotearoa New Zealand, July 2019 to February 2021.

A more granular analysis

However, REACH is able to disaggregate by triage level, allowing different insights into what was happening in our EDs.

First, the increased activity seen since October 2020 was concentrated in the most urgent triage levels of 1 and 2, where it remained consistently above expected levels for the period. Triage levels 1 and 2 cover illness or injury which is “immediately life-threatening” (triage 1), or “imminently life-threatening, or important time-critical” (triage 2), and which require immediate simultaneous triage and treatment or maximum of ten minutes triage time, respectively.[[17]]

Figure 4 shows this higher-than-expected level of urgent triage level 1 and 2 activity nationally after October 2020.

Figure 4: Actual and forecast emergency department presentations, triage levels 1 and 2, Aotearoa New Zealand, July 2019 to February 2021.

During the same period there were in fact fewer than expected presentations at the less urgent triage levels of 4 and 5 (which comprise “potentially serious” or “less urgent” issues requiring maximum triage times of 60 minutes or 120 minutes, respectively).[[17]] Thus, any data where all triage levels are aggregated have the effect of hiding the increase in presentations triaged as being potentially immediately or imminently life-threatening. When aggregated together, the lower than expected “potentially serious” or “less urgent” triage level presentations offset or effectively “hide” the increased more urgent activity.

Figure 5 shows this lower-than-expected level of triage level 4 and 5 activity from approximately January 2021.

Figure 5: Actual and forecast emergency department presentations, triage levels 4 and 5, Aotearoa New Zealand, July 2019 to February 2021.

This emphasises that raw aggregated figures can hide details that are hugely important in understanding what is really happening. In short, as Table 1 shows, while the overall figures in the first two months of 2021 were about the same as expected, those of the most urgent and challenging cases were substantially higher.

Table 1: Excess ED presentations compared with historic trends, by triage level and period.

This pattern is not universal around the country. In fact, stark variation in experience between DHBs is notable. For example, one large DHB has had 5,000 more ED presentations since October 2020 than our model would forecast based on 2018/2019 data, whereas another has seen approximately the expected amount when triage levels are aggregated (Figures 6 and 7). However, there is some evidence the differential between triage levels also plays out at a local level.

Figure 6: Actual and forecast emergency department presentations, July 2019 to February 2021, DHB A.

Figure 7: Actual and forecast emergency department presentations, July 2019 to February 2021, DHB B.

A further concern is whether the increased demands on ED would also exacerbate, or reflect an exacerbation of, inequity in our health system. Table 2 shows that across ethnic groups there are different but generally inconsistent patterns that raise questions more than provide clear answers.

Table 2: Excess Triage 1 and 2 ED presentation by ethnic and age groups, October 2020 to February 2021.

Discussion

By comparing, at a granular level, ED activity since the emergence of COVID-19 with historical trends, we can show with a degree of certainty that, at least in some parts of the country, the pressures felt in EDs were a reflection of real, sustained increases in demand above historical norms and cannot simply be ignored as an artefact. However, further questions are raised by this analysis, including whether specific presenting diagnoses drove the increased demand, and whether patient pathways following presentation at the ED have changed.

To answer these questions, deeper analysis of local data is required. For this reason the Commission has been working with DHBs to apply the tool to local data systems that have the advantage of both more recent data and more detail than that contained in the national datasets. REACH is being made available to DHBs and we would advocate all to apply it to their local systems.

The uses of this type of tool are manifold. It can be used to consider different types of admission. Waiting list backlogs are relatively simple to interpret (a backlog is likely to consist of patients needing treatment who have been denied access), whereas acute admission shortfalls may be avoided or delayed admissions, the ramifications of each being different. Further we can identify divergent patterns for different groups of people, including between different ethnic groups, which in some instances seem to have worsened existing inequities.

One opportunity highlighted in the health system reforms is that of more coherent and consistent use of data. We contend that this tool is a good example of the type of development that all parts of the health system should be using on a routine basis as we look to monitor and indeed improve our health system. Monitoring and action around trends and impacts on inequities in healthcare for Māori and Pacific peoples in particular require close attention.