In May 2009 the Ministry of Health formally announced six national health targets for public hospitals in Aotearoa New Zealand.1 One of these was the Shorter Stays in Emergency Departments target, which states that 95% of patients should be admitted, discharged or transferred from an emergency department (ED) within six hours of arrival.2 Performance against this target would be reported publicly according to district health board (DHB).

This policy was introduced on the basis of international evidence that suggested an association between ED and hospital overcrowding (reflected by long waits for admission to hospital from ED) and poorer outcomes for patients.3–6 Time-based ED targets were initially introduced in the UK’s National Health Service in 2001,7 and have since been introduced both in New Zealand2 and in Australia.8 There is debate as to whether or not ‘Targets’ are helpful or harmful,9 and the effect that time-based targets have upon patient care is uncertain. Some studies suggest better outcomes for patients when such targets were introduced,10,11 and other evidence suggests that targets may distort clinical and management priorities, diverting attention from clinical care.12

The Shorter Stays in ED (SSED) National Research Project is a mixed methods study within New Zealand public hospital EDs, investigating the relationship between the introduction of a time target for the completion of care in ED and quality of care.13 An important goal was to understand the effect that introducing a process measure such as an ED length of stay (LOS) target had on other aspects of care for the healthcare consumer. A key research question for this broader project was: “Is there any change in clinically relevant outcomes after the target was introduced?”13

To explore this research question, a number of clinical indicators of quality of care were identified from a literature review, and a stakeholder analysis process was conducted.14 The indicators were chosen to cover both ED and hospital outcomes to determine whether target implementation had effects on care quality beyond the ED.

The primary outcomes of interest in this study were the times to thrombolysis for ST elevation myocardial infarction (STEMI), time to antibiotics for severe sepsis, time to analgesia for moderate or severe pain, time to theatre for fractured neck of femur (NOF) and time to theatre for appendicitis. The secondary outcomes of interest were the adequacy of care with respect to each indicator condition; appropriate thrombolysis for STEMI, appropriate antibiotics for the site of infection for sepsis, adequate analgesia for pain, time to theatre <24 hours for NOF and perforated appendix at operation for appendicitis.

Methods

Study design and setting

The overall SSED study has previously been described in detail,13 and involved all emergency departments across New Zealand. This sub-study was a retrospective, pre- and post-intervention study, which linked administrative data with chart review using the National Health Index number, a number unique to each New Zealand citizen used to record health visits. Four case study site hospitals were selected based on a combination of factors, including: populations with a higher average proportion of Māori people, geographic diversity and initial target performance.13 There were two urban major referral academic hospitals, one urban district academic hospital and one major regional hospital, serving a combined population of 1.5 million people and a combined annual ED census of 290,000 in 2010.

Selection of participants

All patient visits to the study sites that were recorded in the New Zealand Health Information Service (NZHIS) database from first January 2006 to 31st December 2012 were identified, along with the visit date and demographic data. Data from 2009 were excluded as this was the year in which the target was introduced. Using the random number generator function in Microsoft Excel®, a random sample of visits sufficient to meet the required sample size for each outcome was taken for each of the time periods 2006–08 and 2010–12 (pre- and post-target introduction). For the ‘analgesia’ outcome, we sample data from all ED presentations nationally. Due to a change in coding practice, NZHIS only supplied data from 1/7/2006 for this outcome. To evaluate the remaining condition specific outcomes the International Classification of Diseases (ICD-10-AM) codes for STEMI, sepsis, fractured neck of femur and appendicitis were used to identify potentially eligible cases at one of the case sites (Table 1). The randomly selected visits were linked to site-specific patient information management systems that included data on the times for each patient journey within a hospital from presentation, triage and assessment, to admission and discharge. Finally, the case notes of the selected visits were reviewed, and the relevant clinical data were extracted by researchers. Prior to data collection a data dictionary that defined each data field and how to classify missing or incomplete data was developed. Smart electronic data extraction forms (Microsoft Excel®) were developed based on the data dictionary.15 Built-in validation rules were set for cells in the data extraction form to prevent incorrect entries. To avoid bias in selecting and classifying cases, formulas based upon clinical, laboratory and radiological data were used where relevant to ascertain whether patients met the entry criteria. The data extraction forms were piloted prior to data collection, and 15–20% of the data for the specific condition outcomes were independently checked for accuracy by a second data collector who was blinded to the initial data extract. Due to logistic issues, this step was not possible for the eligible records for the analgesia outcome, where data was collected nationally. Patients were excluded from analysis if the clinical notes were not available or were incomplete, if the patient was transferred from another hospital, if the episode did not involve an ED stay or if the data relevant to the outcome of interest was missing.

Table 1: Definitions of outcomes, clinically important differences and sample size requirements.

‍

ICD = International Classification of Diseases version 10, ED=Emergency Department, STEMI=St Elevation Myocardial Infarction, CT=Computerised Tomography, GCS=Glasgow Coma Scale. SIRS=Systemic Inflammatory Response Syndrome *Based on a literature review and expert opinion.

Sample size

We determined a clinically important difference in time to treatment for each condition, based on a literature review and expert opinion prior to starting the data collection for these outcomes. Sample sizes were calculated for each outcome to detect this difference, with a power of 90% and an alpha of 0.05, based on the mean and standard deviation (SD) of data piloted at a separate site prior to commencing the study (Table 1).

Analysis

Medians with interquartile range (IQR), means with 95% confidence intervals (95%CI) and proportions (95%CI) were used to describe the data. To investigate changes in treatment times before and after target introduction, a general linear model was fitted with the log of the time plus 0.5 as the outcome. The log transformation was necessary, as treatment time data was skewed. The explanatory variables included were period (pre- or post-target), ethnicity (Māori, Pacific or other), deprivation score (a standard measure of socioeconomic deprivation used in New Zealand based on small geographic areas of domicile16), age and severity of condition where appropriate (Pain outcome). Back transformed least square means (LSM) of time were used to describe the effect size. For the binary outcomes, the analyses were the same with the exception that a generalised linear model was used with a binary distribution and a log link. The explanatory variables included were as above. Unadjusted time-based and descriptive analysis was performed using SPSS v21, Armonk, New York, USA. Multivariable analysis was performed using SAS/STAT version 9.3 SAS Institute, Cary, NC, USA.

Ethics approval

The Shorter Stays in Emergency Departments National Research Project was approved by the Multiregional Ethics Committee (MEC 10/06/60).

Results

Site specific population samples, case selection and study inclusion are outlined in Figure 1. Fewer patients were available for selection in the pre-target period for the pain outcome due to the data that was supplied by NZHIS starting in July rather than January 2006 (see methods), and fewer visits being selected from 2007 compared to other years (168 compared to ≈269 in the years 2008–2012), the reason for which is unclear. Similar proportions of screened visits were excluded for the pain, STEMI and appendicitis outcomes. More exclusions occurred in the pre-target period for the sepsis outcome due to the initial data request inadvertently capturing cases of neo-natal sepsis, which were not ED visits. Once this was recognised during data collection, the data request for subsequent records was changed to exclude births. There were fewer notes available in the post-target period for the NOF outcome due to patients still under active treatment during the data collection phase in late 2012 and early 2013, and more cases being incorrectly coded as fractured NOF (Figure 1).

Figure 1: Case selection and reasons for exclusion.

Key: ED = Emergency Department, STEMI=ST Elevation Myocardial Infarction, NOF=Neck of Femur
*Due to a change in the way coding occurred at NZHIS in 2006, there were fewer visits eligible in the pre-target period for the pain outcome, which starts on 1/7/2006 rather than 1/1/2006.  

The yearly SSED target performance is shown in Figure 2. All sites increased the proportion of patients who left ED within six hours after the target was introduced, although the target threshold was not reached by all hospitals.

Figure 2: Target performance over time.

The unadjusted median times are reported in Table 2, and the modelled primary outcomes adjusting for the explanatory variables as described in the methods are shown in Table 3. There were no clinically important or statistically significant differences between the pre- and post-target periods for any of the primary or secondary outcomes.

Table 2: Unadjusted primary and secondary outcomes.

‍

*Adequacy was defined a-priori as appropriate antibiotics for the site of infection for Sepsis, either thrombolysed or not thrombolysed appropriately for ST elevation myocardial infarction, time to theatre <24 hours for neck of femur fracture, perforated appendix at operation for appendicitis, and adequate analgesia for pain. IQR = Interquartile Range, CI = Confidence Interval† indicates time in minutes (all other times are in hours).

Table 3: Adjusted primary outcomes.

‍

CI = Confidence Interval *This was determined prior to commencing the study.

Discussion

This study found that the implementation of the Shorter Stays in ED target was not associated with clinically important or statistically significant differences in the time to treatment or adequacy of care for five acute clinical conditions reflecting care in both the ED and inpatient surgical units in different hospitals. More people left ED within six hours overall, and the median time spent in ED by patients included in the current study either reduced or remained the same after the target was introduced.

Although introduced with a view to reduce unnecessary time patients spend in ED and hence to reduce ED crowding, time targets for ED length of stay have been criticised on the grounds that they may lead to unintended consequences or divert attention away from other aspects of quality of care.9 Concerns have also been raised by inpatient clinicians about the potential for inappropriate patients to be admitted under their care if sufficient time is not spent differentiating patients in the ED prior to admission. This may lead to inefficient care or unnecessary resource use on inpatient medical24 or surgical wards.25

Prior research has reported mixed results with respect to ED time targets. New Zealand research using similar methods also found no change in the time to steroids in acute moderate to severe asthma in four hospitals, while the proportion of patients receiving steroids in ED increased.26 Another study from our group found improvements in the quality of discharge summaries from the ED to primary care in two other New Zealand hospitals.27 In two Australian hospitals, times to theatre for appendectomy were similar to those we found in their pre-National Emergency Access Target (NEAT) period. However, in the subsequent year, the time to theatre increased to 26 hours in the Australian study.25 In Ontario, a further study compared the difference in outcomes one year before implementation of ED LOS targets with one year after in hospitals that succeeded in reducing ED LOS versus those that didn’t succeed or got worse. The authors reported no difference in the time to thrombolysis for STEMI, time to analgesia or splinting for patients with arm fractures and time to steroids in asthma.28

The results from our study alongside others noted above indicate that there is little evidence to suggest that targets focusing on ED length of stay have diverted attention away from other aspects of quality of care in the specific clinical conditions studied. However, whether this represents ‘success’ with respect implementation of such targets is may depend on the perspective of an observer. An alternative view of this data is that despite apparent ED LOS target ‘success’, the quality of care did not improve substantially. One reason for this may be a ceiling effect, ie, given good quality of care pre-target, it is difficult to demonstrate a clinically important improvement. This may be a relevant issue in our study, as the baseline times to treatment were reasonable for all outcomes. It is also possible that the reported improvements in ED length of stay targets at our study sites may not have been sufficient to result in important reductions in crowding, as not all of the sites we studied achieved the 95% target threshold. In a secondary analysis, the authors of the Ontario study found that treatment times were faster in the least crowded ED shifts (average ED LOS <4 hours) compared to those in the most crowded ED shifts (average ED LOS >8 hours). This suggests that reductions of around one hour in median ED LOS from a baseline of four hours similar to those we observed may not necessarily result in reductions in ED crowding sufficient to accelerate treatment times. Further research is currently underway to explore how the implementation of the SSED target may have impacted on hospital length of stay, re-presentation to ED, re-admission to hospital, rates of leaving prior to being seen in ED and acute and elective mortality nationally.29

Limitations

Due to the number of cases notes required for each outcome and logistic constraints for both the research team and the clinical records departments at the participating hospitals, it was not feasible to measure all outcomes at all sites. We therefore measured different outcomes at different sites so that collectively, we covered a range of quality of acute care indicators in relation to the target, both in the ED and the hospital. For ‘pain’ and ‘sepsis’ the estimated sample size was not achieved, weakening the strength of conclusions around the results for these outcomes and increasing the risk of a type II error. However, the observed differences were small even though the estimates were imprecise. As the SSED target was introduced rapidly by the Ministry of Health to all New Zealand public hospitals, we were limited to using a ‘pre and post’ design rather than a prospective study control sites. Consequently, a causal relationship between the introduction of the target and any differences in quality or lack thereof cannot be assumed because other variables not accounted for by the design may have influenced the outcomes. Retrospective data abstraction involving clinical notes may introduce both selection and measurement bias. In an attempt to minimise bias, notes were selected at random, and electronic data was used wherever possible. Electronic data extraction forms with automated logic checks on individual variables minimised typographical errors. These forms also contained built-in validations, such as formulas which calculated whether or not a given patient met the entry criteria for respective outcomes and additional criteria for severity of condition to reduce subjectivity in these assessments. However, data extractors were not blinded to the objectives of the study, and the collection of time variables meant blinding was not possible with respect to the pre- and post-intervention time periods. Finally, the study results for outcomes measured at single sites may not be generalisable to other settings where the baseline care may differ. It is also possible that quality of care for outcomes that we did not explore may have changed. Importantly, the investigations of these clinical indicators were not powered adequately to explore variations in quality of care between different groups, eg, ethnicity or socio-economic status. This should be the focus of future research.

Conclusion

The introduction of the SSED target was not associated with clinically important or statistically significant changes in the time to treatment and adequacy of care for five clinical indicators of quality of care in Aotearoa New Zealand. For those indicators measured at one site only, it is unknown whether these results can be generalised to other sites.

‍