Journal of the New Zealand Medical Association, 13-December-2002, Vol 115 No 1167
Adverse events in New Zealand public hospitals I: occurrence and impact
Peter Davis, Roy Lay-Yee, Robin Briant, Wasan Ali, Alastair Scott and Stephan Schug
The subject of patient safety, and the quality of healthcare, has gained increasing momentum internationally as a major focus of attention in professional and health policy circles. This has been highlighted recently by the report on patient safety from the Institute of Medicine in the United States,1 by an entire issue of the British Medical Journal devoted to medical error,2 and at least two high-profile reports on aspects of patient safety in the United Kingdom’s National Health Service.3,4 The matter has also gained attention in the UK because of the high level of public interest in the Bristol incident.5
In New Zealand, a number of reports have also highlighted quality issues,6,7 yet the question of patient safety has, to date, been the subject of relatively little systematic research. One of the first studies to use a standardised, epidemiological approach was a survey of adverse drug events among over 9000 admissions to Dunedin Hospital in the early 1970s.8 Although useful research since that time has been carried out on surgical audit9 and anaesthetic error,10 no generic, epidemiological data on adverse events have been published in this country. The absence of such data has been recognised as an obstacle to developing proposals for the regulation of safety in health and disability in New Zealand.11
A major scientific stimulus to rigorous epidemiological research on patient safety was the development of standardised procedures for the assessment of adverse events using medical records.12 This methodological approach was tested for its applicability in both the British13 and Australian contexts,14 and has further been tested for its feasibility in New Zealand.15,16
The objective of this study was to assess the occurrence and impact of adverse events in New Zealand public hospitals as revealed in an audit of a representative sample of medical records.
Sampling The survey population was defined as all patients admitted to 20 general hospitals with more than 100 beds for the calendar year 1998 (excluding day, psychiatric and rehabilitation-only cases). A random sample of 13 public hospitals was then selected following stratification by hospital type (ie, service facilities provided) and geographical area. The sampling frame for each hospital was a list of all eligible admissions in that hospital, ordered by admission date. To ensure distribution throughout the year, systematic list sampling from a random start point was used to identify 575 admissions from each hospital. Initially, 7475 records were selected for screening. However, for a variety of reasons – wrongly sampled record, missing record, current inpatient, and other – 744 records could not be screened, leaving 6579 records for assessment (after double admissions were also removed).
To be included in the study, an adverse event had to be related to, or have occurred during, the sampled admission. The feasibility of using this method was tested in three major hospitals in Auckland.16
Administrative data For each sampled admission, the New Zealand Health Information Service (NZHIS) provided admissions information (dates of admission and discharge, admission type (planned or acute), and admission source (routine or transfer)); socio-demographic data (age, gender, ethnicity, domicile code); and clinical data (diagnostic classification).17 NZDep96 quintiles were derived from patient domicile codes as a measure of residential area deprivation.18 Principal diagnosis or reason for admission was classified according to 25 Major Diagnostic Categories (MDCs) derived from Australian AN-DRG 3.1.17
Data collection The core data collection procedure of the study was a two-stage retrospective review of each selected medical record. This involved the use of two protocols. At the first stage, a screening protocol – Review Form 1 (RF1) – was administered by specially trained Registered Nurses (RNs) to determine if the sampled admission met any of 18 screening criteria selected as potentially indicative of an adverse event. The screening criteria included unplanned admission before the sampled admission, and unplanned readmission after discharge from the sampled admission, among others.19
Those records indicating positive on the initial screening stage were passed on for further consideration. The objective of the second stage was to determine whether the sampled admission was associated with an adverse event, and if so, to then characterise that adverse event according to key clinical criteria. Review Form 2 (RF2) guided these judgements – using structured implicit review – and was administered by specially trained and experienced Medical Officers (MOs). Both review forms were closely modelled on the comparable instruments in the American20,21 and Australian studies.14 Structured implicit review is the guided exercise of professional judgement to facilitate reliable detection and determination of adverse events. A series of seven evaluation questions were used to assist reviewers in arriving at this judgement. The degree of certainty accorded to this assessment was translated into a six-point confidence scale of evidence of causation by healthcare management. This ranged from 1 = virtually no evidence, to 6 = virtually certain evidence.19
An expert reviewer arbitrated on discrepant judgements (in which an RN and an MO disagreed), and carried out an independent review of a one in ten sub-sample of medical records. Measures of concurrent validity22 were used to determine the quality of screening and review.
Definitions An adverse event was operationally defined as: 1) an unintended injury; 2) resulting in disability; and 3) caused by healthcare management rather than the underlying disease process. Each of these criteria had to be fulfilled. Figure 1 provides two examples of unwanted outcomes of treatment; one is not classified as an adverse event and the other is an adverse event of low preventability.
Figure 1. Examples of event occurrences synthesised from real cases19
* Preventability defined as an error in healthcare management due to failure to follow accepted practice at an individual or system level
Disability was categorised into one of the following types: temporary; lasting up to one year; permanent impairment of function; or death. Attributable bed days refer to those extra days associated with an adverse event that were spent in the study hospital during one or more admissions.
For over 85% of sampled records, available information was sufficient to complete all aspects of the RF1. Similarly, for approximately 95% of all medical records classed as adverse events, the available information was sufficient to complete all aspects of the MO review using the RF2.19
The representativeness of the sample was assessed by comparing the distribution of key patient characteristics with the pattern for all New Zealand publicly-funded hospital admissions in 1998 (Table 1). Sample figures for age, gender, ethnic group, discharge status and mortality were all closely comparable to national data, whereas length of stay in the sample appeared to be notably shorter.
Table 1. Patient characteristics, 1998; study sample vs all publicly-funded hospital patients
*All publicly-funded hospitalisation in New Zealand (may include private hospital admissions) (Source: New Zealand Health Information Service)
†Excludes day and psychiatric patients
‡Excludes specialist public hospitals, public hospitals with under 100 beds, and rehabilitation-only patients
The frequency of occurrence of adverse events is considered in Table 2. Such events were associated with 12.9% of admissions. This represents all incidents recorded by a healthcare professional over the period 1998 to 2000 (field work date) in a population of hospital patients admitted in 1998. Technically, this is a prevalence figure. Adjusting for the differential probability of selection of admissions (attributable to the stratified cluster sample design) increased this proportion slightly (13.1%). The incidence rate – of 11.2% – represents only cases recorded during the 1998 sampled admission. This again increased slightly on adjustment for sample design (11.3%).
Table 2. Adverse events: prevalence and incidence
*Prevalence defined as all adverse events found by the study review process as a proportion of sampled 1998 admissions
†Incidence rate defined as incidents recorded by a healthcare professional during the 1998 sampled admission (and later assessed to be an adverse event by a study reviewer)
While all the events reported here were recorded and treated in public hospitals, not all had occurred in such institutions (Table 3). While the great majority of adverse events had occurred inside a public hospital (80.4%), nearly one fifth had taken place outside; most commonly in a doctor’s rooms, at the patient’s home, or in a rest home or private hospital.
Table 3. Distribution of adverse events (AEs), by location of occurrence
In Table 4, the impact of adverse events is assessed according to patient disability and hospital workload (number of extra days’ stay). For the great majority of patients, disability was either minimal or moderate. However, it should be noted that nearly 15% of patients either suffered permanent disability or died. On average, an extra 9.3 days was required for treatment (median of 4 days). There was a close association between disability status and length of stay, with the permanently disabled requiring between three and five weeks’ extra stay.
Table 4. Impact of adverse events (AEs): patient disability status and workload (extra hospital stay)
*Extra bed days associated with an adverse event that were spent in the study hospital during one or more admissions
†Period of disability
The impact of adverse events in relation to major diagnostic criteria is represented in Table 5. This shows the distribution of admissions and adverse events, together with two impact criteria – permanent disability and death, and extra stay in hospital. There were few striking discrepancies; adverse events seemed to be over-represented in injury-related and musculoskeletal MDCs, and less common in birth-related admissions. Adverse events associated with digestive, respiratory and nervous systems seemed to show greater patient impact.
Table 5. Distribution of admissions and adverse events (AEs) and impact of AEs, by Major Diagnostic Category (MDC)
*Principal diagnosis was classified according to 25 Major Diagnostic Categories (MDCs) derived from Australian AN-DRG 3.1 and ordered according to percentage of admissions
†Extra bed days associated with an adverse event that were spent in the study hospital during one or more admissions
The same set of data is presented in Table 6 for a range of socio-demographic factors; namely, age group, gender, ethnic group, and area deprivation. Overall, the pattern of adverse events mirrored that of admissions closely for all comparisons, except for patients over the age of 65. While this group accounted for one third of admissions, it was associated with 40% of adverse events. Patient impact and workload was also slightly higher for those over 65.
Table 6. Distribution of admissions, adverse events (AEs) and impact of AEs, by socio-demographic factors
*Extra bed days associated with an adverse event that were spent in the study hospital during one or more admissions
†NZDep96 quintiles were derived from patient domicile codes as a measure of residential area deprivation; quintile 5 represents the highest level of deprivation
‡77 of the admissions could not be coded
§7 of the adverse events could not be coded
A primary objective of this investigation was to establish the occurrence of adverse events in New Zealand public hospitals by assessing a representative sample of admission records according to a standardised audit protocol. Using this methodology, it was estimated that 12.9% of hospital admissions were associated with an adverse event. This rate stands almost midway between the levels recorded in two countries with shared medical traditions in training and practice: Australia (16.6%)14 and the UK (10.8%).13
A second major objective of the investigation was to assess the impact of adverse events, both on patients and on hospital workload. The noteworthy finding here is the quite mixed signals on the magnitude of impact (Tables 4 and 6). Less than 15% of adverse events were associated with permanent disability or death (and the great majority of events resulted in relatively minor impact on patients). This outcome is consistent with findings in other studies.13,14,20 However, when considering the impact on hospital workload, adverse events added an average of over nine days to expected hospital stay, an outcome that was similar to the Australian finding of an average of just over seven days.14 It should also be noted that there were few demographic or clinical patterns in the data, aside from the evident vulnerability of older patients.
Another area of interest that emerged from this study is the significant proportion – about one fifth – of adverse events that originated outside a public hospital (Table 3). This result has not been previously reported in the international literature and points to the potential importance of quality and safety issues in primary and community care and in other institutional settings.
The great strength of the study is its representativeness. On key criteria, the sample of records shows a close approximation to the pattern of admissions for all New Zealand hospitals in 1998. However, it should be noted that the sample draws only on generalist, acute hospitals with 100 beds or more, and that length of stay was on average lower (5.1 in the sample, 6.9 in all publicly-funded hospital admissions). The documentation in sampled medical records was sufficiently detailed and comprehensive to permit full completion of study instruments, and there was evidence of internal consistency in the data on key study variables (for example, the relationship between assessed patient disability and extra hospital workload).
Yet, there still remain questions about the quality of key study measures, such as adverse event status and preventability. Study instruments were directly applied, with little if any modification, from internationally established protocols. These rely on the guided judgement of screeners and reviewers – structured implicit review – and are thus potentially subject to observer variability. Hence, for example, the measure of agreement between MO reviewers and the expert reviewer on adverse event determination in this study – kappa 0.47 – was only of moderate strength,19 although within the range for comparable studies.14,23 Furthermore, while the adverse event rate reported in this study clusters with those for Australia14 and the UK,13 it differs significantly from those reported for the United States.20,21,23
An analysis of this overall discrepancy in adverse event rate between Australia and the United States suggests that slight differences in methodology were partly to account, but that the principal explanation lay in contrasting study purposes – medicolegal in one case (the United States), quality in the other (Australia).24 Nevertheless, there remains an irreducible element of subjectivity to the core study instrument, with the potential for considerable observer variability. This constrains the interpretation of any apparent variations in adverse event rates.
This investigation establishes broad clinical and managerial parameters for our understanding of patient safety and the quality of care in New Zealand public hospitals. The findings suggest that adverse events are as significant a problem in New Zealand as they are in Australia, the UK, and the United States. In essence, about one in eight admissions to a hospital are associated with adverse events (which may have occurred within or outside public hospitals). The majority of such incidents have a relatively minor impact on patients (though there is a significant proportion who suffer permanent disability or death), but their effects on hospital workload, and thus costs to the health system, are substantial.
There remain a number of issues unresolved from this investigation. First, there are still questions about the measurement properties of structured implicit review in identifying adverse events from medical records. Further methodological work is required in this area. Second, more detailed analysis of the data from this study – and others – is required in order to provide insight into the detailed patterns of adverse event occurrence and determination, particularly in relation to preventability. Preliminary work in this area has shown that only 6.3% of admissions to New Zealand public hospitals were associated with adverse events that were both preventable and occurred in hospital.25 From such work may come indications for quality improvement initiatives, together with testable propositions for strategies designed to reduce the level of preventable adverse events.
In summary, the first nationally representative audit of medical records in New Zealand public hospitals has identified a level of medical injury that is similar to that recorded in comparable countries. There is a considerable impact of adverse events on hospital workload, and a significant minority of patients suffers death or permanent disability.
Author information: Peter Davis, Professor, Department of Public Health and General Practice, Christchurch School of Medicine and Health Sciences, University of Otago; Roy Lay-Yee, Analyst; Robin Briant, Clinical Director, Division of Community Health, Faculty of Medical and Health Sciences, University of Auckland; Wasan Ali, Visiting Research Fellow, Department of Public Health and General Practice, Christchurch School of Medicine and Health Science, University of Otago; Alastair Scott, Professor, Department of Statistics, University of Auckland; Stephan Schug, Professor, Department of Anaesthesia, University of Western Australia, Perth, Australia
Acknowledgements: Work on this study was funded by the Health Research Council of New Zealand. We thank the 13 New Zealand hospitals that participated in the study and Dr David Richmond (Chair) and members of the study’s Advisory and Monitoring Committee. We also thank Sandra Johnson and Wendy Bingley, our medical review and data processing teams, and hospital records staff.
Correspondence: Professor Peter Davis, Department of Public Health and General Practice, Christchurch School of Medicine, University of Otago, P O Box 4345, Christchurch. Fax: (03) 364 0425; email: firstname.lastname@example.org
issue | Search journal |
Archived issues | Classifieds
| Hotline (free ads)
Subscribe | Contribute | Advertise | Contact Us | Copyright | Other Journals