Journal of the New Zealand Medical Association, 05-March-2010, Vol 123 No 1310
How well does routine hospitalisation data capture information on comorbidity in New Zealand?
Comorbidities are diseases or disorders that coexist with a disease of interest.1 The importance of comorbidity has long been recognised in the clinical management of patients, but there is now increasing recognition of its importance in health related research and policy. Comorbidity can affect quality of life, increase mortality, influence treatment decisions, prolong hospitalisation and confound analysis.1-5 As the population ages, these issues will become increasingly common and pressing.
To date there has been very little work published on comorbidity in New Zealand. Davis et al published a study in 20022 investigating the burden of comorbid disease in major Auckland hospitals. They found that over a third of patients admitted had at least one comorbid condition, and that comorbidity was associated with length of stay, mortality and the occurrence of adverse events. Similarly Stevens et al found that comorbidity was very common among a cohort of lung cancer patients, and that it was adversely associated with survival.6
Currently it is unclear how common comorbidity is in New Zealand more generally, or how well routine hospitalisation data captures information on important comorbid conditions. This latter point is important as the majority of health policy, service planning and research projects requiring information on comorbidity will rely on secondary data. This paper aims to assess how well data on comorbidity are captured in routine databases in New Zealand by comparing detailed comorbidity data extracted by a physician from hospital records of patients with routinely collected hospitalisation data from these patients.
Data for this study come from two sources, firstly from a cohort study which investigated factors affecting colon cancer survival; and secondly from routine hospitalisation data obtained from New Zealand Health Information Service (NZHIS).
Cohort study—Details of this study are available elsewhere.7 In brief, the cohort was made up of patients with first primary colon cancer diagnosed between 1996 and 2003, and notified to the New Zealand Cancer Registry (ICD-10-AM site codes C18-C19 excluding 18.1). Patients were ineligible if they were less than 25 years at diagnosis, or were diagnosed after death. All Māori patients meeting the above criteria were included along with an approximately equal number of randomly-sampled non-Māori patients. This was to allow an assessment of survival disparities between Māori and non-Māori patients with colon cancer.7
Clinical data were abstracted directly from patients’ hospital medical notes during 2006-07. These were recorded on a standardised form by a physician (SH) and double-entered into an electronic database. Data were collected on all major comorbid conditions present at the time of diagnosis and all conditions included in the Charlson comorbidity index.
The Charlson index was developed in 1987 using data from a cohort of 607 medical patients, and validated with a population of breast cancer patients. Nineteen conditions are allocated a weight of 1 to 6 depending on the adjusted relative risk of 1-year mortality, and summed to give an overall score. 8
In addition to the conditions included in the Charlson Index, data were collected on the following conditions: angina, essential hypertension, cardiac arrhythmias, previous pulmonary embolism, cardiac valvular disease, inflammatory bowel disease, other neurological conditions (including multiple sclerosis, Parkinson’s disease, other abnormal movement disorders, epilepsy, spinocerebellar disease, anterior horn disease, other disease of the spinal cord, other demyelinating diseases of the CNS, cerebral palsy, myoneural disorders and muscular dystrophies) and major psychiatric conditions (including schizophrenia, bipolar disease, and depressive psychosis).
Comorbidities were classed in three different ways:
Administrative data—Routine hospital discharge data coded to ICD-9-CM-A were obtained from NZHIS in 2005 on the cohort specified above. These data are coded routinely from patient discharge records by coders based at District Health Boards and sent electronically in agreed format to NZHIS. We treated the admission for surgical resection of colon cancer as the index admission. Where a patient did not receive surgical resection, we treated the first hospital admission with colon cancer as primary diagnosis as the index admission. Those without such an admission were excluded from the study.
One of the problems with using administrative data to assess comorbidity is deciding on an optimal comorbidity ascertainment lookback period. Shorter periods may be more likely to identify currently active health issues, while longer periods may be more likely to identify all important comorbidity.9 In this study we assessed two lookback periods; 1 and 8 years, 8 years being the longest available time for the earliest cancer registrations.
We used both the principal and secondary diagnoses fields to identify comorbid conditions from the administrative dataset. We used the Deyo et al10 system which provides a method of translating the Charlson index for use on administrative data using ICD coding. The algorithm was modified to take account of the fact that we collected data on additional conditions to those included in the Charlson Index. These are listed in Table 1. Because it can be difficult to differentiate between pre-existing conditions and complications of treatment, some conditions are only included in the definition of comorbidity if they are listed prior to the index admission.
We followed the approach used by Deyo et al10, except that we included non-colorectal malignancies in our definition of comorbidity if they were listed in index or prior hospital discharges.11
Table 1. Diagnostic codes used for mapping
* Included in definition of a comorbidity if they are listed either in the index or prior hospital discharge; other codes only included if they are recorded prior to index admission
‡ Not included as part of Charlson Comorbidity Index
a Includes multiple sclerosis, Parkinson’s disease, other abnormal movement disorders, epilepsy, spinocerebellar disease, anterior horn disease, other diseases of spinal cord, other demyelinating diseases of CNS, cerebral palsy, myoneural disorders, muscular dystrophies.
b Includes schizophrenia, bipolar disease and depressive psychosis
Analysis—To calculate the maximum comorbidity we could identify from all data we had available, we first calculated the total number and proportion of patients who were recorded with each condition either in the medical notes review, or in the administrative data combined (separately for 1 and 8 year lookback). We then compared the proportion of these who had been identified in the notes, the administrative data or both, and calculated p-values using McNemar’s test to test whether the number of people with the condition differed significantly between the medical notes and administrative data.
We calculated the distribution of Charlson score and comorbidity count using medical notes, and administrative data with 1 and 8 year lookback. We then measured cross-source agreement for each condition as well as for the Charlson score and comorbidity count (uncategorised) using the weighted kappa statistic with quadratic (Fleiss-Cohen) weights.12
This statistic approximates the intraclass correlation coefficient and provides a measure of reliability that adjusts for agreement that occurs by chance. We considered scores of <0.40 to suggest poor agreement, 0.40 to 0.74 to suggest moderate agreement and 0.75 or higher to suggest very good agreement.
We assessed the association of comorbidity and all-cause survival among this cohort with colon cancer using Cox proportional hazards regression models. We fitted a baseline model that included sex, age, and ethnicity, year of registration, stage, grade and site of disease. The fit of the baseline model was compared to various models that included comorbidity using the likelihood ratio test. For these models comorbidity was measured using Charlson categories or individual conditions.
The conditions were selected on the basis that they had been previously shown to be related to survival from colon cancer in this cohort4, and that there were a minimum of 10 cases within the cohort (these conditions were previous myocardial infarction, congestive heart failure, diabetes, chronic respiratory disease, renal disease, cardiac arrhythmias, non-cerebrovascular neurological conditions and peripheral vascular disease). We compared results from models that included comorbidity measured using data from medial notes to those using administrative data.
Often comorbidity will not be an exposure of interest, but a potential confounding factor in another putative association. Researchers therefore have an interest in knowing how much of the ‘true’ confounding by comorbidity might be captured when adjusting for a misclassified measure such as that from routine administrative data. We explored this for the putative association of ethnicity with survival, and how much of the association might be due to confounding/ mediation by comorbidity (we know that Māori experience poorer survival from colon cancer than non-Māori, and that some of this association is due to Māori carrying a higher burden of comorbidity than non-Māori7).
We measured the hazard ratio for all-cause mortality of Māori compared with non-Māori having adjusted for sex, age, year of registration, stage, grade and site. We then added to the model comorbidity measured using the individual conditions specified above identified either in the notes, or in the administrative data to assess the extent to which each changed the underlying hazard ratio.
Approval for this study was granted by the New Zealand Multi-Region Ethics Committee.
A total of 685 patients met the eligibility criteria for the cohort study, and full data were obtained for 92% of eligible patients to give an initial study sample of 642 (308 Māori and 334 non-Māori). When these cases were matched to the routine hospitalisation data, 73 were excluded because they did not have an admission that met the criteria for the index admission giving a final cohort for this study of 569 patients, 515 having an admission for surgical resection of colon cancer.
Tables 2 and 3 show the comparison of medical notes data with administrative data with 1- and 8-year lookback respectively. They show that there were considerable differences in the comorbidity data obtained from these two data sources. For most conditions, higher numbers of patients were identified with notes review data than administrative data, and this effect was more marked with 1-year than 8-year lookback.
This pattern was reversed for diabetes and renal disease for both lookback periods, as well as non-colorectal malignancy, cardiac valve disease and hemiplegia with the longer lookback period. There was very good agreement (kappa=0.77 and 0.75 for 1- and 8-year lookback respectively) between the sources of data for only one condition (mild to moderate diabetes).
For the 1-year lookback, 11 conditions showed moderate agreement (kappa 0.40 to 0.74), and the remaining five showed poor agreement (kappa<0.40). Agreement between the two data sources improved with the longer lookback period with 14 conditions showing moderate and two showing poor agreement.
View Table 2 and Table 3 here
As expected, both Charlson scores and comorbidity counts tended to be higher when calculated from data extracted from medical notes than from administrative data with 1 or 8 year lookback, and the highest scores were obtained by combining both data sources (Figures 1 and 2). For the Charlson index, agreement between the medical notes data and the administrative data was somewhat better for the longer lookback period (kappa=0.66; 95% CI: 0.57-0.75) than the shorter one (kappa =0.61; 95% CI: 0.51-0.70).
Figure 1. Comparison of Charlson comorbidity scores calculated using medical notes or administrative data
A similar pattern was seen for comorbidity count, although because more conditions were included in this count, scores were generally higher (Figure 2). The agreement between notes and administrative data was also better with kappa coefficients of 0.66 (95% CI 0.60-0.73) and 0.77 (95% CI 0.72-0.81) for administrative data with 1 and 8 year lookback respectively.
We found that comorbidity measures added significantly to the ability of the base model to explain all-cause survival regardless of whether comorbidity was measured using the Charlson score or individual conditions, or whether data was collected from medical notes, administrative data or both (in all cases likelihood ratio test p<0.0001 for model including comorbidity measured compared with base model).
In this cohort, we found that after adjusting for sex, age, year of registration, stage, grade and site, the baseline hazard ratio of all-cause mortality for Māori compared with non-Māori was 1.34 (95% CI 1.03-1.74). When we adjusted for comorbidity using data from both sources combined, the excess hazard ratio decreased to 1.17 (0.89-1.53). Adjusting for comorbidity using either notes or administrative-based data alone resulted in somewhat less reduction in the hazard ratio to 1.23 (95% CI 0.94-1.60), and 1.26 (95% CI 0.96-1.64) respectively.
Figure 2. Comparison of comorbidity counts calculated using medical notes or administrative data
We found that there were considerable differences in the comorbidity data held in the routine administrative hospitalisation database in New Zealand compared with that collected by a physician from medical records. In general, more comorbidity was identified from medical records, however some conditions were more frequently identified from administrative data notably diabetes and renal failure. Agreement between the two data sources improved with a longer lookback period for the administrative data. Despite these differences, any of the measures of comorbidity that we used, regardless of the source of the data, improved the ability of multivariable model to predict all-cause survival in this cohort of colon cancer patients.
This is the first study in New Zealand to assess the quality of routinely collected comorbidity data, which are being increasingly used for health service funding and planning, and research. This is reasonable because although medical notes review data is generally considered superior to administrative data, it is not gold standard. While there may be concern about the accuracy of diagnoses recorded in administrative data, medical notes are also not entirely complete, standardised or error free.13
Furthermore, the results here and elsewhere clearly show that administrative comorbidity data are not a subset of medical notes data, and it is likely that combining datasets provides less misclassification of comorbidity than either source alone.13-15 This is, of course, rarely possible.
Given that both sources result in misclassification of the (immeasurable) underlying construct of ‘true’ comorbidity, it is also possible, or even likely, that each of the sources of data correlates more strongly with this third measure than they do with each other, assuming that the misclassification errors in administrative and notes review data are independent of each other. That is, the kappa comparing the administrative and note-based comorbidity indices probably underestimate the correlation of each with a ‘true’ measure of comorbidity (unless errors in administrative and notes-based measures are moderately to highly correlated).
Furthermore, routinely collected data are considerably more accessible for large population groups than notes review, and a number of approaches to dealing with administratively collected comorbidity data are possible depending on the purpose of the data, and the outcome being assessed.1, 16-20
Our finding that medical notes review results in higher ascertainment of comorbidity is consistent with other studies. 13-15, 21, 22 The extent of this difference depends on a number of factors including the measure or condition that is being compared, the mapping algorithm used and the lookback period used for administrative data. There is considerable variability between conditions in terms of their ascertainment in administrative compared with medical notes data.
For the administrative data with 8-year lookback, this varied from kappa coefficients of 0.32 to 0.75. This variation is likely to depend in part on the seriousness of the condition, and coding practices relating to administrative data. As a general rule in New Zealand, comorbidities are only coded in administrative data if they co-exist or arise during a given episode of care and that they affect patient management in a way which might extend length of hospital stay. This approach is likely to result in an emphasis on the most active and clinically important conditions, and will explain some of the difference between notes and administrative comorbidity data.
It is not entirely clear how one should map conditions from clinical notes to ICD codes, and there has been dissent expressed on this in the literature.10,11,21,23,24 We employed a commonly used approach, but one that has also been criticised by some authors.11,21 For example, we found that for six of the nine mismatches for diabetes with end organ damage, had been coded as diabetes without mention of complication.
Currently no gold standard mapping approach has been developed. The length of the lookback period also makes a difference, but the ideal lookback period seems to depend on the outcome for which the data is being collected.9,25 For example Preen et al (2006)9 found that a one-year lookback provided better comorbidity data to predict mortality while five-year lookback was better for readmission rates. In our study the longer lookback period seemed to give more comparable data to the notes review.
Both the kappa coefficients for the individual conditions and those for the Charlson and comorbidity count (0.66 and 0.77 respectively) compare favourably with similar comparisons carried out elsewhere.15,22 For example, Kieszak et al22 compared comorbidity derived from medical notes with administrative data in the United States and found that only three of 16 individual conditions had kappa coefficients greater than 0.4 (compared with 15/17 for our data with 8 year lookback), and that the correlation between the notes Charlson index and the administrative index was only 0.47.
Regardless of the source of data used, we found that any measure of comorbidity improved multivariable model fit compared with using none. We also found in this study that using data from both sources combined resulted in somewhat better risk adjustment than either source separately. However, both the notes and administrative-based comorbidity measures substantially reduced the excess mortality hazard for a model comparing Māori with non-Māori for colon cancer survival, although more so for the use of notes-based comorbidity index consistent with an a priori expectation that it is a superior estimate of comorbidity.
Furthermore, given that including both the notes and administrative-based measures of comorbidity resulted in greater reduction again in the hazard ratio, it seems reasonable to conclude that neither measure alone (notes or administrative-based) fully captures the confounding or mediating effects of comorbidity.
Of note, is that this study focused solely on patients with colon cancer. Patients with other primary conditions may have different patterns of comorbidity, but it seems unlikely that this will affect the quality of the recording of their comorbidity data. In that respect, it seems reasonable to be able to generalise the findings of this study to hospital-based comorbidity data in New Zealand.
In conclusion, measuring comorbidity is potentially important for risk adjustment in health service policy, funding and planning, and health-related research. Data from clinical notes review are often considered superior but are rarely available. The correlation between clinical notes and administrative data in New Zealand is moderate and varies considerably between individual conditions. However, administrative data provides a source of relatively accessible comorbidity data which we have found allows for reasonable risk adjustment in the cohort presented here, although not quite as good as for a notes-based or combined comorbidity measure.
Competing interests: None known.
Author information: Diana Sarfati, FAFPHM, Senior Research Fellow, Department of Public Health, University of Otago, Wellington; Sarah Hill, FAFPHM, Senior Research Fellow, Department of Public Health, University of Otago, Wellington; Gordon Purdie, Biostatistician, Department of Public Health, University of Otago, Wellington; Liz Dennett, FRACS, Senior lecturer, Department of Surgery and Anaesthesia, University of Otago, Wellington; Tony Blakely, Research Professor, Department of Public Health, University of Otago, Wellington
Acknowledgements: We thank Bridget Robson, Donna Cormack and Kevin Dew who were investigators in the colon cancer cohort study from which data for this study was collected, and for their helpful comments on earlier drafts of this paper.
The authors also acknowledge the Cancer Society of New Zealand for providing funding for this study (grant 05/16). The Cancer Society of New Zealand had no role in the study design; in the collection, analysis or interpretation of data; in the writing of the manuscript or the decision to submit the manuscript for publication.
Correspondence: Diana Sarfati, Department of Public Health, University of Otago, Wellington, PO Box 7343, Wellington South, New Zealand. Fax: +64 (0)4 3895319; email: firstname.lastname@example.org
issue | Search journal |
Archived issues | Classifieds
| Hotline (free ads)
Subscribe | Contribute | Advertise | Contact Us | Copyright | Other Journals