Journal of the New Zealand Medical Association, 04-April-2008, Vol 121 No 1271
Mental health services for children and adolescents in New Zealand, outcomes, and the Health of the Nation Outcome Scale for Children and Adolescents (HoNOSCA)
Matthew J F Eggleston, William G A Watkins
Figures from the Mental Health Commission’s 2006 Report on Progress,2 which examined progress towards the implementation of the Blueprint for Mental Health Services in New Zealand,3 indicated that access to CAMHS for young people aged 0–19 years remains substantially below the equivalent figure for adults. Access rates to CAMHS for young people ranged between 0.9 to 1.6%, depending on region, which is substantially below the Blueprint’s access target of 3%.
While the Blueprint recommended that 26% of Mental Health Service funding be applied to services for young people, the actual figure for 2004/05 was a disappointing 11%.2
Despite a real increase in funding for all Mental Health Services of 154% over the period 1993/94 to 2004/05 to NZ$866.6 million, commensurate increases in rates of access to services have not been realised.2 While there has been no systematic investigation of this phenomenon, in recent years it is apparent that there has been a proliferation of clinical and non-clinical activities required of clinical staff, together with a burgeoning documentation requirement for each activity.
This increase in “compliance” requirements is a likely contributor to the finding that DHB provider expenditure per individual has increased by an average of 20% over three years for three of the four New Zealand regions.2
Over the last decade, there have been strong international Governmental drives for mental health services to demonstrate the quality and cost-effectiveness of their services.4–7 New Zealand is following this trend with formal national reporting of the HoNOSCA by CAMHS being required by the Ministry of Health (MOH) by June 2008.8 In addition to hoping that this measure will demonstrate the quality and cost-effectiveness of services, other rationales for the introduction of routine outcome measures include embedding measures into the routine process of clinical assessment,8 as well as supporting recovery by promoting and facilitating the development of an outcomes-focused culture in the mental health sector.9
The HoNOSCA is the first of a planned suite of outcome measures to be introduced as part of The New Zealand Health Standard Measures of Assessment and Recovery (MH-SMART) Outcomes Initiative.9
Strong advocates of routine outcome measurement for psychiatry acknowledge that there is little empirical evidence that their introduction is of benefit to patients.10 Harnett et al considered that the effectiveness of the introduction of the HoNOSCA and other outcome measures in providing meaningful information for evaluating the impact of child and youth mental health services remains to be determined.11 Bickman et al stated that, without identifiable clinical processes, outcome data alone are unlikely to enhance services.12
No evidence exists to suggest that the use of routine outcome measures, including the HoNOSCA, assists with demonstrating the quality of a CAMHS, its cost-effectiveness, or that it promotes an outcomes-focused culture. Hence, the proposed national introduction of the HoNOSCA is not evidence-based (as is correctly increasingly being demanded of treatments in the field) and should be considered experimental. It seems unfortunate that the manner of the introduction of the HoNOSCA by the MOH does not allow its usefulness in achieving its intended purposes to be evaluated.
Child and adolescent psychiatry has long had a tradition of embedding routine measures into clinical assessment as well as utilising measures for evaluating outcomes. Indeed, well-established standardised measures such as the Conners’ rating scales constitute an integral part of both the assessment and monitoring of response to treatment for attention deficit hyperactivity disorder (ADHD).13
Our outpatient child psychiatry clinic has systematically embedded measures in the routine process of clinical assessment for some years by routinely utilising the Achenbach System of Empirically Based Assessment (the Child Behaviour Checklist, Teacher Report Form and Youth Self-Report).14 These instruments gather systematic information from multiple informants; not only on a broad range of relevant emotional and behavioural problem areas, but also social and academic competencies. They are extensively utilised in clinical and research settings, have been normed on large samples of children and adolescents, and have sound psychometric properties.14
It is difficult to conceive how a single outcome measure would be considered appropriate for the entire field of paediatrics (from oncology to diabetic endocrinology), or indeed any other broad medical specialty area. Yet, this is exactly what is being implemented in child and adolescent as well as adult psychiatry. In fact, many issues make the understandable aims of demonstrating quality, cost effectiveness of services and evaluation of broad outcomes more difficult in child and adolescent psychiatry than in most other areas of medicine.
Child and adolescent psychiatric diagnostic entities are syndromally-based and are imperfect approximations. The natural course for most disorders is not well defined which adds to the difficulty of determining whether treatment may be making a difference.
The majority of children and adolescents who present to clinics with psychiatric disorder have comorbid conditions, making comparisons between outcomes for children and adolescents with similar primary psychiatric disorders very difficult. For many diagnostic entities, best-practice treatment guidelines are largely based on consensus expert opinion rather than solid empirical evidence.
McClellan and Werry, while strongly advocating for evidence-based treatment approaches, stated that “the notion of a well-defined standard of care for most childhood disorders remain elusive”.15
There are also important aspects of CAMHS functioning which are simply not captured by a narrow focus on outcomes. A significant number of children and adolescents will be assessed by a CAMHS and, appropriately, not be offered treatment. A further group, including those with autistic spectrum disorders, may be assessed and offered appropriate treatment, but large improvement in the core symptoms of their condition would not be expected.
An important minority of children and adolescents presenting with conditions (including schizophrenia) prove to be treatment-resistant despite best-practice treatment approaches, yet denying them and their families ongoing psychiatric treatment and care on the basis that they do not show a positive change on outcome measures would be unethical due to the severity of their condition.
There has been insufficient consideration or exploration of other possible adverse effects of routine outcome measurement. It has been estimated that up to 10% of a clinician’s time can be spent on a simple outcome measurement system.16 Hence, the time-costs associated with the implementation of routine outcome measurement are not insubstantial and are likely to have a negative impact on access to already under-resourced and overburdened CAMHS.
Importantly, no additional funding for clinical or administration time has been allocated to assist with the demands of implementing and maintaining the collecting, processing, interpretation, and feeding back to consumers of routine outcome measurement data.
Because HoNOSCA scores are to be repeated within 2 weeks of every 3-month interval during a consumer’s episode of care, a major increase in outpatient reviews will be needed solely for the purposes of scoring the HoNOSCA, rather than for indicated clinical reasons.
In addition to this further time-cost, consumers and their families are likely to object to attending reviews for the purpose of scoring the HoNOSCA, if this is considered by them to be clinically unnecessary. It is also conceivable that, rather than promoting an outcome-focused culture in CAMHS, embedding potentially suboptimal measures into the routine process of clinical assessment may result in clinicians having less time, commitment, and energy to devote to more relevant clinical measurement, audit, and research.
There are also, potentially, major clinical repercussions of formal national reporting of outcomes to the MOH. Some of these will depend on how the MOH intends to utilise the information and, to date, this has not been articulated. There are valid concerns that clinicians may become “outcome measure-focused” rather than “patient outcome-focused”.
Given the requirement for reporting 3-monthly outcome measures, clinicians may preferentially choose short-term, narrow, and medication-oriented approaches rather than clinically indicated longer-term, systemic, and psychotherapeutic treatment approaches. Services may become reluctant to assess and treat individuals who are unlikely to show improvement on measures, but who nevertheless require significant care, stabilisation or support, such as those with autistic spectrum disorders. It is even possible that clinicians may become reluctant to retain in treatment those patients with complex or poor prognosis conditions whose outcome measures continue to deteriorate despite best-practice treatment.
The process of consultation with stakeholders regarding the choice of outcome measures for CAMHS deserves comment in its own right. The New Zealand Mental Health Research and Development Strategy commissioned a study to examine the use and acceptability of child and youth mental health outcome measures.17 A very significant portion of the questionnaire for clinicians related to only two measures, including the HoNOSCA. In fact, the questionnaire appeared designed to favour these two measures in that no other measures were specifically mentioned.
Clinicians’ understandable general support for the importance of outcome measures and for “routine use of outcome measures” (after all these should already be part of everyday practice, particularly for disorders such as ADHD) was then extrapolated to mean support for the introduction of a “routine outcome measurement system”.
Clinicians’ general endorsement of outcome measures and favouring of the HoNOSCA (despite most having limited experience with the instrument) as one of two clinician-rated outcome measures (the other measure was not specifically mentioned in the questionnaire, but needed to be spontaneously mentioned and endorsed by respondents) was extrapolated to the doubtful conclusion that clinicians considered the HoNOSCA “a worthwhile measure”. The authors recommended the introduction of the HoNOSCA and the second measure as routine outcome measures.
Relevantly, 87.6% of respondents anticipated problems with the introduction of routine measures and, of those providing comment, 84.3% indicated that staff willingness or ability to take on extra work was anticipated to be the major complication.
The HoNOSCA: psychometric properties
In general, studies that have examined inter-rater reliability have found this to be within the moderate to good range for individual HoNOSCA items and for the total HoNOSCA score, although most of these estimates were based on ratings from clinical vignettes rather than actual clinical interviews.1,11,18,19
Harnett et al, however, found poor internal validity of the four main HoNOSCA subscales, “significantly limiting their use in any analysis of outcome”.11 Similarly, a New Zealand study found a weak degree of fit of the subscales with the data from the individual HoNOSCA items and concluded “not too much reliance should be placed on the HoNOSCA subscales”.20 Test-retest reliability was good over a period of 2 to 4 weeks in an inpatient adolescent sample.11
Several studies have examined the convergent validity between the total HoNOSCA score and a range of other clinician rated measures. Moderate correlations have been found between the HoNOSCA and a measure of clinical and environmental complexity, the Paddington Complexity Scale (r of 0.46 1 and 0.640,21 p<0.001).21
Similarly, moderate correlations have generally been found between the HoNOSCA and a very brief global measure of impairment, the Children’s Global Assessment Scale (CGAS) (r significant and between -0.6 and -0.644,11,20–22).23
The Treatment for Adolescents with Depression Study (TADS), however, found only a weak correlation with the CGAS (r= -0.14, p = 0.005).24 Even so, the moderate correlation with the CGAS found by some studies is relevant, given that the CGAS is very brief, has good inter-rater and test-retest reliability, considerable evidence to support convergent validity, is sensitive to change, and has been used extensively in both research and clinical settings.25
While neither the HoNOSCA or CGAS have formal normative data, the CGAS has been utilised in several large epidemiological studies that provide some standardised data25. It certainly could be argued that the CGAS may be a better researched and easier to use proxy measure for the significantly more time-consuming HoNOSCA.
Several studies have suggested that the HoNOSCA is able to detect change over time with total HoNOSCA scores correlating moderately with clinicians’ global impressions of change.1,11,18,19 In one such study “marked improvement” on parents’ and referrers’ global outcome measures were associated with HoNOSCA change scores of four to five points.19 However, the order of change is variable. For example, in an adolescent inpatient sample, the mean HoNOSCA change scores were 1.51 and 1.92 points over 3 and 6 months of inpatient, presumably intensive, treatment respectively.11
One factor limiting the merits of clinicians’ rated outcome measures is the finding that clinicians rate very few patients as deteriorating over time.18 Harnett et al noted a tendency for HoNOSCA scores to fluctuate considerably for patients staying longer than 6 months on an inpatient adolescent unit, which was considered to limit the usefulness of the HoNOSCA as an outcome measure for individuals with psychotic symptoms or self-harming behaviour.11
In terms of the utility of the HoNOSCA in clinical settings, reports are highly variable. Bilenburg reported that 80% of clinicians found the HoNOSCA to have “substantial” clinical utility.4 In contrast, Kisely reported that the HoNOSCA was not useful in providing relevant information.5 Similarly, Patterson et al’s survey found that the vast majority of respondents did not find the New South Wales-mandated routine outcome measures (including the HoNOSCA) clinically useful in their regular reviews of patients.26
Clinicians have also reported that the HoNOSCA is difficult to use with children under 6 years,4 as well as being adolescent–oriented.27 Three of the eight psychiatric symptom scales are of little clinical relevance to the vast majority of pre-adolescents; namely substance use, psychotic symptoms, and non-accidental injury. Two studies reported that a 3-month reporting interval may be too short a period of time for reviews.4,26
Several studies in clinical (rather than research) settings have found poor rates of data completion for the HoNOSCA, particularly at follow-up and discharge.5,20,26 A trend for completion rates to diminish over time has also been noted, raising concerns that completing outcome measures is onerous or is seen by clinicians as not providing quality feedback.20, 26 Of interest, in a head-to-head comparison in clinical practice, completion rates were higher for the CGAS than the HoNOSCA.20
Patients experiencing some very significant disorders, usually associated with significant disability, have had unexpectedly low scores on the HoNOSCA. Garralda and Yates found children with pervasive developmental disorders had a mean total HoNOSCA score of 11.5, while children with eating disorders had a mean total score of 7.6, compared with a mean total score for all disorders of 11.58.19
Children with no disorder had a mean score of 7.46, virtually the same as those with eating disorders. Similarly low scores for patients with eating disorders were found in a further study (mean score of 7.1),20 but not in the Bilenburg et al study (mean score of 16.8).4 Such variability in the few studies to date inevitably leads to doubts about the HoNOSCA’s suitability as a universal measure.
Despite reasonably widespread introduction of the HoNOSCA to the United Kingdom and Australia, it is concerning that only a relatively small number of studies have been published involving the HoNOSCA. Additionally, if the HoNOSCA was a robust outcome measure, it is reasonable to expect it would be more widely utilised in randomised controlled trials.
Instead, many major randomised controlled trials utilise other clinician-rated global measures of change such as the Clinical Global Impression (CGI) scale.28 This very brief, simple and easily completed scale also potentially allows direct comparison of clinician, parent and child or adolescent views on the benefits (or otherwise) of treatment.
Child and adolescent mental health services in New Zealand remain significantly under-funded, and further progress toward improving this appears to have stalled. The mandatory requirement for universal reporting with routine outcome measures such as the HoNOSCA—despite no evidence that this is of benefit to patients or that it improves the quality of services—is premature. The possible adverse effects of introducing routine outcome measurement to CAMHS have been inadequately evaluated. Furthermore, the requirement to utilise the HoNOSCA has not been costed and comes with no additional funding, thus it can be expected to further erode clinical time. This in turn is likely to adversely affect access to CAMHS for children and adolescents.
The HoNOSCA has some significant limitations, particularly for younger children and for some diagnostic groups such as pervasive developmental disorder and eating disorders. A number of other possible outcome measures remain worthy of reconsideration. For example, the CGAS, which compares well psychometrically and appears to have better clinical utility than the HoNOSCA.
Competing interests: None known.
Author information: Matthew J F Eggleston, Director of Training, Christchurch Advanced Child and Adolescent Psychiatry Training Programme, Canterbury District Health Board, Christchurch; William G A Watkins, Senior Lecturer, Department of Psychological Medicine, University of Otago, Christchurch
Correspondence: Matthew Eggleston, Director of Training, Christchurch Advanced Child and Adolescent Psychiatry Training Programme, The Walshe Centre, 1st Floor, 36 Oxford Terrace, Christchurch, New Zealand. Fax: +64 (0)3 3770267; email: firstname.lastname@example.org
issue | Search journal |
Archived issues | Classifieds
| Hotline (free ads)
Subscribe | Contribute | Advertise | Contact Us | Copyright | Other Journals