He mana tō te kupu. Words have great power. This whakataukī (Māori proverb) speaks to the value of words that Patricia Leavy also references when she says: “People must be able to use their voice, tell their stories, have their experiences recognised and their voices heard”.[[1]] As the lead author (AL), and in reflecting upon my own experience of the health system, I have felt seen and validated when services have listened to my feedback, both positive and constructive.

Within the health system, one method of hearing and honouring the voice of the people is to gather and utilise data about people’s experiences of services.[[2]] However, it is not sufficient to just collect satisfaction or experience data.[[3]] To truly recognise and hear the voices of people who experience mental health challenges and substance use issues (tāngata whai ora – people seeking wellness) it is imperative to act on information about people’s service experiences. This is the cycle of continuous quality improvement that evidences more effective services. Quality improvement aims to make a difference to tāngata whai ora by improving the “safety, effectiveness, and experience of care”,[[4]] and the completion of this feedback cycle returns power to the people receiving services. This article advocates that when measures of tāngata whai ora reported experience are an integral component of quality improvement activities, there is potential for powerful change resulting in improved support and outcomes for people.

He Ara Oranga: Report of the Government Inquiry into Mental Health and Addiction concludes that the “consumer voice needs to be supported, strengthened and included in all aspects of the system, from governance to service delivery”.[[5]] The Health and Disability System Review also highlights the importance of centring the health system around the needs of people.[[6]] More recently, Kia Manawanui, the Ministry of Health’s long-term pathway to mental wellbeing, commits to “amplify the voices and strengthen the leadership of [...] people with lived experience”. It further commits to “set expectations that funders, commissioners and providers of mental wellbeing services and supports will proactively seek out the voices of these groups and establish mechanisms to obtain their input”.[[7]]

In line with the international literature,[[2]] these three documents signal an increasing focus on listening to, and acting upon, the voices of tāngata whai ora and their whānau within mental health and addiction services, and health services more broadly. The policy environment in Aotearoa New Zealand is shifting to place greater value on gathering and understanding information from people accessing health services. Gathering experience data and feedback is one way of understanding what tāngata whai ora and their whānau want from services. Importantly, feedback from tāngata whai ora Māori and whānau voices are critical for informing strategies to address the health inequities experienced by Māori.[[8]]

This viewpoint article is written explicitly from a tāngata whai ora perspective, and intends to highlight topical issues for lived experience and other mental health and addiction sector leaders on the subject of quality improvement with specific reference to the use of a locally developed tool, Mārama Real Time Feedback (Mārama). The article describes international practice in the area of person-reported experience measures, with a particular focus on mental health and addiction, then goes on to describe a solution developed and implemented in Aotearoa New Zealand. Finally, we comment on the future direction and need for tāngata whai ora led experience measures.

It is noted that this article takes a strengths-based approach to the use of language, based upon the conventions of Te Reo Hāpai: The Language of Enrichment, a Māori language glossary for the mental health, addiction, and disability sectors.[[9]] As such, words (kupu) in te reo Māori are used in this article to include both Māori and non-Māori.

Experience measures are widely used internationally

Generally, health services can use two types of tools to capture peoples’ voices and perspectives. Patient reported outcome measures (PROMs) gather information about peoples’ perceived physical, mental and emotional health statuses to help services understand how they have impacted on peoples’ outcomes. On the other hand, patient reported experience measures (PREMs) gather information about the perceived quality of care. PREMs help services to understand how people feel about the service or support provided, what is important to them, and what opportunities there are for improvement.[[10,11,12]]

This viewpoint article focuses on PREMS, which the World Health Organization considers as an indicator of the quality and responsiveness of health services and systems.[[13]] Similarly, a recent paper from the Organisation for Economic Co-operation and Development (OECD) highlights that “patient-reported measures are a critical tool for improving policy and practice in mental health care”.[[2]] Many countries have widely implemented PREMS in health care settings, including New Zealand, Australia, the UK, US and Canada.[[14]] Over the last 20 years, there has been a deliberate international shift towards measuring peoples’ experience of services (rather than satisfaction), as this provides greater detail to enhance services’ abilities to make improvements to people’s experiences.[[15]]

PREMs can form one part of a feedback loop to inform service quality improvement. PREMs help identify areas of effective practice, and where and why people report positive and negative experiences.[[16]] In addition to quality improvement, people’s experience of services provides an indication of the quality of support that can be used for benchmarking. Having a standard set of questions and a consistent method of data collection enables people’s experiences to be compared over time and between services.[[16]] Standardisation also enables benchmarking at a national and international level[[16]] and can support shared learning across services.

Several foundational frameworks have guided development of consumer experience measures, including the Picker Institute[[17]] and Institute of Medicine.[[18]] Looking at other countries, there are numerous examples of consumer experience measures used in quality improvement and benchmarking. Examples of surveys designed for large-scale collection in mental health and addiction services include: Your Experience of Service Survey (Australia); Community Mental Health Survey (UK); Consumer Assessment of Healthcare Providers and Systems (CAHPS) – Experience of Care & Health Outcomes (ECHO) Survey (US); National Patient Survey (Sweden); Flemish Mental Health Services Survey (Belgium); and the Psychiatric Inpatient Patient Experience Questionnaire (Norway).[[2,14]]

A tool has been developed for Aotearoa New Zealand

In Aotearoa New Zealand, mental health and addiction services are required to collect and respond to feedback data about tāngata whai ora experiences in accordance with several policies. The Ngā Paerewa Health and Disability Services Standard requires all providers of health and disability services to ensure that people with lived experience of mental health issues or addiction, and whānau, participate in the planning, implementation, monitoring and evaluation of service delivery.[[19]] Under the previous health system structure, district health boards (DHBs) were required as part of the Ministry of Health’s Operational Policy Framework to undertake and report the results of a mental health consumer survey. DHBs were further expected to work with tāngata whai ora to implement systems for capturing consumer experience.[[20]] They must then demonstrate internal and external accountability through reporting to the public and clinical community, including any improvements and achieved changes. However, a limitation of this previous process was that it does not ensure data is translated into quality improvements.[[20]] For tāngata whai ora, it is crucial that services collect this information, implement change, and report on the actions taken as a result of feedback.

In 2013, the Office of the Health and Disability Commissioner (HDC), in its capacity to monitor mental health and addiction services, contracted CBG Health Research (CBG) to develop an electronic tool to enable tāngata whai ora and whānau to provide feedback directly to service providers in “real-time”. CBG led a collaborative process to develop an evidence-informed survey with the HDC and the Ministry of Health, which included involvement of people with lived experience. A literature review was undertaken to inform the survey design, from which CBG developed a set of potential questions.[[15]] These were used to initiate conversations with seven pilot sites, which included Māori and Pasifika services. Feedback gained through the pilot significantly influenced the final question design.[[21]] This led to the development of Mārama Real Time Feedback (Mārama).

Mārama is an online feedback collection tool that can be completed by tāngata whai ora and/or whānau at any point during a person’s contact with services. Feedback is provided using tablets, URL links, and QR codes. Work is also underway to enable mobile phone access to the survey using a URL link. The use of technology means feedback data is captured and reported in real time. Timely access to feedback data through effective use of information management and technology is essential for quality improvement initiatives.[[22]]

Mārama consists of seven questions that tāngata whai ora and whānau rate on a 5-point agreement scale (see Figure 1). The questions were developed to collect feedback from people of varied population groups and are available in seven languages, including te reo Māori. Importantly, Mārama includes a question about whānau involvement and provides the option for whānau to complete the survey from their own perspective. This honours and acknowledges the important role of whānau within Māori health models, such as Te Whare Tapa Whā.[[23]] Services can also add questions customised to their setting or local needs. Some services have used this opportunity to include a broad cultural question, for example “my culture and beliefs are respected”, and others have developed questions specific to Māori experiences of services to meet their local needs.[[24]] To ensure the tool is used to its full potential to meet the needs of Māori and Pasifika peoples, there is a need for further evaluation to be undertaken in this area.

Since its launch in 2014, Mārama has gathered data on the experiences of over 44,240 tāngata whai ora and whānau (as of 2 March 2022). Approximately 80% of the feeback is from tāngata whai ora and 20% from whānau. The aggregated data collected since 2014 indicates around one third (38%) of the feeback was collected from respondents identifying as Māori; 9% Pasifika; 4% Asian; and 49% New Zealand European/Other. These proportions have been fairly consistent over time. There is good represention of Māori perspectives among the feedback data (29% of people who accessed mental health and addiction services in 2019/20 were Māori).[[25]] Mārama is currently being used by a large proportion of DHBs (17) and an increasing number of NGOs (13), along with several primary health services. The trend towards increasing collection and use by NGOs looks likely to continue. Continuous review and improvement will remain essential to support the expansion of Mārama across the sector, including responsiveness to tāngata whai ora Māori and whānau.

Mārama provides a national quality measurement and reporting system for the sector. Mārama itself has been subject to continuous quality improvement, and technology developments have enabled greater uptake of the tool. Increased uptake of Mārama builds a larger data set that can be used by individual organisations, and nationally at an aggregated level. The collected data can be publicly viewed through tracking graphs and report cards available on the Te Pou website (tepou.co.nz) and participating organisations can access more detailed reports through the Mārama website (marama.co.nz). Figures 2 and 3 provide examples of how Mārama data are regularly reported at a national level.

View Figures 1–3.

People with lived experience provide critical leadership

People with lived experience of mental health challenges and or addiction issues who work within mental health and addiction services have been key to the implementation of Mārama. Among services with the highest Mārama collection rates in 2020, people with lived experience of mental health challenges and addiction issues were key to implementation. Lived experience leadership has been demonstrated through various roles in the implementation of Mārama, including involvement in Mārama project lead or champion roles, steering groups, as well as the coordination and direct involvement of feedback collection. Similarly, a sector consultation survey undertaken in 2020 found people working in mental health and addiction services agreed that lived experience leadership is important to the collection and use of Mārama feedback.[[24]] Lived experience leadership also plays a critical role in closing the quality feedback loop.

While leadership for the collection of feedback to inform quality improvement is not restricted to people working in lived experience roles, it is clear that a key consideration for services is how to utilise and demonstrate lived experience leadership in their use of Mārama. However, it is crucial that lived experience leadership is valued and implemented respectfully, and that it is not simply co-opted in to improve completion rates of Mārama questionnaires.

One organisation that meaningfully embeds lived experience leadership for Mārama is Odyssey, an addiction service provider in Aotearoa New Zealand. The organisation’s consumer advisor, Thomas White, has a lead role in taking responsibility for the implementation of Mārama, alongside other colleagues. Lived experience leadership is demonstrated in the collection of information, how it is shared and interpreted, and actions taken as a result. White describes the value of lived experience leadership in terms of “giving [tāngata] whai ora ownership of a direct source of communication [to the organisation]”. The connection to governance and decision making is a key part of closing the feedback loop and demonstrating real change in response to feedback collected through Mārama. White describes the benefits of “being able to bring change that people with lived experience may have experienced themselves in services and being a link to governance and oversight of programmes”.[[26]]

Feedback from tāngata whai ora requires action

While Mārama is a useful method of PREM data collection,[[21,24]] the value of the process is not simply in gathering more information, but in effectively analysing findings and using results to make service improvements. While there are excellent examples of the use of Mārama data, some services are still in the early stages of using the feedback to make improvements. Further work is needed to enhance organisational capacity and capability in using Mārama for quality improvement. Recent sector consultation suggests use of Mārama is maximised when organisations are focused on data use, not just collection rates, and have organisational capacity or tools to analyse the free text responses.[[24]]

The wider literature indicates improving service quality using PREMs requires commitment, an understanding of the data, dissemination of findings, use of quality improvement tools, and co-designed plans to action positive change.[[3,4,27]] At the organisational level, readiness for change and sustainable quality improvement requires senior management support and staff training, as well as infrastructure and processes that will enable change.[[3,4,27]] Implementing change is both a pragmatic and compassionate response to understanding how people experience services, and whether services have made a positive difference to people’s wellbeing during some of their most difficult times. Investment in the Plan-Do-Study-Act (PDSA) cycle can help organisations demonstrate this change and communicate changes made in response to feedback.[[4]]

One such PDSA method used by some organisations in Aotearoa New Zealand is the “you said, we did” approach. This approach presents Mārama results and actions taken in response to findings to both tāngata whai ora and service employees.[[24]] Northland DHB is implementing learnings from Mārama according to this “you said, we did” approach. Staff are involved in presenting Mārama findings using easily understandable graphs, as well as any plans that services have to address the feedback from tāngata whai ora. Paul Clements, a lived experience leader from Northland DHB, describes this process as “very important for people to know that what they have said makes a difference and they can see it, otherwise when we ask them later to provide feedback they will be reluctant and think ‘what’s the point?’”.[[28]] This level of accountability to tāngata whai ora and whānau validates the importance of the knowledge gained through PREMs, in this case Mārama, and demonstrates the utility of the information gathered.

Feedback from tāngata whai ora and whānau gathered through Mārama has been used in multiple ways by services. It is a key mechanism of sharing feedback with staff within some services, including executive leadership teams. Some organisations have developed dashboards which staff can access to support this process. While Mārama helps identify areas for service design and improvement, tāngata whai ora and whānau also use it as a mechanism to share positive feedback to staff. Services are using feedback to identify priority areas that need improvement in strategic plans. Thomas White from Odyssey says that “user commentary and key words have been used in data compilation to inform strategic and organisational pathways, creating objective evidence and robust discussions for change”.[[29]] The introduction of new initiatives, training, and clearer communication in some services has been based on feedback about how tāngata whai ora and whānau are feeling. These improvements are often driven by both Mārama data and co-design with consumer groups. Other services have focused on areas where they have lower agreement scores and identified these as priority focus areas, leading some to explore what this means for their service and what is required to make meaningful improvements in practice. The identification that whānau engagement scores collected through Mārama are low across Aotearoa New Zealand has also led to engagement with family whānau advisors and progress in developing a plan focused on this area.

Conclusion

He aha te kai ō te Rangatira? He kōrero, he kōrero, he kōrero. What is the food of the leader? It is knowledge. It is communication.

Feedback data, specifically PREMs, need to be collected in order to inform and support effective service delivery. As we look to the future there are opportunities to enhance the collection and use of feedback data. With health reforms currently underway towards a more centralised system ( Te Whatu Ora – Health New Zealand and Te Aka Whai Ora – Māori Health Authority), there is potential to move to a nationally consistent approach to mental health and addiction service feedback collection. Strategic leadership is required to ensure that feedback and quality improvement are embedded in the new health structure. There is a substantial amount of feedback that could be collected from tāngata whai ora and whānau through a national approach. In 2019/20, around 184,000 adults accessed specialist mental health and addiction services according to the Programme for the Integration of Mental Health Data.[[25]] This indicates the potential for the tāngata whai ora voice to make important contributions to service improvements.

Aotearoa New Zealand is now in a strong position to collect and respond to the service experiences of tāngata whai ora and whānau with Mārama. Mārama is a viable, useful and effective method for gathering feedback, and has seen good success in implementation across a range of service settings. Aotearoa New Zealand is placing increasing emphasis on hearing and responding to the voices of tāngata whai ora. This leads to opportunities for the growth and embedding of Mārama as a data collection and quality improvement tool. Lived experience leadership is a pivotal factor in the use of Mārama. The pairing of effective quality improvement through PREMs, with lived experience leadership, has the potential for tangible and positive improvements in service delivery for tāngata whai ora. He Ara Oranga directly encourages use of “a real-time feedback tool like Mārama”[[5]] as a method to receive feedback from tāngata whai ora and their whānau. This article further articulates the relevance of this tool.

As lead author, thinking about my own lived experience, I have felt the most personal power when people providing mental health services have changed how they work in response to the feedback I have given them. In a system that inherently detracts from my decision-making and opinion, this feeling of power cannot be undervalued. We have opportunities now to make real changes to services in response to the experience data collected in Aotearoa New Zealand, and these changes will enhance the mana of our tāngata whai ora.