Many New Zealand hospitals are currently undergoing a process of digitisation.[[1]] There is no national roll-out of a single vendor electronic clinical record, but each hospital or region of hospitals has been left to adopt a “best-of-breed” approach to digitising the inpatient hospital record.[[1]] Internationally, there is significant published literature indicating low levels of clinical satisfaction with the digital electronic clinical record.[[2–5]]

Te Whatu Ora – Health New Zealand Waitaha Canterbury (HNZ-C) oversees the delivery of healthcare in the Canterbury region of New Zealand. It is made up of seven hospitals, and upwards of 13,000 staff members;[[6]] Christchurch Hospital is its largest campus, and also the largest hospital in New Zealand’s South Island (bed capacity: 600–650) serving a population upwards of 550,000 people.[[7]]

Christchurch Hospital has taken a hybrid approach to developing a complete inpatient clinical record. It has adopted commercially available solutions for observations, radiology and laboratory results, and medications. However, it has elected to co-design with a commercial vendor (Sense Medical Ltd.) a novel mobile iOS solution, Cortex. Cortex is an application that is used to deliver inpatient documentation and inter-professional communication, and to enable viewing other components of the inpatient clinical record (laboratory results, radiology, and observations).

The key principle of this approach was to co-design the solution and functionality with clinical and technical domain experts. It was acknowledged that provision of healthcare occurs within a complex adaptive system, and so a “perfect” final solution could not be articulated at the start of the process. Rather, it was accepted that “experimenting” would be required, and then to amplify functionality that added clinical value, while dampening functionality that did not. The co-design process involved the description by end-users of the problem they wished to solve. Developers then designed such functionality. The functionality was then deployed often to a small beta group, and further iterations made. This cycle continued until the desired functionality was achieved. Thus, a staged roll-out across the campus was planned after using early adopters (Department of General Surgery, Christchurch Hospital) to iterate and feedback on the product as it was developed. The roll-out process was performed in staged fashion by departments. It involved a small group of multi-profession representatives of the departments to meet on weekly basis over 8-week period. During this period department workflows were discussed and a combination of bespoke or standardised clinical notes designed for each. At the time of writing, the product is now fully deployed across an inpatient tertiary campus (Christchurch Hospital) with the exceptions of the Neonatal Intensive Care Unit (working on a national system), and the Emergency Department (ED) (decided to remain on their current system until a desktop version of Cortex is developed).

The definition of usability has been operationalised to refer to a measure of how effectively a product or application is used to perform the function for which it was designed.[[8]] In fact, the International Organisation for Standardisation (ISO) has published criteria, definitions, and standards for usability (e.g., ISO 9241–210).[[9]] The System Usability Scale (SUS) is a validated scale created in the 1980s,[[8]] and is considered a standard tool for use in usability research (appearing in upwards of 1,500 studies across a wide spectrum of products and industries[[10]]). It is comprised of a 10-item survey for which participants answer each question on a 5-point response scale. The overall score (which involves a complex calculation based upon the participant’s responses) ranges from 0 to 100. Interpretation of the scores is provided in Figure 1.[[11]] It is noteworthy that responses from as few as 20 participants yield valid and reliable results.[[10]]

For a publicly funded health system, especially one with frequent budgetary deficits[[12]] that is required to accommodate an increasingly complex and large volume of patients (due to a system of universal health coverage in New Zealand), the implications of information systems usability are considerable.[[13]] With the recent merging of New Zealand’s 20 district health boards (DHBs) into a single national health service (Te Whatu Ora – Health New Zealand),[[14]] there is an opportunity for broader utilisation of health-related products and applications with high usability, as this may improve the hospital experience for patients and staff alike.[[1]]

The objective of the present study was to determine the usability of the inpatient clinical documentation and communication platform known as Cortex approximately one year after full deployment at Christchurch Hospital. Assessment of its usability may be seen as a way of objectively measuring one facet of the success of the product, as well as identifying areas that needed further enhancement.

Methods

Study setting and participants

For the current study, all HNZ-C staff were invited via their work email to complete an online questionnaire. A link to the survey was also available on the Intranet page. The study ran over a 10-day period (the survey link was included in the daily “Staff Communication Update”). This study is part of a larger project (Studying Usability of Christchurch Electronic health record Systems [SUCCESS]), and was approved by the University of Otago Human Ethics Committee (reference: D22/086).

Study survey

Participants were invited to anonymously complete the study survey online (using Typeform© as the interface). It was comprised of the SUS survey, one question about the participant’s clinical profession within the organisation, and a final open-ended question for any additional comments. The survey was only available in the English language. All respondents had consented to participate in the study.

Data analysis

Descriptive statistics were utilised to present the data (expressed as medians and interquartile ranges [IQR], or proportions). Qualitative data were analysed using a general inductive approach which involved coding responses, and grouping codes into common themes via an iterative process. The first author of the research team coded the data; codes were then reviewed and revised by the whole research team.

Results

A total of 144 responses were received during the study-period. Participants were made up of 45 junior doctors (31%), 17 senior doctors (12%), 2 senior medical students (1%), 12 junior nurses (8%), 37 senior nurses (26%), and 32 allied health staff (22%). The median SUS score was 75 with an IQR of 60–87.5. The median IQR SUS scores did not significantly differ among the different occupation groups: 78 (65–90) for doctors; 70 (57.5–82.5) for nurses; and 73 (55.6–84.4) for allied health staff (p=0.268).

View Figures 1 and 2.

A total of 70 participants (48.6%) provided qualitative responses. Three themes were identified through the analysis of the participants’ responses. These were: the need for integration with other electronic systems, implementation issues, and fine-tuning the functionality of Cortex.

The need for integration with other electronic systems

Several respondents commented on the current poor integration of the various electronic health systems in use at Christchurch Hospital. Many thought that Cortex ought to integrate the data from the other systems/applications.

“Needs to be more integrated with [another application]—Cortex would be a million times more helpful if I could access emergency department notes, clinic letters, etc.” – Participant 15

“It would be easier to use if [3 applications] were linked better. You have to go in and out of the different programmes multiple time in a shift.” – Participant 27

Despite this, many participants found Cortex very useful. Several lamented the fact their departments/areas do not use Cortex.

“Love Cortex. Desperately needs to be introduced at [a different hospital in Canterbury]” – Participant 69

“It would be great for all ED and other hospitals such as [a different hospital in Canterbury] to be on Cortex, so patient info is all in one system, rather than paper and electronic.” –  Participant 44

“Cortex is great! Truly one of the greatest IT developments to happen for [hospital]." – Participant 20

Implementation issues

The second theme, issues with implementing Cortex as the inpatient health system, was highlighted by multiple participants. Such issues included the fact that Cortex currently requires iOS system/Apple Inc. devices, the staggered roll-out of Cortex, and the frequency of soft-ware or clinical note updates.

“I think the biggest deficiency is you can’t write a Cortex note on the computer. It would be a phenomenal system if this was the case.” – Participant 30

“The fact that Cortex requires iOS rather than PC [personal computer] is incredibly frustrating and shortage of hardware to support this leads to poorer patient documentation.” – Participant 16

“Excellent app; [I] like the constant development and improvement.” – Participant 47

“Our department has put in a bunch of work in the last year to be more consistent in our notes. However, due to the staggered roll out of Cortex, this really should have been done at the beginning.” – Participant 7

Fine-tuning the functionality of Cortex

A number of participants expressed frustrations with, or suggested improvements to the functionality of Cortex. These included repetitive information carried over in several notes, asynchronous communications with other team members, and sub-optimal search functionality within the application.

“Just hard to filter previous reviews by allied health many days prior.” – Participant 14

“Cortex is great. Only downside is number of forms on Cortex and inconsistencies with documentation (i.e., progress note, routine note, ward round note, etc) and not all of these coming up as medical notes when searching.” – Participant 27

“I believe that since Cortex has arrived, there is less actual verbal communication with doctors about managing our patients.” – Participant 36

“One of the best features is the ability to message team members and see the responses added to the clinical notes, make it easy to ask for and follow orders.” – Participant 49

“Not very searchable for forms/orders when I don’t know the specific name of the form I’m looking for, and there are lots of forms for one specialty.” – Participant 18

Discussion

The current study revealed a good usability score of the inpatient digital clinical documentation and communication platform, Cortex. A diverse variety of responses across the professions was received. The overall median SUS score of 75 equates to a usability between “good” and “excellent” on the adjective rating of the SUS, indicating superior usability especially among health-related applications (see Figure 2).

The results of this study are encouraging—especially considering the fact that health-related electronic applications have not, in general, had a good track-record of usability. For example, a recent study by Bloom et al. found median SUS scores for the electronic health record system in all the UK’s emergency departments to range between 35 and 65.[[13]] Therefore, none of these systems meet the internationally validated standard of acceptable usability for information technology (IT; see Figure 2).[[6]]

The user experience was equivalent among the various professions of the study’s participants (doctors, nurses, and allied health staff). This is an important aspect as much of the documentation and communication is done by team members other than medical professionals.

The two main user concerns were the dependence of Cortex on the iOS system, and the lack of integration with other electronic health systems within the hospital eco-system. iPads and iPhone were provided for clinical staff to document and communicate within the application. The roadmap of future development includes a web-based version of Cortex. Cortex set out to be mobile first, and so this issue highlights an agile approach to product development which requires trade-offs with prioritisation of desired functionality. However, other systems (e.g., medication charting, outpatient letters, and requesting radiological investigations) remained PC-based, and some could not be accessed by Cortex due to their underlying legacy architecture. As Hira (a national platform, currently under development, that will connect health information systems) comes into evolution as part of the health reforms in New Zealand,[[1]] it is hoped that access to a standard set of patient information will be possible. However, within hospital systems, seamless switching between applications or data transfer between individual vendor systems will be dependent upon the widespread deployment of Fast Healthcare Interoperability Resources standards.[[15]]

Although Cortex documentation may be viewed within the Hospital’s PC-based electronic clinical portal, some users also wished to be able to complete documentation via the desktop-based clinical portal. The lack of this functionality was one of the main drivers why Cortex was not deployed to the ED, which crosses both outpatient- and inpatient-type workflows. Such functionality currently remains within the roadmap, and is a good example of the importance of iterative design and understanding end-users’ requirements for a minimal viable product. Despite this, it must be acknowledged that Cortex’s success, at least in part, could be attributed to its truly “mobile-first” approach. Multiple (if not all) members of the clinical team possess iPad devices which allows for instant access to, and addition/modification of, a patient’s clinical notes. In addition, the tasking and communication component also allow for a quick, yet asynchronous, way of communicating among other members of the team who may be at geographically distant locations (within a large hospital ward, across the campus, or beyond). This is increasingly important as hospitals come under growing occupancy pressures, and the corresponding increase in outlying patients across multiple wards is a regular feature of clinical teams’ daily working life.

Creation of Cortex documentation has largely employed a dual approach. Forms that are intended to be used hospital-wide have been standardised. Department-specific forms, on the other hand, have been left to each clinical department to develop independent of the technical team. Shifting the locus of control to the end-users for content development enables quick and multiple iterations as feedback from both document creators and readers accumulates. Typically, a tension exists between these two groups, and so this shift of creation control to end-users helps to ensure open dialogue, and form consensus and understanding of each other’s needs helps in reaching a better overall solution. As this process evolves, it does result in multiple form deployments. Although some may see this as a negative, others may perceive it as a positive. The quick iteration of feedback is perceived as people listening to the feedback provided. It also allows clinical form builders to go through a learning process, and to enhance workflows as they develop deeper understanding of the functionality of the product.

Reducing the amount of redundancy of information in clinical documentation is an important concept to increasing usability. Within the application functionality, being able to easily import standardised panels of information (e.g., past medical history or the problem list) across notes has been an important step forward within the application. However, the scalable importing of information from other applications [using the Fast Healthcare Interoperability Resources standards; 15] remains elusive. To solve this will require incentives for vendors or legislation to ensure a connected system. This is an important problem to overcome if Te Whatu Ora – Health New Zealand is to continue the concept of “best-of-breed” as its digital strategy.

The approach taken to the development of Cortex did not come from knowing what the solution was going to be at the start of the project. It consisted of significant experimentation, testing and iteration with a group of early adopters. There were several points where a “pivot” was undertaken given the failure of an “experiment” or feedback from users. This is an important concept for the New Zealand health digital ecosystem as it transitions to a national system. Partnership with clinical users from the start is key to ensuring fitness with the clinical workflow. It would seem unwise to “bet big” on knowing the solution before one starts; instead, adopt the principles associated with successful innovation within a complex adaptive system and improve with cumulative experience.

The current study is not without limitations. The number of responses to the survey has been low, although the validity of the SUS tool for responses as few as 20 mitigates this limitation. Additionally, information on the participants’ service/department was not collected, as the usability may have differed among departments. However, there was representation from across the professions further enhancing the validity of the current study. It is possible that responders were more digitally literate/technologically savvy than non-responders. Whilst this could have biased the results towards high usability, it would be prohibitively difficult to distribute the survey to such a large scale on paper, or attempt to complete the surveys in person.

The median SUS score of Cortex seems superior to most other electronic health record systems published in the literature, although direct comparison may not be possible owing to the differences in the platforms. However, the present study provides a useful benchmark for Cortex at a point-of-time, and sets up potential to periodically repeat this survey to see how new functionality has added to (or detracted from) its usability. It could also be incorporated into exit surveys for staff as they rotate through roles in the organisation. Finally, health enterprises (e.g., Te Whatu Ora – Health New Zealand) could integrate assessments of usability (such as SUS scores) into future procurement processes. This would allow ongoing assessment of an increasingly digitalised health system to be assessed. It would seem critical at a time when workforces are struggling to cope with clinical demand.