Cataract surgery is the most frequently performed surgical procedure in New Zealand with approximately 16,500 publicly funded cataract surgeries completed annually.1–3 With limited resources for publicly funded surgery, prioritising patients for cataract surgery is essential to enable equal access to surgery for all New Zealand residents and ensure those who are most likely to benefit from surgery are prioritised highest. The New Zealand public health system currently utilises a standardised Clinical Priority Assessment Criteria (CPAC) that involves priority scoring to determine patient eligibility for publicly funded elective surgical services including cataract surgery.4 The CPAC system aims to improve equity of access to surgical services across New Zealand, enhance transparency around prioritisation for surgery and improve certainty regarding treatment for patients who require surgery.5

Prioritisation for cataract surgery in New Zealand using the CPAC system is based on weighted scores for patient responses to the Impact on Life (IoL) questionnaire, best corrected visual acuity (BCVA) and cataract morphology.6 The IoL questionnaire is intended to quantitatively score patient-reported functional status in six qualitative domains that include safety, social interactions, responsibility to others, personal relationships, personal care and leisure activities. The IoL questionnaire was not designed specifically for use with cataract or ophthalmic surgery, and was initially developed for prioritisation in orthopaedic and other surgical specialities.7 Despite the national adoption of the IoL as an integral component of CPAC prioritisation for cataract surgery in New Zealand, the ability of the IoL questionnaire to assess vision-related quality of life (VRQoL) has not been formally assessed.

The use of patient-reported measures has gained wide acceptance in ophthalmology following development of cataract-related visual disability questionnaires.8–10 The International Consortium for Health Outcomes Measurement (ICHOM) has convened global groups of experts and patient representatives to outline minimum standard outcomes using a structured process for a variety of specific conditions including cataract based on evidence-based measures to assess quality of life related to vision.11 The resulting Catquest-9SF questionnaire has been extensively validated as an accurate tool for assessment of patient-reported visual disability for patients undergoing cataract surgery.12 The Catquest-9SF is well suited for use in clinical practice due to its validity, brevity and ease of use, however, this questionnaire has not been validated in a New Zealand population.13

The aim of the current study is to validate and compare the ability of the IoL and the Catquest-9SF to measure VRQoL for New Zealand patients undergoing cataract surgery.

Methods

Formal approval from the New Zealand Health and Disability Ethics Committee was obtained prior to patient recruitment (16/CEN/132), and this study was registered with the Australian New Zealand Clinical Trials Registry (12616001593426). This is a prospective observational cohort study involving patients enrolled for routine cataract surgery at Greenlane Clinical Centre, Auckland District Health Board, New Zealand.

Patients who were referred for publicly funded surgery at Auckland District Health Board were invited to participate in the study. Patients who agreed to participate in the study completed both questionnaires before surgery and at again three months following surgery. All patients completed the IoL and Catquest-9SF questionnaires while the clinician was not in the room and the questionnaires were collected by an independent investigator.

The six-question IoL questionnaire requires patients to score the degree of difficulty that poor vision affected their social interactions, personal relationships, ability to meet responsibilities to others, personal care, personal safety and leisure activities using an ordinal scale (Figure 1). For each question on the IoL questionnaire, patients are required to select one option from a scale labelled ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’.

Figure 1A: The Impact on Life questionnaire, currently used in the Clinical Priority Assessment Criteria (CPAC) to determine patient eligibility for publicly funded elective cataract surgery in New Zealand.

Figure 1B: The Catquest-9SF questionnaire, developed by the International Consortium for Health Outcomes Measurement to assess quality of life related to vision as a result of cataracts.  

The Catquest-9SF is composed of three sections that require patients to select an option from a five-point Likert scale including one option of ‘cannot decide’ (Figure 1). The questions included: “Do you find that your sight at present in some way causes you difficulty in your everyday life?”; “Are you satisfied or dissatisfied with your sight at present?”; “Do you have difficulty with the following activities because of your sight?”. This last question allowed patients to label their satisfaction with vision in various contexts: reading text in newspapers; recognising the faces of people they meet; seeing the prices of goods when shopping; seeing to walk on uneven surfaces eg, cobblestones; seeing to do handicrafts/woodwork; reading subtitles on television; and seeing to engage in an activity/hobby of interest.

All surgery, and assessments before and after surgery, were completed by a single surgeon who performed the operation using standardised surgical technique, intraocular lens and emmetropic refractive target.

Statistical analysis

A group of statistical models termed the Item Response Theory (IRT) have been developed to instrument questionnaire development, evaluation and refinement. This framework analyses individual components of a questionnaire by a set of properties that describe the relationship of the questionnaire with the underlying construct measured by the model, in addition to how well individual questions fit with respect to the underlying construct. IRT is not dependent on the sample of respondents.14,15 This allows researchers to identify the questions that can most accurately measure the intended purpose of the questionnaire.

The Rasch model is a robust and commonly used form of IRT which can be used to assess functioning of rating scale categories within the Catquest-9SF and IoL questionnaires. This is a mathematical framework that takes into account the ability of participants, the difficulty of questions in the questionnaire, and assumes equal discriminating ability across all questions.16 In the Rasch model, the probability of a particular response to a specific question can be modelled as a logistic function of the difference between the person’s ability (measured by using test questions) and the difficulty of the items being asked.17

All IoL and Catquest-9SF question responses were assessed using the Rasch model to assess the validity of the questions in quantifying VRQoL. If responses to a question successfully fit the Rasch model, it provides evidence that this question adequately measures VRQoL. Two types of mean square fit statistics (infit and outfit) were used to evaluate how well patient responses fit the Rasch model for all of the questions within the IoL and Catquest-9SF questionnaires. Infit and outfit statistics have a chi-square distribution and provide an index of magnitude for the degree of misfit of a question with the model. These fit statistics have an expected value of 1 and suggested acceptable lower and upper thresholds of 0.5 and 1.5 respectively.18 Fit statistics for each question were calculated using an average of the squared residuals between the observed and expected responses from the Rasch model. The infit statistic is an estimate that gives more weight to individual variance of questionnaire responses to minimise the impact of unexpected responses far from the mean. Conversely, the outfit statistic is an unweighted estimate of the average question response variance within the IoL and Catquest-9SF questionnaires, and is more likely to be influenced by unexpected responses.

All statistical analyses were completed using R software.19 IoL questionnaire data were coded 1–6 representing the options of ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’, respectively. Catquest-9SF questionnaire data were coded 1–5 representing the options of ‘no difficulty’, ‘some difficulty’, ‘great difficulty’, ‘very great difficulty’ and ‘cannot decide’, respectively. The mirt package was used to fit models for item response theory analysis.20 Preoperative and postoperative data were combined for model fitting. The corrected Akaike information criterion (AICc) is an estimator of the relative quality of statistical models for a given set of data, and was used to select the best model. The G–test of goodness-of-fit was used to determine if the final model accurately predicted the data. Normalised factor scores for both questionnaires were correlated with visual acuity in the operated eye and age using Pearson’s product-moment correlation. Secondary analyses of normalised factor scores by ethnicity and gender were performed using analysis of variance. A qualified statistician reviewed all statistical methodology and analyses used in this study.

Results

Forty-one patients undergoing cataract surgery were enrolled in the study from March to May 2017. All patients who were approached consented to inclusion in the study and completed the questionnaire at both time points. The mean patient age was 77±8 years (sd), with 20 (49%) female participants. Ethnicity included New Zealand European 29 (71%), Māori 3 (7%), Pacific Island 1 (2%), Asian 4 (10%), Indian 3 (7%), and ‘Other’ 1 (2%). Table 1 shows the preoperative and postoperative visual acuities and spherical equivalent for the patient cohort.

Table 1: Visual acuity and spherical equivalent of 41 patients before and following cataract surgery.

‍

UCVA = uncorrected visual acuity, BCVA = best corrected visual acuity, SPE = refractive error (spherical equivalent) in dioptres. Spherical equivalent = sphere power + (cylinder power x 0.5). Postoperative visual acuity was measured at three months following cataract surgery and all visual acuity was represented in logMAR notation.

Figure 2A: Rasch model category probability curves for the Impact on Life questionnaire.

These curves summarise the probability (y-axis) that a patient with visual difficulty (x-axis) would answer with a given response. P1 to P6 represents the question response options; P1 = no difficulty, P2 = little difficulty, P3 = some difficulty, P4 = quite difficult, P5 = very difficult, P6 = extremely difficult.

Figure 2B: Category frequency responses for 41 patients who completed the Impact on Life questionnaire before surgery (pink) and three months following surgery (blue).  

Table 2: Summary of Rasch model fit statistics for the Impact on Life (IoL) questionnaire. The sample includes responses from 41 patients preoperatively and at three months following cataract surgery.

‍

The model fit statistics for the Catquest-9SF responses are summarised in Table 3. ‘Cannot decide’ responses on the Catquest-9SF questionnaire represented 7 of 738 responses (0.95%) and were assumed equivalent to data missing at random for analysis. Apart from ‘recognising faces’, ‘seeing price of goods when shopping’ and ‘ability to read TV subtitles’ (mean-square fit statistics 0.43, 0.44 and 0.48 respectively), all other Catquest-9SF questions were within the range suitable for measurement (mean-square outfit statistic 0.5 to 1.5). The graphical Rasch categorical probability curves for the Catquest-9SF questions are summarised in Figure 3A. The category frequency of each response in the IoL questionnaire is summarised in Figure 3B.

Table 3: Summary of Rasch model fit statistics for the Catquest-9SF questionnaire from International Consortium for Health Outcomes Measurement (ICHOM). The sample includes responses from 41 patients preoperatively and at three months following cataract surgery.

‍

Figure 3A: Rasch model category probability curves for the Catquest-9SF questionnaire from International Consortium for Health Outcomes Measurement.

These curves summarise the probability (y-axis) that a patient with visual difficulty (x-axis) would answer with a given response. A higher number of question difficulty indicates greater disability (6 = extremely difficult; -6 = no difficulty). P1 to P4 represents the question response options; P1 = no difficulty; P2 = some difficulty; P3 = very difficult; P4 = extremely difficult.

Figure 3B: Category frequency responses for 41 patients who completed the Catquest-9SF questionnaire before surgery (pink) and three months following cataract surgery (blue).

The difference in visual acuity before and after surgery correlated with the change in total F-score for the Catquest-9SF responses (P=0.04), but not the IoL responses (P=0.17). The overall questionnaire score in both IoL and Catquest-9SF questionnaires correlated with worsening visual acuity (P<0.001). There were no statistical differences in quality of life scores between ages or ethnic groups for both questionnaires. The change in F-score was not significantly different for patients who received cataract surgery on their first eye or second eye.

Discussion

The current study uses Rasch analysis to evaluate the validity of the IoL and Catquest-9SF questionnaires for quantifying VRQoL for cataract surgery patients in New Zealand. As far as the authors are aware, this is the first study to statistically assess the validity of the IoL questionnaire for use in cataract surgery, and the first study to compare the IoL with any other questionnaire to assess VRQoL.

The unequal peaks noted in the Rasch analysis for the IoL questionnaire suggest that the response options of ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’, are too numerous and ideally should be collapsed into fewer options with more consistent probability thresholds. This finding was consistent for all six questions in the IoL questionnaire. In contrast, the Catquest-9SF demonstrated relatively uniform peak height for the question response options that remained consistent for all questions in the Catquest-9SF questionnaire, similar to previous studies.21

The IoL questionnaire demonstrates category disordering on Rasch analysis. Category disordering occurs when the ordinal numbering of categories (response options) does not correspond with their substantive meaning. The IoL questionnaire ordered response options are substantively defined to represent increasing levels of disability in VRQoL. In all six questions, there is substantive and step disordering such that ‘little difficulty’ consistently locates below ‘no difficulty’ in the Rasch analysis. This finding suggests that the response options used in the IoL questionnaire are not able to accurately discriminate increasing impairment in VRQoL as intended.

The IoL questionnaire demonstrated unsatisfactory statistical fit of almost all questions (mean-square fit less than 0.5). This finding indicates less variation in participant responses than expected, and that responses are more predictable than the Rasch model expects. The high predictability of responses to IoL questions and overfit to the Rasch model suggests sub-optimal question wording resulting in non-discriminatory patient responses.22 This finding suggests that the IoL questionnaire is likely to lack the required sensitivity to accurately rank patients based on VRQoL.

Catquest-9SF questions demonstrated satisfactory mean fit squares and appropriate category response curves with monotonic increases and decreases in the category thresholds (Figure 3A). This finding was consistent with other studies evaluating the Catquest-9SF using Rasch analysis in Europe and Australia.21,23 These results confirm that the Catquest-9SF questionnaire is valid tool for the assessment of VRQoL in a New Zealand population and can accurately rank patients based on VRQoL.

Questionnaire scores for the Catquest-9SF and the IoL improved with the improvement in visual acuity following surgery. Only the Catquest-9SF questionnaire, however, demonstrated significant correlation between the change in visual acuity and change in questionnaire F-scores following surgery. The F-score is a single indicator that summarises the variance (accuracy and recall ratio) of data points around the mean, which can be used to evaluate and compare the fit of multiple linear models.24 Based on these results, the IoL questionnaire responses appear to be independent to VRQoL and poorly suited for predicting which patients will experience quality of life gains as a result of improved vision following cataract surgery.

There are several limitations to this study. Firstly, patient bias may influence questionnaire responses. Patients may suspect that preoperative questionnaire responses could affect their eligibility for surgery and bias towards over-reporting poor quality of life prior to surgery or after surgery where second eye surgery is required. The lack of significant difference in F-scores between patients receiving first or second eye cataract surgery, however, suggests similar degrees of variance in responses indicating no such bias exists in this data. Secondly, the current study has a relatively small sample size. Reports of Rasch analysis results are considered to be robust to smaller sample size.25 In addition, despite the small sample size, the current study was able to replicate similar findings to previous, larger studies evaluating the Catquest-9SF.21,23

The primary strength of this study is analysis of the qualitative responses using Rasch analysis. The importance of Rasch analysis has been well-recognised for the evaluation of questionnaire quality and there have been numerous requests for the development of Rasch-approved questionnaires within ophthalmology.26–28 The current study offers the first Rasch assessment of the IoL questionnaire. This questionnaire is currently in widespread use to assess eligibility for all patients in the New Zealand public health system that require cataract surgery.

In summary, the current study compared the ability of IoL and Catquest-9SF questionnaires to accurately measure VRQoL. The results of this study demonstrate that the IoL does not accurately assess VRQoL for patients that require cataract surgery in New Zealand. The Catquest-9SF is a domain-specific assessment tool that can accurately measure VRQoL in New Zealand. The convenience of using a single tool, such as the IoL, to allocate healthcare resources across multiple specialities must be carefully weighed against the risk of not allocating resources where they are needed the most.

Despite its widespread use, the current study highlights inadequacies of the IoL questionnaire for the assessment of VRQoL for cataract surgery in New Zealand. In addition to any role in surgical prioritisation, it is increasingly important for quality improvements in healthcare delivery to use standardised patient reported outcome tools, such as the Catquest-9SF. These standardised tools enable international benchmarking and direct comparison with other studies. In conclusion, the Catquest-9SF questionnaire provides a more accurate assessment of VRQoL than the currently used IoL questionnaire for New Zealand patients that require cataract surgery.

‍