View Article PDF

Cataract surgery is the most frequently performed surgical procedure in New Zealand with approximately 16,500 publicly funded cataract surgeries completed annually.1–3 With limited resources for publicly funded surgery, prioritising patients for cataract surgery is essential to enable equal access to surgery for all New Zealand residents and ensure those who are most likely to benefit from surgery are prioritised highest. The New Zealand public health system currently utilises a standardised Clinical Priority Assessment Criteria (CPAC) that involves priority scoring to determine patient eligibility for publicly funded elective surgical services including cataract surgery.4 The CPAC system aims to improve equity of access to surgical services across New Zealand, enhance transparency around prioritisation for surgery and improve certainty regarding treatment for patients who require surgery.5

Prioritisation for cataract surgery in New Zealand using the CPAC system is based on weighted scores for patient responses to the Impact on Life (IoL) questionnaire, best corrected visual acuity (BCVA) and cataract morphology.6 The IoL questionnaire is intended to quantitatively score patient-reported functional status in six qualitative domains that include safety, social interactions, responsibility to others, personal relationships, personal care and leisure activities. The IoL questionnaire was not designed specifically for use with cataract or ophthalmic surgery, and was initially developed for prioritisation in orthopaedic and other surgical specialities.7 Despite the national adoption of the IoL as an integral component of CPAC prioritisation for cataract surgery in New Zealand, the ability of the IoL questionnaire to assess vision-related quality of life (VRQoL) has not been formally assessed.

The use of patient-reported measures has gained wide acceptance in ophthalmology following development of cataract-related visual disability questionnaires.8–10 The International Consortium for Health Outcomes Measurement (ICHOM) has convened global groups of experts and patient representatives to outline minimum standard outcomes using a structured process for a variety of specific conditions including cataract based on evidence-based measures to assess quality of life related to vision.11 The resulting Catquest-9SF questionnaire has been extensively validated as an accurate tool for assessment of patient-reported visual disability for patients undergoing cataract surgery.12 The Catquest-9SF is well suited for use in clinical practice due to its validity, brevity and ease of use, however, this questionnaire has not been validated in a New Zealand population.13

The aim of the current study is to validate and compare the ability of the IoL and the Catquest-9SF to measure VRQoL for New Zealand patients undergoing cataract surgery.

Methods

Formal approval from the New Zealand Health and Disability Ethics Committee was obtained prior to patient recruitment (16/CEN/132), and this study was registered with the Australian New Zealand Clinical Trials Registry (12616001593426). This is a prospective observational cohort study involving patients enrolled for routine cataract surgery at Greenlane Clinical Centre, Auckland District Health Board, New Zealand.

Patients who were referred for publicly funded surgery at Auckland District Health Board were invited to participate in the study. Patients who agreed to participate in the study completed both questionnaires before surgery and at again three months following surgery. All patients completed the IoL and Catquest-9SF questionnaires while the clinician was not in the room and the questionnaires were collected by an independent investigator.

The six-question IoL questionnaire requires patients to score the degree of difficulty that poor vision affected their social interactions, personal relationships, ability to meet responsibilities to others, personal care, personal safety and leisure activities using an ordinal scale (Figure 1). For each question on the IoL questionnaire, patients are required to select one option from a scale labelled ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’.

Figure 1A: The Impact on Life questionnaire, currently used in the Clinical Priority Assessment Criteria (CPAC) to determine patient eligibility for publicly funded elective cataract surgery in New Zealand.

c

Figure 1B: The Catquest-9SF questionnaire, developed by the International Consortium for Health Outcomes Measurement to assess quality of life related to vision as a result of cataracts.  

c

The Catquest-9SF is composed of three sections that require patients to select an option from a five-point Likert scale including one option of ‘cannot decide’ (Figure 1). The questions included: “Do you find that your sight at present in some way causes you difficulty in your everyday life?”; “Are you satisfied or dissatisfied with your sight at present?”; “Do you have difficulty with the following activities because of your sight?”. This last question allowed patients to label their satisfaction with vision in various contexts: reading text in newspapers; recognising the faces of people they meet; seeing the prices of goods when shopping; seeing to walk on uneven surfaces eg, cobblestones; seeing to do handicrafts/woodwork; reading subtitles on television; and seeing to engage in an activity/hobby of interest.

All surgery, and assessments before and after surgery, were completed by a single surgeon who performed the operation using standardised surgical technique, intraocular lens and emmetropic refractive target.

Statistical analysis

A group of statistical models termed the Item Response Theory (IRT) have been developed to instrument questionnaire development, evaluation and refinement. This framework analyses individual components of a questionnaire by a set of properties that describe the relationship of the questionnaire with the underlying construct measured by the model, in addition to how well individual questions fit with respect to the underlying construct. IRT is not dependent on the sample of respondents.14,15 This allows researchers to identify the questions that can most accurately measure the intended purpose of the questionnaire.

The Rasch model is a robust and commonly used form of IRT which can be used to assess functioning of rating scale categories within the Catquest-9SF and IoL questionnaires. This is a mathematical framework that takes into account the ability of participants, the difficulty of questions in the questionnaire, and assumes equal discriminating ability across all questions.16 In the Rasch model, the probability of a particular response to a specific question can be modelled as a logistic function of the difference between the person’s ability (measured by using test questions) and the difficulty of the items being asked.17

All IoL and Catquest-9SF question responses were assessed using the Rasch model to assess the validity of the questions in quantifying VRQoL. If responses to a question successfully fit the Rasch model, it provides evidence that this question adequately measures VRQoL. Two types of mean square fit statistics (infit and outfit) were used to evaluate how well patient responses fit the Rasch model for all of the questions within the IoL and Catquest-9SF questionnaires. Infit and outfit statistics have a chi-square distribution and provide an index of magnitude for the degree of misfit of a question with the model. These fit statistics have an expected value of 1 and suggested acceptable lower and upper thresholds of 0.5 and 1.5 respectively.18 Fit statistics for each question were calculated using an average of the squared residuals between the observed and expected responses from the Rasch model. The infit statistic is an estimate that gives more weight to individual variance of questionnaire responses to minimise the impact of unexpected responses far from the mean. Conversely, the outfit statistic is an unweighted estimate of the average question response variance within the IoL and Catquest-9SF questionnaires, and is more likely to be influenced by unexpected responses.

All statistical analyses were completed using R software.19 IoL questionnaire data were coded 1–6 representing the options of ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’, respectively. Catquest-9SF questionnaire data were coded 1–5 representing the options of ‘no difficulty’, ‘some difficulty’, ‘great difficulty’, ‘very great difficulty’ and ‘cannot decide’, respectively. The mirt package was used to fit models for item response theory analysis.20 Preoperative and postoperative data were combined for model fitting. The corrected Akaike information criterion (AICc) is an estimator of the relative quality of statistical models for a given set of data, and was used to select the best model. The G–test of goodness-of-fit was used to determine if the final model accurately predicted the data. Normalised factor scores for both questionnaires were correlated with visual acuity in the operated eye and age using Pearson’s product-moment correlation. Secondary analyses of normalised factor scores by ethnicity and gender were performed using analysis of variance. A qualified statistician reviewed all statistical methodology and analyses used in this study.

Results

Forty-one patients undergoing cataract surgery were enrolled in the study from March to May 2017. All patients who were approached consented to inclusion in the study and completed the questionnaire at both time points. The mean patient age was 77±8 years (sd), with 20 (49%) female participants. Ethnicity included New Zealand European 29 (71%), Māori 3 (7%), Pacific Island 1 (2%), Asian 4 (10%), Indian 3 (7%), and ‘Other’ 1 (2%). Table 1 shows the preoperative and postoperative visual acuities and spherical equivalent for the patient cohort.

Table 1: Visual acuity and spherical equivalent of 41 patients before and following cataract surgery.

UCVA = uncorrected visual acuity, BCVA = best corrected visual acuity, SPE = refractive error (spherical equivalent) in dioptres. Spherical equivalent = sphere power + (cylinder power x 0.5). Postoperative visual acuity was measured at three months following cataract surgery and all visual acuity was represented in logMAR notation.

Figure 2A: Rasch model category probability curves for the Impact on Life questionnaire.

c

These curves summarise the probability (y-axis) that a patient with visual difficulty (x-axis) would answer with a given response. P1 to P6 represents the question response options; P1 = no difficulty, P2 = little difficulty, P3 = some difficulty, P4 = quite difficult, P5 = very difficult, P6 = extremely difficult.

Figure 2B: Category frequency responses for 41 patients who completed the Impact on Life questionnaire before surgery (pink) and three months following surgery (blue).  

c

Table 2: Summary of Rasch model fit statistics for the Impact on Life (IoL) questionnaire. The sample includes responses from 41 patients preoperatively and at three months following cataract surgery.

The model fit statistics for the Catquest-9SF responses are summarised in Table 3. ‘Cannot decide’ responses on the Catquest-9SF questionnaire represented 7 of 738 responses (0.95%) and were assumed equivalent to data missing at random for analysis. Apart from ‘recognising faces’, ‘seeing price of goods when shopping’ and ‘ability to read TV subtitles’ (mean-square fit statistics 0.43, 0.44 and 0.48 respectively), all other Catquest-9SF questions were within the range suitable for measurement (mean-square outfit statistic 0.5 to 1.5). The graphical Rasch categorical probability curves for the Catquest-9SF questions are summarised in Figure 3A. The category frequency of each response in the IoL questionnaire is summarised in Figure 3B.

Table 3: Summary of Rasch model fit statistics for the Catquest-9SF questionnaire from International Consortium for Health Outcomes Measurement (ICHOM). The sample includes responses from 41 patients preoperatively and at three months following cataract surgery.

Figure 3A: Rasch model category probability curves for the Catquest-9SF questionnaire from International Consortium for Health Outcomes Measurement.

c

These curves summarise the probability (y-axis) that a patient with visual difficulty (x-axis) would answer with a given response. A higher number of question difficulty indicates greater disability (6 = extremely difficult; -6 = no difficulty). P1 to P4 represents the question response options; P1 = no difficulty; P2 = some difficulty; P3 = very difficult; P4 = extremely difficult.

Figure 3B: Category frequency responses for 41 patients who completed the Catquest-9SF questionnaire before surgery (pink) and three months following cataract surgery (blue).

c

The difference in visual acuity before and after surgery correlated with the change in total F-score for the Catquest-9SF responses (P=0.04), but not the IoL responses (P=0.17). The overall questionnaire score in both IoL and Catquest-9SF questionnaires correlated with worsening visual acuity (P<0.001). There were no statistical differences in quality of life scores between ages or ethnic groups for both questionnaires. The change in F-score was not significantly different for patients who received cataract surgery on their first eye or second eye.

Discussion

The current study uses Rasch analysis to evaluate the validity of the IoL and Catquest-9SF questionnaires for quantifying VRQoL for cataract surgery patients in New Zealand. As far as the authors are aware, this is the first study to statistically assess the validity of the IoL questionnaire for use in cataract surgery, and the first study to compare the IoL with any other questionnaire to assess VRQoL.

The unequal peaks noted in the Rasch analysis for the IoL questionnaire suggest that the response options of ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’, are too numerous and ideally should be collapsed into fewer options with more consistent probability thresholds. This finding was consistent for all six questions in the IoL questionnaire. In contrast, the Catquest-9SF demonstrated relatively uniform peak height for the question response options that remained consistent for all questions in the Catquest-9SF questionnaire, similar to previous studies.21

The IoL questionnaire demonstrates category disordering on Rasch analysis. Category disordering occurs when the ordinal numbering of categories (response options) does not correspond with their substantive meaning. The IoL questionnaire ordered response options are substantively defined to represent increasing levels of disability in VRQoL. In all six questions, there is substantive and step disordering such that ‘little difficulty’ consistently locates below ‘no difficulty’ in the Rasch analysis. This finding suggests that the response options used in the IoL questionnaire are not able to accurately discriminate increasing impairment in VRQoL as intended.

The IoL questionnaire demonstrated unsatisfactory statistical fit of almost all questions (mean-square fit less than 0.5). This finding indicates less variation in participant responses than expected, and that responses are more predictable than the Rasch model expects. The high predictability of responses to IoL questions and overfit to the Rasch model suggests sub-optimal question wording resulting in non-discriminatory patient responses.22 This finding suggests that the IoL questionnaire is likely to lack the required sensitivity to accurately rank patients based on VRQoL.

Catquest-9SF questions demonstrated satisfactory mean fit squares and appropriate category response curves with monotonic increases and decreases in the category thresholds (Figure 3A). This finding was consistent with other studies evaluating the Catquest-9SF using Rasch analysis in Europe and Australia.21,23 These results confirm that the Catquest-9SF questionnaire is valid tool for the assessment of VRQoL in a New Zealand population and can accurately rank patients based on VRQoL.

Questionnaire scores for the Catquest-9SF and the IoL improved with the improvement in visual acuity following surgery. Only the Catquest-9SF questionnaire, however, demonstrated significant correlation between the change in visual acuity and change in questionnaire F-scores following surgery. The F-score is a single indicator that summarises the variance (accuracy and recall ratio) of data points around the mean, which can be used to evaluate and compare the fit of multiple linear models.24 Based on these results, the IoL questionnaire responses appear to be independent to VRQoL and poorly suited for predicting which patients will experience quality of life gains as a result of improved vision following cataract surgery.

There are several limitations to this study. Firstly, patient bias may influence questionnaire responses. Patients may suspect that preoperative questionnaire responses could affect their eligibility for surgery and bias towards over-reporting poor quality of life prior to surgery or after surgery where second eye surgery is required. The lack of significant difference in F-scores between patients receiving first or second eye cataract surgery, however, suggests similar degrees of variance in responses indicating no such bias exists in this data. Secondly, the current study has a relatively small sample size. Reports of Rasch analysis results are considered to be robust to smaller sample size.25 In addition, despite the small sample size, the current study was able to replicate similar findings to previous, larger studies evaluating the Catquest-9SF.21,23

The primary strength of this study is analysis of the qualitative responses using Rasch analysis. The importance of Rasch analysis has been well-recognised for the evaluation of questionnaire quality and there have been numerous requests for the development of Rasch-approved questionnaires within ophthalmology.26–28 The current study offers the first Rasch assessment of the IoL questionnaire. This questionnaire is currently in widespread use to assess eligibility for all patients in the New Zealand public health system that require cataract surgery.

In summary, the current study compared the ability of IoL and Catquest-9SF questionnaires to accurately measure VRQoL. The results of this study demonstrate that the IoL does not accurately assess VRQoL for patients that require cataract surgery in New Zealand. The Catquest-9SF is a domain-specific assessment tool that can accurately measure VRQoL in New Zealand. The convenience of using a single tool, such as the IoL, to allocate healthcare resources across multiple specialities must be carefully weighed against the risk of not allocating resources where they are needed the most.

Despite its widespread use, the current study highlights inadequacies of the IoL questionnaire for the assessment of VRQoL for cataract surgery in New Zealand. In addition to any role in surgical prioritisation, it is increasingly important for quality improvements in healthcare delivery to use standardised patient reported outcome tools, such as the Catquest-9SF. These standardised tools enable international benchmarking and direct comparison with other studies. In conclusion, the Catquest-9SF questionnaire provides a more accurate assessment of VRQoL than the currently used IoL questionnaire for New Zealand patients that require cataract surgery.

Summary

Abstract

Aim

The Impact on Life (IoL) questionnaire is used to prioritise publicly funded cataract surgery in New Zealand, however, it has not been formally validated for ophthalmic use. The Catquest-9SF questionnaire is widely used to assess vision-related quality of life (VRQoL) but has not been validated in New Zealand. This study evaluates the validity of the IoL and Catquest-9SF questionnaires for measuring VRQoL in New Zealand.

Method

Formal ethics approval was obtained. Participants completed the IoL and Catquest-9SF questionnaires before and three months after routine cataract surgery. Rasch analysis was used to investigate all qualitative questionnaire responses. Results were correlated with the change in patient visual acuity.

Results

There was a 100% response rate at follow-up (41 participants). Disordered probability thresholds were observed for all IoL questions but no Catquest-9SF questions. All IoL questions demonstrated unsatisfactory mean-square fit statistics. Differences in visual acuity following surgery correlated with the change in total F-score for the Catquest-9SF (P=0.04), but not IoL responses (P=0.17).

Conclusion

Disordered probability thresholds, poor question-model fit and correlation with visual acuity changes indicate the current IoL questionnaire is poorly suited for assessment of VRQoL. In contrast, the Catquest-9SF demonstrated credible results for assessment of VRQoL in New Zealand.

Author Information

Sunny S Li, House Officer, Counties Manukau District Health Board, Auckland;-Stuti Misra, Senior Lecturer, The University of Auckland, Auckland;-Henry Wallace, House Officer, Auckland District Health Board, Auckland;-Lyn Hunt, Senior Lecturer, Academic Programme Convenor (Statistics), The University of Waikato, Hamilton; James McKelvie, Ophthalmologist, Waikato District Health Board, Hamilton.

Acknowledgements

Correspondence

James McKelvie, Department of Ophthalmology, Private Bag 92019, University of Auckland, Auckland.

Correspondence Email

james@mckelvie.co.nz

Competing Interests

Nil.

  1. Elder M, Tarr K, Leaming D. The New Zealand cataract and refractive surgery survey 1997/1998(1). Am J Ophthalmol. 2000 Oct; 130(4):543.
  2. Riley AF, Grupcheva CN, Malik TY, Craig JP, McGhee CN. The Auckland Cataract Study: demographic, corneal topographic and ocular biometric parameters. Clin Experiment Ophthalmol. 2001 Dec; 29(6):381–6.
  3. OECD Health Statistics (database). OECD 2018 [Internet]. Health care utilisation (Edition 2018). [cited 2019 Feb 17]. Available from: http://dx.doi.org/10.1787/732466c2-en
  4. Hadorn DC, Holmes AC. The New Zealand priority criteria project. Part 1: Overview. BMJ. 1997; 314(7074):131–131.
  5. Gauld R, Derrett S. Solving the surgical waiting list problem? New Zealand’s ‘booking system’. Int J Health Plann Manage. 2000; 15(4):259–72.
  6. Wong VWY, Lai TYY, Lam PTH, Lam DSC. Prioritization of cataract surgery: visual analogue scale versus scoring system. ANZ J Surg. 2005 Jul; 75(7):587–92.
  7. Chan G, Bezuidenhout L, Walker L, Rowan R. The Impact on Life questionnaire: validation for elective surgery prioritisation in New Zealand prioritisation criteria in orthopaedic surgery. N Z Med J. 2016 Apr 1; 129(1432):26–32.
  8. Pesudovs K, Coster DJ. An instrument for assessment of subjective visual disability in cataract patients. Br J Ophthalmol. 1998 Jun; 82(6):617–24.
  9. Mangione CM, Lee PP, Gutierrez PR, Spritzer K, Berry S, Hays RD, et al. Development of the 25-item National Eye Institute Visual Function Questionnaire. Arch Ophthalmol. 2001 Jul; 119(7):1050–8.
  10. Roberts HW, Wagh VK, Sullivan DL, Hidzheva P, Detesan DI, Heemraz BS, et al. A randomized controlled trial comparing femtosecond laser–assisted cataract surgery versus conventional phacoemulsification surgery. J Cataract Refract Surg. 2019; 45(1):11–20.
  11. Porter ME, Larsson S, Lee TH. Standardizing Patient Outcomes Measurement. N Engl J Med. 2016 Feb 11; 374(6):504–6.
  12. McAlinden C, Gothwal VK, Khadka J, Wright TA, Lamoureux EL, Pesudovs K. A head-to-head comparison of 16 cataract surgery outcome questionnaires. Ophthalmology. 2011 Dec; 118(12):2374–81.
  13. Lundstrom M, Behndig A, Kugelberg M, Montan P, Stenevi U, Pesudovs K. The outcome of cataract surgery measured with the Catquest-9SF. Acta Ophthalmol. 2011 Dec; 89(8):718–23.
  14. Reeve BB. Item response theory modeling in health outcomes measurement. Expert Rev Pharmacoecon Outcomes Res. 2003 Apr; 3(2):131–45.
  15. Chang C-H, Reeve BB. Item Response Theory and its Applications to Patient-Reported Outcomes Measurement. Eval Health Prof. 2005; 28(3):264–82.
  16. Wolins L, Wright BD, Rasch G. Probabilistic Models for some Intelligence and Attainment Tests. J Am Stat Assoc. 1982; 77(377):220.
  17. Luo X, Cappelleri JC, Cella D, Li JZ, Charbonneau C, Kim ST, et al. Using the Rasch model to validate and enhance the interpretation of the Functional Assessment of Cancer Therapy-Kidney Symptom Index--Disease-Related Symptoms scale. Value Health. 2009 Jun; 12(4):580–6.
  18. Linacre JM. Rasch Measurement Transactions: Based on Rasch measurement transactions Vol. 6:no. 1-Vol. 8:4. 1995.
  19. R Core Team. R: A language and environment for statistical computing [Internet]. 2017. Available from: http://www.R-project.org/
  20. Chalmers RP, Philip Chalmers R. mirt: A Multidimensional Item Response Theory Package for the R Environment. J Stat Softw [Internet]. 2012;48(6). Available from: http://dx.doi.org/10.18637/jss.v048.i06
  21. Lundström M, Pesudovs K. Catquest-9SF patient outcomes questionnaire: nine-item short-form Rasch-scaled revision of the Catquest questionnaire. J Cataract Refract Surg. 2009 Mar; 35(3):504–13.
  22. Pallant JF, Tennant A. An introduction to the Rasch measurement model: An example using the Hospital Anxiety and Depression Scale (HADS) [Internet]. Vol. 46, British Journal of Clinical Psychology. 2007. p. 1–18. Available from: http://dx.doi.org/10.1348/014466506x96931
  23. Gothwal VK, Wright TA, Lamoureux EL, Lundström M, Pesudovs K. Catquest questionnaire: re-validation in an Australian cataract population. Clin Experiment Ophthalmol. 2009; 37(8):785–94.
  24. Goutte C, Gaussier E. A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation. In: Lecture Notes in Computer Science. 2005. p. 345–59.
  25. Smith AB, Rush R, Fallowfield LJ, Velikova G, Sharpe M. Rasch fit statistics and sample size considerations for polytomous data. BMC Med Res Methodol. 2008 May 29; 8:33.
  26. Pesudovs K, Burr JM, Harley C, Elliott DB. The Development, Assessment, and Selection of Questionnaires. Optom Vis Sci. 2007; 84(8):663–74.
  27. Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007 Jan; 60(1):34–42.
  28. Pesudovs K. Patient-centred measurement in ophthalmology – a paradigm shift. BMC Ophthalmol [Internet]. 2006;6(1). Available from: http://dx.doi.org/10.1186/1471-2415-6-25

For the PDF of this article,
contact nzmj@nzma.org.nz

View Article PDF

Cataract surgery is the most frequently performed surgical procedure in New Zealand with approximately 16,500 publicly funded cataract surgeries completed annually.1–3 With limited resources for publicly funded surgery, prioritising patients for cataract surgery is essential to enable equal access to surgery for all New Zealand residents and ensure those who are most likely to benefit from surgery are prioritised highest. The New Zealand public health system currently utilises a standardised Clinical Priority Assessment Criteria (CPAC) that involves priority scoring to determine patient eligibility for publicly funded elective surgical services including cataract surgery.4 The CPAC system aims to improve equity of access to surgical services across New Zealand, enhance transparency around prioritisation for surgery and improve certainty regarding treatment for patients who require surgery.5

Prioritisation for cataract surgery in New Zealand using the CPAC system is based on weighted scores for patient responses to the Impact on Life (IoL) questionnaire, best corrected visual acuity (BCVA) and cataract morphology.6 The IoL questionnaire is intended to quantitatively score patient-reported functional status in six qualitative domains that include safety, social interactions, responsibility to others, personal relationships, personal care and leisure activities. The IoL questionnaire was not designed specifically for use with cataract or ophthalmic surgery, and was initially developed for prioritisation in orthopaedic and other surgical specialities.7 Despite the national adoption of the IoL as an integral component of CPAC prioritisation for cataract surgery in New Zealand, the ability of the IoL questionnaire to assess vision-related quality of life (VRQoL) has not been formally assessed.

The use of patient-reported measures has gained wide acceptance in ophthalmology following development of cataract-related visual disability questionnaires.8–10 The International Consortium for Health Outcomes Measurement (ICHOM) has convened global groups of experts and patient representatives to outline minimum standard outcomes using a structured process for a variety of specific conditions including cataract based on evidence-based measures to assess quality of life related to vision.11 The resulting Catquest-9SF questionnaire has been extensively validated as an accurate tool for assessment of patient-reported visual disability for patients undergoing cataract surgery.12 The Catquest-9SF is well suited for use in clinical practice due to its validity, brevity and ease of use, however, this questionnaire has not been validated in a New Zealand population.13

The aim of the current study is to validate and compare the ability of the IoL and the Catquest-9SF to measure VRQoL for New Zealand patients undergoing cataract surgery.

Methods

Formal approval from the New Zealand Health and Disability Ethics Committee was obtained prior to patient recruitment (16/CEN/132), and this study was registered with the Australian New Zealand Clinical Trials Registry (12616001593426). This is a prospective observational cohort study involving patients enrolled for routine cataract surgery at Greenlane Clinical Centre, Auckland District Health Board, New Zealand.

Patients who were referred for publicly funded surgery at Auckland District Health Board were invited to participate in the study. Patients who agreed to participate in the study completed both questionnaires before surgery and at again three months following surgery. All patients completed the IoL and Catquest-9SF questionnaires while the clinician was not in the room and the questionnaires were collected by an independent investigator.

The six-question IoL questionnaire requires patients to score the degree of difficulty that poor vision affected their social interactions, personal relationships, ability to meet responsibilities to others, personal care, personal safety and leisure activities using an ordinal scale (Figure 1). For each question on the IoL questionnaire, patients are required to select one option from a scale labelled ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’.

Figure 1A: The Impact on Life questionnaire, currently used in the Clinical Priority Assessment Criteria (CPAC) to determine patient eligibility for publicly funded elective cataract surgery in New Zealand.

c

Figure 1B: The Catquest-9SF questionnaire, developed by the International Consortium for Health Outcomes Measurement to assess quality of life related to vision as a result of cataracts.  

c

The Catquest-9SF is composed of three sections that require patients to select an option from a five-point Likert scale including one option of ‘cannot decide’ (Figure 1). The questions included: “Do you find that your sight at present in some way causes you difficulty in your everyday life?”; “Are you satisfied or dissatisfied with your sight at present?”; “Do you have difficulty with the following activities because of your sight?”. This last question allowed patients to label their satisfaction with vision in various contexts: reading text in newspapers; recognising the faces of people they meet; seeing the prices of goods when shopping; seeing to walk on uneven surfaces eg, cobblestones; seeing to do handicrafts/woodwork; reading subtitles on television; and seeing to engage in an activity/hobby of interest.

All surgery, and assessments before and after surgery, were completed by a single surgeon who performed the operation using standardised surgical technique, intraocular lens and emmetropic refractive target.

Statistical analysis

A group of statistical models termed the Item Response Theory (IRT) have been developed to instrument questionnaire development, evaluation and refinement. This framework analyses individual components of a questionnaire by a set of properties that describe the relationship of the questionnaire with the underlying construct measured by the model, in addition to how well individual questions fit with respect to the underlying construct. IRT is not dependent on the sample of respondents.14,15 This allows researchers to identify the questions that can most accurately measure the intended purpose of the questionnaire.

The Rasch model is a robust and commonly used form of IRT which can be used to assess functioning of rating scale categories within the Catquest-9SF and IoL questionnaires. This is a mathematical framework that takes into account the ability of participants, the difficulty of questions in the questionnaire, and assumes equal discriminating ability across all questions.16 In the Rasch model, the probability of a particular response to a specific question can be modelled as a logistic function of the difference between the person’s ability (measured by using test questions) and the difficulty of the items being asked.17

All IoL and Catquest-9SF question responses were assessed using the Rasch model to assess the validity of the questions in quantifying VRQoL. If responses to a question successfully fit the Rasch model, it provides evidence that this question adequately measures VRQoL. Two types of mean square fit statistics (infit and outfit) were used to evaluate how well patient responses fit the Rasch model for all of the questions within the IoL and Catquest-9SF questionnaires. Infit and outfit statistics have a chi-square distribution and provide an index of magnitude for the degree of misfit of a question with the model. These fit statistics have an expected value of 1 and suggested acceptable lower and upper thresholds of 0.5 and 1.5 respectively.18 Fit statistics for each question were calculated using an average of the squared residuals between the observed and expected responses from the Rasch model. The infit statistic is an estimate that gives more weight to individual variance of questionnaire responses to minimise the impact of unexpected responses far from the mean. Conversely, the outfit statistic is an unweighted estimate of the average question response variance within the IoL and Catquest-9SF questionnaires, and is more likely to be influenced by unexpected responses.

All statistical analyses were completed using R software.19 IoL questionnaire data were coded 1–6 representing the options of ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’, respectively. Catquest-9SF questionnaire data were coded 1–5 representing the options of ‘no difficulty’, ‘some difficulty’, ‘great difficulty’, ‘very great difficulty’ and ‘cannot decide’, respectively. The mirt package was used to fit models for item response theory analysis.20 Preoperative and postoperative data were combined for model fitting. The corrected Akaike information criterion (AICc) is an estimator of the relative quality of statistical models for a given set of data, and was used to select the best model. The G–test of goodness-of-fit was used to determine if the final model accurately predicted the data. Normalised factor scores for both questionnaires were correlated with visual acuity in the operated eye and age using Pearson’s product-moment correlation. Secondary analyses of normalised factor scores by ethnicity and gender were performed using analysis of variance. A qualified statistician reviewed all statistical methodology and analyses used in this study.

Results

Forty-one patients undergoing cataract surgery were enrolled in the study from March to May 2017. All patients who were approached consented to inclusion in the study and completed the questionnaire at both time points. The mean patient age was 77±8 years (sd), with 20 (49%) female participants. Ethnicity included New Zealand European 29 (71%), Māori 3 (7%), Pacific Island 1 (2%), Asian 4 (10%), Indian 3 (7%), and ‘Other’ 1 (2%). Table 1 shows the preoperative and postoperative visual acuities and spherical equivalent for the patient cohort.

Table 1: Visual acuity and spherical equivalent of 41 patients before and following cataract surgery.

UCVA = uncorrected visual acuity, BCVA = best corrected visual acuity, SPE = refractive error (spherical equivalent) in dioptres. Spherical equivalent = sphere power + (cylinder power x 0.5). Postoperative visual acuity was measured at three months following cataract surgery and all visual acuity was represented in logMAR notation.

Figure 2A: Rasch model category probability curves for the Impact on Life questionnaire.

c

These curves summarise the probability (y-axis) that a patient with visual difficulty (x-axis) would answer with a given response. P1 to P6 represents the question response options; P1 = no difficulty, P2 = little difficulty, P3 = some difficulty, P4 = quite difficult, P5 = very difficult, P6 = extremely difficult.

Figure 2B: Category frequency responses for 41 patients who completed the Impact on Life questionnaire before surgery (pink) and three months following surgery (blue).  

c

Table 2: Summary of Rasch model fit statistics for the Impact on Life (IoL) questionnaire. The sample includes responses from 41 patients preoperatively and at three months following cataract surgery.

The model fit statistics for the Catquest-9SF responses are summarised in Table 3. ‘Cannot decide’ responses on the Catquest-9SF questionnaire represented 7 of 738 responses (0.95%) and were assumed equivalent to data missing at random for analysis. Apart from ‘recognising faces’, ‘seeing price of goods when shopping’ and ‘ability to read TV subtitles’ (mean-square fit statistics 0.43, 0.44 and 0.48 respectively), all other Catquest-9SF questions were within the range suitable for measurement (mean-square outfit statistic 0.5 to 1.5). The graphical Rasch categorical probability curves for the Catquest-9SF questions are summarised in Figure 3A. The category frequency of each response in the IoL questionnaire is summarised in Figure 3B.

Table 3: Summary of Rasch model fit statistics for the Catquest-9SF questionnaire from International Consortium for Health Outcomes Measurement (ICHOM). The sample includes responses from 41 patients preoperatively and at three months following cataract surgery.

Figure 3A: Rasch model category probability curves for the Catquest-9SF questionnaire from International Consortium for Health Outcomes Measurement.

c

These curves summarise the probability (y-axis) that a patient with visual difficulty (x-axis) would answer with a given response. A higher number of question difficulty indicates greater disability (6 = extremely difficult; -6 = no difficulty). P1 to P4 represents the question response options; P1 = no difficulty; P2 = some difficulty; P3 = very difficult; P4 = extremely difficult.

Figure 3B: Category frequency responses for 41 patients who completed the Catquest-9SF questionnaire before surgery (pink) and three months following cataract surgery (blue).

c

The difference in visual acuity before and after surgery correlated with the change in total F-score for the Catquest-9SF responses (P=0.04), but not the IoL responses (P=0.17). The overall questionnaire score in both IoL and Catquest-9SF questionnaires correlated with worsening visual acuity (P<0.001). There were no statistical differences in quality of life scores between ages or ethnic groups for both questionnaires. The change in F-score was not significantly different for patients who received cataract surgery on their first eye or second eye.

Discussion

The current study uses Rasch analysis to evaluate the validity of the IoL and Catquest-9SF questionnaires for quantifying VRQoL for cataract surgery patients in New Zealand. As far as the authors are aware, this is the first study to statistically assess the validity of the IoL questionnaire for use in cataract surgery, and the first study to compare the IoL with any other questionnaire to assess VRQoL.

The unequal peaks noted in the Rasch analysis for the IoL questionnaire suggest that the response options of ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’, are too numerous and ideally should be collapsed into fewer options with more consistent probability thresholds. This finding was consistent for all six questions in the IoL questionnaire. In contrast, the Catquest-9SF demonstrated relatively uniform peak height for the question response options that remained consistent for all questions in the Catquest-9SF questionnaire, similar to previous studies.21

The IoL questionnaire demonstrates category disordering on Rasch analysis. Category disordering occurs when the ordinal numbering of categories (response options) does not correspond with their substantive meaning. The IoL questionnaire ordered response options are substantively defined to represent increasing levels of disability in VRQoL. In all six questions, there is substantive and step disordering such that ‘little difficulty’ consistently locates below ‘no difficulty’ in the Rasch analysis. This finding suggests that the response options used in the IoL questionnaire are not able to accurately discriminate increasing impairment in VRQoL as intended.

The IoL questionnaire demonstrated unsatisfactory statistical fit of almost all questions (mean-square fit less than 0.5). This finding indicates less variation in participant responses than expected, and that responses are more predictable than the Rasch model expects. The high predictability of responses to IoL questions and overfit to the Rasch model suggests sub-optimal question wording resulting in non-discriminatory patient responses.22 This finding suggests that the IoL questionnaire is likely to lack the required sensitivity to accurately rank patients based on VRQoL.

Catquest-9SF questions demonstrated satisfactory mean fit squares and appropriate category response curves with monotonic increases and decreases in the category thresholds (Figure 3A). This finding was consistent with other studies evaluating the Catquest-9SF using Rasch analysis in Europe and Australia.21,23 These results confirm that the Catquest-9SF questionnaire is valid tool for the assessment of VRQoL in a New Zealand population and can accurately rank patients based on VRQoL.

Questionnaire scores for the Catquest-9SF and the IoL improved with the improvement in visual acuity following surgery. Only the Catquest-9SF questionnaire, however, demonstrated significant correlation between the change in visual acuity and change in questionnaire F-scores following surgery. The F-score is a single indicator that summarises the variance (accuracy and recall ratio) of data points around the mean, which can be used to evaluate and compare the fit of multiple linear models.24 Based on these results, the IoL questionnaire responses appear to be independent to VRQoL and poorly suited for predicting which patients will experience quality of life gains as a result of improved vision following cataract surgery.

There are several limitations to this study. Firstly, patient bias may influence questionnaire responses. Patients may suspect that preoperative questionnaire responses could affect their eligibility for surgery and bias towards over-reporting poor quality of life prior to surgery or after surgery where second eye surgery is required. The lack of significant difference in F-scores between patients receiving first or second eye cataract surgery, however, suggests similar degrees of variance in responses indicating no such bias exists in this data. Secondly, the current study has a relatively small sample size. Reports of Rasch analysis results are considered to be robust to smaller sample size.25 In addition, despite the small sample size, the current study was able to replicate similar findings to previous, larger studies evaluating the Catquest-9SF.21,23

The primary strength of this study is analysis of the qualitative responses using Rasch analysis. The importance of Rasch analysis has been well-recognised for the evaluation of questionnaire quality and there have been numerous requests for the development of Rasch-approved questionnaires within ophthalmology.26–28 The current study offers the first Rasch assessment of the IoL questionnaire. This questionnaire is currently in widespread use to assess eligibility for all patients in the New Zealand public health system that require cataract surgery.

In summary, the current study compared the ability of IoL and Catquest-9SF questionnaires to accurately measure VRQoL. The results of this study demonstrate that the IoL does not accurately assess VRQoL for patients that require cataract surgery in New Zealand. The Catquest-9SF is a domain-specific assessment tool that can accurately measure VRQoL in New Zealand. The convenience of using a single tool, such as the IoL, to allocate healthcare resources across multiple specialities must be carefully weighed against the risk of not allocating resources where they are needed the most.

Despite its widespread use, the current study highlights inadequacies of the IoL questionnaire for the assessment of VRQoL for cataract surgery in New Zealand. In addition to any role in surgical prioritisation, it is increasingly important for quality improvements in healthcare delivery to use standardised patient reported outcome tools, such as the Catquest-9SF. These standardised tools enable international benchmarking and direct comparison with other studies. In conclusion, the Catquest-9SF questionnaire provides a more accurate assessment of VRQoL than the currently used IoL questionnaire for New Zealand patients that require cataract surgery.

Summary

Abstract

Aim

The Impact on Life (IoL) questionnaire is used to prioritise publicly funded cataract surgery in New Zealand, however, it has not been formally validated for ophthalmic use. The Catquest-9SF questionnaire is widely used to assess vision-related quality of life (VRQoL) but has not been validated in New Zealand. This study evaluates the validity of the IoL and Catquest-9SF questionnaires for measuring VRQoL in New Zealand.

Method

Formal ethics approval was obtained. Participants completed the IoL and Catquest-9SF questionnaires before and three months after routine cataract surgery. Rasch analysis was used to investigate all qualitative questionnaire responses. Results were correlated with the change in patient visual acuity.

Results

There was a 100% response rate at follow-up (41 participants). Disordered probability thresholds were observed for all IoL questions but no Catquest-9SF questions. All IoL questions demonstrated unsatisfactory mean-square fit statistics. Differences in visual acuity following surgery correlated with the change in total F-score for the Catquest-9SF (P=0.04), but not IoL responses (P=0.17).

Conclusion

Disordered probability thresholds, poor question-model fit and correlation with visual acuity changes indicate the current IoL questionnaire is poorly suited for assessment of VRQoL. In contrast, the Catquest-9SF demonstrated credible results for assessment of VRQoL in New Zealand.

Author Information

Sunny S Li, House Officer, Counties Manukau District Health Board, Auckland;-Stuti Misra, Senior Lecturer, The University of Auckland, Auckland;-Henry Wallace, House Officer, Auckland District Health Board, Auckland;-Lyn Hunt, Senior Lecturer, Academic Programme Convenor (Statistics), The University of Waikato, Hamilton; James McKelvie, Ophthalmologist, Waikato District Health Board, Hamilton.

Acknowledgements

Correspondence

James McKelvie, Department of Ophthalmology, Private Bag 92019, University of Auckland, Auckland.

Correspondence Email

james@mckelvie.co.nz

Competing Interests

Nil.

  1. Elder M, Tarr K, Leaming D. The New Zealand cataract and refractive surgery survey 1997/1998(1). Am J Ophthalmol. 2000 Oct; 130(4):543.
  2. Riley AF, Grupcheva CN, Malik TY, Craig JP, McGhee CN. The Auckland Cataract Study: demographic, corneal topographic and ocular biometric parameters. Clin Experiment Ophthalmol. 2001 Dec; 29(6):381–6.
  3. OECD Health Statistics (database). OECD 2018 [Internet]. Health care utilisation (Edition 2018). [cited 2019 Feb 17]. Available from: http://dx.doi.org/10.1787/732466c2-en
  4. Hadorn DC, Holmes AC. The New Zealand priority criteria project. Part 1: Overview. BMJ. 1997; 314(7074):131–131.
  5. Gauld R, Derrett S. Solving the surgical waiting list problem? New Zealand’s ‘booking system’. Int J Health Plann Manage. 2000; 15(4):259–72.
  6. Wong VWY, Lai TYY, Lam PTH, Lam DSC. Prioritization of cataract surgery: visual analogue scale versus scoring system. ANZ J Surg. 2005 Jul; 75(7):587–92.
  7. Chan G, Bezuidenhout L, Walker L, Rowan R. The Impact on Life questionnaire: validation for elective surgery prioritisation in New Zealand prioritisation criteria in orthopaedic surgery. N Z Med J. 2016 Apr 1; 129(1432):26–32.
  8. Pesudovs K, Coster DJ. An instrument for assessment of subjective visual disability in cataract patients. Br J Ophthalmol. 1998 Jun; 82(6):617–24.
  9. Mangione CM, Lee PP, Gutierrez PR, Spritzer K, Berry S, Hays RD, et al. Development of the 25-item National Eye Institute Visual Function Questionnaire. Arch Ophthalmol. 2001 Jul; 119(7):1050–8.
  10. Roberts HW, Wagh VK, Sullivan DL, Hidzheva P, Detesan DI, Heemraz BS, et al. A randomized controlled trial comparing femtosecond laser–assisted cataract surgery versus conventional phacoemulsification surgery. J Cataract Refract Surg. 2019; 45(1):11–20.
  11. Porter ME, Larsson S, Lee TH. Standardizing Patient Outcomes Measurement. N Engl J Med. 2016 Feb 11; 374(6):504–6.
  12. McAlinden C, Gothwal VK, Khadka J, Wright TA, Lamoureux EL, Pesudovs K. A head-to-head comparison of 16 cataract surgery outcome questionnaires. Ophthalmology. 2011 Dec; 118(12):2374–81.
  13. Lundstrom M, Behndig A, Kugelberg M, Montan P, Stenevi U, Pesudovs K. The outcome of cataract surgery measured with the Catquest-9SF. Acta Ophthalmol. 2011 Dec; 89(8):718–23.
  14. Reeve BB. Item response theory modeling in health outcomes measurement. Expert Rev Pharmacoecon Outcomes Res. 2003 Apr; 3(2):131–45.
  15. Chang C-H, Reeve BB. Item Response Theory and its Applications to Patient-Reported Outcomes Measurement. Eval Health Prof. 2005; 28(3):264–82.
  16. Wolins L, Wright BD, Rasch G. Probabilistic Models for some Intelligence and Attainment Tests. J Am Stat Assoc. 1982; 77(377):220.
  17. Luo X, Cappelleri JC, Cella D, Li JZ, Charbonneau C, Kim ST, et al. Using the Rasch model to validate and enhance the interpretation of the Functional Assessment of Cancer Therapy-Kidney Symptom Index--Disease-Related Symptoms scale. Value Health. 2009 Jun; 12(4):580–6.
  18. Linacre JM. Rasch Measurement Transactions: Based on Rasch measurement transactions Vol. 6:no. 1-Vol. 8:4. 1995.
  19. R Core Team. R: A language and environment for statistical computing [Internet]. 2017. Available from: http://www.R-project.org/
  20. Chalmers RP, Philip Chalmers R. mirt: A Multidimensional Item Response Theory Package for the R Environment. J Stat Softw [Internet]. 2012;48(6). Available from: http://dx.doi.org/10.18637/jss.v048.i06
  21. Lundström M, Pesudovs K. Catquest-9SF patient outcomes questionnaire: nine-item short-form Rasch-scaled revision of the Catquest questionnaire. J Cataract Refract Surg. 2009 Mar; 35(3):504–13.
  22. Pallant JF, Tennant A. An introduction to the Rasch measurement model: An example using the Hospital Anxiety and Depression Scale (HADS) [Internet]. Vol. 46, British Journal of Clinical Psychology. 2007. p. 1–18. Available from: http://dx.doi.org/10.1348/014466506x96931
  23. Gothwal VK, Wright TA, Lamoureux EL, Lundström M, Pesudovs K. Catquest questionnaire: re-validation in an Australian cataract population. Clin Experiment Ophthalmol. 2009; 37(8):785–94.
  24. Goutte C, Gaussier E. A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation. In: Lecture Notes in Computer Science. 2005. p. 345–59.
  25. Smith AB, Rush R, Fallowfield LJ, Velikova G, Sharpe M. Rasch fit statistics and sample size considerations for polytomous data. BMC Med Res Methodol. 2008 May 29; 8:33.
  26. Pesudovs K, Burr JM, Harley C, Elliott DB. The Development, Assessment, and Selection of Questionnaires. Optom Vis Sci. 2007; 84(8):663–74.
  27. Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007 Jan; 60(1):34–42.
  28. Pesudovs K. Patient-centred measurement in ophthalmology – a paradigm shift. BMC Ophthalmol [Internet]. 2006;6(1). Available from: http://dx.doi.org/10.1186/1471-2415-6-25

For the PDF of this article,
contact nzmj@nzma.org.nz

View Article PDF

Cataract surgery is the most frequently performed surgical procedure in New Zealand with approximately 16,500 publicly funded cataract surgeries completed annually.1–3 With limited resources for publicly funded surgery, prioritising patients for cataract surgery is essential to enable equal access to surgery for all New Zealand residents and ensure those who are most likely to benefit from surgery are prioritised highest. The New Zealand public health system currently utilises a standardised Clinical Priority Assessment Criteria (CPAC) that involves priority scoring to determine patient eligibility for publicly funded elective surgical services including cataract surgery.4 The CPAC system aims to improve equity of access to surgical services across New Zealand, enhance transparency around prioritisation for surgery and improve certainty regarding treatment for patients who require surgery.5

Prioritisation for cataract surgery in New Zealand using the CPAC system is based on weighted scores for patient responses to the Impact on Life (IoL) questionnaire, best corrected visual acuity (BCVA) and cataract morphology.6 The IoL questionnaire is intended to quantitatively score patient-reported functional status in six qualitative domains that include safety, social interactions, responsibility to others, personal relationships, personal care and leisure activities. The IoL questionnaire was not designed specifically for use with cataract or ophthalmic surgery, and was initially developed for prioritisation in orthopaedic and other surgical specialities.7 Despite the national adoption of the IoL as an integral component of CPAC prioritisation for cataract surgery in New Zealand, the ability of the IoL questionnaire to assess vision-related quality of life (VRQoL) has not been formally assessed.

The use of patient-reported measures has gained wide acceptance in ophthalmology following development of cataract-related visual disability questionnaires.8–10 The International Consortium for Health Outcomes Measurement (ICHOM) has convened global groups of experts and patient representatives to outline minimum standard outcomes using a structured process for a variety of specific conditions including cataract based on evidence-based measures to assess quality of life related to vision.11 The resulting Catquest-9SF questionnaire has been extensively validated as an accurate tool for assessment of patient-reported visual disability for patients undergoing cataract surgery.12 The Catquest-9SF is well suited for use in clinical practice due to its validity, brevity and ease of use, however, this questionnaire has not been validated in a New Zealand population.13

The aim of the current study is to validate and compare the ability of the IoL and the Catquest-9SF to measure VRQoL for New Zealand patients undergoing cataract surgery.

Methods

Formal approval from the New Zealand Health and Disability Ethics Committee was obtained prior to patient recruitment (16/CEN/132), and this study was registered with the Australian New Zealand Clinical Trials Registry (12616001593426). This is a prospective observational cohort study involving patients enrolled for routine cataract surgery at Greenlane Clinical Centre, Auckland District Health Board, New Zealand.

Patients who were referred for publicly funded surgery at Auckland District Health Board were invited to participate in the study. Patients who agreed to participate in the study completed both questionnaires before surgery and at again three months following surgery. All patients completed the IoL and Catquest-9SF questionnaires while the clinician was not in the room and the questionnaires were collected by an independent investigator.

The six-question IoL questionnaire requires patients to score the degree of difficulty that poor vision affected their social interactions, personal relationships, ability to meet responsibilities to others, personal care, personal safety and leisure activities using an ordinal scale (Figure 1). For each question on the IoL questionnaire, patients are required to select one option from a scale labelled ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’.

Figure 1A: The Impact on Life questionnaire, currently used in the Clinical Priority Assessment Criteria (CPAC) to determine patient eligibility for publicly funded elective cataract surgery in New Zealand.

c

Figure 1B: The Catquest-9SF questionnaire, developed by the International Consortium for Health Outcomes Measurement to assess quality of life related to vision as a result of cataracts.  

c

The Catquest-9SF is composed of three sections that require patients to select an option from a five-point Likert scale including one option of ‘cannot decide’ (Figure 1). The questions included: “Do you find that your sight at present in some way causes you difficulty in your everyday life?”; “Are you satisfied or dissatisfied with your sight at present?”; “Do you have difficulty with the following activities because of your sight?”. This last question allowed patients to label their satisfaction with vision in various contexts: reading text in newspapers; recognising the faces of people they meet; seeing the prices of goods when shopping; seeing to walk on uneven surfaces eg, cobblestones; seeing to do handicrafts/woodwork; reading subtitles on television; and seeing to engage in an activity/hobby of interest.

All surgery, and assessments before and after surgery, were completed by a single surgeon who performed the operation using standardised surgical technique, intraocular lens and emmetropic refractive target.

Statistical analysis

A group of statistical models termed the Item Response Theory (IRT) have been developed to instrument questionnaire development, evaluation and refinement. This framework analyses individual components of a questionnaire by a set of properties that describe the relationship of the questionnaire with the underlying construct measured by the model, in addition to how well individual questions fit with respect to the underlying construct. IRT is not dependent on the sample of respondents.14,15 This allows researchers to identify the questions that can most accurately measure the intended purpose of the questionnaire.

The Rasch model is a robust and commonly used form of IRT which can be used to assess functioning of rating scale categories within the Catquest-9SF and IoL questionnaires. This is a mathematical framework that takes into account the ability of participants, the difficulty of questions in the questionnaire, and assumes equal discriminating ability across all questions.16 In the Rasch model, the probability of a particular response to a specific question can be modelled as a logistic function of the difference between the person’s ability (measured by using test questions) and the difficulty of the items being asked.17

All IoL and Catquest-9SF question responses were assessed using the Rasch model to assess the validity of the questions in quantifying VRQoL. If responses to a question successfully fit the Rasch model, it provides evidence that this question adequately measures VRQoL. Two types of mean square fit statistics (infit and outfit) were used to evaluate how well patient responses fit the Rasch model for all of the questions within the IoL and Catquest-9SF questionnaires. Infit and outfit statistics have a chi-square distribution and provide an index of magnitude for the degree of misfit of a question with the model. These fit statistics have an expected value of 1 and suggested acceptable lower and upper thresholds of 0.5 and 1.5 respectively.18 Fit statistics for each question were calculated using an average of the squared residuals between the observed and expected responses from the Rasch model. The infit statistic is an estimate that gives more weight to individual variance of questionnaire responses to minimise the impact of unexpected responses far from the mean. Conversely, the outfit statistic is an unweighted estimate of the average question response variance within the IoL and Catquest-9SF questionnaires, and is more likely to be influenced by unexpected responses.

All statistical analyses were completed using R software.19 IoL questionnaire data were coded 1–6 representing the options of ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’, respectively. Catquest-9SF questionnaire data were coded 1–5 representing the options of ‘no difficulty’, ‘some difficulty’, ‘great difficulty’, ‘very great difficulty’ and ‘cannot decide’, respectively. The mirt package was used to fit models for item response theory analysis.20 Preoperative and postoperative data were combined for model fitting. The corrected Akaike information criterion (AICc) is an estimator of the relative quality of statistical models for a given set of data, and was used to select the best model. The G–test of goodness-of-fit was used to determine if the final model accurately predicted the data. Normalised factor scores for both questionnaires were correlated with visual acuity in the operated eye and age using Pearson’s product-moment correlation. Secondary analyses of normalised factor scores by ethnicity and gender were performed using analysis of variance. A qualified statistician reviewed all statistical methodology and analyses used in this study.

Results

Forty-one patients undergoing cataract surgery were enrolled in the study from March to May 2017. All patients who were approached consented to inclusion in the study and completed the questionnaire at both time points. The mean patient age was 77±8 years (sd), with 20 (49%) female participants. Ethnicity included New Zealand European 29 (71%), Māori 3 (7%), Pacific Island 1 (2%), Asian 4 (10%), Indian 3 (7%), and ‘Other’ 1 (2%). Table 1 shows the preoperative and postoperative visual acuities and spherical equivalent for the patient cohort.

Table 1: Visual acuity and spherical equivalent of 41 patients before and following cataract surgery.

UCVA = uncorrected visual acuity, BCVA = best corrected visual acuity, SPE = refractive error (spherical equivalent) in dioptres. Spherical equivalent = sphere power + (cylinder power x 0.5). Postoperative visual acuity was measured at three months following cataract surgery and all visual acuity was represented in logMAR notation.

Figure 2A: Rasch model category probability curves for the Impact on Life questionnaire.

c

These curves summarise the probability (y-axis) that a patient with visual difficulty (x-axis) would answer with a given response. P1 to P6 represents the question response options; P1 = no difficulty, P2 = little difficulty, P3 = some difficulty, P4 = quite difficult, P5 = very difficult, P6 = extremely difficult.

Figure 2B: Category frequency responses for 41 patients who completed the Impact on Life questionnaire before surgery (pink) and three months following surgery (blue).  

c

Table 2: Summary of Rasch model fit statistics for the Impact on Life (IoL) questionnaire. The sample includes responses from 41 patients preoperatively and at three months following cataract surgery.

The model fit statistics for the Catquest-9SF responses are summarised in Table 3. ‘Cannot decide’ responses on the Catquest-9SF questionnaire represented 7 of 738 responses (0.95%) and were assumed equivalent to data missing at random for analysis. Apart from ‘recognising faces’, ‘seeing price of goods when shopping’ and ‘ability to read TV subtitles’ (mean-square fit statistics 0.43, 0.44 and 0.48 respectively), all other Catquest-9SF questions were within the range suitable for measurement (mean-square outfit statistic 0.5 to 1.5). The graphical Rasch categorical probability curves for the Catquest-9SF questions are summarised in Figure 3A. The category frequency of each response in the IoL questionnaire is summarised in Figure 3B.

Table 3: Summary of Rasch model fit statistics for the Catquest-9SF questionnaire from International Consortium for Health Outcomes Measurement (ICHOM). The sample includes responses from 41 patients preoperatively and at three months following cataract surgery.

Figure 3A: Rasch model category probability curves for the Catquest-9SF questionnaire from International Consortium for Health Outcomes Measurement.

c

These curves summarise the probability (y-axis) that a patient with visual difficulty (x-axis) would answer with a given response. A higher number of question difficulty indicates greater disability (6 = extremely difficult; -6 = no difficulty). P1 to P4 represents the question response options; P1 = no difficulty; P2 = some difficulty; P3 = very difficult; P4 = extremely difficult.

Figure 3B: Category frequency responses for 41 patients who completed the Catquest-9SF questionnaire before surgery (pink) and three months following cataract surgery (blue).

c

The difference in visual acuity before and after surgery correlated with the change in total F-score for the Catquest-9SF responses (P=0.04), but not the IoL responses (P=0.17). The overall questionnaire score in both IoL and Catquest-9SF questionnaires correlated with worsening visual acuity (P<0.001). There were no statistical differences in quality of life scores between ages or ethnic groups for both questionnaires. The change in F-score was not significantly different for patients who received cataract surgery on their first eye or second eye.

Discussion

The current study uses Rasch analysis to evaluate the validity of the IoL and Catquest-9SF questionnaires for quantifying VRQoL for cataract surgery patients in New Zealand. As far as the authors are aware, this is the first study to statistically assess the validity of the IoL questionnaire for use in cataract surgery, and the first study to compare the IoL with any other questionnaire to assess VRQoL.

The unequal peaks noted in the Rasch analysis for the IoL questionnaire suggest that the response options of ‘no difficulty’, ‘little difficulty’, ‘some difficulty’, ‘quite difficult’, ‘very difficult’ and ‘extremely difficult’, are too numerous and ideally should be collapsed into fewer options with more consistent probability thresholds. This finding was consistent for all six questions in the IoL questionnaire. In contrast, the Catquest-9SF demonstrated relatively uniform peak height for the question response options that remained consistent for all questions in the Catquest-9SF questionnaire, similar to previous studies.21

The IoL questionnaire demonstrates category disordering on Rasch analysis. Category disordering occurs when the ordinal numbering of categories (response options) does not correspond with their substantive meaning. The IoL questionnaire ordered response options are substantively defined to represent increasing levels of disability in VRQoL. In all six questions, there is substantive and step disordering such that ‘little difficulty’ consistently locates below ‘no difficulty’ in the Rasch analysis. This finding suggests that the response options used in the IoL questionnaire are not able to accurately discriminate increasing impairment in VRQoL as intended.

The IoL questionnaire demonstrated unsatisfactory statistical fit of almost all questions (mean-square fit less than 0.5). This finding indicates less variation in participant responses than expected, and that responses are more predictable than the Rasch model expects. The high predictability of responses to IoL questions and overfit to the Rasch model suggests sub-optimal question wording resulting in non-discriminatory patient responses.22 This finding suggests that the IoL questionnaire is likely to lack the required sensitivity to accurately rank patients based on VRQoL.

Catquest-9SF questions demonstrated satisfactory mean fit squares and appropriate category response curves with monotonic increases and decreases in the category thresholds (Figure 3A). This finding was consistent with other studies evaluating the Catquest-9SF using Rasch analysis in Europe and Australia.21,23 These results confirm that the Catquest-9SF questionnaire is valid tool for the assessment of VRQoL in a New Zealand population and can accurately rank patients based on VRQoL.

Questionnaire scores for the Catquest-9SF and the IoL improved with the improvement in visual acuity following surgery. Only the Catquest-9SF questionnaire, however, demonstrated significant correlation between the change in visual acuity and change in questionnaire F-scores following surgery. The F-score is a single indicator that summarises the variance (accuracy and recall ratio) of data points around the mean, which can be used to evaluate and compare the fit of multiple linear models.24 Based on these results, the IoL questionnaire responses appear to be independent to VRQoL and poorly suited for predicting which patients will experience quality of life gains as a result of improved vision following cataract surgery.

There are several limitations to this study. Firstly, patient bias may influence questionnaire responses. Patients may suspect that preoperative questionnaire responses could affect their eligibility for surgery and bias towards over-reporting poor quality of life prior to surgery or after surgery where second eye surgery is required. The lack of significant difference in F-scores between patients receiving first or second eye cataract surgery, however, suggests similar degrees of variance in responses indicating no such bias exists in this data. Secondly, the current study has a relatively small sample size. Reports of Rasch analysis results are considered to be robust to smaller sample size.25 In addition, despite the small sample size, the current study was able to replicate similar findings to previous, larger studies evaluating the Catquest-9SF.21,23

The primary strength of this study is analysis of the qualitative responses using Rasch analysis. The importance of Rasch analysis has been well-recognised for the evaluation of questionnaire quality and there have been numerous requests for the development of Rasch-approved questionnaires within ophthalmology.26–28 The current study offers the first Rasch assessment of the IoL questionnaire. This questionnaire is currently in widespread use to assess eligibility for all patients in the New Zealand public health system that require cataract surgery.

In summary, the current study compared the ability of IoL and Catquest-9SF questionnaires to accurately measure VRQoL. The results of this study demonstrate that the IoL does not accurately assess VRQoL for patients that require cataract surgery in New Zealand. The Catquest-9SF is a domain-specific assessment tool that can accurately measure VRQoL in New Zealand. The convenience of using a single tool, such as the IoL, to allocate healthcare resources across multiple specialities must be carefully weighed against the risk of not allocating resources where they are needed the most.

Despite its widespread use, the current study highlights inadequacies of the IoL questionnaire for the assessment of VRQoL for cataract surgery in New Zealand. In addition to any role in surgical prioritisation, it is increasingly important for quality improvements in healthcare delivery to use standardised patient reported outcome tools, such as the Catquest-9SF. These standardised tools enable international benchmarking and direct comparison with other studies. In conclusion, the Catquest-9SF questionnaire provides a more accurate assessment of VRQoL than the currently used IoL questionnaire for New Zealand patients that require cataract surgery.

Summary

Abstract

Aim

The Impact on Life (IoL) questionnaire is used to prioritise publicly funded cataract surgery in New Zealand, however, it has not been formally validated for ophthalmic use. The Catquest-9SF questionnaire is widely used to assess vision-related quality of life (VRQoL) but has not been validated in New Zealand. This study evaluates the validity of the IoL and Catquest-9SF questionnaires for measuring VRQoL in New Zealand.

Method

Formal ethics approval was obtained. Participants completed the IoL and Catquest-9SF questionnaires before and three months after routine cataract surgery. Rasch analysis was used to investigate all qualitative questionnaire responses. Results were correlated with the change in patient visual acuity.

Results

There was a 100% response rate at follow-up (41 participants). Disordered probability thresholds were observed for all IoL questions but no Catquest-9SF questions. All IoL questions demonstrated unsatisfactory mean-square fit statistics. Differences in visual acuity following surgery correlated with the change in total F-score for the Catquest-9SF (P=0.04), but not IoL responses (P=0.17).

Conclusion

Disordered probability thresholds, poor question-model fit and correlation with visual acuity changes indicate the current IoL questionnaire is poorly suited for assessment of VRQoL. In contrast, the Catquest-9SF demonstrated credible results for assessment of VRQoL in New Zealand.

Author Information

Sunny S Li, House Officer, Counties Manukau District Health Board, Auckland;-Stuti Misra, Senior Lecturer, The University of Auckland, Auckland;-Henry Wallace, House Officer, Auckland District Health Board, Auckland;-Lyn Hunt, Senior Lecturer, Academic Programme Convenor (Statistics), The University of Waikato, Hamilton; James McKelvie, Ophthalmologist, Waikato District Health Board, Hamilton.

Acknowledgements

Correspondence

James McKelvie, Department of Ophthalmology, Private Bag 92019, University of Auckland, Auckland.

Correspondence Email

james@mckelvie.co.nz

Competing Interests

Nil.

  1. Elder M, Tarr K, Leaming D. The New Zealand cataract and refractive surgery survey 1997/1998(1). Am J Ophthalmol. 2000 Oct; 130(4):543.
  2. Riley AF, Grupcheva CN, Malik TY, Craig JP, McGhee CN. The Auckland Cataract Study: demographic, corneal topographic and ocular biometric parameters. Clin Experiment Ophthalmol. 2001 Dec; 29(6):381–6.
  3. OECD Health Statistics (database). OECD 2018 [Internet]. Health care utilisation (Edition 2018). [cited 2019 Feb 17]. Available from: http://dx.doi.org/10.1787/732466c2-en
  4. Hadorn DC, Holmes AC. The New Zealand priority criteria project. Part 1: Overview. BMJ. 1997; 314(7074):131–131.
  5. Gauld R, Derrett S. Solving the surgical waiting list problem? New Zealand’s ‘booking system’. Int J Health Plann Manage. 2000; 15(4):259–72.
  6. Wong VWY, Lai TYY, Lam PTH, Lam DSC. Prioritization of cataract surgery: visual analogue scale versus scoring system. ANZ J Surg. 2005 Jul; 75(7):587–92.
  7. Chan G, Bezuidenhout L, Walker L, Rowan R. The Impact on Life questionnaire: validation for elective surgery prioritisation in New Zealand prioritisation criteria in orthopaedic surgery. N Z Med J. 2016 Apr 1; 129(1432):26–32.
  8. Pesudovs K, Coster DJ. An instrument for assessment of subjective visual disability in cataract patients. Br J Ophthalmol. 1998 Jun; 82(6):617–24.
  9. Mangione CM, Lee PP, Gutierrez PR, Spritzer K, Berry S, Hays RD, et al. Development of the 25-item National Eye Institute Visual Function Questionnaire. Arch Ophthalmol. 2001 Jul; 119(7):1050–8.
  10. Roberts HW, Wagh VK, Sullivan DL, Hidzheva P, Detesan DI, Heemraz BS, et al. A randomized controlled trial comparing femtosecond laser–assisted cataract surgery versus conventional phacoemulsification surgery. J Cataract Refract Surg. 2019; 45(1):11–20.
  11. Porter ME, Larsson S, Lee TH. Standardizing Patient Outcomes Measurement. N Engl J Med. 2016 Feb 11; 374(6):504–6.
  12. McAlinden C, Gothwal VK, Khadka J, Wright TA, Lamoureux EL, Pesudovs K. A head-to-head comparison of 16 cataract surgery outcome questionnaires. Ophthalmology. 2011 Dec; 118(12):2374–81.
  13. Lundstrom M, Behndig A, Kugelberg M, Montan P, Stenevi U, Pesudovs K. The outcome of cataract surgery measured with the Catquest-9SF. Acta Ophthalmol. 2011 Dec; 89(8):718–23.
  14. Reeve BB. Item response theory modeling in health outcomes measurement. Expert Rev Pharmacoecon Outcomes Res. 2003 Apr; 3(2):131–45.
  15. Chang C-H, Reeve BB. Item Response Theory and its Applications to Patient-Reported Outcomes Measurement. Eval Health Prof. 2005; 28(3):264–82.
  16. Wolins L, Wright BD, Rasch G. Probabilistic Models for some Intelligence and Attainment Tests. J Am Stat Assoc. 1982; 77(377):220.
  17. Luo X, Cappelleri JC, Cella D, Li JZ, Charbonneau C, Kim ST, et al. Using the Rasch model to validate and enhance the interpretation of the Functional Assessment of Cancer Therapy-Kidney Symptom Index--Disease-Related Symptoms scale. Value Health. 2009 Jun; 12(4):580–6.
  18. Linacre JM. Rasch Measurement Transactions: Based on Rasch measurement transactions Vol. 6:no. 1-Vol. 8:4. 1995.
  19. R Core Team. R: A language and environment for statistical computing [Internet]. 2017. Available from: http://www.R-project.org/
  20. Chalmers RP, Philip Chalmers R. mirt: A Multidimensional Item Response Theory Package for the R Environment. J Stat Softw [Internet]. 2012;48(6). Available from: http://dx.doi.org/10.18637/jss.v048.i06
  21. Lundström M, Pesudovs K. Catquest-9SF patient outcomes questionnaire: nine-item short-form Rasch-scaled revision of the Catquest questionnaire. J Cataract Refract Surg. 2009 Mar; 35(3):504–13.
  22. Pallant JF, Tennant A. An introduction to the Rasch measurement model: An example using the Hospital Anxiety and Depression Scale (HADS) [Internet]. Vol. 46, British Journal of Clinical Psychology. 2007. p. 1–18. Available from: http://dx.doi.org/10.1348/014466506x96931
  23. Gothwal VK, Wright TA, Lamoureux EL, Lundström M, Pesudovs K. Catquest questionnaire: re-validation in an Australian cataract population. Clin Experiment Ophthalmol. 2009; 37(8):785–94.
  24. Goutte C, Gaussier E. A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation. In: Lecture Notes in Computer Science. 2005. p. 345–59.
  25. Smith AB, Rush R, Fallowfield LJ, Velikova G, Sharpe M. Rasch fit statistics and sample size considerations for polytomous data. BMC Med Res Methodol. 2008 May 29; 8:33.
  26. Pesudovs K, Burr JM, Harley C, Elliott DB. The Development, Assessment, and Selection of Questionnaires. Optom Vis Sci. 2007; 84(8):663–74.
  27. Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007 Jan; 60(1):34–42.
  28. Pesudovs K. Patient-centred measurement in ophthalmology – a paradigm shift. BMC Ophthalmol [Internet]. 2006;6(1). Available from: http://dx.doi.org/10.1186/1471-2415-6-25

Contact diana@nzma.org.nz
for the PDF of this article

Subscriber Content

The full contents of this pages only available to subscribers.

LOGINSUBSCRIBE