View Article PDF

Vaccinations have produced some of the largest gains in the history of public health interventions. Timely, complete and safe immunisation requires reliable and complete vaccination records at the individual level. Accurate and meaningful evaluation of the coverage, effectiveness and safety of the National Immunisation Schedule (NIS) vaccines also depends on reliable and complete vaccination records at the population level.[[1]] In Aotearoa New Zealand, there are two sources of immunisation records: each child’s Well Child Tamariki Ora parent-held health record book (Health Book) and the National Immunisation Register (NIR), which is automatically fed data every night from the practice management system (PMS) of immunisation providers. Health Books were the only source for immunisation records until the rollout of the NIR. A Health Book is given to parents upon the birth of a child and contains child health and development information.

A section on vaccination in each Health Book allows vaccinators to record details of vaccinations given as per the routine New Zealand childhood NIS, as well as any other preschool vaccinations given as required or privately purchased. When a child is immunised overseas, their Health Book is retrospectively filled out in general practice upon sighting proof of vaccination documentation. Written records in Health Books are seen as the reference standard to which NIR records can be compared because Health Book records are written directly by the vaccinator at the time of vaccination and the Health Book Immunisation Certificate is necessary documentation for enrolment at early childcare centres and school.

The NIR is a computerised information system developed to hold vaccination details of children in New Zealand from six weeks to 12 years (and some adult vaccines) (Figure 1) and has been comprehensively used to record vaccination details for all children in New Zealand born since 2006.[[1]] The NIR is a valuable resource, but there are probably differences between the NIR and the records in each child’s Health Book, and the degree of error is currently unknown. Validation research internationally and locally has found varying levels of misclassification in electronic registries.[[2–6]] Recording errors have the potential to result in bias results: for example, a small study in New Zealand indicated that the NIR may overestimate the number of children meeting national milestone targets.[[5]] Our study proposed to examine, describe and quantify the extent of the error in the NIR using Health Books as a reference standard.

Figure 1: New Zealand National Immunisation Schedule (NIS) for childhood vaccines between 2006 and 2019.

DTaP-IPV: diphtheria, tetanus, acellular pertussis, inactivated poliovirus vaccines. Tdap-IPV: diphtheria, tetanus, acellular pertussis, inactivated poliovirus vaccines. DTaP- IPV-HepB/Hib: diphtheria, tetanus, acellular pertussis, inactivated poliovirus, hepatitis B and Haemophilus influenzae type b vaccines. Tdap: diphtheria, tetanus, acellular pertussis vaccines. Hib-Hep B: Haemophilus influenzae type b and hepatitis B vaccine. Hib-PRP: Haemophilus influenzae type b polyribosylribitol phosphate vaccine. MeNZB: A strain-specific g roup B meningococcal vaccine. MMR: measles, mumps, rubella vaccine. PCV7: 7-valent pneumococcal conjugate vaccine. PCV10: 10-valent pneumococcal conjugate vaccine. PCV13: 13-valent pneumococcal conjugate vaccine.RV5: pentavalent rotavirus vaccine. RV1: monovalent rotavirus vaccine. HPV4: quadrivalent human papillomavirus vaccine. HPV9: 9 valent human papillomavirus vaccine. VV: varicella vaccine (chickenpox; varicella-zoster virus).

Methods

Study population

The inclusion criteria were children born between 2006 and 2019 with vaccination data recorded in their Health Book for vaccinations given from birth until 4 years of age, and where the parent or legal guardian believes the recorded vaccination data is a true record of the vaccinations the child has received. Parents and caregivers of child participants were recruited as a convenience sample via the University of Auckland intranet, posters at appropriate venues (eg, Dunedin kindergartens and Allied Health Plus primary health organisation general practice clinics in Auckland), social media (eg, targeted Facebook advertisements), use of Well Child Tamariki Ora and immunisation provider networks (eg, the Health Book provider newsletter and vaccinator education mailing lists, posters at the New Zealand Immunisation Conference) and word of mouth.

At first contact, parents and caregivers were provided with either a paper or electronic participant information sheet and consent form. Participation required parents and caregivers to return a completed, signed consent form and submit a picture of the vaccination page of their child’s Health Book. Children aged 8 to 14 years had to sign an assent form if they agreed to participate. These documents were submitted either electronically by email (as a scanned or photographed image) or in person as an original hardcopy. In acknowledgement of their time, all guardians who returned participant data and completed recruitment went into the draw to win one of five $100 gift cards to be spent at selected grocery stores, pharmacies and stationery shops. NIR records were identified for each child using their unique National Health Index (NHI) identifier. Subsequently, each child participant was assigned a unique study ID to preserve anonymity, and this was used across all data sources.

Well Child Tamariki Ora parent-held health record books

Data elicited for this study was sourced from the “Immunisation record” section of each Health Book. Relevant fields included vaccine, batch, site, date given, sign/stamp and notes.

National Immunisation Register

Individual-level vaccination data was extracted from the NIR using each child’s NHI identifier. Relevant fields included the NHI, vaccination date, vaccine, vaccine dose, antigen and batch number. Live NHIs were provided to the Ministry of Health Analytical Services team for the data extraction. Demographic information on each child was also requested. Relevant fields included children’s sex, date of birth, ethnicity, New Zealand Index of Deprivation 2013 decile and district health board of residence.

Although the NIR has a field allowing for the indication of high-risk schedules for at least Prevenar and influenza, validation of this variable could not be established as this information is not indicated in Health Books. (High risk schedules are schedule modifications for high-risk groups: for example, additional doses of an already scheduled vaccine or another vaccine, like the 23-valent pneumococcal vaccine for selected high-risk groups such as those with primary immune deficiency or human immunodeficiency virus.) Therefore, this study defines NIS vaccines as the routine New Zealand childhood immunisation schedule vaccines. Any vaccines outside of this definition were considered non-NIS vaccines. Some non-NIR vaccines, such as for rotavirus, have become NIS vaccines over the study period. These changes have been accounted for by comparing NIS with the date of vaccine administration.

Statistical analysis

Vaccination data from the Health Books were entered into an Excel spreadsheet with only the study ID to identify each participant. Records from Health Books were entered by two researchers (AH and HC) and checked by a third (JP). NIR and NHI data were merged with this reference standard data by unique study IDs. Vaccination records were excluded if they occurred after 60 months of age (this was to allow for flexibility in later delivery of the four-year milestone age vaccines). Records were also excluded if they were administration artefacts of the NIR, rather than a vaccine record (eg, records of vaccine declines).

Participant characteristics were described as number and percentage for the total population. Discordance between the Health Book records and the NIR records were investigated and described as percentage of misclassified vaccination events and types of errors found: for example, incorrect vaccine type or date, or missing events. Observed agreement, sensitivity, specificity and positive and negative predictive values were used as an index of agreement between the NIR and the Health Books. These measures perform best as a first step to quantify agreement between measures and have been used in previous literature on this topic.[[7]]

Results were presented by both total records and NIR vaccination records. This was because the authors noted substantial differences in recording concordance between NIR vaccination records and non-NIR vaccination records. All statistical analyses were undertaken using SAS Enterprise Guide (9.4) statistical software (SAS Institute Inc., Cary, NC, USA). This study was approved by the Health and Disability Ethics Committee: ethics reference number 19/CEN/51.

Results

One hundred and one participants were initially recruited, but three were excluded due to data quality concerns (Figure 2). After application of exclusion criteria (Figure 2), the total number of vaccination records was 1,641.

Figure 2: Study flowchart.

Characteristics of the study participants are summarised in Table 1. Of the 98 participants, most were European (75.5%), with 14% identifying as Māori, 9% as Asian and 1% as Pacific Island. Two-thirds lived in low deprivation areas (deciles 1–4), and a third lived in medium to high deprivation areas (deciles 5–10). Although nearly three-quarters of participants were from the Auckland area, there was participation from around the country. Seventeen percent resided in the South Island.

Table 1: Demographics of Health Book study participants.

Most vaccination records (85%) were present in both the Health Book and the NIR (Table 2). The remaining 15% were recorded in either the Health Book or the NIR. Most NIR vaccination records were present in both the Health Book and the NIR. However, only a minority (5%) of the non-NIS vaccinations were recorded in both the Health Book and the NIR. Almost three-quarters of the non-NIS vaccinations were recorded in the Health Book only, and the remaining quarter were recorded in the NIR only.

Table 2: Immunisation records by record source.

NIR: National Immunisation Register. [[1]] Column percentage.

The agreement between the Health Book and NIR records for both the recorded date and vaccine was high (Table 3). The date and vaccine record agreement for NIS and non-NIS vaccines was similarly high; however, the number of non-NIS vaccination records was small. Appendix Table 1 presents the non-NIS vaccine records by the data source. Influenza was the only non-NIS vaccine to be recorded by both sources; however, the majority of influenza vaccine records were contained in the NIR only. Influenza was also the only non-NIS vaccine to be recorded by the NIR only. Of the non-NIS vaccines recorded in Health Books only, varicella and rotavirus vaccines (before their introduction to the NIS) were the most common. Appendix Table 2 presents the comparison of vaccines for each data source where vaccine type did not match. The greatest discrepancy was for PCV10 and PCV13; this was most probably due to NIS changes in pneumococcal vaccine brand/valency and general practice vaccine stock. General practices could have used up previous pneumococcal vaccine stock before distribution of the new pneumococcal vaccine. Alternatively, vaccinators may have incorrectly recorded the PCV brand/valency administered.

Table 3: Immunisation date and vaccine agreement by immunisation schedule for records contained in both sources.

[[1]] Column percentage. A third had a recorded date one day earlier in the NIR and a third had a day or two later recorded in the NIR, with a range between -295 and 247 days.

The predictive accuracy of the NIR compared with the Health Books is reported in Table 4. The NIR had the greatest sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for the NIR vaccination records. For the total records, the NIR demonstrated high sensitivity (92%) and high specificity (81%). The PPV (92%) and NPV (80%) were also high. The NIR had the lowest accuracy for non-NIS vaccinations, and this may in part be due to the small number of records.

Table 4: Predictive accuracy of the National Immunisation Register compared to the Well Child Tamariki Ora parent-held health record books (Health Books).

[[1]] n=1,641. [[2]] n=1,526. [[3]] n=115.

Discussion

The results of this study indicate the NIR replicates information in Health Books with a high level of accuracy for NIS vaccines: sensitivity, specificity, positive predictive value and negative predictive value estimates were 85% and above.

In international comparisons, the NIR generally performs well. The NIR demonstrated greater sensitivity than the Ontario Health Insurance Plan database by approximately 11%.[[2]] The NIR had substantially less error than the Boston Immunization Information System where chart records were used as the comparison; date agreement for the NIR was 97% compared with 66% for the Boston Immunization Information System.[[4]]

Locally, our study indicated a higher level of accuracy than the Reynolds et al (2014) study that compared records of the completion of the five-month vaccination target in the NIR to the PMS records. However, the Reynolds et al study had a small sample restricted to only one general practice. Reynolds et al reported that the PMS recorded 9.7% greater immunisation levels compared with the NIR. In contrast, our study found very high sensitivity and positive predictive value for the NIS vaccination records in the NIR compared to the Health Books.

The reasons for discordance between the NIR and Health Book vaccination records may include: vaccinators entering data into the incorrect field in the PMS; IT errors in the translation of vaccination records from the PMS to the NIR or in the centralised recordings of the NIR; or failures to enter vaccination records into the PMS. However, the time-consuming nature of transcription from overseas vaccination records may account for some discordance. Although country of birth was not available for study participants, children born overseas are less likely to have a record in the NIR compared with New Zealand-born children.[[8]] International records must be added by providers, and frontline providers may prioritise the simpler approach of adding an entry into the Health Book over the complexity of entering international records into the NIR.

This is the first study to the authors’ knowledge that describes the extent of NIR error for non-NIS vaccination records in New Zealand. The number of non-NIS vaccination records was considerably lower than the number of NIS vaccination records. Most non-NIS vaccine records were recorded in Health Books only. Influenza was the only exception, where the NIR contained more records than the Health Books. Although the NIR is not designed or intended for recording non-NIS vaccine records, accurate records of non-NIS vaccines (especially those with known associations and significant adverse events following immunisation, such as for rotavirus) are essential for safety monitoring activities. In addition, silent inequities are possible where inaccurate records in non-NIS vaccinations exist. For example, those most at risk may not be receiving non-NIS vaccinations. Misclassification of vaccination status for non-NIR vaccines is a concern. Estimates of vaccine effectiveness are an important component of the negotiation to move a non-NIR vaccine onto the NIR. NIR data are utilised for studies of vaccine effectiveness, and underestimation of vaccination status may result in reduced vaccine effectiveness estimates.[[9]] Also, misclassification could result in missed or slower recognition of adverse events following vaccination, particularly if these events are rare.

The high sensitivity and specificity reported for NIS vaccinations is encouraging. Our results indicate that national milestone reporting, vaccine effectiveness estimates and safety research using historical NIR data is probably relatively accurate. The lack of substantial ethnic and socioeconomic diversity in the study sample prevents any comment on the accuracy of milestone reporting for equity. Although our non-NIS vaccine sample size was small, our results indicate a propensity for non-NIS vaccine exposure misclassification using historical NIR data. We recommend caution be exercised for non-NIS vaccine research using historical NIR data for exposure misclassification. We recommend researchers explore the implications of exposure misclassification on study outcomes. A larger study is needed to determine the extent of non-NIS vaccine misclassification in the NIR.

Our study sample was small and obtained through convenience sampling. One implication of this is that there were several family clusters. Sibling records are subject to similar demographic characteristics and behaviours of parents and general practice providers. Thus, these have the potential to bias our parameter estimates. Our sample was not representative of the New Zealand paediatric population over the study period. New Zealand Europeans, children living in low-socioeconomic deprivation areas and the Northern District Health Board region were over-represented in our sample. Demographic characteristics may have influenced the completeness of records in both the NIR and the Health Books. There are well-established disparities in the quality of healthcare received between Māori and non-Māori and low- and high-socioeconomic deprivation populations in New Zealand; this may have implications for the quality of vaccination records and the PPV and NPV as they are sensitive to the characteristics of the population in which they are measured.[[10,11]] Investigation of demographic inequalities in NIR vaccination records is an important concern to be addressed in future research. We acknowledge Health Book records are probably not without error and that this could bias the results, but it is unclear in which direction. Completion of each Health Book is influenced by the nature of the healthcare appointment (opportunistic vaccination or scheduled immunisation appointment or immunisation outreach service), whether the Health Book is lost and whether the parent/caregiver remembers to bring it to the appointment or at a later time for updating. However, less than 7% of records were recorded in the NIR only, indicating that this source of error was unlikely to have significantly affected parameter estimates. Some aspects of the process of recording vaccination events are common to Health Book and PMS records and, therefore, are subject to some of the same omission or misclassification errors. It is not possible to determine whether or to what extent this occurred in our study. Population groups with high mobility and/or poor healthcare access are more likely to have incomplete Health Book vaccination records. However, we expect the completeness of Health Book records in our study to have been high, as inclusion criteria stipulated that guardians needed to believe their children’s Health Books were an accurate record of the vaccinations received during childhood. Although it could be argued that sensitivity and specificity were used as a proxy for agreement, these measures represent an initial inquiry and, together with the relatively small sample size, are associated with limitations such as overestimation of agreement.

Conclusion

This study compared two methods of registering vaccination. Neither method is perfect, but each likely has different types of errors. The results of this work indicate the NIR data accuracy generally performs well compared with Well Child Tamariki Ora parent-held health record books (Health Books), especially for NIS vaccine records. To the authors’ knowledge, this is the first study in New Zealand to have also looked at non-NIS vaccination records. Description and quantification of the error in the NIR can be used to improve the accuracy of immunisation research in New Zealand. We recommend further validation research be undertaken, in particular for non-NIS vaccines and for Māori, Pacific and low-socioeconomic deprivation groups. If a new NIR is established, validation of this system will be necessary also.

Appendix

Appendix Table 1: Vaccines of non-scheduled immunisations by record source.

NIR: National Immunisation Register. [[1]] Column percentage.

Appendix Table 2: Comparison of vaccines for each source where vaccine type did not match.

NIR: National Immunisation Register.

Summary

Abstract

Aim

The National Immunisation Register (NIR), which is derived from general practice management systems, is an important tool for the provision of clinical services, national immunisation programme evaluation and immunisation research in New Zealand. However, the accuracy of the NIR data has not yet been quantified. This study aimed to examine, describe and quantify the extent of discrepancy in the NIR compared to Well Child Tamariki Ora parent-held health record books (Health Books).

Method

Immunisation data for vaccinations given between birth and four years old for children born between 2006 and 2019 were compared between the Health Books and the NIR. Health Book records were used as the reference standard to calculate performance measures: sensitivity, specificity, positive and negative predictive values for the NIR.

Results

Overall, NIR performance was high: sensitivity ranged from 90% to 93%, specificity from 78% to 85%, the positive predictive value from 91% to 94% and the negative predictive value from 77% to 84%. NIR performance was higher for National Immunisation Schedule (NIS) vaccines compared with non-NIS vaccines.

Conclusion

This study indicates the NIR data accuracy generally performs well compared with international equivalents, especially for NIS vaccine records. Further work is required to ascertain why discrepancies between the Health Books and NIR continue to occur, with particular attention to important subgroups and translating records across from migrant populations. Also, future work is required to understand the accuracy of vaccination records for groups who experience lower-quality healthcare and a higher burden of infectious diseases.

Author Information

Anna S Howe: Lecturer, School of Health Sciences, University of Canterbury; Research Fellow, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland. Hannah Chisholm: PhD Candidate, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland. Janine Paynter: Senior Research Fellow, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland. Esther Willing: Lecturer, Kōhatu – Centre for Hauora Māori, Division of Health Sciences, University of Otago. Nikki Turner: Director, Immunisation Advisory Centre, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland.

Acknowledgements

We would like to thank Karin Batty for sharing her contextual and technical expertise with us and for reviewing this manuscript. Karin, you are a tremendous resource.

Correspondence

Anna Howe, School of Health Sciences, University of Canterbury, Private Bag 4800, Christchurch 8140

Correspondence Email

a.howe@ auckland.ac.nz

Competing Interests

All authors have been involved in research utilising NIR data for vaccine effectiveness, safety and coverage studies. Additionally, Nikki Turner is a GP.

1) Ministry of Health [Internet]. National Immunisation Register: Ministry of Health; 2015. Available from: https://www.health.govt.nz/our-work/preventative-health-wellness/immunisation/national-immunisation-register

2) Schwartz KL, Tu K, Wing L, et al. Validation of infant immunization billing codes in administrative data. Hum Vaccin Immunother. 2015;11(7):1840-7.

3) MacDonald SE, Schopflocher DP, Golonka RP. The pot calling the kettle black: the extent and type of errors in a computerized immunization registry and by parent report. BMC Pediatr. 2014;14:1-.

4) Samuels R, Appel L, Reddy SI, Tilson RS. Improving accuracy in a computerized immunization registry. Ambul Pediatr. 2002;2 3:187-92.

5) Reynolds G, Timo M, Dev A, et al. Effective general practice: audit and feedback for the primary series of immunisations. J Prim Health Care. 2014;6(1):40-8.

6) Poskitt N, Taft S. National Immunisation Register inaccuracies and duplications. N Z Med J (Online). 2009;122(1288).

7) Hripcsak G, Heitjan DF. Measuring agreement in medical informatics reliability studies. Journal of Biomedical Informatics 2002; 35(2): 99-110.

8) Charania NA, Paynter J, Lee AC, et al. Vaccine-Preventable Disease-Associated Hospitalisations Among Migrant and Non-migrant Children in New Zealand. J Immigr Minor Health. 2020;22(2):223-31.

9) Jackson ML. Use of self-reported vaccination status can bias vaccine effectiveness estimates from test-negative studies. Vaccine: X. 2019;1:100003.

10) Mulholland M. State of the Māori nation : twenty-first century issues in Aotearoa. Auckland, New Zealand: Reed; 2006.

11) Šimundić A-M. Measures of Diagnostic Accuracy: Basic Definitions. EJIFCC. 2009;19(4):203.

For the PDF of this article,
contact nzmj@nzma.org.nz

View Article PDF

Vaccinations have produced some of the largest gains in the history of public health interventions. Timely, complete and safe immunisation requires reliable and complete vaccination records at the individual level. Accurate and meaningful evaluation of the coverage, effectiveness and safety of the National Immunisation Schedule (NIS) vaccines also depends on reliable and complete vaccination records at the population level.[[1]] In Aotearoa New Zealand, there are two sources of immunisation records: each child’s Well Child Tamariki Ora parent-held health record book (Health Book) and the National Immunisation Register (NIR), which is automatically fed data every night from the practice management system (PMS) of immunisation providers. Health Books were the only source for immunisation records until the rollout of the NIR. A Health Book is given to parents upon the birth of a child and contains child health and development information.

A section on vaccination in each Health Book allows vaccinators to record details of vaccinations given as per the routine New Zealand childhood NIS, as well as any other preschool vaccinations given as required or privately purchased. When a child is immunised overseas, their Health Book is retrospectively filled out in general practice upon sighting proof of vaccination documentation. Written records in Health Books are seen as the reference standard to which NIR records can be compared because Health Book records are written directly by the vaccinator at the time of vaccination and the Health Book Immunisation Certificate is necessary documentation for enrolment at early childcare centres and school.

The NIR is a computerised information system developed to hold vaccination details of children in New Zealand from six weeks to 12 years (and some adult vaccines) (Figure 1) and has been comprehensively used to record vaccination details for all children in New Zealand born since 2006.[[1]] The NIR is a valuable resource, but there are probably differences between the NIR and the records in each child’s Health Book, and the degree of error is currently unknown. Validation research internationally and locally has found varying levels of misclassification in electronic registries.[[2–6]] Recording errors have the potential to result in bias results: for example, a small study in New Zealand indicated that the NIR may overestimate the number of children meeting national milestone targets.[[5]] Our study proposed to examine, describe and quantify the extent of the error in the NIR using Health Books as a reference standard.

Figure 1: New Zealand National Immunisation Schedule (NIS) for childhood vaccines between 2006 and 2019.

DTaP-IPV: diphtheria, tetanus, acellular pertussis, inactivated poliovirus vaccines. Tdap-IPV: diphtheria, tetanus, acellular pertussis, inactivated poliovirus vaccines. DTaP- IPV-HepB/Hib: diphtheria, tetanus, acellular pertussis, inactivated poliovirus, hepatitis B and Haemophilus influenzae type b vaccines. Tdap: diphtheria, tetanus, acellular pertussis vaccines. Hib-Hep B: Haemophilus influenzae type b and hepatitis B vaccine. Hib-PRP: Haemophilus influenzae type b polyribosylribitol phosphate vaccine. MeNZB: A strain-specific g roup B meningococcal vaccine. MMR: measles, mumps, rubella vaccine. PCV7: 7-valent pneumococcal conjugate vaccine. PCV10: 10-valent pneumococcal conjugate vaccine. PCV13: 13-valent pneumococcal conjugate vaccine.RV5: pentavalent rotavirus vaccine. RV1: monovalent rotavirus vaccine. HPV4: quadrivalent human papillomavirus vaccine. HPV9: 9 valent human papillomavirus vaccine. VV: varicella vaccine (chickenpox; varicella-zoster virus).

Methods

Study population

The inclusion criteria were children born between 2006 and 2019 with vaccination data recorded in their Health Book for vaccinations given from birth until 4 years of age, and where the parent or legal guardian believes the recorded vaccination data is a true record of the vaccinations the child has received. Parents and caregivers of child participants were recruited as a convenience sample via the University of Auckland intranet, posters at appropriate venues (eg, Dunedin kindergartens and Allied Health Plus primary health organisation general practice clinics in Auckland), social media (eg, targeted Facebook advertisements), use of Well Child Tamariki Ora and immunisation provider networks (eg, the Health Book provider newsletter and vaccinator education mailing lists, posters at the New Zealand Immunisation Conference) and word of mouth.

At first contact, parents and caregivers were provided with either a paper or electronic participant information sheet and consent form. Participation required parents and caregivers to return a completed, signed consent form and submit a picture of the vaccination page of their child’s Health Book. Children aged 8 to 14 years had to sign an assent form if they agreed to participate. These documents were submitted either electronically by email (as a scanned or photographed image) or in person as an original hardcopy. In acknowledgement of their time, all guardians who returned participant data and completed recruitment went into the draw to win one of five $100 gift cards to be spent at selected grocery stores, pharmacies and stationery shops. NIR records were identified for each child using their unique National Health Index (NHI) identifier. Subsequently, each child participant was assigned a unique study ID to preserve anonymity, and this was used across all data sources.

Well Child Tamariki Ora parent-held health record books

Data elicited for this study was sourced from the “Immunisation record” section of each Health Book. Relevant fields included vaccine, batch, site, date given, sign/stamp and notes.

National Immunisation Register

Individual-level vaccination data was extracted from the NIR using each child’s NHI identifier. Relevant fields included the NHI, vaccination date, vaccine, vaccine dose, antigen and batch number. Live NHIs were provided to the Ministry of Health Analytical Services team for the data extraction. Demographic information on each child was also requested. Relevant fields included children’s sex, date of birth, ethnicity, New Zealand Index of Deprivation 2013 decile and district health board of residence.

Although the NIR has a field allowing for the indication of high-risk schedules for at least Prevenar and influenza, validation of this variable could not be established as this information is not indicated in Health Books. (High risk schedules are schedule modifications for high-risk groups: for example, additional doses of an already scheduled vaccine or another vaccine, like the 23-valent pneumococcal vaccine for selected high-risk groups such as those with primary immune deficiency or human immunodeficiency virus.) Therefore, this study defines NIS vaccines as the routine New Zealand childhood immunisation schedule vaccines. Any vaccines outside of this definition were considered non-NIS vaccines. Some non-NIR vaccines, such as for rotavirus, have become NIS vaccines over the study period. These changes have been accounted for by comparing NIS with the date of vaccine administration.

Statistical analysis

Vaccination data from the Health Books were entered into an Excel spreadsheet with only the study ID to identify each participant. Records from Health Books were entered by two researchers (AH and HC) and checked by a third (JP). NIR and NHI data were merged with this reference standard data by unique study IDs. Vaccination records were excluded if they occurred after 60 months of age (this was to allow for flexibility in later delivery of the four-year milestone age vaccines). Records were also excluded if they were administration artefacts of the NIR, rather than a vaccine record (eg, records of vaccine declines).

Participant characteristics were described as number and percentage for the total population. Discordance between the Health Book records and the NIR records were investigated and described as percentage of misclassified vaccination events and types of errors found: for example, incorrect vaccine type or date, or missing events. Observed agreement, sensitivity, specificity and positive and negative predictive values were used as an index of agreement between the NIR and the Health Books. These measures perform best as a first step to quantify agreement between measures and have been used in previous literature on this topic.[[7]]

Results were presented by both total records and NIR vaccination records. This was because the authors noted substantial differences in recording concordance between NIR vaccination records and non-NIR vaccination records. All statistical analyses were undertaken using SAS Enterprise Guide (9.4) statistical software (SAS Institute Inc., Cary, NC, USA). This study was approved by the Health and Disability Ethics Committee: ethics reference number 19/CEN/51.

Results

One hundred and one participants were initially recruited, but three were excluded due to data quality concerns (Figure 2). After application of exclusion criteria (Figure 2), the total number of vaccination records was 1,641.

Figure 2: Study flowchart.

Characteristics of the study participants are summarised in Table 1. Of the 98 participants, most were European (75.5%), with 14% identifying as Māori, 9% as Asian and 1% as Pacific Island. Two-thirds lived in low deprivation areas (deciles 1–4), and a third lived in medium to high deprivation areas (deciles 5–10). Although nearly three-quarters of participants were from the Auckland area, there was participation from around the country. Seventeen percent resided in the South Island.

Table 1: Demographics of Health Book study participants.

Most vaccination records (85%) were present in both the Health Book and the NIR (Table 2). The remaining 15% were recorded in either the Health Book or the NIR. Most NIR vaccination records were present in both the Health Book and the NIR. However, only a minority (5%) of the non-NIS vaccinations were recorded in both the Health Book and the NIR. Almost three-quarters of the non-NIS vaccinations were recorded in the Health Book only, and the remaining quarter were recorded in the NIR only.

Table 2: Immunisation records by record source.

NIR: National Immunisation Register. [[1]] Column percentage.

The agreement between the Health Book and NIR records for both the recorded date and vaccine was high (Table 3). The date and vaccine record agreement for NIS and non-NIS vaccines was similarly high; however, the number of non-NIS vaccination records was small. Appendix Table 1 presents the non-NIS vaccine records by the data source. Influenza was the only non-NIS vaccine to be recorded by both sources; however, the majority of influenza vaccine records were contained in the NIR only. Influenza was also the only non-NIS vaccine to be recorded by the NIR only. Of the non-NIS vaccines recorded in Health Books only, varicella and rotavirus vaccines (before their introduction to the NIS) were the most common. Appendix Table 2 presents the comparison of vaccines for each data source where vaccine type did not match. The greatest discrepancy was for PCV10 and PCV13; this was most probably due to NIS changes in pneumococcal vaccine brand/valency and general practice vaccine stock. General practices could have used up previous pneumococcal vaccine stock before distribution of the new pneumococcal vaccine. Alternatively, vaccinators may have incorrectly recorded the PCV brand/valency administered.

Table 3: Immunisation date and vaccine agreement by immunisation schedule for records contained in both sources.

[[1]] Column percentage. A third had a recorded date one day earlier in the NIR and a third had a day or two later recorded in the NIR, with a range between -295 and 247 days.

The predictive accuracy of the NIR compared with the Health Books is reported in Table 4. The NIR had the greatest sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for the NIR vaccination records. For the total records, the NIR demonstrated high sensitivity (92%) and high specificity (81%). The PPV (92%) and NPV (80%) were also high. The NIR had the lowest accuracy for non-NIS vaccinations, and this may in part be due to the small number of records.

Table 4: Predictive accuracy of the National Immunisation Register compared to the Well Child Tamariki Ora parent-held health record books (Health Books).

[[1]] n=1,641. [[2]] n=1,526. [[3]] n=115.

Discussion

The results of this study indicate the NIR replicates information in Health Books with a high level of accuracy for NIS vaccines: sensitivity, specificity, positive predictive value and negative predictive value estimates were 85% and above.

In international comparisons, the NIR generally performs well. The NIR demonstrated greater sensitivity than the Ontario Health Insurance Plan database by approximately 11%.[[2]] The NIR had substantially less error than the Boston Immunization Information System where chart records were used as the comparison; date agreement for the NIR was 97% compared with 66% for the Boston Immunization Information System.[[4]]

Locally, our study indicated a higher level of accuracy than the Reynolds et al (2014) study that compared records of the completion of the five-month vaccination target in the NIR to the PMS records. However, the Reynolds et al study had a small sample restricted to only one general practice. Reynolds et al reported that the PMS recorded 9.7% greater immunisation levels compared with the NIR. In contrast, our study found very high sensitivity and positive predictive value for the NIS vaccination records in the NIR compared to the Health Books.

The reasons for discordance between the NIR and Health Book vaccination records may include: vaccinators entering data into the incorrect field in the PMS; IT errors in the translation of vaccination records from the PMS to the NIR or in the centralised recordings of the NIR; or failures to enter vaccination records into the PMS. However, the time-consuming nature of transcription from overseas vaccination records may account for some discordance. Although country of birth was not available for study participants, children born overseas are less likely to have a record in the NIR compared with New Zealand-born children.[[8]] International records must be added by providers, and frontline providers may prioritise the simpler approach of adding an entry into the Health Book over the complexity of entering international records into the NIR.

This is the first study to the authors’ knowledge that describes the extent of NIR error for non-NIS vaccination records in New Zealand. The number of non-NIS vaccination records was considerably lower than the number of NIS vaccination records. Most non-NIS vaccine records were recorded in Health Books only. Influenza was the only exception, where the NIR contained more records than the Health Books. Although the NIR is not designed or intended for recording non-NIS vaccine records, accurate records of non-NIS vaccines (especially those with known associations and significant adverse events following immunisation, such as for rotavirus) are essential for safety monitoring activities. In addition, silent inequities are possible where inaccurate records in non-NIS vaccinations exist. For example, those most at risk may not be receiving non-NIS vaccinations. Misclassification of vaccination status for non-NIR vaccines is a concern. Estimates of vaccine effectiveness are an important component of the negotiation to move a non-NIR vaccine onto the NIR. NIR data are utilised for studies of vaccine effectiveness, and underestimation of vaccination status may result in reduced vaccine effectiveness estimates.[[9]] Also, misclassification could result in missed or slower recognition of adverse events following vaccination, particularly if these events are rare.

The high sensitivity and specificity reported for NIS vaccinations is encouraging. Our results indicate that national milestone reporting, vaccine effectiveness estimates and safety research using historical NIR data is probably relatively accurate. The lack of substantial ethnic and socioeconomic diversity in the study sample prevents any comment on the accuracy of milestone reporting for equity. Although our non-NIS vaccine sample size was small, our results indicate a propensity for non-NIS vaccine exposure misclassification using historical NIR data. We recommend caution be exercised for non-NIS vaccine research using historical NIR data for exposure misclassification. We recommend researchers explore the implications of exposure misclassification on study outcomes. A larger study is needed to determine the extent of non-NIS vaccine misclassification in the NIR.

Our study sample was small and obtained through convenience sampling. One implication of this is that there were several family clusters. Sibling records are subject to similar demographic characteristics and behaviours of parents and general practice providers. Thus, these have the potential to bias our parameter estimates. Our sample was not representative of the New Zealand paediatric population over the study period. New Zealand Europeans, children living in low-socioeconomic deprivation areas and the Northern District Health Board region were over-represented in our sample. Demographic characteristics may have influenced the completeness of records in both the NIR and the Health Books. There are well-established disparities in the quality of healthcare received between Māori and non-Māori and low- and high-socioeconomic deprivation populations in New Zealand; this may have implications for the quality of vaccination records and the PPV and NPV as they are sensitive to the characteristics of the population in which they are measured.[[10,11]] Investigation of demographic inequalities in NIR vaccination records is an important concern to be addressed in future research. We acknowledge Health Book records are probably not without error and that this could bias the results, but it is unclear in which direction. Completion of each Health Book is influenced by the nature of the healthcare appointment (opportunistic vaccination or scheduled immunisation appointment or immunisation outreach service), whether the Health Book is lost and whether the parent/caregiver remembers to bring it to the appointment or at a later time for updating. However, less than 7% of records were recorded in the NIR only, indicating that this source of error was unlikely to have significantly affected parameter estimates. Some aspects of the process of recording vaccination events are common to Health Book and PMS records and, therefore, are subject to some of the same omission or misclassification errors. It is not possible to determine whether or to what extent this occurred in our study. Population groups with high mobility and/or poor healthcare access are more likely to have incomplete Health Book vaccination records. However, we expect the completeness of Health Book records in our study to have been high, as inclusion criteria stipulated that guardians needed to believe their children’s Health Books were an accurate record of the vaccinations received during childhood. Although it could be argued that sensitivity and specificity were used as a proxy for agreement, these measures represent an initial inquiry and, together with the relatively small sample size, are associated with limitations such as overestimation of agreement.

Conclusion

This study compared two methods of registering vaccination. Neither method is perfect, but each likely has different types of errors. The results of this work indicate the NIR data accuracy generally performs well compared with Well Child Tamariki Ora parent-held health record books (Health Books), especially for NIS vaccine records. To the authors’ knowledge, this is the first study in New Zealand to have also looked at non-NIS vaccination records. Description and quantification of the error in the NIR can be used to improve the accuracy of immunisation research in New Zealand. We recommend further validation research be undertaken, in particular for non-NIS vaccines and for Māori, Pacific and low-socioeconomic deprivation groups. If a new NIR is established, validation of this system will be necessary also.

Appendix

Appendix Table 1: Vaccines of non-scheduled immunisations by record source.

NIR: National Immunisation Register. [[1]] Column percentage.

Appendix Table 2: Comparison of vaccines for each source where vaccine type did not match.

NIR: National Immunisation Register.

Summary

Abstract

Aim

The National Immunisation Register (NIR), which is derived from general practice management systems, is an important tool for the provision of clinical services, national immunisation programme evaluation and immunisation research in New Zealand. However, the accuracy of the NIR data has not yet been quantified. This study aimed to examine, describe and quantify the extent of discrepancy in the NIR compared to Well Child Tamariki Ora parent-held health record books (Health Books).

Method

Immunisation data for vaccinations given between birth and four years old for children born between 2006 and 2019 were compared between the Health Books and the NIR. Health Book records were used as the reference standard to calculate performance measures: sensitivity, specificity, positive and negative predictive values for the NIR.

Results

Overall, NIR performance was high: sensitivity ranged from 90% to 93%, specificity from 78% to 85%, the positive predictive value from 91% to 94% and the negative predictive value from 77% to 84%. NIR performance was higher for National Immunisation Schedule (NIS) vaccines compared with non-NIS vaccines.

Conclusion

This study indicates the NIR data accuracy generally performs well compared with international equivalents, especially for NIS vaccine records. Further work is required to ascertain why discrepancies between the Health Books and NIR continue to occur, with particular attention to important subgroups and translating records across from migrant populations. Also, future work is required to understand the accuracy of vaccination records for groups who experience lower-quality healthcare and a higher burden of infectious diseases.

Author Information

Anna S Howe: Lecturer, School of Health Sciences, University of Canterbury; Research Fellow, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland. Hannah Chisholm: PhD Candidate, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland. Janine Paynter: Senior Research Fellow, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland. Esther Willing: Lecturer, Kōhatu – Centre for Hauora Māori, Division of Health Sciences, University of Otago. Nikki Turner: Director, Immunisation Advisory Centre, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland.

Acknowledgements

We would like to thank Karin Batty for sharing her contextual and technical expertise with us and for reviewing this manuscript. Karin, you are a tremendous resource.

Correspondence

Anna Howe, School of Health Sciences, University of Canterbury, Private Bag 4800, Christchurch 8140

Correspondence Email

a.howe@ auckland.ac.nz

Competing Interests

All authors have been involved in research utilising NIR data for vaccine effectiveness, safety and coverage studies. Additionally, Nikki Turner is a GP.

1) Ministry of Health [Internet]. National Immunisation Register: Ministry of Health; 2015. Available from: https://www.health.govt.nz/our-work/preventative-health-wellness/immunisation/national-immunisation-register

2) Schwartz KL, Tu K, Wing L, et al. Validation of infant immunization billing codes in administrative data. Hum Vaccin Immunother. 2015;11(7):1840-7.

3) MacDonald SE, Schopflocher DP, Golonka RP. The pot calling the kettle black: the extent and type of errors in a computerized immunization registry and by parent report. BMC Pediatr. 2014;14:1-.

4) Samuels R, Appel L, Reddy SI, Tilson RS. Improving accuracy in a computerized immunization registry. Ambul Pediatr. 2002;2 3:187-92.

5) Reynolds G, Timo M, Dev A, et al. Effective general practice: audit and feedback for the primary series of immunisations. J Prim Health Care. 2014;6(1):40-8.

6) Poskitt N, Taft S. National Immunisation Register inaccuracies and duplications. N Z Med J (Online). 2009;122(1288).

7) Hripcsak G, Heitjan DF. Measuring agreement in medical informatics reliability studies. Journal of Biomedical Informatics 2002; 35(2): 99-110.

8) Charania NA, Paynter J, Lee AC, et al. Vaccine-Preventable Disease-Associated Hospitalisations Among Migrant and Non-migrant Children in New Zealand. J Immigr Minor Health. 2020;22(2):223-31.

9) Jackson ML. Use of self-reported vaccination status can bias vaccine effectiveness estimates from test-negative studies. Vaccine: X. 2019;1:100003.

10) Mulholland M. State of the Māori nation : twenty-first century issues in Aotearoa. Auckland, New Zealand: Reed; 2006.

11) Šimundić A-M. Measures of Diagnostic Accuracy: Basic Definitions. EJIFCC. 2009;19(4):203.

For the PDF of this article,
contact nzmj@nzma.org.nz

View Article PDF

Vaccinations have produced some of the largest gains in the history of public health interventions. Timely, complete and safe immunisation requires reliable and complete vaccination records at the individual level. Accurate and meaningful evaluation of the coverage, effectiveness and safety of the National Immunisation Schedule (NIS) vaccines also depends on reliable and complete vaccination records at the population level.[[1]] In Aotearoa New Zealand, there are two sources of immunisation records: each child’s Well Child Tamariki Ora parent-held health record book (Health Book) and the National Immunisation Register (NIR), which is automatically fed data every night from the practice management system (PMS) of immunisation providers. Health Books were the only source for immunisation records until the rollout of the NIR. A Health Book is given to parents upon the birth of a child and contains child health and development information.

A section on vaccination in each Health Book allows vaccinators to record details of vaccinations given as per the routine New Zealand childhood NIS, as well as any other preschool vaccinations given as required or privately purchased. When a child is immunised overseas, their Health Book is retrospectively filled out in general practice upon sighting proof of vaccination documentation. Written records in Health Books are seen as the reference standard to which NIR records can be compared because Health Book records are written directly by the vaccinator at the time of vaccination and the Health Book Immunisation Certificate is necessary documentation for enrolment at early childcare centres and school.

The NIR is a computerised information system developed to hold vaccination details of children in New Zealand from six weeks to 12 years (and some adult vaccines) (Figure 1) and has been comprehensively used to record vaccination details for all children in New Zealand born since 2006.[[1]] The NIR is a valuable resource, but there are probably differences between the NIR and the records in each child’s Health Book, and the degree of error is currently unknown. Validation research internationally and locally has found varying levels of misclassification in electronic registries.[[2–6]] Recording errors have the potential to result in bias results: for example, a small study in New Zealand indicated that the NIR may overestimate the number of children meeting national milestone targets.[[5]] Our study proposed to examine, describe and quantify the extent of the error in the NIR using Health Books as a reference standard.

Figure 1: New Zealand National Immunisation Schedule (NIS) for childhood vaccines between 2006 and 2019.

DTaP-IPV: diphtheria, tetanus, acellular pertussis, inactivated poliovirus vaccines. Tdap-IPV: diphtheria, tetanus, acellular pertussis, inactivated poliovirus vaccines. DTaP- IPV-HepB/Hib: diphtheria, tetanus, acellular pertussis, inactivated poliovirus, hepatitis B and Haemophilus influenzae type b vaccines. Tdap: diphtheria, tetanus, acellular pertussis vaccines. Hib-Hep B: Haemophilus influenzae type b and hepatitis B vaccine. Hib-PRP: Haemophilus influenzae type b polyribosylribitol phosphate vaccine. MeNZB: A strain-specific g roup B meningococcal vaccine. MMR: measles, mumps, rubella vaccine. PCV7: 7-valent pneumococcal conjugate vaccine. PCV10: 10-valent pneumococcal conjugate vaccine. PCV13: 13-valent pneumococcal conjugate vaccine.RV5: pentavalent rotavirus vaccine. RV1: monovalent rotavirus vaccine. HPV4: quadrivalent human papillomavirus vaccine. HPV9: 9 valent human papillomavirus vaccine. VV: varicella vaccine (chickenpox; varicella-zoster virus).

Methods

Study population

The inclusion criteria were children born between 2006 and 2019 with vaccination data recorded in their Health Book for vaccinations given from birth until 4 years of age, and where the parent or legal guardian believes the recorded vaccination data is a true record of the vaccinations the child has received. Parents and caregivers of child participants were recruited as a convenience sample via the University of Auckland intranet, posters at appropriate venues (eg, Dunedin kindergartens and Allied Health Plus primary health organisation general practice clinics in Auckland), social media (eg, targeted Facebook advertisements), use of Well Child Tamariki Ora and immunisation provider networks (eg, the Health Book provider newsletter and vaccinator education mailing lists, posters at the New Zealand Immunisation Conference) and word of mouth.

At first contact, parents and caregivers were provided with either a paper or electronic participant information sheet and consent form. Participation required parents and caregivers to return a completed, signed consent form and submit a picture of the vaccination page of their child’s Health Book. Children aged 8 to 14 years had to sign an assent form if they agreed to participate. These documents were submitted either electronically by email (as a scanned or photographed image) or in person as an original hardcopy. In acknowledgement of their time, all guardians who returned participant data and completed recruitment went into the draw to win one of five $100 gift cards to be spent at selected grocery stores, pharmacies and stationery shops. NIR records were identified for each child using their unique National Health Index (NHI) identifier. Subsequently, each child participant was assigned a unique study ID to preserve anonymity, and this was used across all data sources.

Well Child Tamariki Ora parent-held health record books

Data elicited for this study was sourced from the “Immunisation record” section of each Health Book. Relevant fields included vaccine, batch, site, date given, sign/stamp and notes.

National Immunisation Register

Individual-level vaccination data was extracted from the NIR using each child’s NHI identifier. Relevant fields included the NHI, vaccination date, vaccine, vaccine dose, antigen and batch number. Live NHIs were provided to the Ministry of Health Analytical Services team for the data extraction. Demographic information on each child was also requested. Relevant fields included children’s sex, date of birth, ethnicity, New Zealand Index of Deprivation 2013 decile and district health board of residence.

Although the NIR has a field allowing for the indication of high-risk schedules for at least Prevenar and influenza, validation of this variable could not be established as this information is not indicated in Health Books. (High risk schedules are schedule modifications for high-risk groups: for example, additional doses of an already scheduled vaccine or another vaccine, like the 23-valent pneumococcal vaccine for selected high-risk groups such as those with primary immune deficiency or human immunodeficiency virus.) Therefore, this study defines NIS vaccines as the routine New Zealand childhood immunisation schedule vaccines. Any vaccines outside of this definition were considered non-NIS vaccines. Some non-NIR vaccines, such as for rotavirus, have become NIS vaccines over the study period. These changes have been accounted for by comparing NIS with the date of vaccine administration.

Statistical analysis

Vaccination data from the Health Books were entered into an Excel spreadsheet with only the study ID to identify each participant. Records from Health Books were entered by two researchers (AH and HC) and checked by a third (JP). NIR and NHI data were merged with this reference standard data by unique study IDs. Vaccination records were excluded if they occurred after 60 months of age (this was to allow for flexibility in later delivery of the four-year milestone age vaccines). Records were also excluded if they were administration artefacts of the NIR, rather than a vaccine record (eg, records of vaccine declines).

Participant characteristics were described as number and percentage for the total population. Discordance between the Health Book records and the NIR records were investigated and described as percentage of misclassified vaccination events and types of errors found: for example, incorrect vaccine type or date, or missing events. Observed agreement, sensitivity, specificity and positive and negative predictive values were used as an index of agreement between the NIR and the Health Books. These measures perform best as a first step to quantify agreement between measures and have been used in previous literature on this topic.[[7]]

Results were presented by both total records and NIR vaccination records. This was because the authors noted substantial differences in recording concordance between NIR vaccination records and non-NIR vaccination records. All statistical analyses were undertaken using SAS Enterprise Guide (9.4) statistical software (SAS Institute Inc., Cary, NC, USA). This study was approved by the Health and Disability Ethics Committee: ethics reference number 19/CEN/51.

Results

One hundred and one participants were initially recruited, but three were excluded due to data quality concerns (Figure 2). After application of exclusion criteria (Figure 2), the total number of vaccination records was 1,641.

Figure 2: Study flowchart.

Characteristics of the study participants are summarised in Table 1. Of the 98 participants, most were European (75.5%), with 14% identifying as Māori, 9% as Asian and 1% as Pacific Island. Two-thirds lived in low deprivation areas (deciles 1–4), and a third lived in medium to high deprivation areas (deciles 5–10). Although nearly three-quarters of participants were from the Auckland area, there was participation from around the country. Seventeen percent resided in the South Island.

Table 1: Demographics of Health Book study participants.

Most vaccination records (85%) were present in both the Health Book and the NIR (Table 2). The remaining 15% were recorded in either the Health Book or the NIR. Most NIR vaccination records were present in both the Health Book and the NIR. However, only a minority (5%) of the non-NIS vaccinations were recorded in both the Health Book and the NIR. Almost three-quarters of the non-NIS vaccinations were recorded in the Health Book only, and the remaining quarter were recorded in the NIR only.

Table 2: Immunisation records by record source.

NIR: National Immunisation Register. [[1]] Column percentage.

The agreement between the Health Book and NIR records for both the recorded date and vaccine was high (Table 3). The date and vaccine record agreement for NIS and non-NIS vaccines was similarly high; however, the number of non-NIS vaccination records was small. Appendix Table 1 presents the non-NIS vaccine records by the data source. Influenza was the only non-NIS vaccine to be recorded by both sources; however, the majority of influenza vaccine records were contained in the NIR only. Influenza was also the only non-NIS vaccine to be recorded by the NIR only. Of the non-NIS vaccines recorded in Health Books only, varicella and rotavirus vaccines (before their introduction to the NIS) were the most common. Appendix Table 2 presents the comparison of vaccines for each data source where vaccine type did not match. The greatest discrepancy was for PCV10 and PCV13; this was most probably due to NIS changes in pneumococcal vaccine brand/valency and general practice vaccine stock. General practices could have used up previous pneumococcal vaccine stock before distribution of the new pneumococcal vaccine. Alternatively, vaccinators may have incorrectly recorded the PCV brand/valency administered.

Table 3: Immunisation date and vaccine agreement by immunisation schedule for records contained in both sources.

[[1]] Column percentage. A third had a recorded date one day earlier in the NIR and a third had a day or two later recorded in the NIR, with a range between -295 and 247 days.

The predictive accuracy of the NIR compared with the Health Books is reported in Table 4. The NIR had the greatest sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for the NIR vaccination records. For the total records, the NIR demonstrated high sensitivity (92%) and high specificity (81%). The PPV (92%) and NPV (80%) were also high. The NIR had the lowest accuracy for non-NIS vaccinations, and this may in part be due to the small number of records.

Table 4: Predictive accuracy of the National Immunisation Register compared to the Well Child Tamariki Ora parent-held health record books (Health Books).

[[1]] n=1,641. [[2]] n=1,526. [[3]] n=115.

Discussion

The results of this study indicate the NIR replicates information in Health Books with a high level of accuracy for NIS vaccines: sensitivity, specificity, positive predictive value and negative predictive value estimates were 85% and above.

In international comparisons, the NIR generally performs well. The NIR demonstrated greater sensitivity than the Ontario Health Insurance Plan database by approximately 11%.[[2]] The NIR had substantially less error than the Boston Immunization Information System where chart records were used as the comparison; date agreement for the NIR was 97% compared with 66% for the Boston Immunization Information System.[[4]]

Locally, our study indicated a higher level of accuracy than the Reynolds et al (2014) study that compared records of the completion of the five-month vaccination target in the NIR to the PMS records. However, the Reynolds et al study had a small sample restricted to only one general practice. Reynolds et al reported that the PMS recorded 9.7% greater immunisation levels compared with the NIR. In contrast, our study found very high sensitivity and positive predictive value for the NIS vaccination records in the NIR compared to the Health Books.

The reasons for discordance between the NIR and Health Book vaccination records may include: vaccinators entering data into the incorrect field in the PMS; IT errors in the translation of vaccination records from the PMS to the NIR or in the centralised recordings of the NIR; or failures to enter vaccination records into the PMS. However, the time-consuming nature of transcription from overseas vaccination records may account for some discordance. Although country of birth was not available for study participants, children born overseas are less likely to have a record in the NIR compared with New Zealand-born children.[[8]] International records must be added by providers, and frontline providers may prioritise the simpler approach of adding an entry into the Health Book over the complexity of entering international records into the NIR.

This is the first study to the authors’ knowledge that describes the extent of NIR error for non-NIS vaccination records in New Zealand. The number of non-NIS vaccination records was considerably lower than the number of NIS vaccination records. Most non-NIS vaccine records were recorded in Health Books only. Influenza was the only exception, where the NIR contained more records than the Health Books. Although the NIR is not designed or intended for recording non-NIS vaccine records, accurate records of non-NIS vaccines (especially those with known associations and significant adverse events following immunisation, such as for rotavirus) are essential for safety monitoring activities. In addition, silent inequities are possible where inaccurate records in non-NIS vaccinations exist. For example, those most at risk may not be receiving non-NIS vaccinations. Misclassification of vaccination status for non-NIR vaccines is a concern. Estimates of vaccine effectiveness are an important component of the negotiation to move a non-NIR vaccine onto the NIR. NIR data are utilised for studies of vaccine effectiveness, and underestimation of vaccination status may result in reduced vaccine effectiveness estimates.[[9]] Also, misclassification could result in missed or slower recognition of adverse events following vaccination, particularly if these events are rare.

The high sensitivity and specificity reported for NIS vaccinations is encouraging. Our results indicate that national milestone reporting, vaccine effectiveness estimates and safety research using historical NIR data is probably relatively accurate. The lack of substantial ethnic and socioeconomic diversity in the study sample prevents any comment on the accuracy of milestone reporting for equity. Although our non-NIS vaccine sample size was small, our results indicate a propensity for non-NIS vaccine exposure misclassification using historical NIR data. We recommend caution be exercised for non-NIS vaccine research using historical NIR data for exposure misclassification. We recommend researchers explore the implications of exposure misclassification on study outcomes. A larger study is needed to determine the extent of non-NIS vaccine misclassification in the NIR.

Our study sample was small and obtained through convenience sampling. One implication of this is that there were several family clusters. Sibling records are subject to similar demographic characteristics and behaviours of parents and general practice providers. Thus, these have the potential to bias our parameter estimates. Our sample was not representative of the New Zealand paediatric population over the study period. New Zealand Europeans, children living in low-socioeconomic deprivation areas and the Northern District Health Board region were over-represented in our sample. Demographic characteristics may have influenced the completeness of records in both the NIR and the Health Books. There are well-established disparities in the quality of healthcare received between Māori and non-Māori and low- and high-socioeconomic deprivation populations in New Zealand; this may have implications for the quality of vaccination records and the PPV and NPV as they are sensitive to the characteristics of the population in which they are measured.[[10,11]] Investigation of demographic inequalities in NIR vaccination records is an important concern to be addressed in future research. We acknowledge Health Book records are probably not without error and that this could bias the results, but it is unclear in which direction. Completion of each Health Book is influenced by the nature of the healthcare appointment (opportunistic vaccination or scheduled immunisation appointment or immunisation outreach service), whether the Health Book is lost and whether the parent/caregiver remembers to bring it to the appointment or at a later time for updating. However, less than 7% of records were recorded in the NIR only, indicating that this source of error was unlikely to have significantly affected parameter estimates. Some aspects of the process of recording vaccination events are common to Health Book and PMS records and, therefore, are subject to some of the same omission or misclassification errors. It is not possible to determine whether or to what extent this occurred in our study. Population groups with high mobility and/or poor healthcare access are more likely to have incomplete Health Book vaccination records. However, we expect the completeness of Health Book records in our study to have been high, as inclusion criteria stipulated that guardians needed to believe their children’s Health Books were an accurate record of the vaccinations received during childhood. Although it could be argued that sensitivity and specificity were used as a proxy for agreement, these measures represent an initial inquiry and, together with the relatively small sample size, are associated with limitations such as overestimation of agreement.

Conclusion

This study compared two methods of registering vaccination. Neither method is perfect, but each likely has different types of errors. The results of this work indicate the NIR data accuracy generally performs well compared with Well Child Tamariki Ora parent-held health record books (Health Books), especially for NIS vaccine records. To the authors’ knowledge, this is the first study in New Zealand to have also looked at non-NIS vaccination records. Description and quantification of the error in the NIR can be used to improve the accuracy of immunisation research in New Zealand. We recommend further validation research be undertaken, in particular for non-NIS vaccines and for Māori, Pacific and low-socioeconomic deprivation groups. If a new NIR is established, validation of this system will be necessary also.

Appendix

Appendix Table 1: Vaccines of non-scheduled immunisations by record source.

NIR: National Immunisation Register. [[1]] Column percentage.

Appendix Table 2: Comparison of vaccines for each source where vaccine type did not match.

NIR: National Immunisation Register.

Summary

Abstract

Aim

The National Immunisation Register (NIR), which is derived from general practice management systems, is an important tool for the provision of clinical services, national immunisation programme evaluation and immunisation research in New Zealand. However, the accuracy of the NIR data has not yet been quantified. This study aimed to examine, describe and quantify the extent of discrepancy in the NIR compared to Well Child Tamariki Ora parent-held health record books (Health Books).

Method

Immunisation data for vaccinations given between birth and four years old for children born between 2006 and 2019 were compared between the Health Books and the NIR. Health Book records were used as the reference standard to calculate performance measures: sensitivity, specificity, positive and negative predictive values for the NIR.

Results

Overall, NIR performance was high: sensitivity ranged from 90% to 93%, specificity from 78% to 85%, the positive predictive value from 91% to 94% and the negative predictive value from 77% to 84%. NIR performance was higher for National Immunisation Schedule (NIS) vaccines compared with non-NIS vaccines.

Conclusion

This study indicates the NIR data accuracy generally performs well compared with international equivalents, especially for NIS vaccine records. Further work is required to ascertain why discrepancies between the Health Books and NIR continue to occur, with particular attention to important subgroups and translating records across from migrant populations. Also, future work is required to understand the accuracy of vaccination records for groups who experience lower-quality healthcare and a higher burden of infectious diseases.

Author Information

Anna S Howe: Lecturer, School of Health Sciences, University of Canterbury; Research Fellow, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland. Hannah Chisholm: PhD Candidate, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland. Janine Paynter: Senior Research Fellow, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland. Esther Willing: Lecturer, Kōhatu – Centre for Hauora Māori, Division of Health Sciences, University of Otago. Nikki Turner: Director, Immunisation Advisory Centre, Department of General Practice and Primary Health Care, School of Population Health, University of Auckland.

Acknowledgements

We would like to thank Karin Batty for sharing her contextual and technical expertise with us and for reviewing this manuscript. Karin, you are a tremendous resource.

Correspondence

Anna Howe, School of Health Sciences, University of Canterbury, Private Bag 4800, Christchurch 8140

Correspondence Email

a.howe@ auckland.ac.nz

Competing Interests

All authors have been involved in research utilising NIR data for vaccine effectiveness, safety and coverage studies. Additionally, Nikki Turner is a GP.

1) Ministry of Health [Internet]. National Immunisation Register: Ministry of Health; 2015. Available from: https://www.health.govt.nz/our-work/preventative-health-wellness/immunisation/national-immunisation-register

2) Schwartz KL, Tu K, Wing L, et al. Validation of infant immunization billing codes in administrative data. Hum Vaccin Immunother. 2015;11(7):1840-7.

3) MacDonald SE, Schopflocher DP, Golonka RP. The pot calling the kettle black: the extent and type of errors in a computerized immunization registry and by parent report. BMC Pediatr. 2014;14:1-.

4) Samuels R, Appel L, Reddy SI, Tilson RS. Improving accuracy in a computerized immunization registry. Ambul Pediatr. 2002;2 3:187-92.

5) Reynolds G, Timo M, Dev A, et al. Effective general practice: audit and feedback for the primary series of immunisations. J Prim Health Care. 2014;6(1):40-8.

6) Poskitt N, Taft S. National Immunisation Register inaccuracies and duplications. N Z Med J (Online). 2009;122(1288).

7) Hripcsak G, Heitjan DF. Measuring agreement in medical informatics reliability studies. Journal of Biomedical Informatics 2002; 35(2): 99-110.

8) Charania NA, Paynter J, Lee AC, et al. Vaccine-Preventable Disease-Associated Hospitalisations Among Migrant and Non-migrant Children in New Zealand. J Immigr Minor Health. 2020;22(2):223-31.

9) Jackson ML. Use of self-reported vaccination status can bias vaccine effectiveness estimates from test-negative studies. Vaccine: X. 2019;1:100003.

10) Mulholland M. State of the Māori nation : twenty-first century issues in Aotearoa. Auckland, New Zealand: Reed; 2006.

11) Šimundić A-M. Measures of Diagnostic Accuracy: Basic Definitions. EJIFCC. 2009;19(4):203.

Contact diana@nzma.org.nz
for the PDF of this article

Subscriber Content

The full contents of this pages only available to subscribers.

LOGINSUBSCRIBE