Several pathways exist for the registration of IMGs into the New Zealand medical workforce; the competent authority pathway (based on the applicant holding a primary qualification from a competent authority—the United Kingdom and the Republic of Ireland—and having completed an internship in a competent authority), the comparable health system pathway (based on comparable experience in a comparable health system), and the temporary special purpose pathways.The NZREX Clinical is the pathway for IMGs who do not meet the other registration pathways. Only those with an acceptable primary medical qualification listed on the Avicenna website are eligible to sit the NZREX Clinical. Prior to sitting the NZREX Clinical, IMGs must have successfully met Council's criteria for English language requirements and must have passed either the United States Medical Licensing Examination Steps 1 and 2 (Clinical Knowledge) or have passed the Australian Medical Council MCQ examination or have passed the Professional and Linguistics Assessment Board Part 1 (United Kingdom) within the last 5 years.Successful NZREX Clinical candidates enter a period of provisional registration in New Zealand, during which time they will be further assessed. To be eligible for registration in a general scope, candidates who pass NZREX Clinical must successfully complete four 3-month runs in a New Zealand public hospital and be signed off by their supervisor. All runs for interns are categorised by the Medical Council of New Zealand according to their educational role.1The compulsory runs are one category A medical run, one category A surgical run and two other runs which may be A, B or C runs. Category D runs are not acceptable. While it is possible to work in primary care during this year, stringent conditions apply to those wishing to do so.Previous research has explored the experiences of IMGs entering the New Zealand workforce through the NZREX Clinical pathway.2 The research demonstrates that these IMGs experience difficulties adjusting to the culture in New Zealand hospitals and society alongside financial and bureaucratic barriers to integration with the New Zealand workforce. The importance of robust orientation processes and adequate support from Resident Medical Officer Units cannot be overemphasised for doctors that have been successful in NZREX Clinical.Prior to September 2005, the NZREX Clinical format was based on the various medical disciplines with little coordination across disciplines regarding content or method of assessment. Some disciplines utilised the traditional 'long case and two short cases structure, others a series of short cases. Both real patients and simulated patients were used.Standard setting was by expert opinion with no psychometric data to inform the process. The examination rotated through a number of centres sequentially, giving little opportunity for examiners to gain experience in assessment. A decision was made in 2005 to redesign the process with respect to best practice in the delivery of assessment in a high stakes medical examination. This paper describes the current structure of the NZREX Clinical.Assessment designAfter a comprehensive literature review to inform both the design process and eventual structure, a working group was formed to create the blueprint from which the examination would be drawn. Those involved in the initial blueprint design were recruited for expertise in education, assessment, content knowledge relevant to the examination and cultural issues relevant to the process.The blueprint specifies the clinical domains (cardiovascular, respiratory, neurological, gastroenterology, genitourinary, musculoskeletal, pregnancy, endocrine/metabolic, haematological, oncological, behavioural and senses) as well as the competencies that are examinable in each domain (history taking, clinical examination, investigating, management, clinical reasoning, communication and professionalism).The domain of clinical reasoning was included to test the capability of a candidate to integrate different aspects of clinical information such as correctly interpreting the combination of history, physical examination and investigations. The ability of a candidate to 'counsel' a patient (such as breaking bad news) is examined under management. The examination stations are drawn exclusively from a blueprint.An 'item writing group' composed of hospital consultants, general practitioners, registrars, interns and trainee interns is used to develop stations. Currently, there are over 100 stations available for selection at any one examination.The current NZREX Clinical is an OSCE format of 16 stations. Each station lasts for 12 minutes. There are 2 minutes allocated for reading the station instructions, followed by 10 minutes in each station. Each domain is tested in at least one station. The competencies that are tested are history (three cases), clinical examination (four cases), investigating (two cases), management (three cases) and clinical reasoning (four cases). There is a specified minimum of one case on child health, one case on mental health and one case on women's health. Stations are termed 'dynamic' where there is a simulated or real patient and 'static' where the station is paper based.On average, three to four of the stations are paper based per examination. Static stations are necessary because some parts of the blueprint do not lend themselves well to the use of simulated or real patients (interpretation of investigations such as ECGs, x rays and blood tests, for example). All dynamic stations assess a candidate's competence in communication and professionalism. Professionalism is defined on the marking sheet as: \"Candidates should display a good sense of professionalism this would encompass areas such as: integrity, respect, cultural competence, ethical practice, non-discrimination and honesty\".The examination is held in a single centre, North Shore Hospital. When recruiting examiners, a strong preference is given to those with prior NZREX Clinical experience. New examiners are provided with one-on-one coaching from either the Examinations Director or the Assistant Examinations Director.Professional actors are used for the vast majority of cases to play the role of a simulated patient. Many of the actors have significant NZREX Clinical experience. Similarly, the management staff remain consistent from examination to examination. It is considered important to maintain a high degree of consistency in the personnel used to run the examination.All actors and examiners attend a training session (mock examination) 1 week prior to the examination to learn the case that they will be involved with. Final year students (trainee interns) are used to assist the examiner and actor to 'learn' the case by taking the role of the candidate. In general, between three and four separate run-throughs with different trainee interns is necessary to adequately prepare a station.The trainee interns also assist as an independent assessment of the 'pitch' of the examination by giving feedback regarding how easy or difficult the stations are in comparison to recent assessments that they have experienced in their New Zealand-based training. During the session, a calibration exercise is also undertaken using a videoed case and standard marking sheets. The objective of the exercise is to develop consensus over how marks should be awarded for various aspects of a case. After separate marking, there is discussion amongst the group concerning the allocation of marks. This process has been valuable in improving consistency in the performance of examiners.Although simulated patients are utilised for the majority of stations, 'real' patients with stable signs are sometimes utilised. While excellent for testing the detection of abnormal clinical signs, the inclusion of 'real' patients presents significant management issues as well as concerns over how both validity and reliability may be influenced.In March 2010, an expert external examiner provided an independent assessment of the NZREX Clinical process. The report was favourable and provided guidance on several areas where further improvements could be made. Recommendations included reviewing the blueprint with increased emphasis on content validity, providing retired stations to candidates, making available information on validity and reliability of the examination as well as methods of standard setting and pass rate variations, running calibration exercises for examiners and using couplet stations where the same clinical scenario was examined in different ways.The report was sent to the Ministry of Health and District Health Board Chief Executives and Chief Medical Officers. All of the recommendations have been actioned or are in the process of being actioned. It is envisaged that the external review will be repeated at 5 year intervals.Standard settingA variety of methods of standard setting have been described with defensibility as a key factor in the choice of method.3 For the NZREX Clinical, the purpose of the examination (to ensure that candidates are competent to enter a period of highly supervised provisional registration in New Zealand, during which time they will be further assessed) influences the process of standard setting.Calculation of the 'cut score' (the score above which a candidate will be deemed to be successful) for the examination utilises a combination of a 'Borderline groups' method 4-7 for the dynamic stations and a modified Angoff 8–10 method for the static stations. The cut score is adjusted for the Standard Error of Measurement (SEM) to allow for uncertainty of scores.The SEM is dependent on both the standard deviation of scores and the reliability of the examination and therefore varies between cohorts of candidates. In general, the SEM adjustment is between 3% and 5 % of the total score. Thus the cut score will vary from examination to examination to adjust for the differences in difficulty of a particular series of stations and the reliability of the examination.All stations have equal weight and there are no 'killer stations', where failure to gain a pass mark in that station results in failing the entire examination. However, a 'critical incident' policy was introduced for instances where there has been a clear breach of expected professional standards.If a candidate has a potential critical incident as judged by an examiner, the case is immediately discussed with the Examinations Director or Assistant Examinations Director and a Senior Examiner. The implications of a critical incident are described in Council's ‘Policy on Critical Incidents'.11Psychometrics of the examinationCronbach's alpha is the most commonly used measurement of reliability in OSCE examinations.12 However, the coefficient is limited in its use by the size of the sample being analysed. In general, a sample of 200 would be considered necessary for a robust analysis with smaller sample sizes resulting in lower alphas.13As the maximum sample size for NZREX Clinical is 28, the alpha coefficients generated as part of the analysis need to be considered from this perspective. The range of alpha over the last 5 years has been 0.75 to 0.85, indicating satisfactory internal consistency.14-16 A range of statistical analyses are undertaken on the results of the examination for quality control; discrimination analysis for each station, Spearman's Rho for each station against all stations combined and 'alpha with item deleted' for each station.Poorly performing stations can thus be identified and rewritten or reviewed. At the end of each examination, anonymous feedback forms are completed by both examiners and candidates.Reporting of resultsReporting of the results is to the senior officers of Council: the Chair, the CEO, the registrar and medical advisers. The report details the method by which the cut score was calculated, the candidates deemed to have passed, significant events that may have occurred and the statistical data detailed above. Relevant information from the candidate and examiner feedback is included.ResultsSince the OSCE format was introduced in 2005 until the end of 2011, approximately 520 candidates have sat the NZREX Clinical as a part of their registration process. The mean of the 'cut scores' was 62.2% and was quite consistent from cohort to cohort with a standard deviation of 1.8%. The pass rate was more variable, ranging from a high of 85% to a low of 45%. The mean pass rate is 60.2% with a standard deviation of 9.1. Comparisons of cut score and passing percent are given in Figure 1.\r\n Figure 1. Comparison of ‘cut score' and passing percentage across exams There have been approximately 320 successful candidates in NZREX Clinical since September 2005. Future developments in the examination process will focus on predictive validity by comparing successful performance in the NZREX with sequential supervisor reports in subsequent years. A comparison between the NZREX and the Australian Medical Council clinical examination for IMGs is underway with the objective of contrasting and improving methods and processes. Conclusion The NZREX Clinical is part of a wider process whereby IMGs who are eligible for the NZREX Clinical pathway can be adequately assessed. The examination is founded on robust educational principles, has comprehensive methods of quality control and utilises internationally accepted methods of standard setting. Because of the small numbers of candidates that take the examination, care must be taken in the interpretation of statistical data for both candidates and the examination process. The pathway provides a small but important source for meeting New Zealand's medical workforce requirements.