Medical schools are under increasing pressure to select medical students who will cope with the rigorous curriculum, have traits desirable in a good doctor and represent the society they serve.1-4 Whilst doing so, there is an expectation of transparency and acceptability of the selection process, given the large numbers of worthy candidates, yet limited places.Amongst others, personality and ability to reason have long been seen as important traits for doctors.5 Various tools have been used to attempt to measure these traits including structured interviews, references, personal statements and personality tests. 3, 4, 6-8 Although these have face validity, with the exception of the Multiple Mini Interview, none is particularly reliable.The Undergraduate Medicine and Health Sciences Admission Test (UMAT) is used widely across Australasia as a tool to select entrants to undergraduate medical programme. In New Zealand, selection for medical school entry occurs after one year at university or prior degree.UMAT was introduced as part of the selection process in NZ in 2003.9 All domestic applicants must sit UMAT, however it is given less weighting in making ranking decisions than prior academic achievement determined by an academic score. Developed by the Australian Council for Educational Research, the UMAT comprises three parts: Section 1 (logical reasoning and problem solving), Section 2 (understanding people) and Section 3 (non-verbal reasoning).The UMAT website states that it's an aptitude test "designed to assess general attributes and abilities gained through prior experience and learning; ... these abilities are considered important to the study, and later practice, of professions in the health sciences."10 The test can be taken only once a year (July), in dedicated test centres.Despite its extensive use, research to date is sparse. A 2011 study suggested that it held limited predictive validity for subsequent academic performance.11 Anecdotally, UMAT is associated with high stress, pressure to attend coaching and purchase preparatory material, and take the test more than once in the hope that scores increase. There is one report from Australia that included student perceptions on the benefit of coaching for interview and UMAT, but none on what medical students think of UMAT, how students prepare, or the cost of this preparation.12This study aimed to explore NZ medical student perceptions of UMAT and its effect upon them. In particular we were interested in students' perceptions of UMAT, their preparation, and its associated costs and impact.Methods Study design—We developed a 35-question survey to investigate medical students' perception of UMAT. It was reviewed by university academic staff and the New Zealand Medical Students' Association (NZMSA) executive. The study was approved by the Human Participants' Ethics Committee at the University of Auckland and the University of Otago Human Ethics Committee. In November 2009, all medical students who were enrolled at the University of Auckland or the University of Otago were invited to complete the questionnaire. Students were informed of the online questionnaire via student email lists, or had the opportunity to fill out a paper version at the end of class congregations. They were asked to complete only one questionnaire. The online version was hosted on the NZMSA website. Both versions were completed anonymously and took about 10 minutes. Response modes included yes/no options or choice of descriptor from a 4 or 5 point Likert scale. The only discrepancy between versions was in selecting ethnicity: students could only select one ethnic category online, whereas students completing the paper version could select more than one group. If more than one ethnicity was listed, this was prioritised in the order Māori, Pacific Island, Asian, then European/Other. We performed a posthoc analysis to compare the paper responses with the online responses in order to assess whether the method of response may have introduced bias. Perception—Importance of UMAT in selection was tested using binary logistic regression. The responses were grouped into two categories (Not at all/Not very or fairly/very/extremely). All demographic data were incorporated as explanatory variables. Additional variables were whether UMAT reflected cultural values and whether UMAT tested for non-cognitive attributes required in doctors. Fairness of UMAT was tested using binary logistic regression. It used the same explanatory variables as above including the additional variables. The stressfulness of sitting UMAT was tested using an ordinal logistic regression. The five Likert categories were reduced to four with 'Not at all' and 'Not really' grouped together. It used the same explanatory variables as above. Responses to the question about the value of each section in selecting medical student were combined into one chi square analysis. Preparation—Costs of preparation were tested using an ordinal logistic regression. For cost, the outcome was grouped into 0, $50-$200, $250-$650, $750+. It used the same explanatory variables as above, with the addition of the student rating of importance, fairness and stress. The effect of re-sitting was tested using a chi square test to determine if the results in the best section increased on taking the UMAT test more than once. Results A total of 1325 of 2043 students responded (65%) with a mean age of 22. Of these, 398 (30%) responded online. There were 569 responders out of 847 students at The University of Auckland (67%) and 756 out of 1196 students from University of Otago (63%). Demographic data are presented in Table 1. Table 1: Demographic distribution of respondents Category Number Percentage Total population 2043 Response population 1325 65 University Auckland Otago 569 756 43 57 Gender Female Male 742 583 56 44 First language English Other 1020 305 77 23 Ethnicity Māori Pacific Asian European/Other 76 47 418 784 6 4 32 59 Entry type School or First Year University University graduate Māori and Pacific Scheme International student Rural entry scheme Other 835 265 80 80 93 146 63 20 6 6 7 11 Year level MBChB 2 MBChB 3 MBChB 4 MBChB 5 Trainee Intern BMedSci/BHB Honours 392 298 179 230 220 6 30 22 13 17 17 <1 Method of response Online Paper 398 927 30 70 Of the 1325 respondents, 1221 (92%) had sat UMAT. The following results are based on this population of 1221 respondents. Perception—4% believed that UMAT was extremely important for selection, 8% said it was very important, 31% said it was fairly important, 56% said it was not or not really important. Otago students were more likely to think it important (P<0.03). When asked "Were there any aspects of your UMAT test(s) that you feel reflected cultural values?" 28% answered "yes". Of these, 76% replied "yes" to the question "Were they culturally appropriate for a New Zealand context." Thirty percent said that what they perceived to be the likely correct answer was against their cultural belief. In response to "Do you think the UMAT test is a fair test?" 67% of students replied "no." Respondents noted high stress associated with the process of UMAT, with 40% and 41% finding it to be stressful or very stressful respectively. There was evidence of an influence of medical school with University of Otago students reporting higher levels of stress (P<0.01). When asked how well it assessed non-cognitive attributes required as doctors, 3% said very much, 26% somewhat, 17% neutral, 31% not really, and 23% said not at all. Those more supportive that it assessed non-cognitive attributes were less likely to consider it unfair (P<0.001). There were differences (P<0.001) in how students perceived the value of each UMAT section in selecting students for medical school. Responses are presented in Figure 1. Figure 1. How valuable each section is in selecting students for medical school (percentage of respondents) Preparation—21% of students reported sitting UMAT more than once; of these, 70% reported gaining higher marks in a subsequent test. Students were asked to indicate in which section they performed the best. Thirty-six percent of students said that Section 2: ‘Understanding people' was their best; 33% Section 1: ‘Logical Reasoning and Problem Solving'; and 27%, Section 3: ‘Non-verbal Reasoning'. There was strong evidence that the best section depended on whether or not they re-sat (P<0.001) with Section 3 more likely to be the better section for those who did so. Thirty-one percent sat a UMAT course prior to their examination, and 7% took more than one course. Of those who sat a course, 52% spent between $0–$400, 46% between $400– $800, and 2% greater than $800. Thirty-two percent read books to prepare them for UMAT. Books did not incur cost for 37% of students. There was a cost of between $0–$100 for 13% of students, between $100–$200 for 45% of students, and more than $200 for 5% of students. Those who sat UMAT preparatory courses were asked if they believed it helped with each section. Forty-nine percent believed it did so for Section 1, 34% for Section 2, and 3% for Section 3. With respect to money spent on UMAT courses, there was evidence of an influence of ethnicity (P<0.001), age (P<0.007), and perceived stress (P<0.001). Asian students, and those who were younger, or feeling more stressed spent more. Method of response—There was no difference between student responses with paper versus online questionnaires in the reporting of costs (p=0.0586), stress (p=0.5377) or fairness of UMAT (p=0.5351). The only response that showed a significant effect (p=0.002) was importance, with those responding online considering UMAT was less important in selecting medical students than those responding via paper. Discussion This study is among the first to assess students' perceptions of the general cognitive test, UMAT. It indicates that the majority of New Zealand medical students who responded to this survey are unconvinced that UMAT is an important test for selecting medical students. This seemed to be for two reasons: first that they were ambivalent that it judged the non-cognitive attributes required of a doctor. Second, it may not be a fair test. Further, we found there was a difference in how students perceived the three UMAT sections. Section 3 was felt to be the least valuable in selecting medical students and often stated as the worst section, yet also was the one most felt they improved on if they re-sat the test. This is consistent with a 2008 study that showed a mild increase in marks for Section 3 after coaching and re-sitting.12 By contrast, students in the present study did not think that preparatory courses helped them with Section 3. The explanation for this discrepancy is not clear, but does raise the possibility that questions in Section 3 are relatively unfamiliar, with the best way to address this being self-directed learning, after they have been seen once in the test setting. The responses may reflect general uncertainty as to what UMAT measures, as there have been so few predictive validity studies to date. Additionally, it could be argued that students within a programme may not be in a position to judge how well UMAT measures traits important for medical practice. Against this is that a third of the respondents were from later years of the programme and will be learning and being assessed on attributes important in medical practice. Our data suggest the weighting of UMAT in the selection process is an important influencing factor in students' perception of stress and how important it is. Over 80% of students feel that UMAT was stressful or very stressful. Of those who felt it was stressful, it is more likely that these students were from the University of Otago. Furthermore, those who believe it is important are also more likely to be from University of Otago. A conclusion is that this is due to UMAT having a weighting of 34% percent in University of Otago selection compared to only 15% percent at The University of Auckland. The paper highlights the cost associated with this admissions test: about half of students spent more than $400 on courses and more than $100 on books. A fifth sat UMAT more than once. UMAT may be a convenient tool to diversify the pool of applicants; 13 however the costs associated may prove to be a barrier to students already disadvantaged socioeconomically, educationally, or by distance.12 A quarter of students indicated that the UMAT test had a cultural dimension, but most were of the view that this was aligned with the cultural values in New Zealand. While a minority felt the likely correct answer was at odds with their own values, this is difficult to interpret as the correct answers are not known. These findings underscore the need to ensure selection tests are developed and validated with a view towards the setting in which they are to be used. Strengths of the study are the relatively high numbers and return rate for a student survey, and that it was a national study with similar return rates between the two universities. The distribution by gender, ethnicity, English as first language and age was generally reflective of the medical school populations in NZ.3 The students who did not sit UMAT were mainly international students, along with several domestic graduates and school leavers. There is internal consistency of the responses, in particular, that more stress was seen in the Otago students for whom UMAT was a higher stakes test. As most medical schools now use a combination of academic achievement and general cognitive tests, as in our study, the results are likely to have relevance in other settings. Weaknesses of this study are methodological. Students were able to take the survey online or in the hard-copy format. Although students were asked to sit this survey only once, we cannot guarantee this took place as surveys were completed anonymously. Most completed a paper survey at the end of a class, making it hard to respond more than once in that format. We argue that any bias from students completing more than one survey is not large: first, a minority (30%) responded on line; second, there were slight differences in the demographic details of the responders; finally, we observed a fairly similar pattern of responses between those done on paper and those on-line, with only one of 30 comparisons suggesting any interaction caused by the response method (p=0.04). The study was retrospective and reliant on students' recollection of events. It is possible that responses reflect more negatively on UMAT given the frequent debates amongst medical schools as to its validity. Results would be strengthened if we had been able to validate student perceptions with the UMAT scores achieved. However, we could not identify students and did not have access to their UMAT scores. It is not known how NZ students perceive the other selection tools used in determining who will be offered a place in medical school; hence it is difficult to judge whether UMAT has undue stress in an inherently stressful process. Nonetheless, we believe schools need to be mindful of the extra stress and cost of sitting UMAT, and seek to minimise this, especially as its validity is uncertain. The fairness of allowing results from re-sits also needs to be considered, given this may potentially disadvantage students from lower socioeconomic backgrounds. It is notable that recently, the UMAT consortium changed the policy of use of test scores so that 2012 UMAT scores can be used for admission only in the year following the test; i.e. for 2013 but not for 2014. 10 Future studies could explore qualitative angles in an attempt to better explain the perceptions outlined in this study. A study of the student view as to how best to select medical students may be in order. In conclusion, this is the first study the authors are aware of that primarily investigates students' perceptions of UMAT. At present, the majority of NZ medical students have reservations about the acceptability of UMAT as a test of non-cognitive attributes. UMAT is costly to students and also stressful; this seems to relate to the degree of weighting. These findings are a stimulus to medical schools to continue to validate UMAT, including the way it is used in selection decisions, and to promulgate this information to current and prospective students.