Medical Teacher

ISSN: 0142-159X (Print) 1466-187X (Online) Journal homepage: http://www.tandfonline.com/loi/imte20

Predictors of professional behaviour and academic outcomes in a UK medical school: A longitudinal cohort study Jane Adam, Miles Bore, Roy Childs, Jason Dunn, Jean Mckendree, Don Munro & David Powis To cite this article: Jane Adam, Miles Bore, Roy Childs, Jason Dunn, Jean Mckendree, Don Munro & David Powis (2015) Predictors of professional behaviour and academic outcomes in a UK medical school: A longitudinal cohort study, Medical Teacher, 37:9, 868-880, DOI: 10.3109/0142159X.2015.1009023 To link to this article: http://dx.doi.org/10.3109/0142159X.2015.1009023

Published online: 10 Feb 2015.

Submit your article to this journal

Article views: 294

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=imte20 Download by: [Rutgers University]

Date: 28 September 2015, At: 00:11

2015, 37: 868–880

Predictors of professional behaviour and academic outcomes in a UK medical school: A longitudinal cohort study JANE ADAM1, MILES BORE2, ROY CHILDS3, JASON DUNN1, JEAN McKENDREE1, DON MUNRO2 & DAVID POWIS2 1

Hull York Medical School, UK, 2University of Newcastle, Australia, 3Team Focus, UK

Downloaded by [Rutgers University] at 00:11 28 September 2015

Abstract Background: Over the past 70 years, there has been a recurring debate in the literature and in the popular press about how best to select medical students. This implies that we are still not getting it right: either some students are unsuited to medicine or the graduating doctors are considered unsatisfactory, or both. Aim: To determine whether particular variables at the point of selection might distinguish those more likely to become satisfactory professional doctors, by following a complete intake cohort of students throughout medical school and analysing all the data used for the students’ selection, their performance on a range of other potential selection tests, academic and clinical assessments throughout their studies, and records of professional behaviour covering the entire five years of the course. Methods: A longitudinal database captured the following anonymised information for every student (n ¼ 146) admitted in 2007 to the Hull York Medical School (HYMS) in the UK: demographic data (age, sex, citizenship); performance in each component of the selection procedure; performance in some other possible selection instruments (cognitive and non-cognitive psychometric tests); professional behaviour in tutorials and in other clinical settings; academic performance, clinical and communication skills at summative assessments throughout; professional behaviour lapses monitored routinely as part of the fitness-to-practise procedures. Correlations were sought between predictor variables and criterion variables chosen to demonstrate the full range of course outcomes from failure to complete the course to graduation with honours, and to reveal clinical and professional strengths and weaknesses. Results: Student demography was found to be an important predictor of outcomes, with females, younger students and British citizens performing better overall. The selection variable ‘‘HYMS academic score’’, based on prior academic performance, was a significant predictor of components of Year 4 written and Year 5 clinical examinations. Some cognitive subtest scores from the UK Clinical Aptitude Test (UKCAT) and the UKCAT total score were also significant predictors of the same components, and a unique predictor of the Year 5 written examination. A number of the non-cognitive tests were significant independent predictors of Years 4 and 5 clinical performance, and of lapses in professional behaviour. First- and second-year tutor ratings were significant predictors of all outcomes, both desirable and undesirable. Performance in Years 1 and 2 written exams did not predict performance in Year 4 but did generally predict Year 5 written and clinical performance. Conclusions: Measures of a range of relevant selection attributes and personal qualities can predict intermediate and end of course achievements in academic, clinical and professional behaviour domains. In this study HYMS academic score, some UKCAT subtest scores and the total UKCAT score, and some non-cognitive tests completed at the outset of studies, together predicted outcomes most comprehensively. Tutor evaluation of students early in the course also identified the more and less successful students in the three domains of academic, clinical and professional performance. These results may be helpful in informing the future development of selection tools.

Introduction Writing in the British Medical Journal in 1946, Smyth observed that ‘‘Existing methods of selection [of medical students] which worked well in the past may no longer be the best possible in changing conditions’’, further suggesting ‘‘we want . . . two independent tests or sets of tests – the one for ability, the other for character’’ (Smyth 1946). Though most medical students do graduate and become professional and capable doctors, the

subsequent and continuing debate creates the impression that medical schools are still selecting unsuitable students (Campbell 1974; Lockhart 1981; Lancet editorial 1984; Best 1989; Barr 2010), ‘‘who, though able to pass examinations, have not the necessary aptitude, character or staying power for a medical career’’ (Goodenough Committee 1944). For decades medical schools have tried to appraise the personal qualities that might underpin students’ future

Correspondence: Professor David Powis, School of Psychology (Psychology Building), The University of Newcastle, Callaghan, New South Wales 2308, Australia. Tel: +61 2 4921 5625; E-mail: [email protected]

868

ISSN 0142-159X print/ISSN 1466-187X online/15/090868–13 ß 2015 Informa UK Ltd. DOI: 10.3109/0142159X.2015.1009023

Predictors of outcome at medical school

Downloaded by [Rutgers University] at 00:11 28 September 2015

Practice points The study uniquely documents  the progress of a complete entry cohort of students throughout a five-year medical course.  The consequences of an extended range of predictor variables (including non-cognitive qualities) measured at the start of the course.  student progress on an extended range of outcome data collected during the course, including.  frequently repeated behavioural observations documented by Year 1 and Year 2 tutors.  standardised observation and reporting of all lapses in professional behaviour.  summative course outcome measures that distinguish between performance in clinical assessments and academic examinations. Together, these features have allowed the creation of a correlation matrix to determine the strength of linkage between entry, progress and outcome variables.

professional behaviour, by evaluating applicants’ personal statements and referees’ reports, measuring cognitive skills and conducting face-to-face interviews, in addition to assessing their academic suitability. These approaches have not as yet shown convincing predictive validity for medical school or later (Gray et al. 2002; Ferguson et al. 2002, 2003; Groves et al. 2007; Lynch et al. 2009; Poole et al. 2012; Kelly et al. 2013). No previous study has examined the predictive power of noncognitive tests, either alone or combined with a range of other selection tools, throughout the length of a medical course. However, demonstrating predictive validity of medical student selection is particularly difficult because most medical schools record course outcomes only in terms of the results of academic and clinical examinations. Such outcomes are seldom in an appropriate form to reflect, or sufficiently robust to evaluate, students’ non-cognitive and behavioural attributes (Schuwirth & Cantillon 2005), and seldom include final summative or barrier assessments of professional behaviour. The aim of our study was to examine what student attributes and qualities, alone or in combination, best predicted a range of outcomes of medical education. We therefore undertook an in-depth longitudinal study of an entire entry cohort of medical students through a five-year medical course at one UK medical school, in whom most potential predictors were measured either before or soon after the start of the course. This longitudinal study explores the question of predictive validity of the selection methods employed, and of other potential selection tests, in relation not only to the final examination outcomes but also to clinical and professional behaviour throughout the course. The approach was not hypothesis-driven but exploratory, thus allowing for emergent relationships between variables. The data included all the initial selection parameters, plus results from cognitive and non-cognitive test results not used in selection, as well as summative written and clinical examination results from each sitting over the five years of

the course. These examination data allowed novel ways of exploring clinical performance, because the clinical examinations in the final two years (Objective Structured Long Examination Reports, OSLERs, and Objective Structured Clinical Examinations, OSCEs) provided not only measures of success but also of deficiencies in various aspects of clinical performance, expressed as ‘‘penalty points’’ (PPs). In addition, data were collected from regular structured observations made by students’ tutors (all of whom were experienced clinicians, the problem-based learning tutors in the first two years as well as the clinical placement tutors in Years 3–5), which included assessments of professional behaviours. Uniquely, the tutor data were collated with the records of the school’s fitness-topractise committee to provide a measure of lapses of professional behaviour, quantified for the purpose of this study on a scale of ‘‘fitness to practise penalty points’’ (FTPPPs). We considered that this wide range of assessments, and in particular the design of the clinical examinations, would provide a range of measures not only of students’ academic prowess, but also of their likely personal and behavioural qualities as clinicians. The exact assessment methods will be described in detail. This complex longitudinal data set was analysed to determine associations between predictor variables (including demography, prior academic qualifications, cognitive and noncognitive/personality and behavioural qualities), and to link these to criterion variables measured during or at the end of the course reflecting academic and clinical performance, and to professional behaviour. The findings from Years 1 and 2 of the five-year course have been reported earlier (Adam et al. 2012); this report deals with Years 3 to 5 inclusive.

Methods Study sample All students admitted to Hull York Medical School (HYMS) in September 2007 participated in the study, which had ethics approval from HYMS’ Medical Education Ethics Committee. HYMS offers a five-year, problem-based, spiral curriculum within which a large proportion of the clinical experience is met in primary care. The cohort comprised 146 individuals, of whom 140 agreed to complete the non-cognitive tests. There were 62 males (43%) in the study sample. One hundred and eleven (76%) of the sample were aged under 21 years at the time of entry and 120 (82%) were British citizens. Ethnic origin was not recorded. The mean age at entry of the study sample was 19.9  S. D. 3.9 years (range: 18–42, median age 18).

Data collection HYMS selection parameters UCAS form. Selection into the HYMS medical school programme for this cohort was based on a score derived from information contained in the Universities and Colleges Admissions Service (UCAS) form submitted centrally by UK university applicants, and an interview score.

869

Downloaded by [Rutgers University] at 00:11 28 September 2015

J. Adam et al.

HYMS academic score. Each applicant’s UCAS form listed their academic results (either predicted or already obtained), usually as either advanced-level grades (or equivalent) or a degree classification. The academic grades likely to be required for entry were published before applications opened. Administrative staff categorised the academic results listed and determined a numerical HYMS academic score, using guidelines that ensured comparability with other systems of examination results such as the International Baccalaureate. The HYMS academic scores, using the example of A level grades, were: Unsatisfactory ¼ 0, predicted 2 A level grades below likely requirement; Just satisfactory ¼ 2, predicted 1 A level grade below likely requirement; Clearly satisfactory ¼ 5, predicted or obtained likely requirement; Outstanding ¼ 8, above likely requirement, either already obtained or predicted (in which case supporting evidence was required from earlier examination results). Only applications with a HYMS academic score 40 were considered further. The mean academic score in the study cohort was 6.1 (range 2–8, median 5 and mode 8). Personal statement and referee’s report. The remaining UCAS applications were assessed by two independent readers using a structured information sheet to assess the referees’ comments and record evidence of a realistic understanding of medicine, examples of self-motivation, acceptance of responsibility, communication skills and team working, and any other distinctive attributes (e.g. social disadvantage or exceptional sporting achievement). The data were then scored by administrative staff. The average score per reader was 25 (range 17–35). The HYMS academic score and mean of the two reader scores were added together to give the UCAS form score, maximum 50 points. Interview scores. The applicants with the top UCAS form scores were invited to interview. Structured, scripted 20minute interviews were conducted by two interviewers, yielding a maximum possible interview score of 50, comprising 40 points based on the answers to eight questions and 10 points from an overall assessment of suitability scored on a 0 to 10-point visual analogue scale. The UCAS form score and interview score were added to contribute equally to the final selection score on which applicants were ranked; places were then offered to the top scoring candidates.

Potential selection instruments Several standardised instruments were administered to this cohort as part of this study, but were not used in selection. Traits and skills measured by the cognitive UK Clinical Aptitude Test (UKCAT) and non-cognitive tests (Personal Qualities Assessment, PQA; Resilience Scales Questionnaire, RSQ; Trait Emotional Intelligence Questionnaire, TEI) are written in italics. Combined traits derived from these are written IN ITALIC CAPITALS.

Cognitive tests UKCAT. The UKCAT is a mandatory standardised test of cognitive ability for those applying to study medicine at the 870

majority of UK medical schools. It was first taken in 2006, by applicants to courses starting in 2007 (see www.ukcat.ac.uk). UKCAT results were available for 131 students; 13 had applied the year before the test was introduced, one student was exempt and one result was unavailable. The UKCAT comprised four cognitive skills subtest scores: verbal reasoning (VR), numerical reasoning (NR), abstract reasoning (AR) and decision analysis (DA) (Childs 2012) and a total score. The UKCAT scores were not revealed to the HYMS selectors at any stage and did not inform selection decisions.

Non-cognitive tests PQA.

This comprised three tests: (a) The Interpersonal Traits Questionnaire, which measures the traits narcissism, aloofness, confidence (in dealing with people) and empathy and produces a summary score for the combined trait INVOLVEMENT (versus DETACHMENT) in which confidence and empathy are positive, narcissism and aloofness negative (Munro et al. 2005). (b) The Interpersonal Values Questionnaire, which measures the extent to which the respondent favours individual freedoms (versus societal rules) as the basis of their moral orientation (Bore et al. 2005). (c) The Self-Appraisal Inventory, which measures the combined traits EMOTIONAL RESILIENCE (comprising scales measuring anxiety, moodiness, neuroticism and irrational thinking) and SELF-CONTROL, in contrast to risk taking tendency, (using the scales of restraint, conscientiousness, permissiveness and anti-social tendencies). The inventory also contains a Lie scale (Bore et al. 2009; www.pqa.net.au).

RSQ.

RSQ is a self-report questionnaire that identifies six cognitive, behavioural and affective components, named self-esteem, optimism, self-discipline, control, emotional nondefensiveness and image management (impression management) (Childs 2012).

TEI. The TEI Short Form is a 30-item questionnaire designed to measure the global trait EMOTIONAL INTELLIGENCE (Petrides 2009; Cooper & Petrides 2010). The non-cognitive tests were all delivered in a paper-based format under examination conditions at the University of Hull and the University of York; PQA and TEI in October 2007 and RSQ in October 2008.

Tutor ratings and grades Problem-based learning tutors in Years 1 and 2 met with their students twice weekly for 90 minutes. They assessed the individual students’ interpersonal skills and professional behaviours, which were recorded on standard scales in Year 1 (May 08) and Year 2 (January 09 and May 09), as described in Adam et al. (2012). ‘‘Tutor ratings’’ represent the sum of the scores for each of 14–17 defined skills or behaviours. At the same time, these experienced problem-based learning tutors made a global assessment, grading each student as either

Predictors of outcome at medical school

‘‘problematic’’, ‘‘average’’ or ‘‘particularly promising’’. This is the ‘‘tutor grade’’.

Course summative assessments

Downloaded by [Rutgers University] at 00:11 28 September 2015

Course assessments were undertaken in accordance with HYMS’ Code of Practice on Assessment and Examination for MB BS in Phases II & III. There was no summative assessment in Year 3. The Year 4 and Year 5 summative assessments are summarised below.

Year 4 written examination. Marks in the written examinations at end of academic year 4 were allocated to one of two HYMS themes: Theme A (Life sciences and Clinical sciences) and Theme C (Evidence-based decision-making, Population health and medicine and Managing resources for quality and efficiency). The papers were mapped to course outcomes including those from pharmacology and therapeutics. Theme C tested not only application of clinical knowledge, but also analytic and numerical evaluation skills across a range of medical contexts. Theme B (Clinical Techniques and Skills; Person-Centred Care) was examined in Year 4 only by the clinical examinations. Year 5 written examination. The written examination at the end of academic year 5 was an integrated paper covering all aspects of the course (Themes A, B and C). The questions were based on clinical scenarios, each one orientated around a common management problem including preventive strategies. Therapeutic issues were a major focus, but other management issues were also examined in this paper. The pass mark was determined by the Hofstee method (McKinley & Norcini 2014). Students sat the paper in March and those who did not achieve the pass mark took another paper in May. This was an independent paper but it was not considered a resit. If the students passed at the second attempt, then they passed the written examination. Year 4 and Year 5 clinical examinations. The Year 4 and Year 5 clinical examinations used a number of Objective Structured Long Examination Records (OSLERs), each being a 45-minute observed clinical assessment in which the student met and talked to a real patient, undertook appropriate examinations, spent 15 minutes alone preparing a written

summary and plan, and then discussed this with the two examiners and the patient. Student performance was assessed in four categories of competence in Year 4: gathering information, clinical examination, problem solving, and relationship with patient. A fifth category, patient management, was added in Year 5. The Year 5 examinations also included Objective Structured Clinical Examinations (OSCEs), which were 7-minute stations assessed by direct observation by one examiner, the majority designed to test students’ high level ‘‘communication skills’’ using simulated patients. Practical clinical procedures were not routinely included in the clinical examinations. Each student was required to have reached a satisfactory standard in every specified practical clinical procedure, assessed earlier in a controlled clinical environment, before being allowed to take the OSCEs. The HYMS clinical examinations used a sequential design and a non-compensatory marking system, fully described elsewhere (Cookson et al. 2011). The key features are summarised below. The sequential design required all students to take the first part of the examination. The best performing candidates (usually around 70%) were found to be clearly satisfactory at this point. The remaining approximately 30% were required to take the second part of the examination; the additional results from the second part, together with the results from the first part, were summed to give increased reliability in determining which side of the pass/fail boundary each student lay. Around 5–8% eventually failed the examination. The Year 4 clinical examinations consisted of two OSLERs in the first part and three OSLERs in the second part. The Year 5 examinations comprised four OSLERs and six OSCEs in the first part, and the same number in the second part. Four of the six OSCEs in the first part and five of the six OSCEs in the second part addressed communications skills; the remainder involved a written task or skill.

Grading and marking system Each category of competence examined in the OSLERs, and every OSCE, was graded by the examiners using the following grade descriptors shown in Table 1. The outcome of these examinations was pass/fail only. On the basis that candidates should not be able to compensate for

Table 1. Grade descriptors, PPs and scores for OSLERs and OSCEs.

Grades A B Cþ C D E

Descriptor Capable in all components to a high standard Capable in all components to a satisfactory standard and a high standard in many Capable in all components to a satisfactory standard Capable in a majority of components to a satisfactory standard, inadequacies in some components Capable in a minority of components. No serious defects Capable in a minority of components. One or more serious defects

Penalty points Year 4

Penalty points Year 5

Research score

– –

– –

6 5

– –

– 1

4 3

2

2

2

3

3

1

871

Downloaded by [Rutgers University] at 00:11 28 September 2015

J. Adam et al.

a serious deficiency in one area by a high standard in another, grades below satisfactory were converted to PPs, and only the PPs were used to make the pass/fail decisions. Candidates accruing more than a fixed number of PPs in the first part were required to take the second part of the examination. HYMS determined the examination outcome from the total number of PPs accrued, across either the first part or both parts of the examination if taken, to give a pass/fail result. The grades A to E form an ordinal categorical scale (Cookson et al. 2011), so for the purpose of this study they were converted to positive scores, as shown in Table 1. For this study we have expressed all examination data as an average per case, derived from the scores from the first part of the examination, and from the second part if taken, whether positive scores or PPs. These are designated OSLER score, OSCE score, OSLER PPs and OSCE PPs. A further summary score and penalty point measure was calculated for the entire Year 5 clinical examination by adding together the OSLER and OSCE scores (equally weighted) and the OSLER and OSCE PPs (equally weighted), called OSLER þ OSCE sum score and OSLER þ OSCE sum PPs, respectively. The maximum number of possible PPs (designated Clinical Examination penalty points, CEPPs) from all the Year 5 OSLERs and OSCEs adding the first part and the second part together was 276; the observed range in this cohort was 0 to 56, mean 11.04, median 6, mode 4.

Other outcome measures Other relevant outcome measures detailed here include subscales derived from the summative examinations that were considered particularly relevant to explore for this study, such as communication skills and Honours awards, and empathy scores provided as formative examination feedback to the students.

OSLER communication scores. The OSLER competence categories ‘‘gathering information’’ and ‘‘relationship with patient’’ rely heavily on communication skills. The scores and PPs accrued for these competences alone were added together and a mean score per case calculated as described above, to give the OSLER communication score and OSLER communication PPs. OSCE communication scores. Four of six OSCE stations in the part 1 examination and 5 of 6 in the part 2 examination assessed communication skills in challenging situations; the mean score and PPs per case accrued from only these stations yielded the OSCE communication score and OSCE communication PPs. OSCE empathy scores. At each communication OSCE station, the examiner and simulated patient undertook independent assessments of the student’s perceived empathy, graded A to E, ranging from ‘‘excellent’’ to ‘‘poor’’ empathy skills (Wright et al. 2014). These grades were collected for research, and provided to students as formative feedback. For the present study the grades were converted to empathy scores (A, B, C, D, E ¼ 5, 4, 3, 2, 1 points) and PPs (D, E ¼ 1, 2 872

PPs) and used to calculate a mean OSCE Empathy score and OSCE Empathy PPs per case, using data from both parts of the examination. The clinical examination variables thus fall into two categories, those that give positive scores for performance, and those that give negative scores for deficient performance. Better performance is denoted by higher totals in the OSLER score, OSLER communication score, OSCE score, OSCE communication score, OSCE Empathy score and OSLER þ OSCE sum score. The indices of deficient clinical performance are OSLER PPs, OSLER communication PPs, OSCE PPs, OSCE communication PPs, OSCE Empathy PPs, OSLER þ OSCE sum PPs and clinical examination PPs.

Criteria for graduation with honours HYMS applied specific criteria for recommending students for graduation with Honours, based on weighted overall performance in summative examinations and student projects throughout the course. For this cohort approximately 6% were awarded Honours.

Fitness to practise penalty points The final important outcome measure was FTPPPs. The HYMS Fitness-to-Practise committee received confidential reports (in which students were identified only by number) not only about serious lapses in professional behaviour, but also about lower level concerns arising from structured formative end-ofblock reviews. These reviews were undertaken regularly between student and their current clinical tutor at approximately two monthly intervals throughout the entire course, and included grading of the student’s professional behaviour (using a structured score sheet with clear grade descriptors), either Excellent, Satisfactory, Borderline or Unsatisfactory, under each of the following headings: ‘‘relationships with patients’’; ‘‘awareness of ethical and moral aspects of subject’’; ‘‘ability to deal with uncertainty and awareness of limitations’’; ‘‘evidence of self-education, enthusiasm and motivation’’; ‘‘teamwork’; ‘‘dress, attendance and punctuality’’. A ‘‘borderline’’ or ‘‘unsatisfactory’’ grade in any aspect of professional behaviour was automatically notified to the Fitness-to-Practise Committee. This allowed HYMS to identify and react when students showed repeated patterns of undesirable behaviour. J. A. reviewed all the reports of the HYMS Fitness-toPractise committee covering the HYMS academic years 2007– 2013 inclusive, and summarised every instance concerning a member of the study cohort. Each instance was given one or more FTPPPs devised for this study; the total number of points accrued by each student across the years of the course was then recorded. For example, a professional behaviour grade of ‘‘borderline’’ was given 1 FTPPP and an ‘‘unsatisfactory’’ grade was given 2 FTPPPs. Other misdemeanours were awarded one or two points, after assessing their seriousness through careful evaluation of all the relevant information including contemporaneous verbal accounts and written records. In total, 45 individuals were identified in Fitness-to-Practise records, of whom 26 had only 1 FTPPP. There were long-term consequences for a significant proportion of those accruing three or

Predictors of outcome at medical school

more FTPPPs, with referral to formal fitness-to-practise procedures that can impose serious penalties, the most serious being either a formal warning reported to the UK General Medical Council on registration as a doctor, or being required to leave the course.

Downloaded by [Rutgers University] at 00:11 28 September 2015

Data handling and statistical analysis All data were anonymised and entered into an SPSS database. All analyses utilised SPSS version 20 (Chicago, IL). Scores on all multiple-component measures (OSCEs, OSLERs, communication and empathy measures, etc.) were computed and checked for internal consistency. Except where noted (e.g. age) no extreme skewness and no extreme outliers were detected. Relationships between continuous variables were computed as Pearson correlation coefficients (r); one-way analysis of variance (ANOVA, F-statistic) was used to compare scores on categorical variables (sex, citizenship, etc.) and on comparisons of extreme groups, such as top versus bottom 20% of scores. Statistical significance was based on unadjusted probabilities (p ¼ 0.05, 0.01 or 0.001). Two-tailed tests of significance were applied in most cases, except where a clear directional relationship was predicted. In order to establish the

best overall predictors of outcomes, a series of linear regression analyses were undertaken and are described in the Results section.

Results The tables show all the statistically significant results. Table 2 shows the significant correlations found between the initial predictor variables (available from the start of the course), intermediate tutor evaluations (from Years 1 and 2), demographic factors and the outcomes from the Year 4 and Year 5 written and clinical examination results. We considered that graduating with Honours or with notably high written and clinical examination results were both desirable selection outcomes, whereas failing to complete the course or having documented instances of significant unprofessional behaviour were undesirable selection outcomes. Table 3 shows which predictors were significant when ANOVA was applied to three pairs of comparisons based on positive exam marks: those students who graduated with Honours (n ¼ 11) versus those who left the course (n ¼ 9); those students with Year 5 written exam scores in the top quintile versus the bottom quintile;

Table 2. Predictors of examination scores.

Initial predictors

HYMS academic

Written exams Year 4 theme A

r 0.174*

Year 4 theme C

r 0.200*

Year 5 final exam

PQA

RSQ

HYMS tutors

Years 1 & 2 examinations

Jan 09g r 0.261** May 09g r 0.257** 08þ09g r 0.236** Jan 09g r 0.246** May 09g r 0.218* 08þ09r r 0.194* 08þ09g r 0.239**

Tot r 0.181*

Demographic predictors

Age

r 0.213*

Year 5 OSLER Communication score Year 5 OSCE score

r 0.170*

y (F)*

UK/Non-UK

UK (F)*

f (F)*

ASoc r 0.181*

r 0.215*

Sex

f (F)**

Tot r 0.175* AR r 0.231**

Clinical exams Year 4 OSLER score Year 4 OSLER communication score Year 5 OSLER score

Year 5 OSCE Communication score Year 5 empathy score Year 5 OSLERþOSCE sum score

UKCAT

Intermediate predictors

UK (F)*

08þ09r r 0.179*

Lie r 0.198* Emp r 0.257**

En-d r 0.219*

Conf r 0.192* Emp r 0.260**

(TEI r 0.201)*

Jan 09g r 0.176* May 09r r 0.176* May 09g r 0.197* 08þ09g r 0.197*

o (F)*

f (F)*

y (F)**

f (F)**

f (F)***

Tot r 0.204* VR r 0.244** AR r 0.250**

y (F)**

UK (F)* UK (F)**

Jan 09r r 0.185* May 09r r 0.188* r 0.256**

Emp r 0.194*

May 09r r 0.191* May 09g r 0.190*

Year Year Year Year Year Year Year Year

1A r 0.254** 1B r 0.228** 1C r 0.215* 1T r 0.296*** 2A r 0.301*** 2B r 0.337*** 2C r 0.271** 2T r 0.359***

y (F)*

f (F)*

UK (F)* UK (F)**

Analysis of variance (F statistic) for categorical variables and Pearson correlations (r) for continuous variables, *p50.05; ** p50.01; *** p50.001 (N ¼ 131–146; significance levels are not adjusted for repeated comparisons). UKCAT Tot, total UKCAT score; VR, verbal reasoning; AR, abstract reasoning; PQA Antisoc, anti-social tendencies; Lie, lie scale score; Emp, empathy; Conf, confidence; RSQ En-d, emotional non-defensiveness; TEI, test of emotional intelligence. HYMS tutors: ratings, r, and grades, g, given in years 1 (May 08) and 2 (January 09, May 09). Years 1 & 2 exams: themes A, B and C, and totals, T, of AþBþC, y: younger; o: older at entry; f: female; UK: UK citizen.

873

J. Adam et al.

Table 3. Predictors of top and bottom performers.

Initial predictors HYMS acad

UKCAT

PQA

Intermediate predictors

RSQ

Year 5 written examination Top 20% versus bottom 20%

Downloaded by [Rutgers University] at 00:11 28 September 2015

Years 1&2 exams

Age

Sex

UK or non-UK

May 08r (F)* Jan 09g (F)** May 09g (F)***

Graduation with Honours versus left course

Year 5 clinical examination (OSLERþOSCE sum score) Top 20% versus bottom 20%

HYMS tutors

Demographic predictors

Tot (F)* VR (F)** AR (F)** QR (F)**

(F)*

Mood (F)* Consc (F)* Conf (F)**

(TEI (F)*) En-d (F)*

May 09r (F)* May 09g (F)* 08þ09r (F)* 08þ09g (F)*

Yr 1A (F)*** Yr 1C (F)*** Yr 1 T (F)*** Yr 2 OSCE (F)* Yr 2A (F)*** Yr 2B (F)*** Yr 2C (F)*** Yr 2 T (F)*** Yr 1 OSCE com (F)** Yr 1 OSCE prac (F)* Yr 1 OSCE T (F)** Yr 1A (F)** Yr 1B (F)** Yr 1C (F)* Year 1 T (F)*** Yr 2 OSCE (F)*** Yr 2A (F)*** Yr 2B (F)*** Yr 2C (F)** Year 2 T (F)***

f (F)*

y (F)*

f (F)**

Analysis of variance (F statistic) for categorical variables, *p50.05; **p50.01; ***p50.001 (N ¼ 131–146; significance levels are not adjusted for repeated comparisons). HYMS acad: computed academic entry score, see Methods section; UKAT Tot, total UKCAT score; VR, verbal reasoning; AR, abstract reasoning; QR, quantitative reasoning; PQA Mood, moodiness; Consc, conscientiousness; Conf, confidence. RSQ En-d, emotional non-defensiveness. TEI, test of emotional intelligence. HYMS tutors: ratings, r, and grades, g, given in years 1 (May 08) and 2 (January 09, May 09). Year 1 & 2 exams: themes A, B and C, and total, T, of AþBþC, and OSCE practical and communication skills stations and sum total, T, of ‘‘prac’’ and ‘‘com’’. y: younger age at entry; f: female.

those students with Year 5 clinical examinations (OSLER þ OSCE sum) scores in the top quintile versus the bottom quintile. Table 4 shows the significant predictors of deficient clinical performance or undesirable outcomes, from correlations with clinical examination PPs, and from comparisons of three groups of students achieving undesirable outcomes: those who left the course versus the rest; those who left the course grouped with those who gained 3 or more FTPPPs versus the rest, and those who gained any FTPPPs versus those who did not. Finally, Table 5 shows the outcome of the multiple regression analyses to determine the best predictors of Year 4 and Year 5 outcomes.

covering Themes A, B and C). They outperformed males also in Year 4 and Year 5 OSLERs, and achieved a better overall score in the combined Year 5 OSLERþOSCE sum score (Table 2). They received fewer OSLER and OSCE PPs than their male peers (Table 4). Females were better represented than males in the top 20% of achievers in the final year, in both written and clinical examinations (Table 3). Males did not outperform females in any component of the Year 4 and 5 examinations.

Demographic data

Citizenship. Non-UK citizens performed at a lower level in Year 4 and 5 written examinations and the Year 5 clinical examinations than UK citizens (Table 2), and gained significantly more PPs and penalty marks in all of the clinical examinations (Table 4).

Age. Students under 21 at the time of entry performed

HYMS selection process

Predictors of achievement and progress

better than older students in the Theme C component of the Year 4 written examination and in the Year 5 clinical examinations (Year 5 OSLER, Year 5 OSCE). Older students outperformed their younger peers only in the Year 4 OSLER examination, specifically in the ‘‘communication’’ segments (Table 2).

Sex. Females performed better than males in the Theme A component of the Year 4 written examination and in the Year 5 final written examination (an integrated examination 874

The HYMS academic score, an index of prior academic performance, was the only item used in the selection procedure that was a useful, significant predictor of progress, correlating with a number of assessment outcomes. Written exam performance (Year 4, Themes A and C) and clinical exam performance were predicted (Year 5 OSLER and Year 5 OSCE) (Table 2). The HYMS academic score also correlated (negatively) with the Year 5 OSLERþOSCE sum PPs and the total clinical exam penalty points (CEPPs) (Table 4). No other

Predictors of outcome at medical school

Table 4. Predictors of undesirable outcomes.

Initial predictors

HYMS acad

HYMS interview

UKCAT

PQA

Intermediate predictors

RSQ

HYMS tutors

Downloaded by [Rutgers University] at 00:11 28 September 2015

Year 5 OSLERþOSCE sum PPs

Neurot r 0.177*

r 0.190* r 0.187*

Year 5 OSLER Communication PPs Year 5 OSCE PPs

Year 5 OSCE Communication PPs Year 5 OSCE Empathy PPs Year 5 CEPPs

Age

Sex

UK or non-UK

m (F)*

Non (F)*

08–09g r 0.192* May 08g r 0.185*

Year 4 OSLER PPs Year 4 OSLER Communication PPs Year 5 OSLER PPs

Years 1&2 exams

Demographic predictors

E n-d r 0.211*

Emp r 0.255** Inv r 0.187* Emp r 0.241** VR r 0.218** AR r 0.274** QR r 0.261** Tot r 0.304*** VR r 0.177* AR r 0.196*

Conf r 0.198*

m (F)*** Self-est r 0.220* Optm r 0.231*

Conf r 0.195*

Non (F)*

May 08g r 0.175* 08–09g r 0.187*

Non (F)* Non (F)*

r 0.235** r 0.208*

VR r 0.214*

Emp r 0.369*** Inv r 0.297** Emp r 0.179*

Left course (versus or not)

Antisoc (F)*

Left course or 43 FTPPPs versus ‘‘not’’ and ‘‘none’’ FTPPPs (versus none)

Antisoc (F)*

Aloof (F)*

08–09r r 0.177*

Optm (F)*

Image (F)*

May 08 (F)* Jan 09r (F)* Jan 09g (F)** May 09r (F)** May 09g (F)** 08–09r (F)* Jan 09r (F)** 08–09r (F)*

Yr Yr Yr Yr Yr Yr Yr Yr Yr Yr Yr

1A r 0.251** 1B r 0.180* 1C r 0.230** 1T r 0.284*** 2A r 0.288*** 2B r 0.338*** 2C r 0.275*** 2T r 0.352*** 2C (F)*** 2T (F)* 2 OSCE (F)**

Yr Yr Yr Yr Yr Yr Yr

2 OSCE (F)* 2C (F)** 2 T (F)* 1B (F)* 1 OSCE (F)* 2B (F)* 2C (F)**

m (F)*

Non (F)*

m (F)**

Non (F)**

o (F)**

o (F)*

m (F)*

Analysis of variance (F statistic) for categorical variables and Pearson correlations (r) for continuous variables, *p50.05; **p50.01; ***p50.001 (N ¼ 131–146; significance levels are not adjusted for repeated comparisons). UKCAT Tot, total UKCAT total score; VR, verbal reasoning; AR, abstract reasoning; QR, quantitative reasoning; PQA Antisoc, anti-social tendencies; Aloof, aloofness; Emp, empathy; Conf, confidence; Neurot, neurotic; Inv, INVOLVED. RSQ En-d, emotional non-defensiveness; Optm, optimism; Image, managing own image. Self-est, self-esteem. HYMS tutors: ratings, r, and grades, g, given in years 1 (May 08) and 2 (January 09, May 09). Year 1&2 exams: themes A, B and C, and totals, T of AþBþC, and OSCE; o: older at entry; m: male; f: female; UK/Non: UK citizen or non-UK citizen. PP: penalty points; FTPPP: fitness to practise penalty points.

variables derived from the UCAS form personal statement & referees’ report score, nor variables from the HYMS interview question scores and visual analogue scale, were useful predictors; indeed, ‘‘interview score’’ correlated significantly only with Year 4 OSLER communication PPs, but in the unexpected direction (Table 4).

FTPPPs. In contrast, none of the PQA test scores predicted written exam performance in Years 4 and 5 (Table 2). The PQA traits confidence, conscientiousness, anti-social tendencies, empathy, moodiness, neuroticism and aloofness and the combined trait INVOLVEMENT were shown to be predictors of clinical examination scores (Tables 2 and 3) and of PPs in these exams (Table 4).

Cognitive tests: UKCAT sub-scores and total score UKCAT total score predicted some written exam scores (Year 4, Theme C and Year 5) and clinical performance assessed by Year 5 OSCE. Some UKCAT subtest scores (Abstract Reasoning and Verbal Reasoning) were also significantly correlated with the written and some clinical exam scores in Year 5, but not Year 4 (Table 2).

Non-cognitive Tests: RSQ Like PQA, some RSQ components (emotional non-defensiveness, self-esteem and optimism) predicted clinical performance both good and bad, but not written exam performance (Table 2).

Non-cognitive tests: PQA

Intermediate predictors: In-course Tutor ratings and grades

Many significant correlations were found between some trait and combined trait scores and clinical exam scores, CEPPs and

Tutor evaluations (both ratings and grades) of students’ capabilities and deficiencies made during three terms in

875

J. Adam et al.

Years 1 and 2 were not only significant predictors of many components of written and clinical examination performance in Years 4 and 5 (Table 2), but also significant predictors of the groups of students who gained Honours or left the course (Tables 3 and 4).

Downloaded by [Rutgers University] at 00:11 28 September 2015

Intermediate predictors: Year 1 and Year 2 examination performance Year 1 Themes A, B and C results and Year 2 Themes A, B and C results each individually predicted Year 5 OSLERþOSCE sum score (Table 2) and Year 5 OSLERþOSCE sum PPs (Table 4). Themes B and C were also found to significantly predict who left the course and FTPPPs (Table 4). The top and bottom 20% of achievers in Year 5 written and clinical examinations (OSLER & OSCE) were generally predicted by Year 1 and 2 results (Table 3). However, Year 1 and 2 examination results did not predict graduation with Honours or leaving the course (Table 3).

Desirable and undesirable outcomes Graduation with honours By 2013, nine individuals (6%) from the study sample had graduated with honours, seven of whom were aged 521 at entry, three were males and eight were UK citizens. Year 1 and 2 tutor ratings (r) and grades (g) were the only significant independent predictors of ‘‘graduation with honours’’ (Table 3).

Leaving the course Eleven individuals from the study sample (8%) left the course without completing their medical degree; six (55%) were aged 521 at entry, two (18%) were males and eight (72%) were UK citizens. Those who left scored more highly on RSQ optimism and PQA anti-social tendencies. Year 1 and 2 tutor ratings and grades (May 08, January 09 and May 09) were consistent predictors of those who left the course (Table 4).

Comparison of those who gained Honours with those who left the course The only significant predictors of leaving the course versus graduating with honours were Year 1 (May 08) tutor ratings and Year 2 (January 09, May 09) tutor grades (Table 3).

Comparison of top 20% with the bottom 20% of achievers in the Year 5 final examinations Year 5 written examination. Being among the top 20% of achievers was significantly associated with being female, or having a higher UKCAT total score (and better UKCAT quantitative reasoning, verbal reasoning and abstract reasoning scores). Students who had performed to a higher standard in Year 1 and Year 2 examinations were significantly more likely to be represented in the top scoring group in the Year 5 written examination (Table 3). Year 5 clinical examination. The following variables significantly indicated the likelihood of being among the top 20% of achievers: being female, younger and scoring better in 876

some of the non-cognitive tests (PQA: lower moodiness, higher conscientiousness and higher confidence; RSQ: higher emotional non-defensiveness; TEI: higher emotional intelligence). Better tutor ratings and grades in Year 1 and 2 predicted higher achievement, as did better overall marks and individual components by theme of Year 1 and Year 2 examinations (Table 3).

Comparison of those with extreme high or low noncognitive PQA scores with central scores Bore et al. (2009) proposed that extreme scores (both high and low) on the PQA non-cognitive tests would be likely to predict low scores on positive course outcomes and/or high scores on negative course outcomes. In order to test this hypothesis, outcome data from the students in the top and bottom 20% of scores on each non-cognitive PQA trait were combined and compared with the middle 60%, using analysis of variance. The hypothesis was confirmed for the PQA combined trait INVOLVEMENT in relation to the Year 5 OSCE score (p ¼ 0.036), Year 5 OSCE empathy score (p ¼ 0.042) and also Year 5 OSCE empathy PPs (p ¼ 0.035).

Comparison of those reported to the fitness-to-practise committee with the rest of the cohort None of the HYMS selection criteria predicted incidents or behaviours of concern to the Fitness-to-Practise committee. Males were significantly more likely to gain FTPPPs, but neither age at entry nor UK citizenship were significant predictors. Those students who had underperformed in components of the Year 1 and Year 2 examinations were more likely to be those who gained FTPPPs. Neither UKCAT sub-scores nor total scores predicted FTPPPs, but PQA aloofness and RSQ managing image did predict FTPPPs (Table 4).

Individual characteristics of the subgroup of students reported to the fitness-to-practise committee Overall, 45 students gained FTPPPs. ANOVA (F-test) showed that several predictors characterised this subgroup: UKCAT higher verbal reasoning scores (p ¼ 0.016); PQA lower conscientiousness (p ¼ 0.023); PQA higher impulsiveness (p ¼ 0.041); PQA higher confidence (p ¼ 0.015); RSQ lower self-discipline (p ¼ 0.015); RSQ lower control (p ¼ 0.011). Lower Year 1 and 2 tutor ratings were also a significant predictor (sum of May 08 þ January 09þMay 09 ratings; p ¼ 0.018). A stepwise regression which included all of the above predictors found that RSQ control was the predominant predictor (beta ¼ 0.539; p ¼ 0.004). The subgroup with FTPPPs was also associated with poorer clinical performance: in Year 4 a lower OSLER score, p ¼ 0.004; lower OSLER communication score, p ¼ 0.004; more OSLER PPs and OSLER communication PPs, p ¼ 0.000 and p ¼ 0.003, respectively, and in Year 5 more OSCE PPs (p ¼ 0.010) and clinical examination PPs (p ¼ 0.018). A further comparison of interest from the selectors’ perspective is reported in Table 4. The worst selection outcomes are students who either leave the course or manifest

Predictors of outcome at medical school

variables were then entered, to show how much additional variance might be accounted for by the other factors they represent. The results show that the prior ability variables alone would be useful predictors of Year 4 examinations, Year 5 OSCE and OSLER scores and most of the associated penalty point scores, but not the Year 4 OSLERs or Year 5 OSCE empathy score. The HYMS academic score appears to be a better predictor than the UKCAT total, which in the case of Year 5 OSLER scores showed a negative relationship with the outcomes. The addition of the non-cognitive variables would make a significant contribution in the case of the Year 5 OSLER scores and PPs, with PQA empathy and RSQ emotional non-defensiveness contributing about equally. The difference between the value of the adjusted R-squared (percentage of the variance) at stage 1 and 2 and after stage 3 shows that the additional demographic variables, particularly sex and citizenship, would attenuate the predictive usefulness of the prior ability and non-cognitive variables.

serious professional misbehaviour. We therefore grouped together those students who had either left the course prematurely or who had gained three or more FTPPPs (n ¼ 22), and compared them with the rest of the cohort, to look for significant predictors of these undesirable outcomes. This group were older at entry, had scored higher on the PQA anti-social tendencies measure, and all had scored lower on the summed Year 1 and 2 tutor ratings (May 08þJanuary 09þMay 09). These students had also performed less well in components of the Year 2 examination (Table 4).

Downloaded by [Rutgers University] at 00:11 28 September 2015

Best predictors of Year 4 and Year 5 outcomes: Regression analyses To clarify the complex relationships between predictors and outcomes, a series of linear regression analyses were undertaken using the SPSS routines, which revealed that the best overall predictors of all outcomes were the demographic variables (age, sex and citizenship), the prior ‘‘ability’’ indicators (HYMS academic score and total UKCAT score) and the non-cognitive test scores for empathy (PQA) and emotional non-defensiveness (RSQ). The apparent selection power of the demographic variables is likely to be mediated by other confounding variables, so the two ability variables (the most readily available for selection purposes) were entered into the first stage of a multiple regression analysis, and the two noncognitive variables (which might be added to the selection procedures) in the second stage. The results are shown in Table 5, which also shows the percentage of variance in each key outcome variable accounted for by the ability and noncognitive predictors. In a third stage, the three demographic

Discussion The aim of our study was to examine what student attributes and qualities, or combination thereof, best predicted outcomes of medical education. This was undertaken longitudinally in the context of a specific medical school cohort with predictors being measured some four to five years prior to measurement of the outcome variables. The approach was not hypothesesdriven but exploratory thus allowing the data to reveal relationships between variables. The principal findings were:

Table 5. Best selectors for Year 4 & Year 5 examination outcomes.

1st stage

Outcome variable Years 4 & 5 – positive outcomes Year 4 Theme A exam Year 4 Theme C exam Year 4 OSLER total Year 4 OSLER communication Year 5 OSCE total Year 5 OSCE communication Year 5 OSLER total Year 5 OSLER communication Year 5 OSLERþOSCE sum score Year 5 OSCE empathy score Years 4 & 5 – penalty points Year 4 OSLER PPs Year 4 OSLER communication PPs Year 5 OSCE PPs Year 5 OSCE communication PPs Year 5 OSLER PPs Year 5 OSLER communication PPs Year 5 Clinical exam PPs Year 5 OSCE Empathy PPs Year5 OSLERþOSCE sum PPs

2nd stage

3rd stage

UCAS acad

UKCAT total

PQA empathy

RSQ e n-d

Stages 1þ2 adjusted R-squared (% of variance)

0.22* 0.31** 0.02 0.01 0.14 0.21* 0.30** 0.24* 0.29** 0.11

0.08 0.04 0.07 0.11 0.18 0.05 0.22* 0.29** 0.01 0.01

0.05 0.06 0.13 0.01 0.05 0.01 0.21* 0.21* 0.17 0.05

0.12 0.14 0.12 0.16 0.02 0.18 0.25** 0.20* 0.17 0.11

4.3 8.8 0.0 0.0 2.3 4.2 20.1 17.7 11.2 0.0

0.30** 0.14 0.22* 0.29** 0.04 0.01 0.19* 0.26** 0.09 0.07

0.00 0.14 0.19 0.22* 0.22* 0.08 0.12 0.13 0.08 0.08

0.00 0.09 0.07 0.06 0.24* 0.18 0.21* 0.10 0.20*

0.17 0.14 0.31** 0.13 0.07 0.10 0.01 0.01 0.15

0.01 0.04 0.04 0.14 0.18 0.26* 0.39** 0.01 0.21*

0.11 0.28** 0.06 0.03 0.29** 0.18 0.08 0.10 0.16

0.0 6.8 7.6 0.0 15.6 11.2 16.3 0.0 12.2

0.12 0.13 0.10 0.05 0.08 0.24* 0.06 0.06 0.11

0.20 0.06 0.17 0.13 0.07 0.03 0.16 0.03 0.15

Sex

Age

UK/Non

Stages 1þ2þ3 adjusted R-squared (% of variance)

0.14 0.15 0.15 0.11 0.24* 0.17 0.16 0.16 0.28** 0.21*

10.9 10.5 2.9 8.0 11.6 4.1 23.3 23.7 17.9 1.4

0.06 0.08 0.18 0.21* 0.23* 0.15 0.23* 0.13 0.27**

0.0 6.0 9.6 0.2 18.2 15.0 19.7 0.0 17.6

Cells in columns 2–5, 7–9 contain standardised coefficients; cells in columns 6 and 10 give percentage of variance accounted for. Statistical significance: *50.05 **50.01 (t-test, two-tailed), N ¼ 131–146. Abbreviations as elsewhere.

877

Downloaded by [Rutgers University] at 00:11 28 September 2015

J. Adam et al.

Prior academic achievement was related to written Year 4 (but not Year 5) exams, and also to Year 5 (but not Year 4) clinical exams and penalty marks. Being younger, or female, or having UK citizenship were each associated with better performance on many outcomes. Cognitive reasoning ability, as assessed by the UKCAT tests, was related to the Year 5 written exam and OSCE performance. Non-cognitive variables were not significantly related to written exams, but were shown to be related to performance in clinical examinations and other aspects of the course in a number of ways, both positive and negative, as summarised below:  Measures of empathy, emotional non-defensiveness, confidence and emotional intelligence were related to Year 5 OSLER performance.  Students with high Year 5 OSLER and OSCE PPs tended to have low empathy and low confidence, with low selfesteem and low optimism also associated with low Year 5 OSCE scores.  Conscientiousness, confidence, moodiness, emotional intelligence and emotional non-defensiveness all differentiated the top 20% of Year 5 clinical examination achievers from the bottom 20%.  Anti-social tendencies and optimism were related to leaving the course, while aloofness and (poor) image management related to FTPPPs. Two general observations can be made. First, the correlations between the different assessment outcomes within the HYMS course were generally strong, indicating a high degree of assessment coherence. Second, our findings suggest that tutors have considerable insight into their students’ behaviour which eventually correlates with their academic and professional performance. Tutor ratings in Years 1 and 2 were positively related to Year 4 performance in written exams, Year 5 OSLERs, and graduating with Honours, and negatively related to leaving the course. This consistency may depend on HYMS’ use of experienced clinicians as PBL tutors, a role tutors had chosen to undertake as part of a ‘‘portfolio career’’. The combination of early examination results and such tutor assessments could thus be useful for identifying those students who might benefit from targeted professionalism mentoring as well as academic interventions later in the course. The breadth and depth of this study is, to date, unique. The entire cohort of 146 students was tracked through the course, 94% of whom had completed all the initial non-cognitive tests. The small size of the cohort is compensated to an extent by the lack of range restriction in the cognitive and non-cognitive parameters and the completeness of follow-up. The outcome data covered not only examination results and tutor assessments, but also evidence of problematic professional behaviour ranging from minor to serious, collected under HYMS’ system for monitoring fitness-to-practise. Although the large number of relationships studied means that about 1 in 20 will appear significant by chance at the 5% level, their consistency with each other and with the underlying meanings of the constructs leaves us confident in the validity of our findings. This approach has yielded evidence that measures of past academic performance, cognitive skills, and also personality 878

traits that reflect emotional engagement with people, plus a positive disposition, all make weak but significant contributions to the prediction of a range of both desired and undesirable outcomes from the medical degree course. A large percentage of complaints about medical practitioners arise from breakdowns in the doctor’s communication with patients, relatives or colleagues rather than from failings in procedural skills (see, e.g. the recent annual reports of the General Medical Council of Great Britain). Communication is a key skill for a doctor, but is difficult to quantify in students. This prompted us, a priori, to devise various ways of measuring different aspects of students’ communication abilities using the clinical examination data, by breaking down the marks into components directly related to communication skills. The different indices derived from the OSLER and OSCE examinations were thus designed as important investigative tools to delve into this difficult area, rather than an attempt to proliferate statistical indices. We expected to find strong relationships between prior academic achievement and outcomes, given that McManus et al. (2013) found that measures of past academic performance were the main predictors of future academic performance. This was indeed the case: prior academic performance (the HYMS academic score) had significant predictive value for some outcomes. However, none of the other student selection parameters used by HYMS (UCAS personal statement, referees’ reports and interview scores) predicted performance in any of the outcome measures in this study, although of course their use as selection parameters inevitably introduces significant range restriction. In contrast, there was no restriction of range for the cognitive tests and non-cognitive tests as they were not used to select applicants into the cohort. The non-cognitive variables that were found to be significant describe aspects of being able to relate well to other people: empathy, confidence with others, not being aloof, being able to manage one’s emotions, being optimistic and having high self-esteem. This is consistent with the findings of a number of recent studies (e.g., Lambe & Bristow 2010, Haight et al. 2012, Koenig et al. 2013, Simpson et al. 2014), which together suggest that such variables may provide suitable proxy measures for the desired elements of the ‘‘character’’ advocated by Smyth (1946). Negative course outcomes such as leaving the courses or having three or more FTPPPs were related to high scores on the trait anti-social tendencies. The questions that contribute to the anti-social tendencies score reflect a disregard for the laws and norms of one’s society; perhaps, this predicted a disregard for, or intolerance of, the rules and norms of the medical educative experience leading to leaving the course or earning PPs. The variables of moral decision making (individual freedom versus societal rules), and the traits of narcissism, anxiety, irrational thinking, restraint, and permissiveness were found to be unrelated to outcome as were the scores of self-discipline and control. Given the numerous literature reports in both medical education (e.g., Ferguson et al. 2014) and organisational psychology research that describe the importance of conscientiousness and other related traits such as self-control

Downloaded by [Rutgers University] at 00:11 28 September 2015

Predictors of outcome at medical school

and self-discipline, we had expected to find these traits better predictors. The relationships described here differ from those reported when the cohort had completed Year 2 (Adam et al. 2012). For example, in the first two years of medical education the variables narcissism, aloofness and irrational thinking predicted poorer performance, while conscientiousness, confidence and INVOLVEMENT predicted better performance in Year 1 and Year 2 OSCEs. UKCAT overall scores and the decision analysis subtest score also predicted Year 1 and Year 2 overall exam scores. Over the complete medical course (five years) reported here, only the three UKCAT reasoning tests and the traits of INVOLVEMENT (confidence plus empathy) have remained as significant predictors. The demographic factors co-varied with some of the predictor variables, for example females producing higher empathy scores compared to males (as is commonly found in review, e.g. Munro et al. 2005; Wright et al. 2014) even though there was no evidence of gender difference in prior academic achievement. The likelihood that demographic factors (age, gender, citizenship) were confounding variables is supported by their accounting for additional variance in examination outcomes over and above that due to ability and personality variables (Table 5). The present study demonstrates that it is possible to identify attributes and qualities, or combination of attributes and qualities, which might predict outcomes in the academic and clinical domains from a medical education. As the ultimate purpose of our research is to assist medical school selectors, we have also addressed the question of whether the preexisting selection method (scrutiny of application form and interview), other standardised selection tests of cognitive abilities and non-cognitive traits, intermediate indicators from the early medical school years (e.g. tutor evaluations) or demographic variables can reliably predict those more likely to achieve desirable versus undesirable outcomes. In summary, our findings show that the qualities that contribute to good outcomes from a medical course are academic ability, reasoning ability and a stable and positive personality. This study is the first to show that measurement of non-cognitive traits, in combination with measures of academic and cognitive ability, can predict desirable (and undesirable) outcomes from medical education. Refinement of the specific traits and their measurement, together with medical schools developing assessments in the non-cognitive domains of medical education, are now the challenges for future researchers in medical education. There is, after all, an ethical imperative to ensure we choose the best candidates to educate as future doctors.

Notes on contributors JANE ADAM, MA, PhD, MB, BChir, MPH, FFPHM, was an Associate Dean for Admissions and a PBL tutor at Hull York Medical School from 2003 to 2011, with a research interest in methods for selecting medical students. MILES BORE gained his PhD in Psychology in 2002, is a registered psychologist and a senior lecturer at the University of Newcastle, Australia. His teaching and research interests are in the areas of psychometrics,

personality, moral orientation and the selection of applicants to health professional education. ROY CHILDS, BSc, PGCFE, AFBPsS, is the Managing Director of Team Focus and a Chartered Occupational Psychologist. He works as a coach, facilitator, trainer and researcher and his main focus is building sustainable relationships that enhance well-being and performance. His challenge to orthodox thinking is embodied in an innovative range of psychometric instruments. JASON DUNN studied for a PhD in Human Sciences at the Hull York Medical School, graduating in 2013. He remains an honorary research fellow of HYMS. JEAN McKENDREE, PhD, is a senior lecturer in medical education at Hull York Medical School. She is a cognitive psychologist whose research involves applying cognitive science principles to improving educational theory and practice. DON MUNRO, PhD, is a former staff member and now conjoint associate professor in the School of Psychology, University of Newcastle. His interests are in personality and motivation, psychometrics and selection testing. DAVID POWIS, PhD, has been a university teacher of, and researcher in, physiology and medical education since 1972. He is currently conjoint professor in the School of Psychology at the University of Newcastle. Over the past three decades, he has worked particularly in the area of medical student selection with the aim of establishing fair principles and appropriate strategy for selecting students for health professional courses.

Acknowledgements We are grateful to Professor Barry Wright for allowing us to use the simulated patients’ ratings of empathy. We express grateful thanks also to Emeritus Professor John Cookson and to Professor Jonathan Bennett both of whom read and commented on earlier drafts of the article. Declaration of interest: Drs. Bore, Munro and Powis are joint authors of the Personal Qualities Assessment battery of tests and receive royalty payments when the PQA is used commercially. Mr. Childs is the owner/manager of Team Focus who are the commercial publishers of the RSQ. Dr. Adam was an unpaid member of the UKCAT executive board from 2005 to 2010. Dr. McKendree and Dr. Dunn report no declarations of interest.

References Adam J, Bore M, McKendree J, Munro D, Powis D. 2012. Can personal qualities of medical students predict in-course examination success and professional behaviour? An exploratory prospective cohort study. BMC Med Ed 12:69ff. Barr DA. 2010. Science as superstition: Selecting medical students. Lancet 376:678–679. Best J. 1989. The politics of the sand pit. Med J Australia 150:158–161. Bore M, Munro D, Kerridge I, Powis D. 2005. Selection of medical students according to their moral orientation. Med Ed 39:266–275. Bore M, Munro D, Powis D. 2009. A comprehensive model for the selection of medical students. Med Teach 31:1066–1072. Campbell EF, Rank BK, Sinclair AJM. 1974. Selection of medical students – A burning question. Med J Australia 1:785–788. Childs R. 2012. [Accessed 28 January 2015]. Available from: http:// www.teamfocus.co.uk/tests-and-questionnaires/understanding-motivation/resilience-scales.php. Cookson J, Crossley J, Fagan G, McKendree J, Mohsen A. 2011. A final clinical examination using a sequential design to improve costeffectiveness. Med Educ 45:741–747.

879

Downloaded by [Rutgers University] at 00:11 28 September 2015

J. Adam et al.

Cooper A, Petrides KV. 2010. A psychometric analysis of the Trait Emotional Intelligence Questionnaire-Short Form (TEIQue-SF) using Item Response Theory. J Person Assess 92:449–457. Ferguson E, James D, Madeley L. 2002. Factors associated with success in medical school: Systematic review of the literature. BMJ 324: 952–957. Ferguson E, James D, O’Hehir F, Sanders A. 2003. Pilot study of the roles of personality, references, and personal statements in relation to performance over the five years of a medical degree. BMJ 326:429–432. Ferguson E, Semper H, Yates J, Fitzgerald JE, Skatova A, James D. 2014. The ‘dark side’ and ‘bright side’ of personality: When too much conscientiousness and too little anxiety are detrimental with respect to the acquisition of medical knowledge and skill. PLOS One 9(2):e88606, 1–11. Goodenough Committee. 1944. Report of the Interdepartmental Committee on Medical Schools, HMSO, London, p 106. Cited in Smyth DH 1946. Gray SA, Deem LP, Straja SR. 2002. Are traditional cognitive tests useful in predicting clinical success? J Dental Educ 66(11):1241–1245. Groves M, Gordon J, Ryan G. 2007. Entry tests for graduate medical programs: Is it time to rethink? Med J Australia 186:120–123. Haight SJ, Chibnall JT, Schindler DL, Slavin SJ. 2012. Associations of medical student personality and health/wellness characteristics with their medical school performance across the curriculum. Acad Med 87: 476–485. Kelly ME, Regan D, Dunne F, Henn P, Newell J, O’Flynn S. 2013. To what extent does the Health Professions Admission Test-Ireland predict performance in early undergraduate tests of communication and clinical skills? – An observational cohort study. BMC Med Ed 13: 68ff. Koenig TW, Parrish SK, Terregino CA, Williams JP, Dunleavy DM, Volsch JM. 2013. Core personal competencies important to entering students’ success in medical school: What are they and how could they be assessed early in the admission process? Acad Med 88:603–613.

880

Lambe P, Bristow D. 2010. What are the most important non-academic attributes of good doctors? A Delphi survey of clinicians. Med Teach 32: e347–e354. Lancet [editorial]. 1984. Medical student selection in the UK. Lancet 2:1190. Lockhart LB. 1981. Why aren’t they choosing the right candidates for medicine? Lancet 7:546–548. Lynch B, MacKenzie R, Dowell J, Cleland J, Prescott G. 2009. Does UKCAT predict year 1 performance in medical school? Med Ed 43: 1203–1209. McKinley DW, Norcini JJ. 2014. How to set standards on performance-based examinations: AMEE Guide No. 85. Med Teach 36(2):97–110. McManus IC, Woolf K, Dacre J, Paice E, Dewberry C. 2013. The academic backbone: Longitudinal continuities in educational achievement from secondary school and medical school to MRCP (UK) and the specialist register in UK medical students and doctors. BMC Med 11:242ff. Munro D, Bore M, Powis D. 2005. Personality factors in professional ethical behaviour: Studies of empathy and narcissism. Australian J Psychol 57: 49–60. Petrides KV. 2009. Technical manual for the Trait Emotional Intelligence Questionnaires (TEIQue). London: London Psychometric Laboratory. Poole P, Shulruf B, Rudland J, Wilkinson T. 2012. Comparison of UMAT scores and GPA in prediction of performance in medical school: A national study. Med Ed 46:163–171. Schuwirth L, Cantillon P. 2005. The need for outcome measures in medical education. BMJ 331:977–978. Simpson PL, Scicluna HA, Jones PD, Cole AMD, O’Sullivan AJ, Harris PG, Velan G, McNeill HP. 2014. Predictive validity of a new integrated selection process for medical school admission. BMC Med Educ 14:86. Smyth DH. 1946. Some principles in the selection of medical students. BMJ Sep 14:357–367. Wright B, McKendree J, Morgan L, Allgar VL, Brown A. 2014. Simulated patients’ ratings of empathy in medical student final year clinical examination. Are they useful? BMC Med Educ 14:199.