CJEM Debate Series: #BetterSelection – Medical school acceptance tests select the wrong doctors: We need fewer memorizers and more thinkers and communicators in modern medicine

INTRODUCTION

Paul Atkinson (@eccucourse)

This series of editorials will provide CJEM readers with the opportunity to hear differing perspectives on topics pertinent to the practice of emergency medicine. The debaters have been allocated opposing arguments on topics where there is some controversy or perhaps scientific equipoise.

We continue with the topic of medical school selection, a source of much anxiety for prospective students and medical school faculty alike. Do you ever wonder whether you would be accepted into a medical school through today’s selection process? Do current selection criteria and systems simply carry on the traditional approach of awarding academic success over life experience and broader skill sets? Do the privileged elite who control the system continue to replenish the system with younger versions of themselves? Or has the selection process perhaps veered too far to accommodate those with varied academic and social backgrounds, potentially threatening the ability of future doctors to continue to innovate and deal with increasingly complex problems? Most importantly, is there an optimal way to select doctors? Have we studied enough different models to tackle this issue with an evidence-based approach? Or is it time to give up on our ability to select successful medical students and doctors and instead offer seats at the table through a lottery process?

Dr. John Steeves, the founding Associate Dean of the regional medical campus, Dalhousie Medicine, New Brunswick, makes the argument that medical students should be selected at random, with Dr. David Petrie, provincial lead in emergency medicine and Professor at Dalhousie University, countering that continuing with a structured approach is better.

[Readers can follow the debate on Twitter and vote for either perspective, by going to @CJEMonline or by searching #CJEMdebate.]

For: John Steeves

A lottery is the most appropriate selection tool for entrance to medical school (or a career in medicine).

The history of medical school candidate selection is one of bias and exclusivity, whether intentional or not. If we are to meet the responsibilities of our social contract to identify the best candidates for the study and ultimate practice of medicine, social diversity (not exclusivity) will be key. Leave the search for the psychologically perfect medical student to those who feel that the Holy Grail exists. We need to put our faith in human diversity. The best way to ensure selection diversity is to abandon all of the subjective tools for ranking candidates and use a lottery to select at random from those applicants who have demonstrated the ability to meet the academic rigours necessary for the study of medicine.

One of the time-honoured responsibilities of the profession of medicine is to identify and select those individuals who will be educated to become the future physicians who will address the health needs of society. The history of selection bias by medical schools (by accident or design) in identifying their pool of entering medical students is embarrassing. Selection biases for sexual orientation, gender and specific religions, races, and cultures or ethnicity are an entrenched part of our medical school historical records. These biases have existed, at least up to recent times, in most Canadian medical schools. The schools have tried to diminish bias by diminishing subjectivity in their selection processes, adding more structured approaches and a sophisticated psychometric analysis of evidence-based selection tools. ¹ However, we all can still cite examples of individuals whom we believe would have made excellent physicians but who were not accepted into a medical school, as well as some that compel us to ask: “How did that person ever get accepted into medical school?”

There can be little doubt, based on current candidates, that the diversity of the applicant pool is already limited, for example, financially societally stratified in favour of the “1 per cent” and against marginalized members of society, as well as favouring urban over rural candidates. Admission committee policy requirements for university prerequisites and the insistence on 2 full years of full-time university attendance impact decision-making of potential rural applicants and those requiring the income of regular employment prior to entering a medical school.

Yet the popularity of a career in medicine remains, such that there are many more applicants than seats available in Canadian medical schools even while there are many whose societal circumstances have effectively denied them access to the applicant pool. In spite of the challenges, a transparent, fair selection process that serves society as well as the profession is clearly needed.

As keepers of the public purse, Canadian provincial governments have medical training demands, yet academic requirements for entrance and the process for selection have been left to Canadian faculties of medicine. Process oversight is handled by external independent accrediting bodies such as the Committee on Accreditation of Canadian Medical Schools (CACMS).

There is general comfort in the use of achievement testing, such as a high undergraduate grade point average (GPA), in predicting success on medical school testing and licensure exams. However, “there is concern that the discriminatory power of prior academic attainment may be diminishing as increasing numbers of medical applicants have top grades.” ¹ There is less agreement in Canadian medical schools regarding the use of aptitude test assessments such as the MCAT. Also, there is a rising interest in personality assessment in spite of “a relative dearth of evidence regarding the long-term predictive validity of personality assessment beyond medical school, and that there has been some concern that personality assessment may narrow the diversity of types of individuals entering medical school.” ¹ Indeed, “the areas of consensus for assessment for selection are small in number.” ²

On the other hand, there is a clear consensus that the ability to prepare for and write undergraduate exams should not be the only criteria in selecting for a career in medicine. What those criteria should be, which ones are needed, at what part of the continuum of medical education into independent practice they are essential, and how they should be identified remain unclear. Achievement in different selection methods may differentially predict performance at various stages of medical education and clinical practice.” ¹ Traditional tools such as references, unstructured interviews, and personal statements have largely been dismissed as ineffectual performance predictors. Even if the nine core personal competencies rated as important for medical students ³ and the 87 qualities of successful doctors could be selected for, their predictive validity has not been established.

“There is a gap in research with respect to the long-term follow-up of trainees that links performance on different selection methods with subsequent performance in clinical practice.” ¹ An appropriate balance of preferred characteristics in the individual and the consequences if preferentially selected for is also unknown. ⁴ Furthermore, to presume that the personal characteristics predictive for effective practice at the time of entrance to a medical school will remain constant through the societal and healthcare practice changes of the following 4 decades would seem presumptuous at best.

Admission committees take some comfort in the use of selection tools and processes applied to numerical “ranking” of applicants even while a few have a theoretical assessment basis for the selection approach that they are using. ² Even if there were evidence as to what individual or group of characteristics should be selected, the specificity and reliability of the tools and processes to do so are weak regarding outcome predictability. With the exception of those students at either end of the ranking list, the challenge for most admission committees is that there is minimal evidence-based, discernible differentiation for the remaining bulk of applicants that might indicate which of them possess the ability to be effective medical students and, ultimately, practicing physicians.

One interesting case study highlights the challenge of selecting by ranking in the typical selection process in a U.S. medical school. “Faced with a major physician shortage, the government legislated an immediate seat increase at the medical school from 150 to 200. The decision came after the applicants had been ranked and preferred applicants for offered positions. Fifty lower-ranked students were then admitted. However, a subsequent analysis revealed that the medical school academic performance of the higher ranked students and lower ranked “late acceptance” cohort were statistically identical.” ⁵

We also must bear in mind that the medical student applicant pool is the primary source of residency programs and, ultimately, physician human resources in all locations in which healthcare is provided. So, how should we honestly and fairly select applicants, keeping in mind societal and healthcare future needs while using a transparent process?

I would suggest that all students who have met the objective requirements for admission to a given medical school using the few tools we have with psychometric credibility regarding effectiveness, ease of utilization, acceptability, and cost-effectiveness (GPA, Multi Mini Interview or CASPER interview score, ⁶ and so forth) be placed in a lottery. Where tools have been developed with evidence of clear predictive value or where policies require it (e.g., number of seats assigned to a geographic location or special preparation-“pipeline” programs), the lottery may be “weighted,” using criteria known in advance by applicants.

Although this approach might seem novel in Canada, until 2017, the Netherlands had many years of experience with a lottery selection process. Hubbeling states that “people – and applicants, in particular – should remember that the reliability and validity of all known selection methods remain questionable. Furthermore, one should not underestimate the value of diversity once a minimum academic standard is achieved. It is time to seriously reconsider using lotteries for medical school entry.” ⁷ It will be interesting to see what the Netherlands’ “product” will be like. Now that the lottery has ended, what challenges will they face in using more complex and expensive tools and admission policy changes to prevent the social and financial elitism that so troubles Canadian medical schools?

We could expect that adding a lottery to the process might help address the consequences of unintended selection bias, socioeconomic elitism, and geographic and career choice maldistribution of physicians that have plagued the human resource management of medicine for so long.

We should be under no illusion that those who have benefitted from meritocracy bias in academic career advancement would cry foul with powerful voices, if a lottery were used. However, for most of society, the process would be fairer, more transparent, and academically honest. The use of a lottery would widen career access to applicants who meet the academic requirements predictably linked with career success. At the very least, we would be able to look an applicant in the eye and tell her or him that we did not decide their entrance into a career in medicine based on unreliable selection tools or unproven criteria. Every applicant would be given an equal chance with all other qualified applicants. For the profession of medicine, the resultant human diversity will become our strength, not our challenge, in preparing for an unpredictable future. Perhaps we need to heed Walter Gretzky’s advice to young Wayne: “Skate to where the puck is going not to where it has been.”

Against: David Petrie

Mistakes in medical school selection have been made in the past; can we learn from them, or is a lottery the only answer?

Quit? Give up? Raise the white flag? Throw in the towel? Throw the baby out with the bathwater? That is what we will be doing if we move to a lottery format to select medical students. Although there is evidence that some selection methods (academic records, multiple mini-interviews, and situational judgment tests) perform better than others (traditional interviews, personal statements, and references),¹ it is also true that, in a robust learning model, ⁸ failure is information, and the goal-experience-feedback cycle can improve future performance. In fact, just as we should be selecting for students with the capacity and courage to learn from their mistakes, students with resilience, ⁹ curiosity, ¹⁰ and an ability to adapt to uncertainty and complexity, we must rise to the challenge of continuing to learn, research, and evolve with the patients and populations whom we serve, in selecting for and supporting the development of good doctors.

Interestingly, the Netherlands’ experience with a lottery has created a natural experiment that we are only now learning from. ¹¹ Stegers-Jager reports that, in 2000, 50% of medical students were selected by lottery (weighted for pre-university grades) and 50% by site-specific structured approaches (consideration of pre-university grades was not allowed). The proportion selected by lottery has slowly sunk over the years. Starting in the academic year 2017-2018, the lottery was abolished, and 100% of students are now selected by structured approaches. This does not prove anything, but one of the observations is a higher dropout rate in lottery-chosen candidates (possibly due to the self-selection bias in the application process). The report concluded that medical schools should “carefully think about a combination and weighting of academic and non-academic selection instruments that would fit both the needs of validity and diversity.” ¹¹

The question then becomes: What combination of academic and non-academic selection instruments is the best? My colleague and other commentators ^{12 , 13} have pointed out many of the problems with traditional selection tools, so I will not dwell on those at length here. The peer-reviewed literature is mixed with regards to the reliability and validity of our approaches. ¹ Past academic performance (GPAs) and MCAT scores have been shown to be good (but not perfect) predictors of future academic performance in medical school and at licensing and specialist exams, ¹ but there is less evidence as to whether they predict good clinical care and more relevant longer-term patient and health–system-oriented outcomes. ¹ There is even less evidence indicating how to evaluate non-academic personal attributes and motivational qualities. ¹ It has also been argued that, in our quest to ensure reliability (are we consistently measuring what we purport to measure?), we have lost track of validity (is what we are measuring relevant to our goal?). In fact, one of the biggest problems in evaluating selection processes is that it is hard to test the tests, to establish “what works,” because there has not been a consensus on defining exactly what “what works” means. ¹⁴ How do we define success in the selection process? What is a good doctor? Is it the same now as it was 50 years ago or even 20 years ago? ¹⁵ – or will be 20 years from now? Cleland has recently made three arguments that challenge the status quo; firstly, we need to broaden our thinking and include other stakeholders in who decides on admissions processes and selections outcomes; next, fundamental systems changes are needed to align input (medical students) with output (meeting the public’s needs); and, finally, that shifting outcomes measures to be more population/systems responsive will have huge implications for selection research. ¹⁵ In keeping with those arguments, what may have been (reasonably) appropriate in the past “may not cut it” today. One hundred years after the influential report by Abraham Flexner entitled, “Medical Education in the United States and Canada: A Report to the Carnegie Foundation,” which brought medicine into the modern scientific era, medical education is again going through an era of reform. More than a decade of analysis and reports has recommended transformational change that includes, but goes beyond, a rational scientific approach to medical care. ¹⁶ Although a level of cognitive capacity is widely deemed to be necessary, it is not sufficient. Scientific reductionism has led to incredible successes in medicine, but it may also contribute to putting less emphasis on the importance of empathy and interpersonal skills, treating diseases and organs rather than whole patient care ¹⁷ and, at a systems level, unintentionally leading to the fragmentation of care-delivery models over integration. ¹⁸

However, this does not mean that we should throw out the (cognitive capacity, scientific understanding) baby with the bathwater and move to a lottery. In fact, the stage theory of critical thinking ¹⁹ suggests that a comfort with “both/and” thinking over “either/or” thinking ²⁰ is a marker of cognitive development. ²¹ What is the best way to evaluate cognitive development in our selection process? Having a good memory for facts and learning how to take tests can often give you good marks in undergraduate courses, but are these the skills that make a good doctor? Pros and cons of the MCAT for Canadian medical schools in the context of the Future of Medical Education in Canada (FMEC) white papers have been argued elsewhere ²² and have provided no definitive answer on their utility. In 2015, the MCAT added a new section on critical analysis and reasoning, which some medical schools now use exclusively as their standardized test component in their selections toolkit. Increasingly, there is evidence that well-designed situational judgment tests ²³ can also be used as part of a structured approach to cognitive capacity.

As we evaluate, iterate, and improve testing on the cognitive side of the coin, there is also significant and increasingly excellent peer-reviewed literature ²⁴ on testing for other personal attributes and interpersonal skills that may increase our ability to identify candidates’ probability of becoming a good doctor. Perhaps one of the most promising areas to consider for more emphasis is the science and practice of developmental psychology. ²⁵ While including these methodologies in a medical school selection bundle is rare, it has been suggested before that we should include testing of moral reasoning. ²⁶ An interesting study in one Canadian medical school used similar methodology in testing first-year medical students and final-year medical students, and found that there was no difference between those groups on a six-point scale, but that the average stage of development in this domain was only moderate on admission. ²⁷

Along with moral reasoning, there are similar models and standardized tests for other domains of development (emotional intelligence, reflective judgment, empathy, and so forth) with varying degrees of evidence. Even more intriguing is the well-argued and studied contention that domain general (rather than domain specific) development can be reliably measured. ²⁸ Imagine the implications of that to medical school selection if it is true – and researchers in this area argue that it is. Given the increasing complexity of medical practice and the significant demands of an interdisciplinary environment, ²⁹ growth and development in many “intelligences” are required to fulfill the CanMEDS roles. ³⁰

At the risk of oversimplification, a useful construct, which may re-frame the standardized testing debate, is the concept of vertical and horizontal development. ³¹ “Neither teachers nor students nor policy-makers nor ordinary citizens fully appreciate the difference between subject matter and discipline.” ³² Yes, subject matter learning (horizontal development) is important and can likely be reliably measured, but the capacity for mature growth and complex thinking within a discipline (vertical development) may be even more important and may be akin to upgrading our Personal Operating System. ³³ If we keep loading new software/subject matter onto a stalled, out-of-date personal operating system, be prepared for bugs, chokes, and dysfunction in performance. If testing for a mature, domain general, stage of cognitive/psychological development is part of a valid way to select for medical students (with the aim of developing good doctors), can we do it in a way that is reliable and non-biased and cost-effective? Good question, and one we should rise to the challenge of in the context of medical school selection and ongoing self-development. There is a growing body of literature ²⁵ and experience from other disciplines (business, military, government), suggesting that we can. ^{33 , 34}

In conclusion, to move forward, any given medical school must balance what is likely to work best in their context and then implement, study, and improve over time. Moving to an exclusively lottery-based system may improve diversity and fairness at the risk of undervaluing cognitive and interpersonal development, and any conscious drive and commitment to mastery of the discipline. However, relying on tests and approaches that have embedded biases and demonstrated problems risks undervaluing fairness and diversity and will perpetuate the identified issues being raised. The answer, it seems, is to improve our testing and approaches based on the improving science in this area, and in line with what all stakeholders suggest makes a good doctor in these rapidly changing times, incorporating minimum standards for both vertical and horizontal developmental scores.

Just as my colleague invokes Einstein’s observation that doing the same thing over and over expecting different results is insanity, Einstein also said: “We can’t solve problems by using the same level of thinking we used when we created them.” I have not heard a more cogent argument for incorporating vertical development testing into medical school selection methods.

Keyword: medical school selection, lottery, debate