Two years ago, I described a new way we interview applicants for our residency program in which we blind ourselves to test scores before choosing a rank position.
We’ve now matched two classes using this technique and have just completed another interview season for a third.
Here is what happened (in no particular order):
1) We absolutely could not determine a person’s test scores by interviewing them (when scores were finally revealed they often felt surprising, high or low)
2) It was not difficult to construct a match list largely based on non-cognitive attributes
3) The average entering United States Medical Licensing Examination (USMLE) score dropped slightly compared with historical averages, but In-Training Exams (ITE) scores showed the usual gains over this baseline when comparing USMLE percentiles to ITE percentiles. It’s not an exact apple to apples comparison, but our residents gain approximately 30-40 percentile points on average as a class from starting USMLE scores to final year ITE scores in our program (the past four years our graduating classes have scored above the 89th percentile or higher on the ITE). We feel these gains are largely due to applying adult learning theory in daily practice.
4) The residents we matched are incredible, marked by growth mindset and service to others
(Regarding this last point – it’s not that previous classes didn’t have these things, just that we’ve seen the current residents manage COVID with continuous grace, heart, and courage.)
We think we’ve attracted residents with these attributes because we:
- asked about failures and response to failure (we even interviewed some people who had failed at some point in medical school – imagine that!)
- paid special attention to ‘distance traveled’ and met incredible people who persevered in the most difficult times and spaces imaginable
- looked for authentic service to others
Once students attain a certain level of medical knowledge, success in residency often depends on noncognitive attributes, such as conscientiousness, empathy, and grit. And yet, medical schools rarely transmit meaningful information about these things.
My colleagues and I recently wrote a paper entitled What Behaviors Define a Good Physician? Assessing and Communicating About Noncognitive Skills, in which we created a list of noncognitive attributes to help medical schools collect assessment data that can be turned into valuable information.
Here are attributes from this paper we find valuable in our residency program:
- advocacy
- affability
- compassion
- conscientiousness
- courage
- cultural sensitivity
- curiosity
- empathy
- equanimity
- equity
- grit
- growth mindset
- holism
- humility
- open-mindedness
- resilience
- self-awareness
- teamwork
It’s not a perfect list (what do you feel might be missing or over-emphasized?), but it’s a great one nonetheless. When these things are present, things tend to go well.
We try to find evidence for these attributes in our applicants. Finding is an active process. We have to ask the right questions and listen in the right way.
Of course, the term ‘right questions’ is vague and potentially rife with bias. Our interviewers undergo bias mitigation training, we try to make our selection committee as diverse as possible, and no single person gets to decide where people go on the list. This is a work in progress, but we constantly strive to learn and be better.
What happens when a program director doesn’t know your test scores? Two years ago, I didn’t know the answer, but now I do: the program and residents can thrive.
I don’t worry about people with lower scores that also have self-awareness, curiosity, and growth mindset. They will learn what they need to learn if the circumstances allow it. To succeed, programs should have the capability to provide the learning environment these residents need.
I do worry about people with high test scores that lack humility, empathy, or equanimity. Change might be possible, but it’s a lot more difficult than gaining medical knowledge.
Be careful about a false dichotomy here. People with high test scores can also have the attributes above, as surely as people with low test scores might not. The point is, scores can get in the way of looking for the other things.
A number of authors have recently called for medical education to be pass/fail. In the United States, at least one of the major exams has already done that.
My colleagues and I have argued that to be truly successful, efforts like this must make the bar of pass high enough that it makes the best irrelevant, and that ‘good enough’ must not be a pejorative.
Until that day happens, our experience shows that it is possible to find and build great value without knowing test scores ahead of time.
The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The Royal College of Physicians and Surgeons of Canada. For more details on our site disclaimers, please see our ‘About’ page