Have you ever had a GREAT coach? …how would you know? Can we effectively measure someone’s coaching skills? That was the idea behind the tool in today’s study: the Wisconsin Surgical Coaching Rubric (WiSCoR) was created to peer surgical coaching. In this week’s paper, they put their tool to the test to determine its reliably and effectiveness.
What do you think? Can we actually efficiently measure coaching? Click here to listen to the co-hosts discuss!
KeyLIME Session 264
Vande Walle et. al., Development and Assessment of the Wisconsin Surgical Coaching Rubric. JAMA Surg. 2020 Apr 22
Jason R. Frank (@drjfrank)
In his classic and hugely popular 2011 essay in the New Yorker “Personal Best”, surgeon-author-influencer Atul Gawande asks why doctors don’t have formal coaches. Comparing our pursuit of world-class expertise to other human endeavours—athletes, singers, etc—he sets out to find his own, pioneering surgical coach to take his game to a new level. It is a must read.
Have you had an amazing coach? In what part of your life? Was it a coach to enhance your medical expertise? Was your coach “good”? Effective? How would you know? It is nearly a decade since Gawande recruited his coach, so have we progressed in our #meded coaching enterprise?
Enter Dr. Vande Walle & co from the University of Wisconsin Department of Surgery…They designed a tool to measure the performance of a surgical coach, the Wisconsin Surgical Coaching Rubric (WiSCoR). In this study, they set out to “evaluate the validity” of the WiSCoR to assess peer surgical coaching.
Key Points on the Methods
The authors defined peer surgical coaching as “an approach to continuous professional development that uses adult learning theory to support a surgeon’s individual performance improvement…” via pairing surgeons with a trained surgeon coach. The process involved collaborative analysis of practice, and constructive feedback for enhanced performance. Core competencies were identified through the development of the coaching programs. The authors shared that they used Cook & Messick as their validity framework to evaluate the novel assessment tool called WiSCoR. [Reminder: “Validity arguments” are made across multiple domains of evidence: Content, Response Process, Internal Structure (reliability), Associations with other variables, and Consequences.)
The authors assessed the performance of surgical coaches in 2 statewide surgical coaching programs (Wisconsin Surgical Coaching Program & Michigan Bariatric Surgery Collaborative Coaching Program) between Nov 2014-Feb 2018, as rated on the WiSCoR. There were 8 (35%) coaches in the Wisconsin and 15 (65%) in the Michigan programs, respectively. The programs coached 38 surgeons during the study period.
Sessions were video recorded and transcribed so that multiple raters could score the coaching performances.
Using Messick and Cook, they described how they developed the “content evidence” of the instrument: literature review, field observations, and interviews of coaches in various fields. These were then reviewed by surgical experts and refined. The subsequent 4 domains of the WiSCoR include:
- Shared responsibility & equal exchange
- Uses guided self-reflection
- Provides constructive feedback
- Guides goal setting & action-planning.
Each domain was rated 1-5, with 5 the level expected of a professional coach.
Validity of ratings was assured via rater training, scoring discussions, and scoring calibrations.
Mean score using the instrument was 3.23 (range 1-5). 282 WiSCoR scores demonstrated high interrater reliability (Gwet weighted agreement coefficient) and modest correlation with coachee ratings (r=0.22) over 106 sessions.
The authors found no other measure to correlate with except coachee ratings.
Consequential evidence consisted of providing feedback to the coaches via sharing the scores.
The authors conclude…The WiSCoR is a reliable and valid measure of surgical coaching.
Spare Keys – other take home points for clinician educators
- It is nice to see the use of Cook/Messick/Kane as a contemporary validity framework
- Note that it is difficult to publish a complete set of “validity arguments” in a single paper using such frameworks
- Coaching is an emerging area of continuing professional development
Access KeyLIME podcast archives here