Accepted Type
Oral
Code
OF1-3-1
Acceptance Declaration
Accept
Additional Information
Yes, I have/had in the past 2 years, a financial interest, arrangement, or affiliation with one or more organizations that could be perceived as a direct/indirect conflict of interest in the context/content of the subject of this or any other presentation.
Was this work accepted for CCME 2020?
yes
Category
General Call (Workshop, Oral Presentation, Poster Presentation)
Type
Oral
Sub Type
Education Research
Will the presenter be a:
Other
Presenter Other
Psychometrician
Affiliation
Considered for Poster
yes
Title
Gaming the Medical School Application System: Revealing the Coaching Effect Size of a Constructed Response SJT
Length of Presentation
Background/Purpose
Situational judgment tests (SJTs) are increasingly used for medical trainee selection, but the effects of test preparation strategies on these tests, especially open-ended SJTs, has not been explored. The greatest concern is coaching effects, as they threaten score enhancement driven by construct-irrelevant factors like response distortion and test-wiseness, and advantage high SES applicants with access to these resources. This study sought to examine the role of coaching effects on SJT performance.
Methods
We invited test-takers to indicate whether they used any of the following preparation strategies: read the tips for applicants on the test website, completed the free practice test, participated in a commercial test preparation course, studied potential questions based on assessment competencies, rehearsed responses with technology, and rehearsed responses without technology. We conducted a multiple regression analysis to compare the additive effect of each preparation method on SJT scores.
Results
Of the six preparation strategies, only completing the free practice test on the test website (b = 0.16, p < .001), studying potential questions based on the assessment competencies (b = 0.13, p = 0.02), and rehearsing responses with technology (b = 0.18, p < .001) provided significant additive benefit to test scores. Test preparation method only accounted for 2% of the overall variance in test scores (R2 = 0.02, F(6,2887) = 14.44, p < .001).
Conclusion
Results suggest coaching effects are extremely small. These results highlight the importance of ensuring equitable access to practice tests, and relieves concerns over potential socially regressive impact of commercial test preparation.
Keyword 1
Admissions
Keyword 2
Assessment
Keyword 3
Situational Judgement Test
Level of Training
Post Graduate
Abstract Themes
Assessment
Assessment
- Admission/selection
- Psychometrics & Measurement
- Technology
Additional Theme (First choice)
Admissions
Additional Theme (Second Choice)
Assessment
Additional Theme (Third Choice)
Professionalism
Authors
Presenter
Heather Davidson
Term 1
Yes
Term 2
Yes
Term 3
Yes
Term 4
Yes
Term 5
Yes
