Submission ID 69448
| Code | OA 3-5 |
|---|---|
| Category | General Call (Workshop, Oral Presentation, Poster Presentation) |
| Type | Oral |
| Sub Type | Education Research |
| Will the presenter be a: | Other |
| Presenter Other | Psychometrician |
| Title | Improving Fairness and Equity in a Situational Judgement Test: An Alternate Response Format |
| Background/Purpose | Gaining admission into medical school is a difficult task, particularly for minority applicants who face additional barriers with admissions assessments that often favour majority groups. While situational judgement tests (SJTs) commonly exhibit smaller demographic differences in test performance relative to other admissions assessments, evidence suggests that changing from typed response (TR) to audio-visual response (AVR) format may further reduce these differences. Therefore, we aimed to determine whether changing an open-response SJT (Casper) from TR to AVR would reduce demographic differences for a sample of medical school and health science program applicants. |
| Methods | 1,947 participants completed a twelve-scenario TR Casper test followed by two voluntary AVR scenarios. Mean scores for TR and AVR scenarios were calculated for each demographic group. Group means were compared and effect sizes reported using Cohen's d. Applicant feedback was also collected via surveys and interviews to inform future use of AVR. |
| Results | When comparing performance of White applicants to Black or African American applicants, AVR produced the lowest group difference of d=0.03. This is compared to the group difference produced by TR (d=0.78). The change to AVR also reduced group differences between native and non-native English speakers (d=0.74 to 0.14), rural and non-rural applicants (d=0.19 to d=0.04), and high- and low-income applicants (d=0.36 to 0.12). |
| Conclusion | Demographic differences in test performance reduced considerably when response format changed from TR to AVR. These results may help to inform how other assessments can be modified to reduce barriers to medical education. Future research will examine psychometric properties and predictive validity. |
| Keyword 1 | Equity |
| Keyword 2 | Situational Judgement Test |
| Keyword 3 | Audio-Visual Response |
| Level of Training | Undergraduate |
| Abstract Themes | Assessment |
| Assessment | Admission/selection |
| Additional Theme (First choice) | Admissions |
| Authors | Colleen Robb, Other Colleen Robb, Other Gill Sitarenios, Other Kelly Dore, Other Cynthia Au Yong, Other Jill Derby, Other Heather Davidson, Other Harold Reiter, Other |
