Submission ID 77931

Code OH-5-2
At the end of this workshop, participants will be able to:
Category Medical Education
Type Oral
Will the presenter be a: Other
Presenter Other Research Technician
Title Proof-of-Concept of Using Natural Language Processing to Score Narrative Assessments of Undergraduate Medical Students.
Background/Purpose There are tens of thousands of narrative statements with undifferentiating sentiment provided for the clerkship in-training evaluation reports (ITERs) in each cohort in our program. Here, we describe a proof-of-concept study on the use of natural language processing (NLP) to effectively review and flag student ITER assessments.
Methods Narrative assessments were collected for the top ten and bottom ten students (determined by ITER scores) within the classes of 2018-2021. Student identifiers were anonymized to prevent coding bias. Using a predetermined set of key words that are associated with "favourable cohort comparison" (FCC; n = 23) and "overall unfavourable cohort comparison" (O-UCC; n = 26) sentiment, narrative comments were coded and categorized using NVivo. Statistical analysis was performed using SPSS.
Results A total of 8859 ITER comments were included in the analysis, where 504 (5.7%) were coded as FCC or O-UCC sentiment. More than half (n = 290; 57.5%) of the coded sentiments belonged to the top students, where 53.6% and 4.0% were FCC and O-UCC sentiment, respectively, while the opposite pattern was true for the bottom students (9.3% and 33.1% were coded as FCC, and O-UCC, respectively). The ROC analysis demonstrated that the words categorized as FCC had an AUC of 0.781 (p < 0.0001, S.E. 0.027, 95% C.I. 0.728-0.834), while O-UCC had an AUC of 0.817 (p < 0.0001, S.E. 0.024, 95% C.I. 0.770-0.864).
Discussion This proof-of-concept study demonstrates the potential use of natural language processing to efficiently and effectively analyze the sentiment of tens of thousands of undergraduate medical student statements.
Keyword 1 Clerkship student evaluations
Keyword 2 Narrative assessments
Keyword 3 Natural language processing
Abstract content most relevant to: (check all that apply) Undergraduate Medical Education
Abstract Track - First Choice AI and Data Science
Authors Irene Ma
Irene Ma
Kevin McLaughlin
Mike Paget
Adrian Harvey
Janeve Desy
Christopher Naugler

Loading . . .
please wait . . . loading