Submission ID 77931
| Code | OH-5-2 |
|---|---|
| At the end of this workshop, participants will be able to: | |
| Category | Medical Education |
| Type | Oral |
| Will the presenter be a: | Other |
| Presenter Other | Research Technician |
| Title | Proof-of-Concept of Using Natural Language Processing to Score Narrative Assessments of Undergraduate Medical Students. |
| Background/Purpose | There are tens of thousands of narrative statements with undifferentiating sentiment provided for the clerkship in-training evaluation reports (ITERs) in each cohort in our program. Here, we describe a proof-of-concept study on the use of natural language processing (NLP) to effectively review and flag student ITER assessments. |
| Methods | Narrative assessments were collected for the top ten and bottom ten students (determined by ITER scores) within the classes of 2018-2021. Student identifiers were anonymized to prevent coding bias. Using a predetermined set of key words that are associated with "favourable cohort comparison" (FCC; n = 23) and "overall unfavourable cohort comparison" (O-UCC; n = 26) sentiment, narrative comments were coded and categorized using NVivo. Statistical analysis was performed using SPSS. |
| Results | A total of 8859 ITER comments were included in the analysis, where 504 (5.7%) were coded as FCC or O-UCC sentiment. More than half (n = 290; 57.5%) of the coded sentiments belonged to the top students, where 53.6% and 4.0% were FCC and O-UCC sentiment, respectively, while the opposite pattern was true for the bottom students (9.3% and 33.1% were coded as FCC, and O-UCC, respectively). The ROC analysis demonstrated that the words categorized as FCC had an AUC of 0.781 (p < 0.0001, S.E. 0.027, 95% C.I. 0.728-0.834), while O-UCC had an AUC of 0.817 (p < 0.0001, S.E. 0.024, 95% C.I. 0.770-0.864). |
| Discussion | This proof-of-concept study demonstrates the potential use of natural language processing to efficiently and effectively analyze the sentiment of tens of thousands of undergraduate medical student statements. |
| Keyword 1 | Clerkship student evaluations |
| Keyword 2 | Narrative assessments |
| Keyword 3 | Natural language processing |
| Abstract content most relevant to: (check all that apply) | Undergraduate Medical Education |
| Abstract Track - First Choice | AI and Data Science |
| Authors | Irene Ma Irene Ma Kevin McLaughlin Mike Paget Adrian Harvey Janeve Desy Christopher Naugler |
