Accepted Type
Oral
Code
OB2-1-1
Acceptance Declaration
Accept
Additional Information
I declare I have no actual or potential conflict of interest in relation to this program.
Was this work accepted for CCME 2020?
no
Category
General Call (Workshop, Oral Presentation, Poster Presentation)
Type
Oral
Sub Type
Education Research
Will the presenter be a:
Other
Presenter Other
Faculty
Affiliation
Considered for Poster
yes
Title
Validation of a Novel Resident Assessment Tool to Support an Anesthesiology Competency-Based Medical Education Curriculum
Length of Presentation
Background/Purpose
Workplace-based assessment (WBA) is key to a competency-based assessment strategy. Concomitantly with our program's launch of Competence by Design, we developed a new formative WBA, the Anesthesia Clinical Encounter Assessment (ACEA), to assess readiness for independence (i.e. entrustability) for competencies essential for perioperative patient care. This study aimed to examine validity evidence of the ACEA during postgraduate anesthesiology training.
Methods
The ACEA comprises an 8-item global rating scale (GRS), an 8-item checklist, an overall independence rating, and case details. ACEA data were extracted for the University of Toronto anesthesia residents from July 2017 to January 2020. Validity evidence was generated from sources based on the unified theory of validity, including internal structure, relations with other variables, and consequences.
Results
We analyzed 8536 assessments for 137 residents completed by 341 assessors. From generalizability analysis, ten observations (2 assessments each from 5 assessors) were sufficient to achieve the reliability threshold of 0.70 for in-training evaluations. A mean GRS score of 3.65/5 provided optimal sensitivity (94%) and specificity (91%) for determining competency on ROC analysis. Test-retest reliability was high (ICC=0.81) for matched assessments within 14 days of each other. Mean GRS scores differed significantly between residents based on their training level (p<0.0001) and correlated highly with overall independence (0.91, p<0.001). The internal consistency of the GRS (α=0.96) was excellent.
Conclusion
This study provides evidence supporting validity of the ACEA for assessing the competence of residents performing perioperative care and supports its use in competency-based anesthesiology training.
Keyword 1
Anesthesia competency-based training
Keyword 2
Workplace-based assessment
Keyword 3
Entrustment
Level of Training
Post Graduate
Abstract Themes
Assessment
Assessment
- Formative
- Psychometrics & Measurement
- Work-place based
- Competency-based assessment
Additional Theme (First choice)
Postgraduate
Additional Theme (Second Choice)
Additional Theme (Third Choice)
Authors
Presenter
Alayne Kealey
Term 1
Yes
Term 2
Yes
Term 3
Yes
Term 4
Yes
Term 5
Yes
