Accepted Type
Oral
Code
OG2-3-3
Was this work accepted for CCME 2020?
no
Category
General Call (Workshop, Oral Presentation, Poster Presentation)
Type
Oral
Sub Type
Education Research
Will the presenter be a:
Other
Presenter Other
Scientist
Affiliation
Considered for Poster
no
Title
Validity Evidence in Support of Work Place Based Assessment Programs
Length of Presentation
Background/Purpose
Designing workplace-based assessments (WBA) in competency based medical education involves translation of outcome frameworks to assessment plans. Appreciation for the real-world contexts in which they are enacted is also essential. Numerous validity threats exist and conducting validity evaluations can be challenging in these settings, leaving validity more intended than certain. In this study we examined the validity evidence of a novel dimension-based WBA model in emergency medicine (EM).
Methods
Faculty translated the EM outcome framework into a dimension-based assessment model that used colloquial language to better align the tool with how faculty naturally think and talk about practice and competence. After implementation we applied Kane's validity framework (using retrospective data), focusing on the generation of assumptions, translation to hypotheses, alignment of data collection and analyses, followed by interpretations/inferences.
Results
We extracted 3,059 assessments over 2 years for 20 residents, 413 faculty and 7 sites at one institution. We tested 8 assumptions related to the "scoring" inference and found: (1) learners are assessed across multiple observers; (2) observers use range of scale; (3) observers differentiate between learners; (4) observers avoid common rater biases; (5) site is not a source of bias; (6) learner scores demonstrate predictable trajectory; (7) learner data is appropriately convergent/divergent (i.e., dimension-based models augment EPA data).
Conclusion
A WBA that uses colloquial dimension-based language, translated from outcomes frameworks, demonstrates numerous sources of validity at the "scoring" inference level. A model for examining validity is demonstrated. Future research will need to explore assumptions related to other inferences in Kane's framework.
Keyword 1
Assessment
Keyword 2
Validity
Keyword 3
Competence
Level of Training
Post Graduate
Abstract Themes
Assessment
Assessment
- Psychometrics & Measurement
- Work-place based
Additional Theme (First choice)
Assessment
Additional Theme (Second Choice)
Postgraduate
Additional Theme (Third Choice)
Undergraduate
Authors
Presenter
Walter Tavares
Term 1
Yes
Term 2
Yes
Term 3
Yes
Term 4
Yes
Term 5
Yes
