Keynote Speaker
Dr. Stefanie Sebok-Syer
Dr. Stefanie Sebok-Syer is an Assistant Professor in the Department of Emergency Medicine at Stanford School of Medicine. Prior to joining Stanford, Dr. Sebok-Syer was postdoctoral fellow at the Centre for Education, Research and Innovation at the Schulich School of Medicine and Dentistry at Western University. She received her PhD in Measurement, Assessment, and Evaluation from Queen’s University in Kingston, Ontario. Her research interests broadly include examining psychometrics aspects of performance-based assessments, capturing the interdependence that exists within teams, and developing innovative approaches such as using electronic health record data to support the shift to competency-based medical education.
Contemporary Assessment Challenges: Measuring Interdependence in Healthcare Systems and Teams
Dr. Stefanie Sebok-Syer
Individual and collective competence are required to provide effective patient care in our healthcare systems. But, assessing both types of competence remains problematic because individual assessment disregards the team and team assessment disregards the individual. In order to meaningfully connect medical education and team-based performance with patient outcomes, we need approaches to assess interdependence. Interdependence, as it relates to teamwork, is focused on complex interactions among human and material agents that exist within teams.
In this talk, Dr. Sebok-Syer will present the current conceptualization of interdependence, which has expanded to distinguish between supportive and collaborative interdependence. She will also engage the audience in thinking about opportunities for assessing interdependence within healthcare teams. Finally, she will present data on how faculty currently view interdependence and understand its relationship to other assessments such as entrustable professional activities and milestones.
Education Scholarship Funded Research Presentations
Ilana Bayer
Design and Evaluation of a Virtual Emergency Department Simulation Space
Bayer, Ilana; Shaikh, Iffah; Cai, Charles; Callisto, Demi; Chan, Teresa
Introduction: Simulation-based education is used to develop healthcare professional learners’ skills. However, there are few accessible environments where learners can easily emulate patient care spaces to improve or redesign systems. This project aims to design and evaluate a quasi-immersive, 3D virtual space emergency department where learners can simulate the care of multiple patients and learn and interact with hospital systems.
Methods: A design-based research approach is being used to design the metaverse ED. Focus groups were conducted with stakeholders to guide design of assets and spaces needed for a virtual ED. The assets and designs were used to develop a 3D metaverse ED using the SoWork® platform. Scenario-based focus groups and simulation-based testing of the prototype will be conducted with stakeholders to refine the spaces and equipment and test the platform’s usability. Health professional learners will be recruited to simulate the care of multiple patients and interact with hospital systems. Participant interactions will be evaluated during simulations using direct observation and focus groups will be conducted after the simulations. Thematic qualitative data analysis will be used for stakeholder and participant data.
Results: Stakeholder input and feedback informed the development and refinement of 3D assets for patient care spaces and a blueprint for the metaverse ED. The metaverse ED was built in the SoWork® virtual platform. Stakeholder feedback will be analyzed to refine the metaverse ED and case-based patient scenarios. Observational and focus group data from participants will provide information on virtual simulation interactions, user experience, and affordances and limitations of the virtual ED.
Conclusion: Our user-driven approach will result in a prototype of a metaverse clinical space with the required elements to engage in a multi-patient, interprofessional simulation. This type of virtual simulation space not only allows for medical teaching in a manner that complements video chat with a workspace that should be familiar to students but can also be used for medical care. Other applications for our medical metaverse would be for use in the digital prototyping of clinical spaces/systems before costly physical implementation or renovations.
Irena Rebalka
A comparison of bellringer performance and preference in three visual modalities for the assessment of anatomny knowledge
Rebalka, Irena; Zhang, Yiming; Sun, Jeffrey; Marshall, Kaitlin; Fu, Nicole; Mitchell, Josh; Laprade, Judi; Wainman, Bruce; Nederveen, Joshua; Helli, Peter; Rebalka, Irena
Introduction: A key component in the testing of anatomy knowledge is the use of cadaveric specimens. These evaluations, however, have an intense resource requirement. As a result, many anatomy tests have shifted to using 2D printed specimen images on paper, and 3D specimen reconstructions within virtual reality headsets, as these formats are more resource efficient. The purpose of this study was to investigate differences in testing performance and preference on anatomy bellringer exam questions presented in these three formats – specimens, 2D, and VR.
Methods: Within a mock examination setting, undergraduate students at McMaster University answered 12 Bellringer-style questions, alternating between visualizing anatomical structures of interest on a physical specimen, in 2D, or in VR. All questions were graded as correct or incorrect, and item difficulty and points biserial were subsequently calculated. Each student completed a post-BR survey on their experience, reported cybersickness symptoms, and completed a visuospatial test.
Results: A comparison of testing performance between all three modalities yielded no statistical difference between question difficulty or points biserial. Neither reported cybersickness nor visuospatial ability influenced student test success. Moreover, cybersickness severity was low and lacked correlations across student characteristics. No correlations were found between modality preference and exam performance, both in the aggregate and when examining scores within each modality. Additionally, student perception of modality difficulty was not correlated with overall BR performance or performance within individual modalities.
Conclusion: While additional data will be collected in Spring 2024, the results above suggest no difference in user or question performance in the three evaluated visual modalities in an anatomy BR exam. Furthermore, although BR questions using cadaveric specimens were preferred, this preference did not contribute to test success. Overall, results from this study will enable educators to make evidence-based decisions for incorporating feasible yet effective testing strategies into their anatomy courses.
Sibbald, Matthew
Professionalism lapses: How transformative can they be?
Sibbald, Matthew; Sheth, Urmi; Last, Nicole; Keuhl, Amy; McPherson, Isla; Wojkowski, Sarah; Bakker, Dorothy; Rowland, Paula
Introduction
Health professions training programs face increasing reports of professionalism lapses which can delay, or end, trainee progression. How programs respond to professionalism lapses to facilitate professional identity development has not been clarified. The objective of this study is to identify factors that facilitate and impair transformations around professionalism lapses in health professions training programs.
Methods
We conducted a qualitative study interviewing 5 faculty and 20 students with first or second hand experience with professionalism lapses from a range of health professional training programs at McMaster University. Using direct content analysis, we coded verbatim transcripts informed by the lenses of social and transformative learning theories. We constructed themes through iterative and comparative analysis, seeking meaningful variation across professions and triangulating faculty and student perspectives.
Results
Five themes were constructed. First, lapses are in the eye of the beholder with personal definitions intersecting with institutional and situation norms. Difficulties exist in recognizing and convincing learners to respond to lapses that are minor or subject to interpretation. Second, trainees often provide explanations: unaware of standard, unclear or variable standard, unique or one time situation, miscommunication, suboptimal learning environment and debating evidence. Third, fostering transformation involves building learner confidence, agency, trust and engagement. Focused support and advocacy for students can enhance the ability to tackle disorienting lapses. Fourth, perspective shifts involve deep engagement over time, including but not limited to self reflection, structured discussion and seeking support. Fifth, transformative intent needs to be embedded with a tiered approach involving clinical site leadership with time pressured decisions to identify and respond to safety issues.
Conclusion
Addressing professionalism lapses should contextualize the varied perceptions around lapses due to personal, institutional, and situational factors. While true learner transformation may not be routine, it can be facilitated by building learner confidence, agency, trust, and engagement, and benefit from systematic approaches that include sustained engagement, self-reflection, structured discussion, and ongoing learner support.