Beyond Hopes and Dreams: AI-enhanced Feedback in Support of Blended Learning

Blended learning could theoretically mean any combination of educational methods to meet a learning objective. Definitions typically include face to face blended with asynchronous or synchronous e-learning.^1,2 For this discussion we refer to anything that blends simulation with pre- or post-simulation methods (typically e-learning). Caveat: most studies show gains in knowledge, not in skill acquisition or translation to practice.² Blended learning might also simply be called curricular integration, enable aspects of mastery learning, or present an opportunity for repetitive practice, all of which have been demonstrated to have value in learning.³

Blended learning advantages:

Blended learning typically creates at least one spaced repetition, which has proven beneficial across online, simulation, and classroom settings.⁴ And spaced retrieval in health professions education specifically shows pushing retrieval intervals up to 29 days improves long-term memory outcomes compared with shorter intervals.⁵

Also, because simulation time tends to be resource intensive, pre-training theoretically maximizes the utility of expensive simulation lab and faculty instructor time. The asynchronous components can be assigned and done at the learner’s pace and preference of timing. It allows for theory and background knowledge learning objectives to be met so that procedural or team skills can be practiced using psychomotor and communication skills on site in a faculty-led environment.

Barriers to blended learning in simulation include:

• Faculty and coordinator (sim or course or both) time demands for logistics and presence, along with trainee time to complete pre-work, simulation, and post-simulation follow-ups.
• Greater difficulty in curricular integration within the clinical curriculum as opposed to the pre-clinical curriculum (fewer logistics resources, competition with clinical demands of faculty, etc.). These challenges can be even greater for interprofessional efforts.⁶
• The technical challenge of linking between learning management systems (LMS) and what is happening in the simulation space. Likewise, the Simulation Information Management Systems (or Virtual Simulation Systems) may not connect back to the larger LMS.

Examples of success and factors to consider in implementation:

We searched for blended learning incorporating any method of simulation (actors, virtual patients, manikins, task trainers) to find just a few good examples of blended learning.

Advanced Cardiac Life Support (ACLS) is taught with online virtual knowledge checks and simulations followed by in person simulations to demonstrate teamwork including taking lead on at least one scenario.⁷ The Neonatal Resuscitation Program (NRP) of the American Academy of Pediatrics similarly pairs online testing and case-based work with an instructor-led event for hands-on practice and debriefing.⁸

Outside the certification-course model, a smaller set of published curricula reports learner outcomes. Reed and colleagues applied a flipped simulation model to six core clinical skills during a senior emergency medicine clerkship: 135 students were pretested, viewed online videos asynchronously followed by a computer-based quiz, received one-on-one deliberate practice with feedback, and were post tested repeatedly until a preset minimum passing standard was met.⁹ The same group extended the model to interprofessional education with nursing and medical students, reporting gains in self-efficacy, TeamSTEPPS® knowledge, and team performance in a simulated patient care environment.¹⁰

Debriefing and Feedback:

Most debriefing contains feedback, but many times feedback is given outside of a debriefing format (e.g. note grading, physiology logs, actor feedback, etc.).

There is consensus in the simulation community that debriefing is important, should be done by those trained in communication methods, and may benefit from following a structured framework.¹² Success with any debriefing framework needs to be framed within a rich description of context. To date, the evidence favoring any one feedback or debriefing method over another remains limited,¹³ and the characteristics that would let us compare them are inconsistently reported in the primary studies.¹⁴

Technology assisted debriefing, using methods such as video review and annotation, has been limited by many factors including faculty comfort with the technology, and time for annotation. Outcomes of video-assisted debriefing have been mixed.^15–17 This is likely to change as AI systems begin to inform the debriefing process through video and transcript review, along with information coming from simulation systems (monitors, manikins, etc.).

Scheduled reinforcement is a third route: low-dose, high-frequency programs replace biennial certification with brief quarterly manikin sessions delivering real-time performance feedback, with reported improvement in psychomotor compression and ventilation performance and a decreasing number of attempts needed to pass across quarters.²² The online components can also close the loop immediately, as when a knowledge check flags an incorrect answer and links the learner directly back to the relevant content.⁷

The challenge to this point is that formative feedback on individual learner patient notes, written reflections, or conversation transcripts gets delayed to allow faculty time to grade (apart from some virtual patient systems). AI-enhanced systems now allow for immediate or near immediate feedback depending on the amount and method of faculty oversight. Even with automated grading, seeing group performance on simulations requires a cohort to complete the simulation and the other blended associated activities. Modern AI-enhanced grading systems now allow this early aggregate view to create data-informed group feedback that can be woven into debriefing.

Feedback Integration:

Feedback from AI-enhanced systems with faculty oversight to ensure optimal grading represents a rich new area for exploration. Such systems should seamlessly transfer data into simulation information management systems and/or larger institutional learning management systems. Doing so allows direct curricular linkage for easier creation of blended learning and competency tracking, to name just two benefits.

Conclusions

There is evidence that blended learning works. Systems that work to support easy curricular integration are needed. Such systems should ideally support debriefing in the moment and individual reflection after simulation. Feedback should address the most salient issues and ideally be individualized or at least relevant to that team’s shared experience. Modern AI-enhanced feedback systems will enable richer feedback from integrated blended learning with linkage to curriculum and competency tracking, so that we can move beyond hopes and dreams.