AI, Faculty, and Inter-Rater Reliability Challenges in Healthcare Simulation

Grading clinical communication is ambiguous: did the learner elicit the patient's illness narrative and demonstrate empathic communication? Standardizing these assessments across multiple raters has challenged health professions education for decades. Now, with generative AI entering clinical grading, institutions face a fundamental question: how reliable are AI graders compared to human faculty?

This article argues the answer hinges on what is measured and how, mapping evidence across six related questions and situating it within an emerging AI benchmarking literature. The conclusion is that the near future is not AI versus humans — it is AI calibrated to humans.

1. What's the Best Achievable Inter-rater Reliability (IRR) in Research Settings?

When conditions are tightly controlled — trained raters, calibration sessions, video-based scoring — inter-rater reliability in healthcare simulation can be excellent, with studies reporting agreement above 0.84–0.90 across rater pairings ^[1]. Research using generalizability theory reinforces this, showing that rater variance can be minimized with proper training and enough cases ^[2]. These highly controlled situations require substantial infrastructure investment and are unlikely to reflect what most programs experience day to day.

2. How Much Effort Does That Require?

Substantial, ongoing effort. Rater training works by optimizing standardized tool use and reducing variation in individual preconceptions, but its effects are not permanent. Rater drift — the tendency of raters to shift their scoring standards over time, typically toward increasing leniency due to fatigue, familiarity, or reduced attention to scoring anchors — occurs without ongoing recalibration ^[3,4]. In one controlled OSCE study, Cohen's κ improved from 0.42 to 0.78 following structured faculty workshops ^[5]. In a separate study, repeated structured rater discussions aimed at reaching scoring consensus improved kappa from 0.49 to 0.82 across 40 scored videos ^[6].

In considering alternate raters, research suggests appropriately trained standardized patient actors (SPs) achieve inter-rater reliability comparable to or better than explicitly trained faculty for behavioral checklist items ^[7]. However, for subtler judgments of clinical competency — particularly using rubrics, global scales, or competency frameworks — faculty clinical expertise remains important. Park et al. (2016) demonstrated that trained faculty raters using a locally developed scoring rubric for patient notes achieved reproducible scores with acceptable reliability but required significant faculty training ^[8]. Bond et al. (2023) found that natural language processing (NLP)-based automated scoring of post-encounter patient notes approached faculty-faculty reliability at substantially lower cost, with faculty effort focused on initial machine scoring calibration ^[9].

3. Is It Fair to Compare IRR Across Checklists, Rubrics, and Competency Scales?

No. Assessment formats differ fundamentally in what they ask raters to do, yielding meaningfully different IRR profiles as a result.

A systematic review by Ilgen et al. (2015) analyzed 45 simulation-based assessment studies and found that pooled IRR was similar for global rating scales (GRS: ICC 0.78, 95% CI 0.71–0.83) and checklists (ICC 0.81, 95% CI 0.75–0.85) ^[10]. This may seem counterintuitive — the apparent objectivity of checklists does not yield markedly higher IRR than holistic global ratings. The explanation lies in what each scale measures: GRS showed higher inter-item reliability (0.92 vs. 0.66) and inter-station reliability (0.80 vs. 0.69) than checklists, meaning global ratings generalize better across cases and items, while each checklist is task-specific by design ^[10].

Analytic rubrics that combine structured criteria with behavioral anchors occupy a middle ground. Park et al. (2016) found Cronbach's α = 0.65–0.77 per case for trained faculty raters and patient note rubric ^[8]. Hassanein et al. (2026), studying dental student clinical short-answer responses using a 12-point rubric, found that three calibrated expert raters achieved ICC = 0.84 ^[11] — toward the higher end of what analytic rubrics typically yield for IRR.

The Dreyfus model of skill acquisition occupies a different level of abstraction. Rather than scoring discrete behaviors, raters must locate a learner on a novice-to-expert continuum — a judgment requiring integrative expertise. In one Group OSCE (GOSCE) study setting with trained SPs, Dreyfus-scale kappa values ranged from 0.61 to 0.72 across four years of observation ^[12].

EPA entrustment decisions also sit at the most demanding end of the spectrum. Rather than 'what did they do?', raters must answer: 'would you trust them to do this unsupervised?' A generalizability study of an EPA-based OSCE at the transition to internship found that a moderately reliable score (G=0.76) required 12 cases, more than some OSCE programs typically deploy, and that faculty raters were more reliable than SP raters for this task ^[2]. Real-world entrustment IRR data outside of structured pilots remains sparse.

Table 1 summarizes the IRR evidence by format, including available AI benchmarks. Note that routine production environment educational IRR data is sparse and rarely collected. Thus, estimates are from the pre-rater training state of existing studies.

Table 1: Inter-Rater Reliability (IRR) by Assessment Format and Setting.
Acronym Legend: acc. = accuracy; EPA = entrustable professional activity; G = generalizability coefficient; GOSCE = group objective structured clinical examination; GRS = global rating scale; ICC = intraclass correlation coefficient; κ = Cohen's kappa; LLM = large language model; SME = subject matter expert.

4. What's the Likely IRR in Real Production Education Settings?

Considerably lower. Research standards for rater training are rarely sustained in production environments. Calibration is often a one-time event, and IRR is seldom monitored in normal operations. The research-to-practice gap is real and underacknowledged.

Within medical education, Yoo and Yoo (2024) found that untrained faculty raters using a portfolio assessment tool in a medical school Introduction to Clinical Medicine course achieved ICC = 0.38, improving modestly to 0.44 after excluding extreme outlier raters — well below the thresholds observed in research settings ^[13]. Rater drift without ongoing recalibration is well documented across both clinical and educational assessment contexts ^[3,4].

This matters for interpreting AI benchmarking studies: when AI is compared to 'expert consensus,' the comparison group is research-quality human performance. The more relevant comparison for many programs is everyday faculty grading — and AI may already meet or exceed that standard in certain assessment formats.

5. How Do AI Agents Fare? Intra-Rater and Inter-Rater Reliability

Busch et al. (2025) ^[14] offers the clearest clinical education benchmark to date. Four large language models (LLMs) — GPT-4o, Claude 3.5, Llama 3.1, and Gemini 1.5 Pro — were scored against expert consensus on all 28 items of the Master Interview Rating Scale (MIRS) across 10 objective structured clinical examination (OSCE) cases, using varied methods to train the AI. MIRS is an anchored rubric: for example, Item 3 ('Negotiates priorities and sets agenda') is scored 1–5, with a score of 5 defined as 'The interviewer fully negotiates priorities of patient concerns, listing all of the concerns and sets the agenda at the onset of the interview. The patient is invited to participate in making an agreed plan' ^[14].

The key finding: AI was highly self-consistent (intra-rater reliability) but inter-rater accuracy against expert consensus remained modest: exact accuracy 0.27–0.44, off-by-one accuracy 0.67–0.87 ^[14]. AI scored reliably with itself; it just was not scoring exactly like human experts. Table 2 summarizes additional references, some of which shared exact match, off-by-one match, or other markers of reliability.

Table 2: Selected AI vs. Human IRR Studies in Clinical and Educational Assessment.
Acronym Legend: ASAG = automated short answer grading; CoT = chain-of-thought; COM = College of Medicine; EFL = English as a Foreign Language; ICC = intraclass correlation coefficient; LLM = large language model; MIRS = Master Interview Rating Scale; NLP = natural language processing; OSCE = objective structured clinical examination; SME = subject matter expert; SP = standardized patient; temp = temperature parameter; UME = undergraduate medical education.

The pattern that emerges: AI reliability is highly format- and calibration-dependent, paralleling what we observe with human raters. Performance on simpler, more structured tasks is more readily approximated by AI than holistic clinical communication assessment.

6. Calibrating AI to Faculty: Human-in-the-Loop and the Path Forward

The current state of the art for closing the AI-to-human gap combines three levers: prompt engineering (i.e., better instructions for the AI), few-shot calibration, and selective human re-review. When using off-the-shelf commercial LLMs, prompt engineering and retrieval-augmented generation (RAG) — providing the AI model with relevant reference materials — are the primary calibration tools available. In contrast, fine tuning the AI model itself requires access to model weights that most programs will not have.

Providing the AI model with detailed rubric criteria, worked examples of strong and weak performances, and explicit behavioral anchors substantially improves alignment with human raters ^[14]. This mirrors frame-of-reference training in human rater calibration, which is considered the most effective approach for improving human IRR ^[18].

The SURE (Selective Uncertainty-based Re-Evaluation) pipeline — where repeated LLM prompting identifies low-certainty cases flagged for human re-review — reduced manual grading time by 40–90% while maintaining accuracy ^[19]. This selective human-in-the-loop model represents one practical option, and this area remains ripe for exploration.

How long will human-in-the-loop be needed? Until AI can reliably replicate expert-level holistic judgment — particularly for Dreyfus-scale placements and EPA entrustment decisions — some form of human oversight will remain essential for high-stakes assessment. For lower-stakes formative assessment using checklists and structured rubrics, AI-assisted grading with appropriate calibration protocols may already be appropriate in many programs.

Conclusion

The question is not whether AI can grade clinical performance. In certain formats and with appropriate calibration, it can — sometimes more consistently than untrained human raters in production settings. The harder questions are: consistent with what, calibrated how, and for which decisions?

The IRR literature tells us that reliability is a function of format, training, and context — for both humans and AI. The path forward is not replacement but a structured partnership: AI handles volume and consistency; humans provide the expert judgment that anchors the system. The goal is not merely to ask whether AI reaches the research-quality human ceiling. It is to ensure that whatever grading system we deploy — human, AI, or hybrid — meets the standard the learner and the future patient deserves.