CliniSense AI for automated clinical skills assessment with real-time feedback

Abstract

Medical education requires scalable, reliable assessment of clinical skills, yet faculty observation cannot meet trainee volume, manual OSCE evaluation yields inter-rater reliability of only κ = 0.35–0.58 with days-to-weeks feedback latency, and no existing system provides real-time, competency-aligned assessment from raw clinical audio. We developed CliniSense AI, an end-to-end pipeline, integrating automatic speech recognition, speaker diarization, role detection, multi-label competency classification, and four-dimensional conversation-level scoring, that transforms raw audio into OSCE-aligned resident performance assessments with actionable in-moment feedback on edge hardware without manual transcription. On 270 publicly available simulated consultations, the classification ensemble achieved Cohen’s κ = 0.824 against majority-vote ground truth, exceeding mean annotator agreement ( κ = 0.721), with sub-1.5-s streaming latency. The system surfaced competency patterns invisible to periodic observation: unexpressed diagnostic reasoning in 10.3% of encounters and a quantifiable trade-off between information gathering and patient-centered communication. CliniSense enables faculty to remotely monitor, flag, and approve trainee feedback across multiple examination rooms simultaneously. Prospective clinical validation is underway (IRB #5521) at the University of Tennessee Medical Center.