A Real time Face Emotion Detection System Based on YOLO11

Abstract

Modern Human-Computer Interaction (HCI) trends demand a shift toward emotionally intelligent systems capable of recognizing user affect. Standard computer vision setups typically process environment tasks sequentially, remaining blind to human psychological and physiological variations. This research presents an optimized, end-to-end computer vision engine that unifies a single-stage regression model (YOLO11), a custom multi-layered Convolutional Neural Network (CNN), and dense topological point cloud extractions (MediaPipe FaceMesh) into an asynchronous behavioral analytics framework. The core structural asset of this design centers on a decoupled dual-stream execution loop: ● Stream A utilizes a pruned YOLO11 object boundary network optimized for low-latency face tracking under unconstrained ambient lighting conditions. Once isolated, the sub matrix is normalized and forwarded to a specialized deep CNN to sort the user's expression across seven key human categories: angry, disgust, fear, happy, neutral, sad, and surprise. ● Stream B executes simultaneously on independent hardware threads, establishing a geometric landmark coordinate array consisting of 468 distinct points. By evaluating spatial vectors across this 3D topographical mesh, the system extracts precise physiological indicators, including involuntary eye closures via the Eye Aspect Ratio (EAR), directional gaze trajectories (LEFT, RIGHT, STRAIGHT), mandibular separation tracking for yawn validation, and relative head orientation changes (UP, DOWN, CENTER). These tracking parameters are aggregated by a multi-parametric fusion engine to calculate a continuous attention score from 0% to 100%. Empirical validation shows that this integrated engine achieves a face-bounding accuracy of 95.1%, an expression classification score of 92.4%, and an EAR blink tracking consistency rate of 93.2%. Natively hosted on commercial edge hardware, the system maintains a stable processing speed of 24 Frames Per Second (FPS) with an end-to-end inference latency bounded tightly at 41 milliseconds. This lightweight footprint removes any reliance on external cloud APIs, ensuring data privacy and making the framework highly viable for driver drowsiness detection, smart learning management systems, and automated medical monitoring.