Real-time emotion recognition pipeline in videogames using physiological signals from a wearable device and facial action labels

Abstract

Advancing real-time emotion recognition in dynamic, naturalistic environments is pivotal for applications in gaming, mental health, and personalized education. This study introduces and evaluates an intelligent emotion recognition pipeline using physiological data from a wearable device, designed for this purpose. This system integrates continuous physiological sensing via the Empatica EmbracePlus wristband. Data were collected from 25 university students engaged in naturalistic gameplay across three commercial video games, forming the basis of the PaGER-Sync ADICVIDEO dataset. Emotions were labeled at a frame-level using FaceReader, providing high-granularity. The proposed classification pipeline employs a Random Forest classifier trained directly on short, lagged windows of raw physiological signals (EDA and BVP), eliminating the need for handcrafted feature extraction and achieving sub-50 ms inference latency, confirming suitability for real-time deployment. Addressing class imbalance with SMOTE-based data augmentation, the system achieved a robust performance, yielding 86.01% accuracy and a macro-averaged AUC of 0.98 across six discrete emotions. Model interpretability, crucial for building trustworthy intelligent systems, is quantified through SHAP-based analyses, revealing the individual contributions of physiological features. These compelling results demonstrate the feasibility of non-invasive, privacy-preserving, real-time emotion recognition using minimal wearable sensors, thereby supporting the development of scalable, interpretable, and computationally efficient affective computing applications in diverse real-world contexts.