CALM: Multimodal Cognitive Load Assessment Framework via Engineered and Explainable Features

Cahree Myrick, Indrajeet Ghosh, Kasthuri Jayarajah, Nirmalya Roy

Abstract: Physiological sensing modalities, such as Electroencephalography (EEG), Galvanic Skin Response (GSR), and Photoplethysmography (PPG), provide valuable insights into cognitive and physiological states. These multimodal signals are applied across diverse domains, including brain-computer interfaces, affective computing, etc., enabling the development of systems to adapt dynamically to an individual’s state, enhancing performance, emotional regulation, and personalization. Similarly, assessing cognitive load using multimodal EEG, GSR, and PPG data presents a promising approach to capturing complex cognitive states. However, challenges remain in developing engineered interpretable features that accurately reflect the cognitive load across the multimodal signals and enhance assessment accuracy. Motivated by this, we propose CALM, a multimodal cognitive load assessment framework that leverages modality-specific engineered features (e.g., GSR features such as Skin Conductance Response Amplitude and Recovery Time, which are highly significant) to enhance robustness and interpretability. The framework incorporates a convolutional neural network for cognitive load prediction, utilizing 21 extracted features across the EEG, GSR, and PPG modalities. Additionally, we employ explainable AI techniques using the shapley additive explanations (SHAP) library, enabling analytical reasoning by examining feature-wise contributions to the model’s predictions. We evaluate CALM on two public and in-house multimodal datasets, and empirical results indicate that CALM achieves 95% macro F1 score in cognitive load assessment.