This project aims to develop a multimodal emotion recognition system that integrates audio (speech) and visual (facial expression) cues to accurately detect and classify human emotional states. The research will leverage deep learning techniques—specifically convolutional neural networks (CNNs) for facial analysis and recurrent neural networks (RNNs) or transformers for speech prosody analysis. By fusing features from both modalities, the system aims to achieve higher accuracy and robustness compared to single-modality approaches.

Potential applications include human-robot interaction, mental health monitoring, and intelligent virtual assistants. Students will work with pre-collected datasets (e.g., RAVDESS, AFEW) and may have the opportunity to collect custom data using lab equipment.