Spanish End-to-End Empathic HCI

By: Alex Mares

Summary of best results and SOTA comparison
The architecture follows an ASR → SER → Retrieval → Reasoning LLM → TTS pipeline.

This study presents the first fully integrated empathic HCI pipeline in Spanish, combining ASR, multitask SER, protocol-driven reasoning via LLMs, and emotionally expressive TTS.

Read the full paper on MDPI

System in Action

Audio Input
Voice input
→
ASR output
Whisper transcription (ASR)
→
Emotion detection
Emotion recognition (Multitask SER)
→
TTS Audio
Retrieval
→
LLM response
Assistant's response (LLM)
→
TTS Audio
Emotional speech output (TTS)
▶ Try the HCI Demo on Gradio View the Code on GitHub
← Back to Main