Bubble coloring to visualize speech emotion in voice messaging

Abstract

Emotion visualization in text messaging has fascinated HCI researchers for decades. While voice messages are widely used, the emotion visualization of voice messaging remains unexplored. To fill this research gap, we present and explore the concept of visualizing speech emotion of voice messages by altering the chat bubble colors. First, we conducted a formative study that adopted the ``Wizard of Oz" approach to explore the emotional effect of chat bubble colors on voice messaging. Based on the formative study, we conducted our main study to investigate how users respond to color-based speech emotion visualization in voice messaging. We developed a working prototype that allows users to interact with the color-based automatic speech emotion visualization system in the real stage. Our findings show that participants showed general enthusiasm and optimistic attitude towards the method as both senders and receivers. The color-based visualization method delivered a brand new and lively communication experience. Surprisingly, the participants also showed great tolerance for speech emotion recognition failures. Based on the findings, we discussed design considerations for integrating color-based speech emotion visualization into voice messaging and the opportunities for the proposed method in other voice-based communication.