Designing an augmented mask for enhancing clarity of communication using machine learning

Abstract

Face masks have become increasingly common with the rise of global health issues, such as fine-dust storms and pandemics. By occluding the mouth, face masks potentially degrade the information delivery through the lips. This problem is likely to affect not only face-to-face conversations but also remote video communications. Although numerous studies have demonstrated the key role of visual cues from lips in communication, the audio-visual impact of wearing a mask on speech intelligibility has not been thoroughly studied. In this paper, we aim to quantify the audio-visual impact that masks have on the intelligibility of speech and mitigate this problem with possible technological interventions such as an augmented mask that provides visual cues of lips motion. Through a user study, we discovered that wearing masks significantly decreases speech intelligibility and users' confidence in their level of understanding. We designed the MAScreen, an augmented mask that shows lips motion that is detected with reflective object sensors and machine learning classifier. We evaluate it comparing with original mask following the previous study setting. For better performance with an augmented mask in communication, we suggest points of improvement and possible applications.