One-shot multi-speaker text-to-speech using RawNet3 speaker representation

Abstract

The recent advancements in Text-to-Speech (TTS) technology have significantly improved the Speech Quality and Naturalness of synthesized speech, reaching a level where it can produce more naturalsounding voices resembling human speech. Consequently, TTS systems find applications in various fields such as automated response system (ARS), mobile phone voice assistants, AI tutors, advertisements, movie or content dubbing, and even in the development of models for language disorder therapy. In light of this, there is a growing need for TTS models that can exhibit diverse acoustic characteristics and synthesize voices based on a single speech file (One-shot) of unseen speaker to secure the speaker’s unique characteristics. In this regard, this dissertation proposes a one-shot multi-speaker TTS model, leveraging the FastSpeech2 acoustic model and HifiGAN vocoder, supplemented by an additional speaker encoder. The speaker encoder utilizes a pre-trained RawNet3 model to extract speaker-related information, ensuring that the speaker’s characteristics are incorporated into both the training and synthesis processes. This enables the generation of speech with the unique voice attributes of unseen speakers during training. Objective and subjective evaluations reveal that the proposed model outperforms other comparative models in terms of both Naturalness and Speaker Similarity. Furthermore, this paper extends the proposed approach to include not only an English one-shot multi-speaker TTS model but also a Korean counterpart.