(A) unified voice model for data-efficient singing voice synthesis

Abstract

In this thesis, we propose a unified voice model for singing voice synthesis focusing on data-efficiency. Singing voice synthesis (SVS) is a task that generates natural vocal sounds from lyrics and melody. SVS systems used to be based on concatenative synthesis or statistical parametric methods but they have limitations in synthesizing a natural singing voice. In recent years SVS models based on deep neural networks have made great improvements in synthesizing natural singing voices. To generate high-quality singing voices with deep neural networks, a large amount of data containing high-quality singing voices paired with lyrics and melodies are required. In addition, aligning melody labels to the corresponding singing voice audio is a time-exhausting manual work. Furthermore, there are a limited amount of publicly available singing voice data with melody labels so it is hard to improve a SVS model and synthesize a variety of singers' voice. To address these issues, we propose a unified voice model which can be trained with either speech or singing voice. They distill pronunciation and timbre information from speech data which are much more abundant than singing voice but they can generate singing voice. The first is a melody-unsupervision model which can be trained without melody information and phoneme length information. We evaluate the proposed model on singing voice synthesis and show that it can generate singing voice with low amount of singing voice. In addition, we improve the model by disentangling global timbre to synthesize a variety of singing voice timbre. Our model has the potential to be applied for personalized singing voice synthesizer with a small amount of data.