Korean Singing Voice Synthesis Based on Auto-Regressive Boundary Equilibrium GAN

Abstract

Singing voice synthesis is a generative task that involves not only multidimensional controls of a singer model such as phonetic modulation by lyrics and pitch control by music score but also expressive elements such as breath sounds and vibrato. Recently, end-to-end learning models based on generative adversarial network (GAN) have drawn much interest as it requires less domain-specific processing but provides high sound quality. When GAN is applied to the audio domain, it entails several issues: the choice of audio representation to generate, handling temporal continuity between two adjacent outputs, and finding an effective loss metric for the audio representation. In this paper, we propose a Korean singing voice synthesis system that addresses the issues using an auto-regressive algorithm that generates spectrogram with the boundary equilibrium GAN objective. Through the qualitative test, we show the proposed methods are superior to the original GAN objective and non-auto-regressive model. We also show that our proposed method can render natural expressions such as continuous pitch contours and breath sounds.