Multi-speaker and multi-domain emotional voice conversion using factorized hierarchical variational autoencoder

Abstract

Due to the complexity of emotional features, there has been limited success in emotional voice conversion. One major challenge is that conversion between more than two kinds of emotions often accompanies distortion of voice signal. The factorized hierarchical variational autoencoder (FHVAE) [1] was previously shown to have an ability, called sequence-level regularization, to generate disentangled representations of both sequence-level (such as speaker identity) and segment-level features. This study exploits the FHVAE pipeline to produce disentangled representations of emotion, making it possible to greatly facilitate emotional voice conversion. We propose three versions of algorithms for improving the quality of disentangled representation and audio synthesis. We conducted three mean opinion score (MOS) surveys to assess the performance of our models in terms of 1) speaker's voice preservation, 2) emotion conversion, and 3) audio naturalness.