MelGAN-Waveform-synthesis

Pytorch re-implementation of MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis.

What is Text To Speech (TTS)?

• Voice synthesis via converting text input into audio output

• Audio modeling

  Stage 1. Models the intermediate representation given text as input

  Stage 2. Transforms the intermediate representation back to audio (i.e., vocoder)

• Representation

  • Typically chosen to be easier to model than raw audio while preserving enough information to allow faithful inversion back to audio

Basic Knowledge

• Mel-spectrogram (example data from LJ Speech dataset [LJ001-0001])

• LJ Speech dataset:

  This is consisting of 13,100 short audio clips of a single speaker reading passages from 7 non-fiction books.

  Clips vary in length from 1 to 10 seconds and have a total length of approximately 24 hours.

MelGAN

• Generator

  • The generator is a fully convolutional feed-forward network with mel-spectrogram as input and raw waveform as output.
  • Stack of transposed convolutional layers are used to upsample the input sequence and each trnasposed convolutional layers is followed by a stack of residual blocks with dilated convolutions.

• Discriminator

  • Multi-scale architecture with three discriminators that have identical network structure but operate on different audio scales are adopted.
  • This structure has an inductive bias that each discriminator learns features for different frequency range of the audio.