Planned TODOs

[r9y9]

This is an umbrella issue to track progress for my planned TODOs. Comments and requests are welcome.

Goal

• achieve higher speech quality than conventional vocoder (WORLD, griffin-lim, etc)
• provide pre-trained model of WaveNet-based mel-spectrogram vocoder

Model

• 1D dilated convolution
• batch forward
• incremental inference
• local conditioning
• global conditioning
• upsampling network (by transposed convolutions)

Training script

• Local conditioning
• Global conditioning
• Configurable maximum number of time steps (to avoid out of memory error). 58ad07f

Experiments

• unconditioned WaveNet trained with CMU Arctic
• conditioning model on mel-spectrogram (local conditioning) with CMU Arctic
• conditioning model on mel-spectrogram and speaker id with CMU Arctic
• conditioning model on mel-spectrogram (local conditioning) with LJSpeech
• DeepVoice3 + WaveNet vocoder WIP: Support for Wavenet vocoder deepvoice3_pytorch#21

Misc

• [ ] Time sliced data generator?
• Travis CI
• Train/val split
• README

Sampling frequency

• 4kHz
• 16kHz
• 22.5kHz
• 44.1kHz
• 48kHz

Advanced (lower priority)

• Mixture of logistic distributions Experimental: Mixture of logistic distributions #5
• polyak averaging https://discuss.pytorch.org/t/how-to-apply-exponential-moving-average-decay-for-variables/10856 Experimental: Mixture of logistic distributions #5
• Faster generation
• Parallel WaveNet

[r9y9]

At the moment, I think I finished to implement basic features (batch/incremental inference, local/global conditioning) and confirmed that unconditioned WaveNet trained on CMU Arctic (~1200 utterances, 16kHz) can generate sounds like speech. Audio samples are attached.

step80000.zip

top: real speech, bottom: generated speech. The only first one sample of real-speech was fed to the WaveNet decoder as an initial input.

step90000.zip

[geneing]

For reference, these are other wavenet projects I know of:
https://github.com/ibab/tensorflow-wavenet
https://github.com/tomlepaine/fast-wavenet - faster version of the original wavenet paper.

[r9y9]

Other projects I know of:

• https://github.com/vincentherrmann/pytorch-wavenet
• https://github.com/musyoku/wavenet
• https://github.com/dhpollack/fast-wavenet.pytorch
• https://github.com/tensorflow/magenta/tree/master/magenta/models/nsynth/wavenet

[r9y9]

Still not quite high quality, but vocoder conditioned on mel-spectrogram started to work. Audio samples from a model trained 10 hours are attached.

step90000.zip

step95000.zip

[r9y9]

Finished transposed convolution support at 8c0b5a9. Started training again.

[jamestang0219]

Hi, I've already tried to use linguistic features as local features, but I found there might be a problem that linguistic features are based on phoneme class, mel-specs are based on frame class, but the local features of wavenet inputs are based on sample point class.

Here is a case, if a phoneme's duration is 0.25s, and its sample rate is 16k, in order to create the wavenet inputs, I have to duplicate the single phoneme's linguistic feature to int(0.25 * 16000) times as their samples' local features. Do you think my practice is right or not? How do you process the mel-spec features while they are frame class?

Thanks for answering me.

[jamestang0219]

Wavenet can capture the differences even if many samples' local features are same as long as its receptive field is wide?

[r9y9]

@jamestang0219 I think you are right. In the paper http://www.isca-speech.org/archive/Interspeech_2017/pdfs/0314.PDF, they use log-f0 and mel-cepstrum as conditional features and duplicate them to adjust time resolution. I also tried this idea and got reasonable result.

[r9y9]

Latest audio sample attached. Mel-spectrogram are repeated to adjust time resolution. See

wavenet_vocoder/audio.py

Lines 39 to 40 in b8ee2ce

	upsample_factor = quantized.size // mel.shape[0]
	mel = np.repeat(mel, upsample_factor, axis=0)

. In this case upsample_factor was always 256.

step70000.zip

[jamestang0219]

@r9y9 In your source code, you use transposed convolution to implement the upsample process? Have you ever checked which method is better for upsampling?

[r9y9]

@jamestang0219 I implemented transposed convolution but haven't got success yet. I wonder 256x upsampling is hard to train, especially for small dataset which I'm experimenting with now. WaveNet authors reported transposed convolution is better, though.

[r9y9]

wavenet_vocoder/hparams.py

Lines 43 to 47 in 3c9deb1

	# If True, use transposed convolutions to upsample conditional features,
	# otherwise repeat features to adjast time resolution
	upsample_conditional_features=False,
	# should np.prod(upsample_scales) == hop_size
	upsample_scales=[16, 16],

For now I am not using transposed convolution.

[jamestang0219]

@r9y9 May I know your hyper parameters for extracting mel spectrogram? Frame shift is 0.0125s and frame width is 0.05s? If this is your parameters, but why you use 256 as the upsample factor instead of sr(16000) * frame_shift(0.0125) = 200? Any tricks here? Forgive me for many questions:( because I also wanna reproduce tacotron2 result