AudioVAE V2 Training Details Needed

[JeneeeF]

Hi VoxCPM team,

Thanks a lot for the continued work on VoxCPM. In #145 and #175 I saw the full training recipe for the AudioVAE V1(16 kHz output). Those details were extremely helpful to the community.

The latest version Voxcpm2 updated the AudioVAE to V2, so I study the related codes and find V2 realized 48KHz output through the design of asymmetric input-output structure and a SampleRateConditionLayer inserted before every CausalDecoderBlock. The audio quality is very impressive. Since neither the technical report nor the closed issues cover how V2 was actually trained, could you share a bit more? Specifically:

1. About Discriminator
   1.1 Is V2 trained with the same DAC-style (MPD + MRD) as v1, just scaled up to 48 kHz?
   1.2 Were the MPD periods kept at [2, 3, 5, 7, 11], or extended for the higher output SR?
   1.3 Were the MRD STFT resolutions kept the same as DAC's defaults, or retuned for 48 kHz (larger n_fft / different hop sizes)?
   1.4 Any other Discriminators(MSD) used in AudioVAE's training?

2. About Losses and their weights
   2.1 Are the loss weights still: mel=15 / feat=2 / adv=1 / kl=5e-5, or were they retuned for the 48 kHz decoder?
   2.2 Is the multi-scale mel config still n_mels=[5,10,20,40,80,160,320], win=[32,64,128,256,512,1024,2048] or has it been extended (e.g. more scales, higher fmax) for the 48 kHz target?
   2.3 Any additional auxiliary losses used for V2 (STFT L1, envelope, phase)?

3. About Sample-rate conditioning during training
   3.1 The pretrained decoder exposes 4 SR buckets (<20k / 20–30k / 30–40k / ≥40k). What was the sampling ratio across SRs, and how was the 16 kHz → target-SR paired data constructed?
   3.2 Was sr_cond_model trained jointly from step 0, or introduced after a fixed-SR warmup phase?

4. About Data and training schedule for V2
   4.1 Roughly how many hours of speech, at what SR distribution, and how many steps in total for V2?
   4.2 Was V2 initialized from the v1 encoder + a fresh decoder, or trained end-to-end from scratch? If the encoder was warm-started, was it frozen at any point?
   4.3 Any KL warmup / annealing this time, or is kl=5e-5 still a constant throughout training?
   4.4 Did you find any tricks particularly important for stability at 48 kHz (gradient penalty, R1, EMA, warmup for the new sub Discriminator)?

5. About Community reproducibility
   5.1 We fully understand that releasing full training code of AudioVAE may not be considered. But I wanna ask, would it be possible to share
   just the AudioVAE V2 training configuration file (D architecture choices + loss weights + optimizer schedule + SR sampling policy). Because some people may find AudioVAE V2 pretrained model is limited for some special speech reconstruction scenario , that level of detail alone would let the community reproduce or fine-tune V2 reliably, similar to what Inquiry regarding loss functions, hyperparameters, and weights for Causal Audio VAE training #145 did for v1.

Any pointers would be hugely appreciated. Thanks again for the great work!

[Numanor]

Hi, thank you for your concern.

AudioVAE V2 largely reuses the previous training recipe disclosed at #145, just extending from 44.1kHz to 48kHz.
Here are the incremental changes:

1. Increase the constant KL loss weight to 5e-4 for stronger regularization but acceptable reconstruction.
2. We reframe the mel-spec loss as the combination of a normal term and an additional mel_16k loss (i.e., fmax=8000) term to strengthen lower-band fidelity, and assign the loss weights as mel/loss=5, mel/loss_16k=10.

The underlying SR distribution displays a significant discrepancy among various data sources.

1. The valid SR of dialects and low-resource languages hits the roof at 32kHz, with over half of the data under 24kHz.
2. For large-scale languages like English and Mandarin, the roof lifts up to 44.1kHz or 48kHz, with half of the data over 32kHz.

In the end, the new AudioVAE V2 is trained from scratch on tens of thousands of hours of compound datasets, including speech, non-verbal sounds, and songs. The training ends after ~1.5M iterations, where the 1st M steps use exponential lr decay from 1e-4 to 4e-6, and the remaining 0.5 M steps use constant lr with increased data sampling prob from the high-SR datasets.

Thank you again for your feedback!