Shouldn't fix_noise_scheduler_betas_for_zero_terminal_snr be applied BEFORE prepare_scheduler_for_custom_training?

[67372a]

Hello,

Reference code:

sd-scripts/train_network.py

Line 195 in 0778dd9

def get_noise_scheduler(self, args: argparse.Namespace, device: torch.device) -> Any:

I noticed that fix_noise_scheduler_betas_for_zero_terminal_snr modifies the alphas_cumprod for the noise scheduler, and prepare_scheduler_for_custom_training uses alphas_cumprod to calculate all_snr. The issue with the current order is that all_snr will reflect NON-ztsnr snrs, not ztsnr, causing anything that uses all_snr to use snrs that do not align with ztsnr.

As such, shouldn't fix_noise_scheduler_betas_for_zero_terminal_snr always be applied before prepare_scheduler_for_custom_training?

Thank you.

[67372a]

sd_snr.ods

Here is a sheet with both snrs dumped from the cache, can see they start varying significantly as they approach timestep 999.

[67372a]

To clarify impact, all_snr is used by the following standard features that I know of:

• min snr gamma
• debiased loss
• scale_v_prediction_loss_like_noise_prediction
• v_prediction_like_loss

In addition, for those using something like my personal fork:

• edm2 loss weighting
• sangoi loss
• laplace timestep sampling
• Any snr based scheduling that uses the all_snr cache