Bug in preparing latents

[OrangeSodahub]

Hi, thanks for your great contributions! When I train this model myself, there might a bug in your script:
According to:
https://github.com/huggingface/diffusers/blob/df1d7b01f18795a2d81eb1fd3f5d220db58cfae6/src/diffusers/pipelines/cogvideo/pipeline_cogvideox_image2video.py#L391-L396
CogVideoX model should not involve the vae_scaling_factor in latents, also this script is used in DaS's inference (without scale factor); however DaS let the latents multiplied with this factor in training (which is misaligned with the inference):

DiffusionAsShader/training/cogvideox_image_to_video_sft.py

Lines 892 to 894 in 897fc58

	image_latents = image_latent_dist.sample() * VAE_SCALING_FACTOR
	image_latents = image_latents.permute(0, 2, 1, 3, 4) # [B, F, C, H, W]
	image_latents = image_latents.to(memory_format=torch.contiguous_format, dtype=weight_dtype)

This way, the output video will have the first frame (reference) slightly brighter than the subsequent frames.
An elegant solution is to remove this multiplication in training script.

Consider the results of DaS are good, I think that is because you only train the side branch? But once someone train the pretrained parameters (main branch) under this condition, it will hurt the results greatly.

[skygoo2000]

Thank you very much for your feedback. We will fix this bug within 24 hours, and we sincerely welcome further suggestions and oversight. If you discover any other issues, feel free to submit a PR—let’s work together to make DaS even better.

Additionally, in the past 24 hours, we have implemented the following updates and bug fixes. You’re welcome to pull the latest changes:

1. We’ve added 4D camera control—now you can re-frame videos in Camera Ctrl (refer to the GIF example in the README for details).

2. We’ve integrated CoTracker3 for providing tracking videos.

3. Regarding updates to the MoGe submodule, we’ve revised the loading method to prevent certain crashes.