Performance Misaligned

[guyuchao]

I test the given checkpoint SIT-XL/2 using command specified in README.
with cfg=1.5: torchrun --nnodes=1 --nproc_per_node=8 sample_ddp.py ODE --cfg-scale 1.5 --model SiT-XL/2 --num-fid-samples 50000
without cfg: torchrun --nnodes=1 --nproc_per_node=8 sample_ddp.py ODE --model SiT-XL/2 --num-fid-samples 50000

In my testing, the FID for cfg=1.0 (w/o cfg) is 9.5 (I try several times and all > 9.3). But in paper, the SIT-XL/2 7M (w/o cfg) achieve FID=8.3. I wonder where will cause this performance misalignment of sampling from pretrained SIT models without cfg.

	FID (this repo)	FID (paper)
SIT-XL/2 (w/o. cfg)	9.5	8.3
SIT-XL/2 (w/. cfg=1.5)	2.08	2.06

[guyuchao]

I also tested DIT-XL/2 using their official codebase, and the performance matches what was reported in the SIT paper.

	FID (DIT repo)	FID (paper)
DIT-XL/2 (w/o. cfg)	9.5	9.6
DIT-XL/2 (w/. cfg=1.5)	2.27	2.27

[zelaki]

I have the same issue with B/2 model. I trained a B/2 model for 400K iterations.

	FID (my reproduction)	FID (paper)
SiT-B/2 (w/o cfg)	34.7	33.0

[willisma]

Hi @guyuchao

the 8.3 FID number is reported from sampling with SDE instead of ODE. The 9.5 FID number you reported aligns with our reported results in Tab 8.

@zelaki I'm not sure the sampling config you are using, but if you use ODE as well the number is also close with our reported results in Tab 8.

[zelaki]

Thank you for your prompt response!

I am using SDE for sampling with the default arguments specified in train_utils.py
--sampling-method="Euler"
--diffusion-form"="sigma"
--diffusion-norm=1.0
--last-step="Mean"
--last-step-size=0.04

and --num-sampling-steps=250

So I expected that these args correspond to the sampling scheme used to obtain 33.0 FID on SiT-B/2

[zelaki]

I also downloaded the SiT-XL and tried to reproduce 8.3 FID (reported in the paper) with SDE sampling but got 8.65 FID

[simulateuser538]

I also downloaded the SiT-XL and tried to reproduce 8.3 FID (reported in the paper) with SDE sampling but got 8.65 FID
hello ，did this problem solved? I got 9.1 FID with SDE sampling.This is my config.
SiT-XL-2-pretrained-cfg-1.0-4-SDE-250-Euler-sigma-Mean-0.04