Regarding results

[yoohyewony]

Thank you for sharing the code!

I generated and tested 50,000 samples with the LDM-4 checkpoint that you provided, but I got IS 58.2. I think I did something wrong, and I hope you can provide some guidance.

[wlfeng0509]

@yoohyewony Hi, I also encountered this problem, the IS metrics of all different bits are generally low, but FID and sFID are good. Did you find any way to reproduce the IS results in the paper?

[yoohyewony]

No I still have the same problem :(

[ThisisBillhe]

Hey guys, do the images you generated look normal? The results of IS and FID reported in paper are obtained by this tool

[yoohyewony]

I used the same tool (which is provided by openAI guided diffusion) and generated images look normal. I don't know why only IS value is far below the paper.

[wlfeng0509]

Yes, the images are also good, and only IS is far below than report. Can you provide the exact setting to reproduce the results? Like ddim_steps, ddim_eta, and scale for sampler, and do you generate 50 images for each class? Thanks a lot!

[ThisisBillhe]

Yes, the images are also good, and only IS is far below than report. Can you provide the exact setting to reproduce the results? Like ddim_steps, ddim_eta, and scale for sampler, and do you generate 50 images for each class? Thanks a lot!

It's 20 steps, eta=0, scale=3, 50 images for each class. However, since the images look proper, I don't think the generation configuration is the reason for the low IS...

[wlfeng0509]

Yes, the images are also good, and only IS is far below than report. Can you provide the exact setting to reproduce the results? Like ddim_steps, ddim_eta, and scale for sampler, and do you generate 50 images for each class? Thanks a lot!

It's 20 steps, eta=0, scale=3, 50 images for each class. However, since the images look proper, I don't think the generation configuration is the reason for the low IS...

Thanks a lot! Looks like i use the same setting, any idea how to fix this? Or could you provide the scripts for generating the .npz file, in case that i miss anything important. Also did you modify the config in https://github.com/openai/guided-diffusion/blob/main/evaluations/evaluator.py line 23-24?
FID_POOL_NAME = "pool_3:0" FID_SPATIAL_NAME = "mixed_6/conv:0"

[ThisisBillhe]

Yes, the images are also good, and only IS is far below than report. Can you provide the exact setting to reproduce the results? Like ddim_steps, ddim_eta, and scale for sampler, and do you generate 50 images for each class? Thanks a lot!

It's 20 steps, eta=0, scale=3, 50 images for each class. However, since the images look proper, I don't think the generation configuration is the reason for the low IS...

Thanks a lot! Looks like i use the same setting, any idea how to fix this? Or could you provide the scripts for generating the .npz file, in case that i miss anything important. Also did you modify the config in https://github.com/openai/guided-diffusion/blob/main/evaluations/evaluator.py line 23-24? FID_POOL_NAME = "pool_3:0" FID_SPATIAL_NAME = "mixed_6/conv:0"

No, I did not modify this config. I update the script to generate 50k samples in scripts/generate_samples_4_evaluation_FP.py, you may have a try.

[wlfeng0509]

Sadly, I followed the script you provided to test again and the results were still the same, which really confused me.

[ThisisBillhe]

Do you use the imagenet reference batch provided here?

[wlfeng0509]

Do you use the imagenet reference batch provided here?

Yes, I use the ImageNet 256x256 reference batch.

[BienLuky]

@cantbebetter2 Hi, can you get the good FID and sFID? I have a problem to reproduce the results. I doubt it's my wrong settings. I try to set 20 steps, eta=0, scale=3 for the LDM-4 ImageNet, and keep the epoch of training as 160 (This is the default setting in open source code). Can you share your setting? Very much looking forward to your reply! Thanks!