Patch diffusion for own model. Reusing diffinfinite?

[miquel-espinosa]

Hi @marcoaversa ! Thanks for your great work.

Given an already pretrained diffusion model, I would like to add patch-diffusion to allow the generation of larger images.

• Is this codebase suitable for this? Do you know what are the best tools/libraries for achieving this?
• What are the important files I should consider? (Will it involve a lot of coding/experimenting?)

Thanks a lot in advance!

Miguel

[marcoaversa]

Hi @miquel-espinosa ! Thanks a lot for your interest in our work! It's your lucky day because we already did it and we shared a codebase ready to go (easy pip install with pretrained models):

Github page: https://github.com/Thanos-DB/Pixelsmith
Project page: https://thanos-db.github.io/Pixelsmith/

Pixelsmith uses StableDiffusionXL as backbone. Unfortunately, I believe the idea won't match nicely with newest foundation models which use Flow Matching, since the forward and backward pass cannot be aligned that easily as in the score-based or denoising diffusion methods. I hope it will be useful anyway for your projects!

Best,
Marco

[miquel-espinosa]

Thanks @marcoaversa for the pointers! Will take a look and let you know how it goes.
Our model is a ViT backbone that uses a dpm solver.
Currently, it is restricted to fixed output resolution like most diffusion models, but I'm hoping that with patch diffusion we can output larger generations without having to retrain.