Fix SDXL Diffusers Format Support & Add Manual Model Type Override#1285
Open
fewtarius wants to merge 3 commits intoleejet:masterfrom
Open
Fix SDXL Diffusers Format Support & Add Manual Model Type Override#1285fewtarius wants to merge 3 commits intoleejet:masterfrom
fewtarius wants to merge 3 commits intoleejet:masterfrom
Conversation
Problem: When loading SDXL models in diffusers directory format, text encoders were loaded with prefixes "te." and "te.1." which don't match the expected tensor names in the model graph. The model expects "cond_stage_model.transformer." for clip_l and "cond_stage_model.1.transformer." for clip_g. This caused "tensor not in model file" errors for all text encoder tensors when loading SDXL diffusers models. Solution: - Changed text_encoder prefix from "te." to "cond_stage_model.transformer." - Changed text_encoder_2 prefix from "te.1." to "cond_stage_model.1.transformer." - These prefixes now match what's used when loading separate clip_l/clip_g files - Added early return in get_sd_version() when SDXL is detected to prevent later components (VAE) from overriding the version - Added version caching to prevent re-detection from changing SDXL version Testing: - SDXL diffusers models now load successfully - SD 1.5 models continue to work (regression tested) - All text encoder tensors are found and loaded correctly Files changed: - model.cpp: Updated diffusers text encoder prefixes and SDXL detection logic - stable-diffusion.cpp: Added version caching to preserve SDXL detection
Adds a new --model-type CLI parameter that allows users to manually specify the model version instead of relying on auto-detection. This is useful when: - Auto-detection fails or is ambiguous - Testing model behavior with different version settings - Working with modified/custom models Usage: --model-type sdxl # Force SDXL version --model-type sd1 # Force SD 1.x version --model-type flux # Force FLUX version Supported values: sd1, sd2, sdxl, sdxl_inpaint, sdxl_pix2pix, flux, sd3, svd Implementation: - Added version_override field to sd_ctx_params_t struct - Added model_type string parameter to SDContextParams - Added string-to-enum conversion in to_sd_ctx_params_t() - Updated model loading to check for manual override before auto-detection - Auto-detection still works when --model-type is not specified Testing: - Tested manual override with --model-type sdxl (works) - Tested auto-detection without parameter (still works) - Tested with SD 1.5 model and --model-type sd1 (works) Files changed: - stable-diffusion.h: Added version_override field to sd_ctx_params_t - stable-diffusion.cpp: Added version override logic and initialization - examples/common/common.hpp: Added CLI parameter and string-to-enum conversion
Problem: SDXL models in diffusers directory format fail to load with "unknown tensor" errors Solution: Added te. and te.1. prefixes to cond_stage_model conversion list Testing: SDXL diffusers models now load and generate successfully Root cause: When loading diffusers SDXL models, text_encoder uses "te." prefix and text_encoder_2 uses "te.1." prefix. These weren't in the name conversion prefix list, so tensors weren't being converted to checkpoint format names. This fix enables diffusers-format SDXL models to work alongside single-file checkpoint models without requiring format conversion. Fixes: Models like duchaiten-pony-real-v20-sdxl in diffusers directory layout
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes loading of SDXL models in diffusers directory format and adds a new
--model-typeCLI parameter for manual model version control.Problem
When loading SDXL models in diffusers directory format (e.g.,
duchaiten-pony-real-v20-sdxl), the text encoder tensors failed to load with "unknown tensor" or "tensor not in model file" errors. This occurred because:te.andte.1.prefixes used by diffusers format weren't in the name conversion listSolution
1. Name Conversion Support (
name_conversion.cpp)Added
te.andte.1.prefixes to the cond_stage_model conversion list, enabling proper tensor name translation from diffusers format to checkpoint format.2. Correct Tensor Prefixes (
model.cpp)text_encoderprefix fromte.tocond_stage_model.transformer.text_encoder_2prefix fromte.1.tocond_stage_model.1.transformer.get_sd_version()when SDXL is detected to prevent VAE from overriding version3. Version Caching (
stable-diffusion.cpp)Added version caching to preserve SDXL detection across component loading phases.
4. Manual Model Type Override (
--model-typeparameter)New CLI parameter allowing users to manually specify the model version:
Supported values:
sd1,sd2,sdxl,sdxl_inpaint,sdxl_pix2pix,flux,sd3,svdUse cases:
Files Changed
src/name_conversion.cppte.andte.1.prefix mappingssrc/model.cppsrc/stable-diffusion.cppinclude/stable-diffusion.hversion_overridefield tosd_ctx_params_texamples/common/common.hpp--model-typeCLI parameter and string-to-enum conversionTesting
--model-type sdxlworks--model-typeis not specifiedCommits
Compatibility
Related Issues
Fixes loading of diffusers-format SDXL models that previously failed with tensor loading errors.