Replies: 3 comments
-
cc: @sayakpaul |
Beta Was this translation helpful? Give feedback.
0 replies
-
I think you are asking for debugging guidance as this is not a direct issue with the library. So, I am going to turn this into a discussion. |
Beta Was this translation helpful? Give feedback.
0 replies
-
I would try to investigate the shape of the concerned tensors ( |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Describe the bug
I want to create a class that inherits from StableDiffusion3ControlNetPipeline. When I rewrite the call function, I meet a problem that encoder_hidden_states and context_attn_output have different shapes. The pre-trained model of StableDiffusion3ControlNetPipeline is "stabilityai/stable-diffusion-3-medium-diffusers", and for the ControlNet is "InstantX/SD3-Controlnet-Pose". Where could this problem come from? Thanks for the help.
Reproduction
def call(
self,
latent_list,
prompt: Union[str, List[str]] = None,
prompt_2: Optional[Union[str, List[str]]] = None,
prompt_3: Optional[Union[str, List[str]]] = None,
pose_map: Union[torch.Tensor, PIL.Image.Image, List[PIL.Image.Image]] = None,
image: Union[torch.Tensor, PIL.Image.Image, List[PIL.Image.Image]] = None,
clip_image: Union[torch.Tensor, PIL.Image.Image, List[PIL.Image.Image]] = None,
target_pose = None,
source_pose = None,
height: Optional[int] = None,
width: Optional[int] = None,
num_inference_steps: int = 50,
timesteps: List[int] = None,
sigmas: List[float] = None,
guidance_scale: float = 7.0,
control_guidance_start: Union[float, List[float]] = 0.0,
control_guidance_end: Union[float, List[float]] = 1.0,
controlnet_conditioning_scale: Union[float, List[float]] = 1.0,
controlnet_pooled_projections: Optional[torch.FloatTensor] = None,
negative_prompt: Optional[Union[str, List[str]]] = None,
negative_prompt_2: Optional[Union[str, List[str]]] = None,
negative_prompt_3: Optional[Union[str, List[str]]] = None,
num_images_per_prompt: Optional[int] = 1,
eta: float = 0.0,
generator: Optional[Union[torch.Generator, List[torch.Generator]]] = None,
latents: Optional[torch.Tensor] = None,
prompt_embeds: Optional[torch.Tensor] = None,
negative_prompt_embeds: Optional[torch.Tensor] = None,
pooled_prompt_embeds: Optional[torch.FloatTensor] = None,
negative_pooled_prompt_embeds: Optional[torch.FloatTensor] = None,
output_type: Optional[str] = "pil",
return_dict: bool = True,
joint_attention_kwargs: Optional[Dict[str, Any]] = None,
clip_skip: Optional[int] = None,
callback_on_step_end: Optional[Callable[[int, int, Dict], None]] = None,
callback_on_step_end_tensor_inputs: List[str] = ["latents"],
max_sequence_length: int = 256,
):
r"""
Function invoked when calling the pipeline for generation.
Logs
System Info
absl-py==2.1.0
accelerate==0.33.0
anaconda-anon-usage @ file:///croot/anaconda-anon-usage_1710965072196/work
anyio==4.4.0
archspec @ file:///croot/archspec_1709217642129/work
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-lru==2.0.4
attrs==23.2.0
Babel==2.15.0
beautifulsoup4==4.12.3
bleach==6.1.0
boltons @ file:///work/perseverance-python-buildout/croot/boltons_1698851177130/work
Brotli @ file:///croot/brotli-split_1714483155106/work
certifi @ file:///croot/certifi_1707229174982/work/certifi
cffi @ file:///croot/cffi_1714483155441/work
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
comm==0.2.2
conda @ file:///croot/conda_1714403036266/work
conda-content-trust @ file:///croot/conda-content-trust_1714483159009/work
conda-libmamba-solver @ file:///croot/conda-libmamba-solver_1706733287605/work/src
conda-package-handling @ file:///croot/conda-package-handling_1714483155348/work
conda_package_streaming @ file:///work/perseverance-python-buildout/croot/conda-package-streaming_1698847176583/work
contourpy==1.2.1
controlnet-aux==0.0.9
cryptography @ file:///croot/cryptography_1714660666131/work
cycler==0.12.1
debugpy==1.8.1
decorator==5.1.1
defusedxml==0.7.1
diffusers==0.30.2
distro @ file:///croot/distro_1714488253808/work
einops==0.8.0
executing==2.0.1
fastjsonschema==2.19.1
filelock==3.14.0
fonttools==4.53.0
fqdn==1.5.1
fsspec==2024.5.0
grpcio==1.64.0
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
huggingface-hub==0.24.5
idna @ file:///croot/idna_1714398848350/work
imageio==2.34.2
importlib_metadata==8.2.0
ipykernel==6.29.4
ipython==8.25.0
ipywidgets==8.1.3
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.4
json5==0.9.25
jsonpatch @ file:///croot/jsonpatch_1714483231291/work
jsonpointer==2.1
jsonschema==4.22.0
jsonschema-specifications==2023.12.1
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter_client==8.6.2
jupyter_core==5.7.2
jupyter_server==2.14.1
jupyter_server_terminals==0.5.3
jupyterlab==4.2.1
jupyterlab-language-pack-zh-CN==4.2.post1
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.2
jupyterlab_widgets==3.0.11
kiwisolver==1.4.5
lazy_loader==0.4
libmambapy @ file:///croot/mamba-split_1714483352891/work/libmambapy
Markdown==3.6
MarkupSafe==2.1.5
matplotlib==3.9.0
matplotlib-inline==0.1.7
menuinst @ file:///croot/menuinst_1714510563922/work
mistune==3.0.2
mpmath==1.3.0
nbclient==0.10.0
nbconvert==7.16.4
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.3
notebook_shim==0.2.4
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.5.40
nvidia-nvtx-cu12==12.1.105
opencv-python-headless==4.10.0.84
overrides==7.7.0
packaging @ file:///croot/packaging_1710807400464/work
pandas==2.2.2
pandocfilters==1.5.1
parso==0.8.4
pexpect==4.9.0
pillow==10.3.0
platformdirs @ file:///work/perseverance-python-buildout/croot/platformdirs_1701732573265/work
pluggy @ file:///work/perseverance-python-buildout/croot/pluggy_1698805497733/work
prometheus_client==0.20.0
prompt_toolkit==3.0.45
protobuf==5.27.0
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
pycosat @ file:///croot/pycosat_1714510623388/work
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
Pygments==2.18.0
pyparsing==3.1.2
PySocks @ file:///work/perseverance-python-buildout/croot/pysocks_1698845478203/work
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
pytz==2024.1
PyYAML==6.0.1
pyzmq==26.0.3
referencing==0.35.1
regex==2024.7.24
requests @ file:///croot/requests_1707355572290/work
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.18.1
ruamel.yaml @ file:///work/perseverance-python-buildout/croot/ruamel.yaml_1698863605521/work
safetensors==0.4.3
scikit-image==0.24.0
scipy==1.14.0
Send2Trash==1.8.3
sentencepiece==0.2.0
setuptools==69.5.1
six==1.16.0
sniffio==1.3.1
soupsieve==2.5
stack-data==0.6.3
supervisor==4.2.5
sympy==1.12.1
tensorboard==2.16.2
tensorboard-data-server==0.7.2
terminado==0.18.1
tifffile==2024.7.24
timm==0.6.7
tinycss2==1.3.0
tokenizers==0.19.1
torch==2.4.0
torchaudio==2.4.0
torchvision==0.19.0
tornado==6.4
tqdm @ file:///croot/tqdm_1714567712644/work
traitlets==5.14.3
transformers==4.44.0
triton==3.0.0
truststore @ file:///work/perseverance-python-buildout/croot/truststore_1701735771625/work
types-python-dateutil==2.9.0.20240316
typing_extensions==4.12.1
tzdata==2024.1
uri-template==1.3.0
urllib3 @ file:///croot/urllib3_1707770551213/work
wcwidth==0.2.13
webcolors==1.13
webencodings==0.5.1
websocket-client==1.8.0
Werkzeug==3.0.3
wheel==0.43.0
widgetsnbextension==4.0.11
xformers==0.0.27.post2
zipp==3.19.2
zstandard @ file:///croot/zstandard_1714677652653/work
Who can help?
@yiyixuxu @DN6 @saya
Beta Was this translation helpful? Give feedback.
All reactions