Skip to content

Commit 06b7a51

Browse files
Bluear7878lmxyy
Bluear7878
and
lmxyy
authored
feat: enable IP-Adapter (XLabs-AI/flux-ip-adapter-v2) support (#418)
* feat: support IP-adapter * FBCache and comfyUI * fixing conflicts * update * update example * update example * style: make linter happy * update * update ipa test * add docs and rename IP to ip * docs: add docs for ipa * docs: add docs for ipa * add an example for pulid * update * save gpu memory * change the threshold to 0.8 --------- Co-authored-by: Muyang Li <[email protected]>
1 parent 24c2f92 commit 06b7a51

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+1481
-36
lines changed

‎docs/source/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ Check out `DeepCompressor <github_deepcompressor_>`_ for the quantization librar
2424
usage/attention.rst
2525
usage/fbcache.rst
2626
usage/pulid.rst
27+
usage/ip_adapter.rst
2728

2829
.. toctree::
2930
:maxdepth: 1

‎docs/source/links/huggingface.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,4 @@
88
.. _hf_nunchaku-flux1-dev-int4: https://huggingface.co/mit-han-lab/nunchaku-flux.1-dev/blob/main/svdq-int4_r32-flux.1-dev.safetensors
99
.. _hf_depth_anything: https://huggingface.co/LiheYoung/depth-anything-large-hf
1010
.. _hf_nunchaku_wheels: https://huggingface.co/nunchaku-tech/nunchaku
11+
.. _hf_ip-adapterv2: https://huggingface.co/XLabs-AI/flux-ip-adapter-v2

‎docs/source/python_api/nunchaku.models.ip_adapter.diffusers_adapters.flux.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
nunchaku.models.ip_adapter.diffusers_adapters.flux
2+
==================================================
3+
4+
.. automodule:: nunchaku.models.ip_adapter.diffusers_adapters.flux
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

‎docs/source/python_api/nunchaku.models.ip_adapter.diffusers_adapters.rst

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
nunchaku.models.ip_adapter.diffusers_adapters
2+
=============================================
3+
4+
.. automodule:: nunchaku.models.ip_adapter.diffusers_adapters
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:
8+
9+
.. toctree::
10+
:maxdepth: 4
11+
12+
nunchaku.models.ip_adapter.diffusers_adapters.flux

‎docs/source/python_api/nunchaku.models.ip_adapter.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
nunchaku.models.ip_adapter
2+
==========================
3+
4+
.. toctree::
5+
:maxdepth: 4
6+
7+
nunchaku.models.ip_adapter.diffusers_adapters
8+
nunchaku.models.ip_adapter.utils

‎docs/source/python_api/nunchaku.models.ip_adapter.utils.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
nunchaku.models.ip_adapter.utils
2+
================================
3+
4+
.. automodule:: nunchaku.models.ip_adapter.utils
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

‎docs/source/python_api/nunchaku.models.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,5 @@ nunchaku.models
77
nunchaku.models.transformers
88
nunchaku.models.text_encoders
99
nunchaku.models.pulid
10+
nunchaku.models.ip_adapter
1011
nunchaku.models.safety_checker

‎docs/source/usage/ip_adapter.rst

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
IP Adapter
2+
==========
3+
4+
Nunchaku supports `IP Adapter <hf_ip-adapterv2_>`_, an adapter achieving image prompt capability for the FLUX.1-dev
5+
6+
.. literalinclude:: ../../../examples/flux.1-dev-IP-adapter.py
7+
:language: python
8+
:caption: IP Adapter Example (`examples/flux.1-dev-IP-adapter.py <https://github.com/nunchaku-tech/nunchaku/blob/main/examples/flux.1-dev-IP-adapter.py>`__)
9+
:linenos:
10+
11+
The IP Adapter integration in Nunchaku follows these main steps:
12+
13+
**Model Initialization**:
14+
15+
- Load a Nunchaku FLUX.1-dev transformer model using :meth:`~nunchaku.models.transformers.transformer_flux.NunchakuFluxTransformer2dModel.from_pretrained`.
16+
- Initialize the FLUX pipeline with :class:`diffusers.FluxPipeline`, passing the transformer and setting the appropriate precision.
17+
18+
**IP Adapter Loading**:
19+
20+
- Use ``pipeline.load_ip_adapter`` to load the IP Adapter weights and the CLIP image encoder.
21+
22+
- ``pretrained_model_name_or_path_or_dict``: Hugging Face repo or local path for the IP Adapter weights.
23+
- ``weight_name``: Name of the weights file (e.g., ``ip_adapter.safetensors``).
24+
- ``image_encoder_pretrained_model_name_or_path``: Name or path of the CLIP image encoder.
25+
- Apply the IP Adapter to the pipeline with :func:`~nunchaku.models.ip_adapter.diffusers_adapters.apply_IPA_on_pipe`, specifying the adapter scale and repo ID.
26+
27+
**Caching (Optional)**:
28+
29+
Enable caching for faster inference and reduced memory usage with :func:`~nunchaku.caching.diffusers_adapters.apply_cache_on_pipe`. See :doc:`fbcache` for more details.
30+
31+
**Image Generation**:
32+
33+
- Load the image to be used as the image prompt (IP Adapter reference).
34+
- Call the pipeline with:
35+
36+
- ``prompt``: The text prompt for generation.
37+
- ``ip_adapter_image``: The reference image (must be RGB).
38+
- The output image will reflect both the text prompt and the visual style/content of the reference image.

‎docs/source/usage/pulid.rst

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,20 @@
11
PuLID
22
=====
33

4-
Nunchaku integrates `PuLID <_pulid_paper>`_, a tuning-free identity customization method for text-to-image generation.
4+
.. image:: https://huggingface.co/datasets/nunchaku-tech/cdn/resolve/main/ComfyUI-nunchaku/workflows/nunchaku-flux.1-dev-pulid.png
5+
6+
Nunchaku integrates `PuLID <paper_pulid_>`_, a tuning-free identity customization method for text-to-image generation.
57
This feature allows you to generate images that maintain specific identity characteristics from reference photos.
68

79
.. literalinclude:: ../../../examples/flux.1-dev-pulid.py
810
:language: python
911
:caption: PuLID Example (`examples/flux.1-dev-pulid.py <https://github.com/nunchaku-tech/nunchaku/blob/main/examples/flux.1-dev-pulid.py>`__)
1012
:linenos:
1113

12-
Implementation Overview
13-
-----------------------
14-
1514
The PuLID integration follows these key steps:
1615

1716
**Model Initialization** (lines 12-20):
18-
Load a Nunchaku FLUX.1-dev model using :class:`~nunchaku.models.transformers.transformer_flux.NunchakuFluxTransformer2dModel`
17+
Load a Nunchaku FLUX.1-dev model using :meth:`~nunchaku.models.transformers.transformer_flux.NunchakuFluxTransformer2dModel.from_pretrained`
1918
and initialize the FLUX PuLID pipeline with :class:`~nunchaku.pipeline.pipeline_flux_pulid.PuLIDFluxPipeline`.
2019

2120
**Forward Method Override** (line 22):

‎examples/flux.1-canny-dev.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
precision = get_precision() # auto-detect your precision is 'int4' or 'fp4' based on your GPU
1010
transformer = NunchakuFluxTransformer2dModel.from_pretrained(
11-
f"mit-han-lab/nunchaku-flux.1-canny-dev/svdq-{precision}_r32-flux.1-canny-dev.safetensors"
11+
f"nunchaku-tech/nunchaku-flux.1-canny-dev/svdq-{precision}_r32-flux.1-canny-dev.safetensors"
1212
)
1313
pipe = FluxControlPipeline.from_pretrained(
1414
"black-forest-labs/FLUX.1-Canny-dev", transformer=transformer, torch_dtype=torch.bfloat16

0 commit comments

Comments
 (0)