Skip to content

Commit 9112028

Browse files
a-r-r-o-wDN6
andauthored
FreeInit (#6315)
* freeinit * update freeinit implementation based on review Co-Authored-By: Dhruv Nair <dhruv.nair@gmail.com> * fix * another fix * refactor * fix timesteps missing bug * apply suggestions from review Co-Authored-By: Dhruv Nair <dhruv.nair@gmail.com> * add test for freeinit * apply suggestions from review Co-Authored-By: Dhruv Nair <dhruv.nair@gmail.com> * refactor * fix test * fix tensor not on same device * update * remove return_intermediate_results * fix broken freeinit test * update animatediff docs --------- Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
1 parent dce0668 commit 9112028

File tree

3 files changed

+485
-67
lines changed

3 files changed

+485
-67
lines changed

docs/source/en/api/pipelines/animatediff.md

+58
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,62 @@ export_to_gif(frames, "animation.gif")
235235
</tr>
236236
</table>
237237

238+
## Using FreeInit
239+
240+
[FreeInit: Bridging Initialization Gap in Video Diffusion Models](https://arxiv.org/abs/2312.07537) by Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, Ziwei Liu.
241+
242+
FreeInit is an effective method that improves temporal consistency and overall quality of videos generated using video-diffusion-models without any addition training. It can be applied to AnimateDiff, ModelScope, VideoCrafter and various other video generation models seamlessly at inference time, and works by iteratively refining the latent-initialization noise. More details can be found it the paper.
243+
244+
The following example demonstrates the usage of FreeInit.
245+
246+
```python
247+
import torch
248+
from diffusers import MotionAdapter, AnimateDiffPipeline, DDIMScheduler
249+
from diffusers.utils import export_to_gif
250+
251+
adapter = MotionAdapter.from_pretrained("guoyww/animatediff-motion-adapter-v1-5-2")
252+
model_id = "SG161222/Realistic_Vision_V5.1_noVAE"
253+
pipe = AnimateDiffPipeline.from_pretrained(model_id, motion_adapter=adapter, torch_dtype=torch.float16).to("cuda")
254+
pipe.scheduler = DDIMScheduler.from_pretrained(
255+
model_id,
256+
subfolder="scheduler",
257+
beta_schedule="linear",
258+
clip_sample=False,
259+
timestep_spacing="linspace",
260+
steps_offset=1
261+
)
262+
263+
# enable memory savings
264+
pipe.enable_vae_slicing()
265+
pipe.enable_vae_tiling()
266+
267+
# enable FreeInit
268+
# Refer to the enable_free_init documentation for a full list of configurable parameters
269+
pipe.enable_free_init(method="butterworth", use_fast_sampling=True)
270+
271+
# run inference
272+
output = pipe(
273+
prompt="a panda playing a guitar, on a boat, in the ocean, high quality",
274+
negative_prompt="bad quality, worse quality",
275+
num_frames=16,
276+
guidance_scale=7.5,
277+
num_inference_steps=20,
278+
generator=torch.Generator("cpu").manual_seed(666),
279+
)
280+
281+
# disable FreeInit
282+
pipe.disable_free_init()
283+
284+
frames = output.frames[0]
285+
export_to_gif(frames, "animation.gif")
286+
```
287+
288+
<Tip warning={true}>
289+
290+
FreeInit is not really free - the improved quality comes at the cost of extra computation. It requires sampling a few extra times depending on the `num_iters` parameter that is set when enabling it. Setting the `use_fast_sampling` parameter to `True` can improve the overall performance (at the cost of lower quality compared to when `use_fast_sampling=False` but still better results than vanilla video generation models).
291+
292+
</Tip>
293+
238294
<Tip>
239295

240296
Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers) to learn how to explore the tradeoff between scheduler speed and quality, and see the [reuse components across pipelines](../../using-diffusers/loading#reuse-components-across-pipelines) section to learn how to efficiently load the same components into multiple pipelines.
@@ -248,6 +304,8 @@ Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers)
248304
- __call__
249305
- enable_freeu
250306
- disable_freeu
307+
- enable_free_init
308+
- disable_free_init
251309
- enable_vae_slicing
252310
- disable_vae_slicing
253311
- enable_vae_tiling

0 commit comments

Comments
 (0)