New student proposed method for LCM, "Faster-LCM", said to allow for 30fps 512x512 SD generation (Code/Blog Up this morning)

[CCpt5]

"[LCM] Real-time style conversion of 512 x 512 px images in 0.02 seconds"

Blog:
https://zenn.dev/discus0434/articles/12427b887b4082

Code:
https://github.com/discus0434/faster-lcm

Twitter (Dev announcing):
https://twitter.com/IMG_5955/status/1729441031119348070

Could this actually work? Thanks for any insight!!

Start of blog:

Hello.

The day before yesterday, I spent a day on Saturday and Sunday working on speeding up the inference speed of Latent Cosistency Model (LCM), and it worked out pretty well.In the task of image-to-image (img2img) for a 512 x 512 px image, I was able to achieve 26fps and A100 on RTX3090 . The inference speed is now 33fps .

[Additional note] I heard that RTX4090 produced 45fps, so I slightly updated the title of the article. Please note that at the time of writing this article, we were testing it with A100.

This means that each image can be processed in 0.03 seconds, so it can be said that the conversion is done almost in real time.


img2img of one image is completed every time the progress bar advances by 1%. It feels good.

Therefore, in this article, I would like to write about tips for accelerating LCM inference that I obtained empirically (not theoretically) while working on accelerating this process .

[wkpark]

it will work with sd-webui 1.6.0 without issue. (currently, Im using sd-webui dev branch)

• download lcm lora. and use "<lora:lcm_blah_blah:0.6>" in your prompt
• select "Eular a" sampler
• set CFG Scale "1.5~2"
• sampling steps: ~8

[CCpt5]

Are you sure that's the same thing as discussed here? I've used the LCM LoRAs they have been available for a couple of weeks, but this person is presenting this as a vast improvement on even those models. I'll do more research (sadly I have zero coding ability). Thanks for the reply!

Here's the code from their repo:

import json
import os
import time

import numpy as np
import torch
import torch._dynamo
from diffusers import AutoencoderTiny, StableDiffusionImg2ImgPipeline
from PIL import Image
from tqdm import tqdm

from config import Config
from lib import LCMScheduler

torch._dynamo.config.suppress_errors = True
torch.backends.cuda.matmul.allow_tf32 = True


class FastLCM:
    def __init__(self, config: Config) -> None:
        """
        Constructs a FastLCM object.

        Parameters
        ----------
        config : Config
            The configuration object.
        """
        self.config = config
        self.lcm_pipeline = self._init_lcm_pipeline()
        self._prompt_embedding = self._precompute_prompt_embedding()
        self._image = self.lcm_pipeline.image_processor.preprocess(
            Image.open(self.config.image_path)
            .convert("RGB")
            .resize((self.config.resolution,) * 2, Image.Resampling.LANCZOS)
        )
        self._warm_up()

        if self.config.save_image:
            os.makedirs("assets/output", exist_ok=True)

    def run(self):
        """
        Runs 100 times the faster LCM pipeline, and calculates the
        average time and FPS.
        """
        times = []
        for i in tqdm(range(100)):
            start_time = time.time()
            image = self.lcm_pipeline(
                image=self._image,
                prompt_embeds=self._prompt_embedding,
                num_inference_steps=int(
                    self.config.num_inference_steps / self.config.strength
                )
                + 1,
                strength=self.config.strength,
                guidance_scale=self.config.guidance_scale,
                original_inference_steps=self.config.original_inference_steps,
                output_type="pil",
            ).images[0]
            times.append(time.time() - start_time)

            if self.config.save_image:
                image.save(f"assets/output/output_{i}.png")

        print(f"Average time: {np.mean(times)}")
        print(f"FPS: {1 / np.mean(times)}")

    def _init_lcm_pipeline(self) -> StableDiffusionImg2ImgPipeline:
        """
        Initializes the LCM pipeline.

        Returns
        -------
        StableDiffusionImg2ImgPipeline
            The LCM pipeline.
        """
        # Initialize the LCM pipeline
        lcm_pipeline = StableDiffusionImg2ImgPipeline.from_pretrained(
            self.config.model_id_or_path,
            safety_checker=None,
            feature_extractor=None,
        )

        # Reload the scheduler with custom config
        lcm_pipeline.scheduler = LCMScheduler.from_config(
            json.loads(open(self.config.config_path).read())
        )

        # Load the tiny VAE
        lcm_pipeline.vae = AutoencoderTiny.from_pretrained(self.config.vae_id)

        # Move the pipeline to the GPU
        lcm_pipeline.to(device=self.config.device, dtype=self.config.dtype)

        # Set the unet to channels last
        lcm_pipeline.unet = lcm_pipeline.unet.to(memory_format=torch.channels_last)

        # Disable the progress bar
        lcm_pipeline.set_progress_bar_config(disable=True)

        # Compile the pipeline
        if self.config.compile:
            lcm_pipeline.unet = torch.compile(
                lcm_pipeline.unet, mode="reduce-overhead", fullgraph=True
            )
            lcm_pipeline.vae = torch.compile(
                lcm_pipeline.vae, mode="reduce-overhead", fullgraph=True
            )

        return lcm_pipeline

    def _warm_up(self):
        """
        Warms up the LCM pipeline.
        """
        for _ in range(3):
            self.lcm_pipeline(
                image=self._image,
                prompt_embeds=self._prompt_embedding,
                num_inference_steps=int(
                    self.config.num_inference_steps / self.config.strength
                )
                + 1,
                strength=self.config.strength,
                guidance_scale=self.config.guidance_scale,
                original_inference_steps=self.config.original_inference_steps,
            )

    def _precompute_prompt_embedding(self) -> torch.FloatTensor:
        """
        Precomputes the prompt embedding to speed up the inference.

        Returns
        -------
        torch.FloatTensor
            The prompt embedding.
        """
        prompt_embedding, _ = self.lcm_pipeline.encode_prompt(
            device=self.config.device,
            prompt=self.config.prompt,
            do_classifier_free_guidance=False,
            num_images_per_prompt=1,
            clip_skip=2,
        )
        return prompt_embedding


if __name__ == "__main__":
    config = Config()

    lcm = FastLCM(config)
    lcm.run()

[wkpark]

wow... (and sorry I thought you were referring to LCM.)
Runs 100 times the faster LCM pipeline, ???
unbelievably fast!!

[CCpt5]

I know - hoping it works (and quality is relatively good) - If I find an implementation of it pop up I'll add it to this thread.