How to use A1111 LORA

[ShylockGao]

I downloaded a LoRA model from Civitai, but I encountered two issues during use:

1. When I set the weight to 0.8, the lora seems to stop working.
2. When I set the weight to 1.0, the LoRA is effective even without using the trigger word "oil painting"
Could there be an issue with my code? How should I fix it?

lora
https://civitai.com/models/84542/oil-paintingoil-brush-stroke
model
https://civitai.com/models/4201?modelVersionId=245598

diffusers V0.30.1
@sayakpaul

pipeline = StableDiffusionPipeline.from_single_file(
    'd:\\stable-diffusion-webui-1.7.0\\models\\Stable-diffusion\\realisticVisionV60B1_v60B1VAE.safetensors',
    torch_dtype=torch.float16,
)
pipeline.safety_checker = None
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(pipeline.scheduler.config,use_karras_sigmas=True)
pipeline.load_lora_weights(".", weight_name="bichu-v0612.safetensors")
pipeline.enable_sequential_cpu_offload()
generator = torch.Generator(device="cuda").manual_seed(33)
prompt = 'interior architecture'
images = pipeline(
    prompt=prompt,
    vae=vae,
    negative_prompt="(deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime), text, cropped, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck",
    num_inference_steps=45,
    generator=generator,
    guidance_scale = 7,
    width=1024,
    height=640,
    num_images_per_prompt = 4,
    clip_skip=2,
    # cross_attention_kwargs={"scale": 1.0}
).images

[sayakpaul]

Good question. Does it work without the trigger keyword on A1111 with the exact parameters?

Also possible to pass a Hugging Face Hub URL instead of "d:\stable-diffusion-webui-1.7.0\models\Stable-diffusion\realisticVisionV60B1_v60B1VAE.safetensors"?

[ShylockGao]

Sorry, I updated the question and included the model link. I tested it again in A1111 and found that the trigger word is "oil painting." The LoRA performs well on the web.

Here’s how it behaves in A1111:

• Without the trigger word and weight values, the LoRA does not activate.
• The LoRA only activates when the trigger word is present.

[sayakpaul]

So, I take it that this issue was solved. Did you try with the new trigger word in diffusers?

[ShylockGao]

When I set the weight to 1.0 or 0.9 in diffusers, the LoRA model remains effective even without using the trigger word "oil painting"

[sayakpaul]

And when using the scale of 1.0 the LoRA isn't effective without the trigger word?

[ShylockGao]

When the ratio is greater than 0.9, the LoRA will still take effect even without using the trigger words
cross_attention_kwargs={"scale": 0.9}