Move the LoRA extraction from prompt step to examples/cli/main.cpp?

[stduhpf]

Right now the prompt is processed for LoRA detection in the internal generate_image() function, which is called from the exposed txt2img() or img2img() functions.
https://github.com/leejet/stable-diffusion.cpp/blob/master/stable-diffusion.cpp#L1221-L1230
While I agree that using the stable-diffusion-webui format to load loras from the CLI is nice, I'm not convinced this should be forced at the API level.
Would it complicate things to accept the lora path/multiplier mappings as arguments to txt2img() and img2img() (and possibly expose the extract_and_remove_lora() function)?
I'm hesitant to start working on implementing it, since this would probably be a breaking change for downstream apps.

What do you think @leejet @fszontagh @piallai ?

[fszontagh]

On the GUI side it's easier to handle the loras in the prompts (because no need to handle :) )
But eg. the online services will have the possibility to limit the used resources with this feature. (eg. limit how many lora is used in one job, or check the used lora's size). So it will give more control and no need to implement in client side these checks. (eg. no need to pre-parse the text prompts)

The other downside of this feature is the non-reproductivable text prompts. The used lora's hashes and names must be stored into exif / png meta data as always do. But if no lora tag in the prompt, it will be "invisible" for the user side at a point.

So, if maybe both feature will be implemented? Eg if you add at least one or more lora from parameter, then the lora tags ignored from the prompt.

Anyway, I support the idea, but if you implement this on the loras, then need to be implemented on text embeddings too.

[stduhpf]

Text embeddings need to be in the prompt though, since they need to be proceessed as would any token somewhere in the prompt.

[stduhpf]

Another problem with the current LoRA implementation is that once a lora is loaded and applied onto the model, it won't get un-applied until the model is fully reloaded, and if free_params_immediately is set to false, the model might not be unloaded in between two prompts. Which is very bad for having reproductible text prompts, since loras can be loaded in the backend without being specified in the prompt.

[piallai]

I believe this is the main problem indeed. There's quite an inherent ambiguity in managing some of the model inputs (the lora configuration) in the prompt. IMO, a clear separation between the model parameters (including the lora) and the prompt would make more sense. However, not sure it's worth the development since having the lora configuration in the prompt seems like a convention now.

[piallai]

Potentially interesting to link with #410

[stduhpf]

Another point, I think there is a potential security/privacy issue with online servers. Letting users put arbitraty paths in the payload and parsing it to check if it correspond to a LoRA would at least leak some information about the file system of the server (right now it's possible to access files outside of the lora-model-dir by adding ../ before the lora path), and maybe even lead to worse attacks?