Function calling, thousands of functions - best practice

[toresenneseth]

Hi all,

I'm just getting started with SK, and realize that what I'm about to ask is not strictly a SK "issue", but more of a general question about how to best implement function calling with LLMs if you have a large number of functions.

The following article says that I should keep my function schema as small as possible:
https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/chat-completion/function-calling/?pivots=programming-language-csharp

My company has a product for building business process automations. We support about 2000+ different automation tasks. I made a small prototype using SK and running Ollama locally with different models. I was able to successfully auto generate configurations for all the business process automations through function calling, where each automation consisted or multiple individual tasks.

However, what I'm concerned about is whether this approach will work when I scale this up to 2000+ functions, and how expensive it becomes when I move it to the cloud (based on the article above).

Note that there's no easy way to categorize my functions so that I can say that "in this context, only functions X,Y,Z will be applicable".

• Should I look into using agents instead, where each agent has their own set of functions?
• Can I "pre-load" / initialize a model somehow (I don't think so)
• Should I train / fine-tune my own model so it already has knowledge about my function calls?
• Am I going about this completely wrong?? :)

Any thoughts or advice from the community would be greatly appreciated.
-Thanks

[sorin-costea]

My experience is quite limited, but if you can group the functions by business meaning, you can probably assign them to agents - also easier to teach those about functions usage.

[markwallace-microsoft]

@toresenneseth straight away I can say that presenting the model with 2K functions is problematic because the model will typically get confused and will produce poor responses. Below is some information which should help you:

1. Some guidance on function calling limitations: https://learn.microsoft.com/en-us/semantic-kernel/concepts/plugins/adding-openapi-plugins?pivots=programming-language-csharp#tips-and-tricks-for-adding-openapi-plugins
2. Sample showing one strategy to handle large number of function: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Optimization/PluginSelectionWithFilters.cs
3. Agents may be a good choice, we have internal teams who use teams of agents to handle very complex sets of tasks where each Agent just focusses on one very specific task, see: https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/?pivots=programming-language-csharp

I'd recommend joining our community office hours so you can discuss this with the entire team https://github.com/microsoft/semantic-kernel/blob/main/COMMUNITY.md

[toresenneseth]

Thank you for the suggestions.

I need to investigate the plugin filter approach, which seems like a viable strategy.
I've tried using the agent approach using AgentGroupChat, but I haven't been able to trigger more than one function to be called (the first one). I probably need to work on my prompting skills.

What I got working was somewhat of a "hack" I guess, by using two chat completion services working together. It's somewhat similar to using plugin filtering, however less elegant.

• Chat completion service 1 figures out which plugins to use based on the user input.
• I parse the output from the first service (a list of plugin names) and dynamically load those plugins into the kernel of the second chat completion service.
• I then invoke the second chat completion service with the same user input.

This produce the desired result, but I'm not proud of my work :)

Will investigate the plugin selection approach you mention above, and joining the community office hours seems like a good idea.

-Thanks