Pydantic integration with a Storagecontext and Query Engine

[theholymath]

I have code that I feel should work but I keep getting a tool error. I know the vector store is written and populated. When I query without passing the pydantic class it works just fine.

from llama_index.core import StorageContext, VectorStoreIndex, load_index_from_storage
from settings import set_llm_config
from custom_data_models import ProductTableEntry

chat_llm, embed_llm = set_llm_config()

# Load your index
storage_context = StorageContext.from_defaults(persist_dir='embeddings/my_index')
index = load_index_from_storage(storage_context)

query_engine = index.as_query_engine(
    output_cls=ProductTableEntry,
    response_mode="compact",
    llm=chat_llm,
)

response = query_engine.query("What is the product order number?")
print(response)

This results in a ValueError: Expected at least one tool call, but got 0 tool calls. error.

When I simply use index.as_query_engine(llm=chat_llm) it works fine.

I am trying to fill a predefined pydantic data model. I am loading from llama_index.core.bridge.pydantic import BaseModel, Field as well.

[dosubot[bot]]

Hey @theholymath, I'm here to help you with any bugs, questions, and contributions you might have. Let's tackle this together!

To resolve the ValueError: Expected at least one tool call, but got 0 tool calls. error and successfully integrate Pydantic with the QueryEngine, you need to ensure that your response contains at least one valid tool call. This error occurs when the tool_calls list is empty, and the error_on_no_tool_call flag is set to True.

Here's how you can modify your code to include a tool call:

1. Create a Tool Call: Define a function to create a tool call.
2. Ensure Tool Call in Response: Modify the chat_with_tools function to ensure at least one tool call is present in the response.

Here's an example:

from llama_index.core.llms.function_calling import ToolSelection
from llama_index.core.types import ChatMessage, MessageRole
from llama_index.legacy.tools.query_engine import QueryEngineTool, ToolMetadata
import json

# Example function to create a tool call
def create_tool_call():
    return {
        "input": json.dumps({"param1": "value1"}),  # Example input
        "id": "tool_id_1",
        "name": "example_tool",
        "type": "tool_use"
    }

# Example usage in your chat function
def chat_with_tools(
    self,
    tools: List["BaseTool"],
    user_msg: Optional[Union[str, ChatMessage]] = None,
    chat_history: Optional[List[ChatMessage]] = None,
    verbose: bool = False,
    allow_parallel_tool_calls: bool = False,
    **kwargs: Any,
) -> ChatResponse:
    chat_history = chat_history or []

    if isinstance(user_msg, str):
        user_msg = ChatMessage(role=MessageRole.USER, content=user_msg)
        chat_history.append(user_msg)

    tool_dicts = []
    for tool in tools:
        tool_dicts.append(
            {
                "name": tool.metadata.name,
                "description": tool.metadata.description,
                "input_schema": tool.metadata.get_parameters_dict(),
            }
        )

    response = self.chat(chat_history, tools=tool_dicts or None, **kwargs)

    # Ensure at least one tool call is present
    if not response.message.additional_kwargs.get("tool_calls"):
        response.message.additional_kwargs["tool_calls"] = [create_tool_call()]

    if not allow_parallel_tool_calls:
        force_single_tool_call(response)

    return response

Additionally, you can use the QueryEngineTool class from the llama_index.legacy.tools.query_engine module to create a tool call. Here is an example of how to instantiate and use it:

from llama_index.legacy.tools.query_engine import QueryEngineTool, ToolMetadata
from llama_index.legacy.core.base_query_engine import BaseQueryEngine

# Create a QueryEngineTool instance
query_engine = BaseQueryEngine()  # Replace with your actual query engine instance
metadata = ToolMetadata(name="example_tool", description="Example tool description")
query_engine_tool = QueryEngineTool(query_engine=query_engine, metadata=metadata)

# Example function to create a tool call using QueryEngineTool
def create_tool_call():
    return {
        "input": json.dumps({"param1": "value1"}),  # Example input
        "id": "tool_id_1",
        "name": query_engine_tool.metadata.name,
        "type": "tool_use"
    }

This ensures that the response always contains at least one tool call, thus preventing the ValueError ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

To resolve the ValueError: Expected at least one tool call, but got 0 tool calls. error when integrating Pydantic with StorageContext and QueryEngine using the LlamaIndex library, you need to ensure that there is at least one tool call in the response. This error occurs when the tool_calls list is empty, and the error_on_no_tool_call flag is set to True.

Here's how you can modify your code to handle this:

1. Ensure that tool_calls is not empty: Modify the response handling to ensure at least one tool call is present.

2. Set error_on_no_tool_call to False: If it's acceptable to proceed without any tool calls, set this flag to False.

Here's an example of how you might modify your code:

from llama_index.core import StorageContext, VectorStoreIndex, load_index_from_storage
from settings import set_llm_config
from custom_data_models import ProductTableEntry

chat_llm, embed_llm = set_llm_config()

# Load your index
storage_context = StorageContext.from_defaults(persist_dir='embeddings/my_index')
index = load_index_from_storage(storage_context)

query_engine = index.as_query_engine(
    output_cls=ProductTableEntry,
    response_mode="compact",
    llm=chat_llm,
)

# Ensure that tool_calls is not empty
content, tool_calls, tool_call_ids, status = query_engine._get_content_and_tool_calls(response)

if len(tool_calls) < 1:
    # Handle the case where there are no tool calls
    print("Warning: No tool calls found in the response.")
    tool_calls = [{"name": "default_tool", "input": "default_input", "toolUseId": "default_id"}]

response = query_engine.query("What is the product order number?")
print(response)

Alternatively, you can set error_on_no_tool_call to False:

query_engine.error_on_no_tool_call = False

response = query_engine.query("What is the product order number?")
print(response)

By ensuring that there is at least one tool call or by handling the absence of tool calls appropriately, you can resolve the ValueError and proceed with your integration ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[theholymath]

Ok, my LLM is the Azure OpenAI. I don't know if it supports function calling. This could be the problem. is there a way to do use function calling with Azure Open AI?

[dosubot[bot]]

Yes, Azure OpenAI supports function calling. You can enable or use it in the context of integrating Pydantic with a StorageContext and QueryEngine using the LlamaIndex library. Here is an example of how you can define and use function calling with Azure OpenAI in your integration:

1. Define the function call in your chat messages:

   from llama_index.core.base.llms.types import ChatMessage, MessageRole
   from openai.types.chat.chat_completion_message_tool_call import ChatCompletionMessageToolCall, Function
   
   chat_messages = [
       ChatMessage(
           role=MessageRole.ASSISTANT,
           content=None,
           additional_kwargs={
               "tool_calls": [
                   ChatCompletionMessageToolCall(
                       id="0123",
                       type="function",
                       function=Function(
                           name="search_hotels",
                           arguments='{\n  "location": "San Diego",\n  "max_price": 300,\n  "features": "beachfront,free breakfast"\n}',
                       ),
                   )
               ],
           },
       ),
   ]

2. Convert chat messages to OpenAI message dicts:

   from llama_index.llms.openai.utils import to_openai_message_dicts
   
   openai_message_dicts = to_openai_message_dicts(chat_messages)

3. Define the Pydantic model and convert it to an OpenAI tool:

   from pydantic import BaseModel
   from llama_index.llms.openai.utils import to_openai_tool
   from llama_index.core.tools import FunctionTool
   
   class TestOutput(BaseModel):
       test: str
   
   def generate_test_output(test: str) -> TestOutput:
       return TestOutput(test=test)
   
   tool = FunctionTool.from_defaults(fn=generate_test_output)

4. Use the converted message dicts and tools in your StorageContext and QueryEngine:

   from llama_index.llms.openai import OpenAI
   from llama_index.core.storage import StorageContext
   from llama_index.core.query_engine import QueryEngine
   
   # Assuming you have a StorageContext and QueryEngine setup
   storage_context = StorageContext()
   query_engine = QueryEngine(storage_context=storage_context)
   
   # Use the openai_message_dicts and tool in your query engine
   llm = OpenAI(model="gpt-3.5-turbo")
   response = llm.predict_and_call([tool], openai_message_dicts)
   print(str(response))

This setup demonstrates how to enable and use function calling with Azure OpenAI in the context of integrating Pydantic with a StorageContext and QueryEngine using the LlamaIndex library. For more detailed examples and additional functionalities, you can refer to the LlamaIndex documentation ^[1].

Additionally, the LlamaIndex library supports integrated structured responses using various response_modes such as refine, compact, tree_summarize, accumulate, and compact_accumulate. These modes allow for the conversion of LLM responses into Pydantic objects, ensuring reliable structured outputs. This can work with any LLM, although non-OpenAI support is still in development and considered beta ^[2][3].

To continue talking to Dosu, mention @dosu.

[theholymath]

Still getting the same errors. Thanks anyways

[pierson-we]

Have you found a way to make this work? I'm having the same issue and have not had success. The simple functionality you detailed in your original post used to work just fine for me in previous llama-index versions, but when updating llama-index to try to use gpt-4o models it broke.

[AbdullahKIRMAN]

You can try using llama-index version 0.10.56. Additionally, version 0.10.56 supports gpt-4o-mini and gpt-4o models.