error with gemini tool call

[licycle]

using gemini and tool call func, return empty response content

async def get_voice() -> float:
    # Returns a random stock price for demonstration purposes.
    print("tool using")
    return random.randint(10, 200)

async def run_tool_agent():
    # Create a runtime.
    runtime = SingleThreadedAgentRuntime()
    # Create the tools.
    tools: List[Tool] = [FunctionTool(get_voice, description="get voice")]
    # Register the agents.
    await ToolAgent.register(runtime, "tool_executor_agent", lambda: ToolAgent("tool executor agent", tools))
    await ToolUseAgent.register(
        runtime,
        "tool_use_agent",
        lambda: ToolUseAgent(
            GeminiChatCompletionClient(model="gemini-1.5-pro", api_key="apikey"),
            [tool.schema for tool in tools], "tool_executor_agent"
        ),
    )

[vballoli]

Hi @licycle , can you paste the entire code along with the autogen and autogen-openaiext-client versions?

[licycle]

sure, @vballoli, the code working when i use gpt-4o-mini

pip list | grep auto
autogen-agentchat                  0.4.3
autogen-core                       0.4.3
autogen-ext                        0.4.2
autogen_openaiext_client           0.0.3
autogenstudio                      0.4.0.3

---

import asyncio
import os
import random
from dataclasses import dataclass
from typing import List, Annotated

from autogen_core import (
    AgentId,
    MessageContext,
    RoutedAgent,
    SingleThreadedAgentRuntime,
    message_handler,
)
from autogen_core.models import (
    ChatCompletionClient,
    LLMMessage,
    SystemMessage,
    UserMessage,
)
from autogen_core.tool_agent import ToolAgent, tool_agent_caller_loop
from autogen_core.tools import FunctionTool, Tool, ToolSchema

from autogen_openaiext_client import GeminiChatCompletionClient
import asyncio

# Initialize the client
client = GeminiChatCompletionClient(model="gemini-1.5-flash", api_key="API_KEY")


@dataclass
class Message:
    content: str


class ToolUseAgent(RoutedAgent):
    def __init__(self, model_client: ChatCompletionClient, tool_schema: List[ToolSchema], tool_agent_type: str) -> None:
        super().__init__("An agent with tools")
        self._system_messages: List[LLMMessage] = [SystemMessage(content="You are a helpful AI assistant.")]
        self._model_client = model_client
        self._tool_schema = tool_schema
        self._tool_agent_id = AgentId(tool_agent_type, self.id.key)

    @message_handler
    async def handle_user_message(self, message: Message, ctx: MessageContext) -> Message:
        # Create a session of messages.
        session: List[LLMMessage] = [UserMessage(content=message.content, source="user")]
        # Run the caller loop to handle tool calls.
        messages = await tool_agent_caller_loop(
            self,
            tool_agent_id=self._tool_agent_id,
            model_client=self._model_client,
            input_messages=session,
            tool_schema=self._tool_schema,
            cancellation_token=ctx.cancellation_token,
        )
        # Return the final response.
        assert isinstance(messages[-1].content, str)
        return Message(content=messages[-1].content)


async def get_voice() -> float:
    # Returns a random stock price for demonstration purposes.
    print("tool using")
    return random.randint(10, 200)

async def run_tool_agent():
    # Create a runtime.
    runtime = SingleThreadedAgentRuntime()
    # Create the tools.
    tools: List[Tool] = [FunctionTool(get_voice, description="get voice num")]
    # Register the agents.
    await ToolAgent.register(runtime, "tool_executor_agent", lambda: ToolAgent("tool executor agent", tools))
    await ToolUseAgent.register(
        runtime,
        "tool_use_agent",
        lambda: ToolUseAgent(
            # OpenAIChatCompletionClient(model="gpt-4o-mini", api_key="API_KEY")
            GeminiChatCompletionClient(model="gemini-1.5-pro", api_key="apikey"),
            [tool.schema for tool in tools], "tool_executor_agent"
        ),
    )

    # Start processing messages.
    runtime.start()
    # Send a direct message to the tool agent.
    tool_use_agent = AgentId("tool_use_agent", "default")
    response = await runtime.send_message(Message("Call the voice tool to return the number, and then reply to the user in Chinese. You need to bring the tool to return the number.：hi"), tool_use_agent)
    print(response.content)
    # Stop processing messages.
    await runtime.stop()


if __name__ == '__main__':
    asyncio.run(run_tool_agent())

[vballoli]

I'll take a look at it in some time and update you. Meanwhile, @licycle, you've attached your keys here; don't forget to rotate them out.

[ykgoon]

I need this fixed. If you haven't gotten to it @vballoli, how would you advise approaching this? How would the test case look like?

[ykgoon]

@vballoli I've been trying to reproduce the problem with this test case but unable to.

In other words the test case passes just fine.

I think I'm not understanding the problem correctly.

@licycle could you also help elaborate how yours is failing? Does my test case reflect it?

In actual use case though, I do experience WebSurfer agent not being able to browse the web when using Gemini, presumably because Gemini client isn't able to call functions.

[vballoli]

Hey @ykgoon and @licycle, sorry for the late replies. I've been busy with some deadlines - I believe this is a Gemini API issue and probably should post it here: https://discuss.ai.google.dev/c/gemini-api/4. I'd encountered something similar and added caution in the v0.0.1 of this extension, but I subsequently removed it because I had not faced it again. I can check in on this next week with more concrete answers, but this is all I have for now. Apologies again for not being entirely helpful.