Does this work with Agents and Agent_Executor?

[hgoona]

I've currently got the following kind of Custom LLM Agent Executor setup, and I'm wondering if your Streaming can be used with this type of agent? Would greatly appreciate any pointers as to how to implement!

... 
# AGENT🤖
tool_names = [tool.name for tool in tools]
agent = LLMSingleActionAgent(
    llm_chain=llm_chain, 
    output_parser=output_parser,
    stop=["\nObservation:"], 
    allowed_tools=tool_names,
)


# AGENT EXECUTOR🤖▶
agent_executor = AgentExecutor.from_agent_and_tools(
    agent=agent,
    tools=tools,
    verbose=True,
    memory=memBuffer,  # 🧠
    max_iterations=3,  # 🔄
)


# API STUFF
from pydantic import BaseModel

class HumanMsg(BaseModel):
    message: str
    chat_history: str | None = None
    
class AIMsg(BaseModel):
    status: str | None = "🤔"
    message: str
    # description: str | None = None
    # price: float
    # tax: float | None = None

# TOKEN COUNT🎫
from langchain.callbacks import get_openai_callback

# EXPORT FUNCTION FOR API
def bettyAI(input: HumanMsg):
    
    with get_openai_callback() as cb:
        result = agent_executor.run(input.message)
        
        print("$", cb.total_cost)
        return { 
                "message": result,
                }

[ajndkr]

Hi! I will check this out. The way how streaming works is we use callback handlers. The async callback handler has a method call on_llm_new_token where we directly stream the input tokens.

In case of agents, there are 2 scenarios:

• you want to stream everything, i.e., intermediate steps + final answer
• you want to stream only the final answer

first case is easy. we can simply reuse the available callback handler to stream everything. The second one is a bit tricky because we need to detect when the agent has reached the final answer. It will take some time but if you already figure something out, you're more than welcome to contribute with a pull request!

[hgoona]

btw @ajndkr Have you seen this for streaming?

I'm trying to implement it to a JS frontend in this manner: https://youtu.be/dXsZp39L2Jk?t=1141

[ajndkr]

what would scenario 1 look like??

the first scenario is similar to how chains works with this package. instead of a chain, you can switch it with an agent or agent executor and it should work. let me know if it doesn't.

btw @ajndkr Have you seen this for streaming?

I'm trying to implement it to a JS frontend in this manner: https://youtu.be/dXsZp39L2Jk?t=1141

i believe they are using the openai API directly without langchain. not sure.

[rbbby]

I think scenario 2 may have been solved here?
langchain-ai/langchain@0fc762f

[ajndkr]

It seems so! Will check this out!

[ajndkr]

@rbbby i've added support for Langchain agents based on your suggestion!

[hgoona]

def bettyAI(input: HumanMsg):
    
    with get_openai_callback() as cb:
        result = agent_executor.run(input.message)
        
        print("$", cb.total_cost)
        return { 
                "message": result,
                }

@ajndkr Really? Can I use it in the above where the agent_executor is within a callback for Token counts?

def bettyAI(input: HumanMsg):
    
    with get_openai_callback() as cb:
        result = agent_executor.run(input.message)
        
        print("$", cb.total_cost)
        return { 
                "message": result,
                } 

I re-wrote everything earlier but it failed so i changed it back to the above. Possible to integrate into the above?? or do i need to abandon the get_openai_callback in order to achieve streaming??

[ajndkr]

@hgoona I've recently added support for Langchain Agents. You can check it out and let me know if it solves your use case.

Demo example: https://github.com/ajndkr/lanarky/tree/main/examples#zero-shot-agent
Code: https://github.com/ajndkr/lanarky/blob/main/examples/app/zero_shot_agent.py

Make sure you install the correct version!

pip install lanarky==0.6.3

[rbbby]

@ajndkr it works nicely now thx

[shenghann]

Hi @ajndkr, thanks for adding streaming support to agents. I'm running the zero shot agent example on 0.6.5 and got the following error when trying out using the chatbot UI with websockets:

"Error! Chain type 'AgentExecutor' is not currently supported by 'AsyncWebsocketCallback'."

Any ideas? Did the latest commits break the example?

[ajndkr]

@shenghann maybe. I will look into this!

[ajndkr]

@shenghann Can you please install the latest version of lanarky and try again? It should be working now.

[shenghann]

@ajndkr Thanks a lot, no longer seeing the error on 0.6.6!