How to use StreamingResponse Callbacks with Fastapi

[Rohith295]

Hello, I have an existing FastAPI application that integrates LangChain. I'm looking to add a new endpoint to support conversation streams. Currently, I'm using the ChatOpenAI class within the ConversationalRetrievalChain. I need guidance on setting up an endpoint that, when triggered with a user prompt, initiates streaming from this chain. Despite searching through various discussions, I haven't found examples that specifically use FastAPI with this setup. Could someone provide assistance on how to achieve this?

[ajndkr]

Hi @Rohith295! thanks for checking out lanarky!

To setup a streaming response with fastapi, you can check out the docs: https://lanarky.ajndkr.com/learn/adapters/langchain/fastapi/#streaming

Let me know if it helps!

[ajndkr]

@Rohith295 super!

If you don't mind me asking how did you approach this? The langchain's documentation is so confusing. I hardly find it useful.

Yeah sure! I agree the documentation can be confusing at times. Especially for this chain type, there's no example code/notebook. There was only some code in the API reference but it wasn't super useful. In such cases when the documentation requires more work, the best option is to just dive into the source code. It's quite easy with an IDE (i use vscode). There I found that the from_llm classmethod reuses the same llm instance for all sub-chains unless you pass the additional parameter i mentioned above.

About debugging the callback handlers, I always use langchain's StreamingStdOutCallbackHandler callback handler. It helps me understand if the lanarky callback handlers contain bugs or the chain. In this case, I found that the chain had the bug.

Beyond that, I have been using Langchain since the start of 2023 so I have developed a good enough intuition about knowing where the bug(s) might be 😅

[Rohith295]

Haha. Thanks for the pointers @ajndkr .

[Rohith295]

@ajndkr Hello Again, I have another question. I have source documents as part of the vector db. How can i include them while streaming the tokens? Is there any recommended way, because i have enabled return_source_documents=True, but only see them when i am doing chain.invoke() as it returns source_documents key in the output, but in the streaming api, its only tokens which are generated by the llm.

[ajndkr]

@Rohith295 you'll need to pass the SourceDocumentStreamingCallbackHandler along with the token streaming callback. You can find all the callback handlers provided by lanarky here: https://lanarky.ajndkr.com/learn/adapters/langchain/callbacks/

[Rohith295]

Can you provide with sample code. I did added as part callbacks handler list, but still running into run time errors.

Issue fixed.

[ajndkr]

@Rohith295 please feel free to open new issues/discussions for new topics. I will close this discussion for now as the main question has been answered.