Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): add EventSource protocol #150

Merged
merged 8 commits into from
Nov 16, 2023
Merged

feat(core): add EventSource protocol #150

merged 8 commits into from
Nov 16, 2023

Conversation

ajndkr
Copy link
Owner

@ajndkr ajndkr commented Nov 16, 2023

Description

Fixes #121
Fixes #112

This PR adds the EventSource protocol for streaming LLM output. We still respect the text and JSON streaming modes but all events will now stream as follows:

data: <llm_token_1>

data: <llm_token_2>

This also allows the possibility of implementing named events for special use cases:

event: <event_a>
data: <llm_token_1>

event: <event_b>
data: <llm_token_2>

Finally, the StreamingResponse also improves error handling by sending a 500 error event when chain execution fails (Fixes #146)

Changelog:

  • ➕ added sse-starlette dependency
  • ♻️ refactored StreamingResponse class and base callback handlers

➕ add httpx-sse dependency for unit tests
use sse_starlette.EventSourceResponse instead of fastapi.responses.StreamingResponse

send 500 error event when chain execution fails
use ServerSentEvent to construct message
@ajndkr ajndkr added feature New feature or request dependencies labels Nov 16, 2023
@ajndkr ajndkr self-assigned this Nov 16, 2023
@ajndkr ajndkr changed the title feat(core): add EventSource compliance feat(core): add EventSource protocol Nov 16, 2023
@ajndkr ajndkr merged commit 3c67a51 into main Nov 16, 2023
2 checks passed
@ajndkr ajndkr deleted the feat/sse-response branch November 16, 2023 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies feature New feature or request
Projects
None yet
1 participant