🤖 feat: OpenAI Assistants v2 (initial support) #2781

danny-avila · 2024-05-18T19:18:08Z

Summary

This PR introduces initial OpenAI Assistant v2 support, a separate endpoint for Azure Assistants and includes several bug fixes and improvements. The major changes are summarized below:

Changes

New Features

GPT-4o Support for Assistants (v2)
Native image vision support for Assistants (v2)
Images with Code Interpreter now supported by LibreChat (v2)
- Note: local copies are also made so the image can be displayed
Azure Assistants Endpoint (v1)
- Added a separate endpoint for Azure Assistants, which still uses v1

Notes

Attaching vector stores for File Search is not yet supported. You can attach them from the Provider Playground or attach files to messages for file search on a thread basis.

Bug Fixes

Resolved the issue where the UI stays in an "analyzing" state if the run ends without a response for Code Interpreter.
Fixed an issue with annotations not being applied correctly with multiple Assistant file outputs
Fixed an issue with Assistant avatars becoming "obtuse" for images without squared dimensions
Fixed issue with new function name validation regex, which was affecting the naming strategy for Actions
Fixed issue where database records of files would be prematurely deleted if one of the file deletions failed
- Now checks for each individual file deletion resolution to determine deletion from file records

Testing

Routes tested:

API Versions 1 & 2

GET / - List assistants
POST /avatar/:assistant_id - Upload/update avatar
GET /documents - User's assistant docs
DELETE /:id - Delete assistant
POST / - Create assistant
GET /:id - Fetch assistant (Not used)
PATCH /:id - Update assistant
All v1/actions
GET /tools

Additional Improvements

Added "source" icons for OpenAI storage files to indicate whether they are from Azure or OpenAI.
Adds "unknown" documents to Assistants Panel if uploaded elsewhere
Improved query invalidation logic for assistant-related queries.
Localized text across the application.
Fixed layout shift issues related to Clipboard Icon app-wide
Removed emboldened text from message results, closes [Bug]: Markdown Syntax Included in Copied Search Results #2780

Known Issues

Assistant Deletion Issue
- Issue where deleting the last assistant leaves it selected in the UI form.
Azure Assistants Rate limiting errors.
- Azure is strict with rate limits on polling, so the default rate was increased
- Currently, there will be no message displayed when this happens, needs an error message
When switching between Assistants and Azure Assistants for the first time, an assistant from the other endpoint may incorrectly still be selected in the builder form (but not the conversation state)

Planned Updates

Vector store management UI for File Search
- Addition of this feature will close the V2 issue: Enhancement: OpenAI Assistants API V2 support #2490
Private Assistants - Enhancement: Private Assistants #1926
- Additional metadata, endpoint, author ID, are now added on assistant creation from LibreChat in anticipation of this update
Code Interpreter Image Flag
- A flag to indicate whether to include images to code interpreter
- This is working by default if attached, regardless of assistant access to the tool.
Optimize image upload with web URL handling for assistant vision.
Fetch unknown document metadata
Add/Delete/Fork Messages

- Separated Azure Assistants to its own endpoint - File Search / Vector Store integration is incomplete, but can toggle and use storage from playground - Code Interpreter resource files can be added but not deleted - GPT-4o is supported - Many improvements to the Assistants Endpoint overall data-provider v2 changes copy existing route as v1 chore: rename new endpoint to reduce comparison operations and add new azure filesource api: add azureAssistants part 1 force use of version for assistants/assistantsAzure chore: switch name back to azureAssistants refactor type version: string | number Ensure assistants endpoints have version set fix: isArchived type issue in ConversationListParams refactor: update assistants mutations/queries with endpoint/version definitions, update Assistants Map structure chore: FilePreview component ExtendedFile type assertion feat: isAssistantsEndpoint helper chore: remove unused useGenerations chore(buildTree): type issue chore(Advanced): type issue (unused component, maybe in future) first pass for multi-assistant endpoint rewrite fix(listAssistants): pass params correctly feat: list separate assistants by endpoint fix(useTextarea): access assistantMap correctly fix: assistant endpoint switching, resetting ID fix: broken during rewrite, selecting assistant mention fix: set/invalidate assistants endpoint query data correctly feat: Fix issue with assistant ID not being reset correctly getOpenAIClient helper function feat: add toast for assistant deletion fix: assistants delete right after create issue for azure fix: assistant patching refactor: actions to use getOpenAIClient refactor: consolidate logic into helpers file fix: issue where conversation data was not initially available v1 chat support refactor(spendTokens): only early return if completionTokens isNaN fix(OpenAIClient): ensure spendTokens has all necessary params refactor: route/controller logic fix(assistants/initializeClient): use defaultHeaders field fix: sanitize default operation id chore: bump openai package first pass v2 action service feat: retroactive domain parsing for actions added via v1 feat: delete db records of actions/assistants on openai assistant deletion chore: remove vision tools from v2 assistants feat: v2 upload and delete assistant vision images WIP first pass, thread attachments fix: show assistant vision files (save local/firebase copy) v2 image continue fix: annotations fix: refine annotations show analyze as error if is no longer submitting before progress reaches 1 and show file_search as retrieval tool fix: abort run, undefined endpoint issue refactor: consolidate capabilities logic and anticipate versioning frontend version 2 changes fix: query selection and filter add endpoint to unknown filepath add file ids to resource, deleting in progress enable/disable file search remove version log

🎹 fix: Autocompletion Chrome Bug on Action API Key Input chore: remove `useOriginNavigate` chore: set correct OpenAI Storage Source fix: azure file deletions, instantiate clients by source for deletion update code interpret files info feat: deleteResourceFileId chore: increase poll interval as azure easily rate limits fix: openai file deletions, TODO: evaluate rejected deletion settled promises to determine which to delete from db records file source icons update table file filters chore: file search info and versioning fix: retrieval update with necessary tool_resources if specified fix(useMentions): add optional chaining in case listMap value is undefined fix: force assistant avatar roundedness fix: azure assistants, check correct flag chore: bump data-provider

* 🤖 Assistants V2 Support: Part 1 - Separated Azure Assistants to its own endpoint - File Search / Vector Store integration is incomplete, but can toggle and use storage from playground - Code Interpreter resource files can be added but not deleted - GPT-4o is supported - Many improvements to the Assistants Endpoint overall data-provider v2 changes copy existing route as v1 chore: rename new endpoint to reduce comparison operations and add new azure filesource api: add azureAssistants part 1 force use of version for assistants/assistantsAzure chore: switch name back to azureAssistants refactor type version: string | number Ensure assistants endpoints have version set fix: isArchived type issue in ConversationListParams refactor: update assistants mutations/queries with endpoint/version definitions, update Assistants Map structure chore: FilePreview component ExtendedFile type assertion feat: isAssistantsEndpoint helper chore: remove unused useGenerations chore(buildTree): type issue chore(Advanced): type issue (unused component, maybe in future) first pass for multi-assistant endpoint rewrite fix(listAssistants): pass params correctly feat: list separate assistants by endpoint fix(useTextarea): access assistantMap correctly fix: assistant endpoint switching, resetting ID fix: broken during rewrite, selecting assistant mention fix: set/invalidate assistants endpoint query data correctly feat: Fix issue with assistant ID not being reset correctly getOpenAIClient helper function feat: add toast for assistant deletion fix: assistants delete right after create issue for azure fix: assistant patching refactor: actions to use getOpenAIClient refactor: consolidate logic into helpers file fix: issue where conversation data was not initially available v1 chat support refactor(spendTokens): only early return if completionTokens isNaN fix(OpenAIClient): ensure spendTokens has all necessary params refactor: route/controller logic fix(assistants/initializeClient): use defaultHeaders field fix: sanitize default operation id chore: bump openai package first pass v2 action service feat: retroactive domain parsing for actions added via v1 feat: delete db records of actions/assistants on openai assistant deletion chore: remove vision tools from v2 assistants feat: v2 upload and delete assistant vision images WIP first pass, thread attachments fix: show assistant vision files (save local/firebase copy) v2 image continue fix: annotations fix: refine annotations show analyze as error if is no longer submitting before progress reaches 1 and show file_search as retrieval tool fix: abort run, undefined endpoint issue refactor: consolidate capabilities logic and anticipate versioning frontend version 2 changes fix: query selection and filter add endpoint to unknown filepath add file ids to resource, deleting in progress enable/disable file search remove version log * 🤖 Assistants V2 Support: Part 2 🎹 fix: Autocompletion Chrome Bug on Action API Key Input chore: remove `useOriginNavigate` chore: set correct OpenAI Storage Source fix: azure file deletions, instantiate clients by source for deletion update code interpret files info feat: deleteResourceFileId chore: increase poll interval as azure easily rate limits fix: openai file deletions, TODO: evaluate rejected deletion settled promises to determine which to delete from db records file source icons update table file filters chore: file search info and versioning fix: retrieval update with necessary tool_resources if specified fix(useMentions): add optional chaining in case listMap value is undefined fix: force assistant avatar roundedness fix: azure assistants, check correct flag chore: bump data-provider * fix: merge conflict * ci: fix backend tests due to new updates * chore: update .env.example * meilisearch improvements * localization updates * chore: update comparisons * feat: add additional metadata: endpoint, author ID * chore: azureAssistants ENDPOINTS exclusion warning

danny-avila added 3 commits May 18, 2024 15:22

fix: merge conflict

6082e26

danny-avila force-pushed the v2-assistants branch from 8fe30be to 6082e26 Compare May 18, 2024 19:29

danny-avila added 5 commits May 18, 2024 15:44

ci: fix backend tests due to new updates

3eabbd5

chore: update .env.example

e55a892

meilisearch improvements

5c15b60

localization updates

bce0584

chore: update comparisons

fb6e87c

danny-avila mentioned this pull request May 19, 2024

Enhancement: OpenAI Assistants API V2 support #2490

Open

danny-avila added 2 commits May 19, 2024 12:41

feat: add additional metadata: endpoint, author ID

84f68f9

chore: azureAssistants ENDPOINTS exclusion warning

1c5e827

danny-avila merged commit 1a45212 into main May 19, 2024
4 checks passed

danny-avila deleted the v2-assistants branch May 19, 2024 16:56

danny-avila restored the v2-assistants branch May 21, 2024 14:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🤖 feat: OpenAI Assistants v2 (initial support) #2781

🤖 feat: OpenAI Assistants v2 (initial support) #2781

danny-avila commented May 18, 2024 •

edited

Loading

🤖 feat: OpenAI Assistants v2 (initial support) #2781

🤖 feat: OpenAI Assistants v2 (initial support) #2781

Conversation

danny-avila commented May 18, 2024 • edited Loading

Summary

Changes

New Features

Notes

Bug Fixes

Testing

API Versions 1 & 2

Additional Improvements

Known Issues

Planned Updates

danny-avila commented May 18, 2024 •

edited

Loading