Enhancement: Dedicated Reverse Proxy Endpoint

[eburnette]

Contact Details

Redacted

What features would you like to see added?

Support models served by Anyscale (https://app.endpoints.anyscale.com/).

More details

Anyscale supports a number of open source llms as fully managed API endpoints. The first million tokens are free so all you need is to sign up and get an API key. They support the OpenAI API, including function calls.

Which components are impacted by your request?

No response

Pictures

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

https://docs.anyscale.com/endpoints/model-serving/openai-migration-guide

It should already work if you set OPENAI_REVERSE_PROXY to ANYSCALE_API_BASE

[fuegovic]

It seems to be working indeed:

OPENAI_API_KEY=esecret_xxxxxxxxxxxxxxxxja79 # Anyscale API key
OPENAI_REVERSE_PROXY=https://api.endpoints.anyscale.com/v1/chat/completions

[eburnette]

That works, but it still says OpenAI in the UI, and once you have this set up you can't switch to an OpenAI model.

BTW I found a similar site, portkey.ai (https://portkey.ai/docs/welcome/what-is-portkey) that supports more models. For all but OpenAI, though, you have to use their sdk.

[fuegovic]

That works, but it still says OpenAI in the UI, and once you have this set up you can't switch to an OpenAI model.

This has been discussed on discord. Looks like it's planned, but it's not a priority.

[fuegovic]

@eburnette @danny-avila I changed the title to better reflect the nature of the feature request, and re-opened it since a dedicated reverse proxy endpoint was requested more than once

[danny-avila]

Indeed this is planned and should actually be pretty simple to implement once I put the time in, thank you for clarifying @fuegovic @eburnette

[danny-avila]

Making progress with the Custom Endpoint (aka Dedicated Reverse Proxy Endpoint)

[danny-avila]

I'm also thinking about how to allow multiple custom endpoints. For example, if i want openrouter, mistral api, and openai available all at once