-
Notifications
You must be signed in to change notification settings - Fork 74
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* ✨ add llm cache to LangchainRouter * ➕ add `cache` extras * 📝 update README * ♻️ refactor `register` module * ✅ update unit tests * 📝 update docs * ➕ add `gptcache` extras * ✨ add gptcache integration * 📝 update docs * 📝 update README
- Loading branch information
Showing
13 changed files
with
275 additions
and
43 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
Deploy Applications with LLM Caching | ||
===================================== | ||
|
||
Langchain offers multiple LLM cache solutions. Reference: `How to cache LLM calls <https://python.langchain.com/en/latest/modules/models/llms/examples/llm_caching.html>`_ | ||
|
||
To simplify the use of these solutions for Lanarky users, the ``LangchainRouter`` can be used to setup LLM caching for your application. | ||
|
||
We'll use a simple ``LLMChain`` application as an example: | ||
|
||
.. code-block:: python | ||
from dotenv import load_dotenv | ||
from fastapi import FastAPI | ||
from langchain import LLMChain | ||
from langchain.llms import OpenAI | ||
from lanarky.routing import LangchainRouter | ||
load_dotenv() | ||
app = FastAPI() | ||
langchain_router = LangchainRouter( | ||
langchain_url="/chat", | ||
langchain_object=LLMChain.from_string( | ||
llm=OpenAI(temperature=0), template="Answer the query.\n{query}" | ||
), | ||
streaming_mode=0, | ||
) | ||
app.include_router(langchain_router) | ||
The ``LangchainRouter`` class uses the ``llm_cache_mode`` parameter to setup LLM caching. | ||
There are three available modes: | ||
|
||
- ``llm_cache_mode=0``: No LLM caching | ||
- ``llm_cache_mode=1``: In-memory LLM caching | ||
- ``llm_cache_mode=2``: Redis LLM caching | ||
- ``llm_cache_mode=3``: GPTCache LLM caching | ||
|
||
In-Memory Caching | ||
----------------- | ||
|
||
To setup in-memory caching, use the following ``LangchainRouter`` configuration: | ||
|
||
.. code-block:: python | ||
langchain_router = LangchainRouter( | ||
langchain_url="/chat", | ||
langchain_object=LLMChain.from_string( | ||
llm=OpenAI(temperature=0), template="Answer the query.\n{query}" | ||
), | ||
streaming_mode=0, | ||
llm_cache_mode=1, | ||
) | ||
Redis Caching | ||
------------- | ||
|
||
To setup Redis caching, first install the required dependencies: | ||
|
||
.. code-block:: bash | ||
pip install "lanarky[redis]" | ||
Next, setup a Redis server. We recommend using Docker: | ||
|
||
.. code-block:: bash | ||
docker run --name redis -p 6379:6379 -d redis | ||
Finally, use the following ``LangchainRouter`` configuration: | ||
|
||
.. code-block:: python | ||
langchain_router = LangchainRouter( | ||
langchain_url="/chat", | ||
langchain_object=LLMChain.from_string( | ||
llm=OpenAI(temperature=0), template="Answer the query.\n{query}" | ||
), | ||
streaming_mode=0, | ||
llm_cache_mode=2, | ||
llm_cache_kwargs={"url": "redis://localhost:6379/"}, | ||
) | ||
GPTCache Caching | ||
---------------- | ||
|
||
To setup GPTCache caching, first install the required dependencies: | ||
|
||
.. code-block:: bash | ||
pip install "lanarky[gptcache]" | ||
Then, use the following ``LangchainRouter`` configuration: | ||
|
||
.. code-block:: python | ||
langchain_router = LangchainRouter( | ||
langchain_url="/chat", | ||
langchain_object=LLMChain.from_string( | ||
llm=OpenAI(temperature=0), template="Answer the query.\n{query}" | ||
), | ||
streaming_mode=0, | ||
llm_cache_mode=3, | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -12,4 +12,5 @@ your Langchain application. | |
:maxdepth: 1 | ||
|
||
deploy | ||
cache | ||
custom_callbacks |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
from .callbacks import ( | ||
STREAMING_CALLBACKS, | ||
STREAMING_JSON_CALLBACKS, | ||
WEBSOCKET_CALLBACKS, | ||
register_streaming_callback, | ||
register_streaming_json_callback, | ||
register_websocket_callback, | ||
) | ||
|
||
__all__ = [ | ||
"STREAMING_CALLBACKS", | ||
"STREAMING_JSON_CALLBACKS", | ||
"WEBSOCKET_CALLBACKS", | ||
"register_streaming_callback", | ||
"register_streaming_json_callback", | ||
"register_websocket_callback", | ||
] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
from typing import Any, Callable, Union | ||
|
||
from .base import register | ||
|
||
STREAMING_CALLBACKS: dict[str, Any] = {} | ||
WEBSOCKET_CALLBACKS: dict[str, Any] = {} | ||
STREAMING_JSON_CALLBACKS: dict[str, Any] = {} | ||
|
||
|
||
def register_streaming_callback(key: Union[list[str], str]) -> Callable: | ||
"""Register a streaming callback handler.""" | ||
|
||
def _register_cls(cls: Any) -> Callable: | ||
register(key, STREAMING_CALLBACKS)(cls=cls) | ||
return cls | ||
|
||
return _register_cls | ||
|
||
|
||
def register_websocket_callback(key: Union[list[str], str]) -> Callable: | ||
"""Register a websocket callback handler.""" | ||
|
||
def _register_cls(cls: Any) -> Callable: | ||
register(key, WEBSOCKET_CALLBACKS)(cls=cls) | ||
return cls | ||
|
||
return _register_cls | ||
|
||
|
||
def register_streaming_json_callback(key: Union[list[str], str]) -> Callable: | ||
"""Register an streaming json callback handler.""" | ||
|
||
def _register_cls(cls: Any) -> Callable: | ||
register(key, STREAMING_JSON_CALLBACKS)(cls=cls) | ||
return cls | ||
|
||
return _register_cls |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,4 @@ | ||
from .langchain import LangchainRouter | ||
from .utils import LLMCacheMode, StreamingMode | ||
|
||
__all__ = ["LangchainRouter"] | ||
__all__ = ["LangchainRouter", "StreamingMode", "LLMCacheMode"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.