langchain/docs/extras/integrations/chat/litellm.ipynb

🚅 LiteLLM

LiteLLM is a library that simplifies calling Anthropic, Azure, Huggingface, Replicate, etc.

This notebook covers how to get started with using Langchain + the LiteLLM I/O library.

In [1]:

from langchain.chat_models import ChatLiteLLM
from langchain.prompts.chat import (
    ChatPromptTemplate,
    SystemMessagePromptTemplate,
    AIMessagePromptTemplate,
    HumanMessagePromptTemplate,
)
from langchain.schema import AIMessage, HumanMessage, SystemMessage

In [2]:

chat = ChatLiteLLM(model="gpt-3.5-turbo")

In [3]:

messages = [
    HumanMessage(
        content="Translate this sentence from English to French. I love programming."
    )
]
chat(messages)

Out [3]:

AIMessage(content=" J'aime la programmation.", additional_kwargs={}, example=False)

ChatLiteLLM also supports async and streaming functionality:

In [4]:

from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

In [5]:

await chat.agenerate([messages])

Out [5]:

LLMResult(generations=[[ChatGeneration(text=" J'aime programmer.", generation_info=None, message=AIMessage(content=" J'aime programmer.", additional_kwargs={}, example=False))]], llm_output={}, run=[RunInfo(run_id=UUID('8cc8fb68-1c35-439c-96a0-695036a93652'))])

In [6]:

chat = ChatLiteLLM(
    streaming=True,
    verbose=True,
    callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),
)
chat(messages)

Out [6]:

 J'aime la programmation.

AIMessage(content=" J'aime la programmation.", additional_kwargs={}, example=False)

In [ ]: