0 Fork 0

mirror of https://github.com/langchain-ai/langchain.git synced 2026-10-15 06:09:21 +03:00

Files

T

Jonathan Pedoeem 05ad399abe Update PromptLayerOpenAI LLM to include support for ASYNC API (#1066 )

This PR updates `PromptLayerOpenAI` to now support requests using the
[Async
API](https://langchain.readthedocs.io/en/latest/modules/llms/async_llm.html)
It also updates the documentation on Async API to let users know that
PromptLayerOpenAI also supports this.

`PromptLayerOpenAI` now redefines `_agenerate` a similar was to how it
redefines `_generate`

2023-02-15 22:48:09 -08:00

4.2 KiB

Raw Blame History

Async API for LLM

LangChain provides async support for LLMs by leveraging the asyncio library.

Async support is particularly useful for calling multiple LLMs concurrently, as these calls are network-bound. Currently, only OpenAI and PromptLayerOpenAI is supported, but async support for other LLMs is on the roadmap.

You can use the agenerate method to call an OpenAI LLM asynchronously.

In [5]:

import time
import asyncio

from langchain.llms import OpenAI

def generate_serially():
    llm = OpenAI(temperature=0.9)
    for _ in range(10):
        resp = llm.generate(["Hello, how are you?"])
        print(resp.generations[0][0].text)


async def async_generate(llm):
    resp = await llm.agenerate(["Hello, how are you?"])
    print(resp.generations[0][0].text)


async def generate_concurrently():
    llm = OpenAI(temperature=0.9)
    tasks = [async_generate(llm) for _ in range(10)]
    await asyncio.gather(*tasks)


s = time.perf_counter()
# If running this outside of Jupyter, use asyncio.run(generate_concurrently())
await generate_concurrently() 
elapsed = time.perf_counter() - s
print('\033[1m' + f"Concurrent executed in {elapsed:0.2f} seconds." + '\033[0m')

s = time.perf_counter()
generate_serially()
elapsed = time.perf_counter() - s
print('\033[1m' + f"Serial executed in {elapsed:0.2f} seconds." + '\033[0m')


I'm doing well. How about you?


I'm doing well, thank you. How about you?


I'm doing well, thank you. How about you?


I'm doing well, thank you. How about you?

I am doing quite well. How about you?


I'm doing well, thank you. How about you?


I'm doing great, thank you! How about you?


I'm doing well, thanks for asking. How about you?


I'm doing well, thank you. How about you?


I'm doing well, thank you. How about you?
[1mConcurrent executed in 1.93 seconds.[0m


I'm doing well, thank you. How about you?


I'm doing well, thank you. How about you?


I'm doing well, thank you. How about you?


I'm doing well, thank you. How about you?


I'm doing well, thank you. How about you?


I'm doing well, thank you. How about you?

I'm doing well, thank you. How about you?


I'm doing well, thank you. How about you?


I'm doing well, thank you. How about you?


I'm doing great, thank you. How about you?
[1mSerial executed in 10.54 seconds.[0m

4.2 KiB Raw Blame History

Async API for LLM

4.2 KiB

Raw Blame History