0 Fork 0

mirror of https://github.com/langchain-ai/langchain.git synced 2026-10-09 19:35:20 +03:00

Files

T

Jiayi NiandBagatur 1efb9bae5f FEAT: Integrate Xinference LLMs and Embeddings (#8171 )

- [Xorbits
Inference(Xinference)](https://github.com/xorbitsai/inference) is a
powerful and versatile library designed to serve language, speech
recognition, and multimodal models. Xinference supports a variety of
GGML-compatible models including chatglm, whisper, and vicuna, and
utilizes heterogeneous hardware and a distributed architecture for
seamless cross-device and cross-server model deployment.
- This PR integrates Xinference models and Xinference embeddings into
LangChain.
- Dependencies: To install the depenedencies for this integration, run
    
    `pip install "xinference[all]"`
    
- Example Usage:

To start a local instance of Xinference, run `xinference`.

To deploy Xinference in a distributed cluster, first start an Xinference
supervisor using `xinference-supervisor`:

`xinference-supervisor -H "${supervisor_host}"`

Then, start the Xinference workers using `xinference-worker` on each
server you want to run them on.

`xinference-worker -e "http://${supervisor_host}:9997"`

To use Xinference with LangChain, you also need to launch a model. You
can use command line interface (CLI) to do so. Fo example: `xinference
launch -n vicuna-v1.3 -f ggmlv3 -q q4_0`. This launches a model named
vicuna-v1.3 with `model_format="ggmlv3"` and `quantization="q4_0"`. A
model UID is returned for you to use.

Now you can use Xinference with LangChain:

```python
from langchain.llms import Xinference

llm = Xinference(
    server_url="http://0.0.0.0:9997", # suppose the supervisor_host is "0.0.0.0"
    model_uid = {model_uid} # model UID returned from launching a model
)

llm(
    prompt="Q: where can we visit in the capital of France? A:",
    generate_config={"max_tokens": 1024},
)
```

You can also use RESTful client to launch a model:
```python
from xinference.client import RESTfulClient

client = RESTfulClient("http://0.0.0.0:9997")

model_uid = client.launch_model(model_name="vicuna-v1.3", model_size_in_billions=7, quantization="q4_0")
```

The following code block demonstrates how to use Xinference embeddings
with LangChain:
```python
from langchain.embeddings import XinferenceEmbeddings

xinference = XinferenceEmbeddings(
    server_url="http://0.0.0.0:9997",
    model_uid = model_uid
)
```

```python
query_result = xinference.embed_query("This is a test query")
```

```python
doc_result = xinference.embed_documents(["text A", "text B"])
```

Xinference is still under rapid development. Feel free to [join our
Slack
community](https://xorbitsio.slack.com/join/shared_invite/zt-1z3zsm9ep-87yI9YZ_B79HLB2ccTq4WA)
to get the latest updates!

- Request for review: @hwchase17, @baskaryan
- Twitter handle: https://twitter.com/Xorbitsio

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>

2023-07-27 21:23:19 -07:00

3.3 KiB

Raw Blame History

Xorbits inference (Xinference)

This notebook goes over how to use Xinference embeddings within LangChain

Installation

Install Xinference through PyPI:

In [ ]:

%pip install "xinference[all]"

Deploy Xinference Locally or in a Distributed Cluster.

For local deployment, run xinference.

To deploy Xinference in a cluster, first start an Xinference supervisor using the xinference-supervisor. You can also use the option -p to specify the port and -H to specify the host. The default port is 9997.

Then, start the Xinference workers using xinference-worker on each server you want to run them on.

You can consult the README file from Xinference for more information.

Wrapper

To use Xinference with LangChain, you need to first launch a model. You can use command line interface (CLI) to do so:

In [8]:

!xinference launch -n vicuna-v1.3 -f ggmlv3 -q q4_0

Model uid: 915845ee-2a04-11ee-8ed4-d29396a3f064

A model UID is returned for you to use. Now you can use Xinference embeddings with LangChain:

In [9]:

from langchain.embeddings import XinferenceEmbeddings

xinference = XinferenceEmbeddings(
    server_url="http://0.0.0.0:9997",
    model_uid = "915845ee-2a04-11ee-8ed4-d29396a3f064"
)

In [10]:

query_result = xinference.embed_query("This is a test query")

In [11]:

doc_result = xinference.embed_documents(["text A", "text B"])

Lastly, terminate the model when you do not need to use it:

In [12]:

!xinference terminate --model-uid "915845ee-2a04-11ee-8ed4-d29396a3f064"

3.3 KiB Raw Blame History

Xorbits inference (Xinference)

Installation

Deploy Xinference Locally or in a Distributed Cluster.

Wrapper

3.3 KiB

Raw Blame History