0 Fork 0

mirror of https://github.com/langchain-ai/langchain.git synced 2026-10-09 19:35:20 +03:00

Files

T

Michael GoinandBagatur 621da3c164 Adds DeepSparse as an LLM (#9184 )

Adds [DeepSparse](https://github.com/neuralmagic/deepsparse) as an LLM
backend. DeepSparse supports running various open-source sparsified
models hosted on [SparseZoo](https://sparsezoo.neuralmagic.com/) for
performance gains on CPUs.

Twitter handles: @mgoin_ @neuralmagic


---------

Co-authored-by: Bagatur <baskaryan@gmail.com>

2023-08-13 22:35:58 -07:00

2.3 KiB

Raw Blame History

DeepSparse

This page covers how to use the DeepSparse inference runtime within LangChain. It is broken into two parts: installation and setup, and then examples of DeepSparse usage.

Installation and Setup

Install the Python package with pip install deepsparse
Choose a SparseZoo model or export a support model to ONNX using Optimum

There exists a DeepSparse LLM wrapper, that provides a unified interface for all models:

In [ ]:

from langchain.llms import DeepSparse

llm = DeepSparse(model='zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none')

print(llm('def fib():'))

Additional parameters can be passed using the config parameter:

In [ ]:

config = {'max_generated_tokens': 256}

llm = DeepSparse(model='zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none', config=config)

2.3 KiB Raw Blame History

DeepSparse

Installation and Setup

2.3 KiB

Raw Blame History