0 Fork 0

mirror of https://github.com/langchain-ai/langchain.git synced 2026-10-13 13:19:22 +03:00

Files

T

Leonid Ganeline c998569c8f docs: text splitters improvements (#4490 )

#docs: text splitters improvements

Changes are only in the Jupyter notebooks.
- added links to the source packages and a short description of these
packages
- removed " Text Splitters" suffixes from the TOC elements (they made
the list of the text splitters messy)
- moved text splitters, based on the length function into a separate
list. They can be mixed with any classes from the "Text Splitters", so
it is a different classification.

## Who can review?
        @hwchase17 - project lead
        @eyurtsev
        @vowelparrot

NOTE: please, check out the results of the `Python code` text splitter
example (text_splitters/examples/python.ipynb). It looks suboptimal.

2023-05-17 21:33:34 -07:00

2.6 KiB

Raw Blame History

tiktoken (OpenAI) tokenizer

tiktoken is a fast BPE tokenizer created by OpenAI.

We can use it to estimate tokens used. It will probably be more accurate for the OpenAI models.

How the text is split: by character passed in
How the chunk size is measured: by tiktoken tokenizer

In [ ]:

#!pip install tiktoken

In [1]:

# This is a long document we can split up.
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()
from langchain.text_splitter import CharacterTextSplitter

In [2]:

text_splitter = CharacterTextSplitter.from_tiktoken_encoder(chunk_size=100, chunk_overlap=0)
texts = text_splitter.split_text(state_of_the_union)

In [3]:

print(texts[0])

Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  

Last year COVID-19 kept us apart. This year we are finally together again. 

Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. 

With a duty to one another to the American people to the Constitution.

2.6 KiB Raw Blame History

tiktoken (OpenAI) tokenizer

2.6 KiB

Raw Blame History