langchain/docs/examples/integrations/textsplitter.ipynb

Text Splitter

When you want to deal wit long pieces of text, it is necessary to split up that text into chunks. This notebook showcases several ways to do that.

At a high level, text splitters work as following:

1. Split the text up into small, semantically meaningful chunks (often sentences).
2. Start combining these small chunks into a larger chunk until you reach a certain size (as measured by some function).
3. Once you reach that size, make that chunk its own piece of text and then start creating a new chunk of text with some overlap (to keep context between chunks).

In [1]:

from langchain.text_splitter import CharacterTextSplitter, NLTKTextSplitter, SpacyTextSplitter
# This is a long document we can split up.
with open('../state_of_the_union.txt') as f:
    state_of_the_union = f.read()

Character Text Splitting

Let's start with the most simple method: let's split based on characters (by default "\n\n") and measure chunk length by number of characters.

In [2]:

text_splitter = CharacterTextSplitter(        
    separator = "\n\n",
    chunk_size = 1000,
    chunk_overlap  = 200,
    length_function = len,
)

In [3]:

texts = text_splitter.split_text(state_of_the_union)
texts[0]

Out [3]:

'Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n\nLast year COVID-19 kept us apart. This year we are finally together again. \n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n\nWith a duty to one another to the American people to the Constitution. \n\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. \n\nGroups of citizens blocking tanks with their bodies. Everyone from students to retirees teachers turned soldiers defending their homeland. '

HuggingFace Length Function

Most LLMs are constrained by the number of tokens that you can pass in, which is not the same as the number of characters. In order to get a more accurate estimate, we can use HuggingFace tokenizers to count the text length.

In [4]:

from transformers import GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")

None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

In [5]:

text_splitter = CharacterTextSplitter.from_huggingface_tokenizer(tokenizer, chunk_size=100, chunk_overlap=0)
texts = text_splitter.split_text(state_of_the_union)

In [6]:

print(texts[0])

Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  

Last year COVID-19 kept us apart. This year we are finally together again. 

Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. 

With a duty to one another to the American people to the Constitution. 

And with an unwavering resolve that freedom will always triumph over tyranny. 

tiktoken (OpenAI) Length Function

You can also use tiktoken, a open source tokenizer package from OpenAI to estimate tokens used. Will probably be ore accurate for their models.

In [7]:

text_splitter = CharacterTextSplitter.from_tiktoken_encoder(chunk_size=100, chunk_overlap=0)
texts = text_splitter.split_text(state_of_the_union)

In [8]:

print(texts[0])

Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  

Last year COVID-19 kept us apart. This year we are finally together again. 

Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. 

With a duty to one another to the American people to the Constitution. 

And with an unwavering resolve that freedom will always triumph over tyranny. 

NLTK Text Splitter

Rather than just splitting on "\n\n", we can use NLTK to split based on tokenizers.

In [14]:

text_splitter = NLTKTextSplitter(chunk_size=1000)

In [16]:

texts = text_splitter.split_text(state_of_the_union)
texts[0]

Out [16]:

'Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.\n\nMembers of Congress and the Cabinet.\n\nJustices of the Supreme Court.\n\nMy fellow Americans.\n\nLast year COVID-19 kept us apart.\n\nThis year we are finally together again.\n\nTonight, we meet as Democrats Republicans and Independents.\n\nBut most importantly as Americans.\n\nWith a duty to one another to the American people to the Constitution.\n\nAnd with an unwavering resolve that freedom will always triumph over tyranny.\n\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.\n\nBut he badly miscalculated.\n\nHe thought he could roll into Ukraine and the world would roll over.\n\nInstead he met a wall of strength he never imagined.\n\nHe met the Ukrainian people.\n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.\n\nGroups of citizens blocking tanks with their bodies.\n\nEveryone from students to retirees teachers turned soldiers defending their homeland.'

Spacy Text Splitter

Another alternative to NLTK is to use Spacy.

In [17]:

text_splitter = SpacyTextSplitter(chunk_size=1000)

In [19]:

texts = text_splitter.split_text(state_of_the_union)
texts[0]

Out [19]:

'Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.\n\nMembers of Congress and the Cabinet.\n\nJustices of the Supreme Court.\n\nMy fellow Americans.  \n\n\n\nLast year COVID-19 kept us apart.\n\nThis year we are finally together again.\n\n\n\n\n\nTonight, we meet as Democrats Republicans and Independents.\n\nBut most importantly as Americans.\n\n\n\n\n\nWith a duty to one another to the American people to the Constitution. \n\n\n\nAnd with an unwavering resolve that freedom will always triumph over tyranny.\n\n\n\n\n\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.\n\nBut he badly miscalculated.\n\n\n\n\n\nHe thought he could roll into Ukraine and the world would roll over.\n\nInstead he met a wall of strength he never imagined.\n\n\n\n\n\nHe met the Ukrainian people.\n\n\n\n\n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.\n\n\n\n\n\nGroups of citizens blocking tanks with their bodies.\n\nEveryone from students to retirees teachers turned soldiers defending their homeland.'

In [ ]: