langchain/docs/modules/indexes/text_splitters/examples/python.ipynb

Python Code

PythonCodeTextSplitter splits text along python class and method definitions. It's implemented as a simple subclass of RecursiveCharacterSplitter with Python-specific separators. See the source code to see the Python syntax expected by default.

1. How the text is split: by list of python specific separators
2. How the chunk size is measured: by number of characters

In [1]:

from langchain.text_splitter import PythonCodeTextSplitter

In [2]:

python_text = """
class Foo:

    def bar():
    
    
def foo():

def testing_func_with_long_name():

def bar():
"""
python_splitter = PythonCodeTextSplitter(chunk_size=40, chunk_overlap=0)

In [3]:

docs = python_splitter.create_documents([python_text])

In [4]:

docs

Out [4]:

[Document(page_content='class Foo:\n\n    def bar():', metadata={}),
 Document(page_content='def foo():', metadata={}),
 Document(page_content='def testing_func_with_long_name():', metadata={}),
 Document(page_content='def bar():', metadata={})]

In [ ]: