0 Fork 0

mirror of https://github.com/langchain-ai/langchain.git synced 2026-10-09 11:25:21 +03:00

Files

T

ByronHsuandbyhsu 9d658aaa5a Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171 )

As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.

Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170

## Who can review?

Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17

---------

Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>

2023-05-30 11:04:05 -04:00

3.7 KiB

Raw Blame History

CodeTextSplitter

CodeTextSplitter allows you to split your code with multiple language support. Import enum Language and specify the language.

In [6]:

from langchain.text_splitter import (
    CodeTextSplitter,
    Language,
)

Choose a language to use

In [7]:

python_splitter = CodeTextSplitter(
    language=Language.PYTHON, chunk_size=16, chunk_overlap=0
)
js_splitter = CodeTextSplitter(
    language=Language.JS, chunk_size=16, chunk_overlap=0
)

Split the code

In [8]:

PYTHON_CODE = """
def hello_world():
    print("Hello, World!")

# Call the function
hello_world()
"""

python_docs = python_splitter.create_documents([PYTHON_CODE])
python_docs

Out [8]:

[Document(page_content='def', metadata={}),
 Document(page_content='hello_world():', metadata={}),
 Document(page_content='print("Hello,', metadata={}),
 Document(page_content='World!")', metadata={}),
 Document(page_content='# Call the', metadata={}),
 Document(page_content='function', metadata={}),
 Document(page_content='hello_world()', metadata={})]

In [9]:

JS_CODE = """
function helloWorld() {
  console.log("Hello, World!");
}

// Call the function
helloWorld();
"""

js_docs = js_splitter.create_documents([JS_CODE])
js_docs

Out [9]:

[Document(page_content='function', metadata={}),
 Document(page_content='helloWorld() {', metadata={}),
 Document(page_content='console.log("He', metadata={}),
 Document(page_content='llo,', metadata={}),
 Document(page_content='World!");', metadata={}),
 Document(page_content='}', metadata={}),
 Document(page_content='// Call the', metadata={}),
 Document(page_content='function', metadata={}),
 Document(page_content='helloWorld();', metadata={})]

In [ ]:

3.7 KiB Raw Blame History

CodeTextSplitter

Choose a language to use

Split the code

3.7 KiB

Raw Blame History