0 Fork 0

mirror of https://github.com/langchain-ai/langchain.git synced 2026-10-06 09:55:21 +03:00

Files

T

Kwuang Tang a508afa91c Add file filter param to Git loader (#2904 )

Allows users to specify what files should be loaded instead of
indiscriminately loading the entire repo.

extends #2851 

NOTE: for reviewers, `hide whitespace` option recommended since I
changed the indentation of an if-block to use `continue` instead so it
looks less like a Christmas tree :)

2023-04-14 10:45:54 -07:00

3.6 KiB

Raw Permalink Blame History

Git

This notebook shows how to load text files from Git repository.

Load existing repository from disk

In [5]:

from git import Repo

repo = Repo.clone_from(
    "https://github.com/hwchase17/langchain", to_path="./example_data/test_repo1"
)
branch = repo.head.reference

In [ ]:

from langchain.document_loaders import GitLoader

In [6]:

loader = GitLoader(repo_path="./example_data/test_repo1/", branch=branch)

In [8]:

data = loader.load()

In [ ]:

len(data)

In [9]:

print(data[0])

page_content='.venv\n.github\n.git\n.mypy_cache\n.pytest_cache\nDockerfile' metadata={'file_path': '.dockerignore', 'file_name': '.dockerignore', 'file_type': ''}

Clone repository from url

In [5]:

from langchain.document_loaders import GitLoader

In [2]:

loader = GitLoader(
    clone_url="https://github.com/hwchase17/langchain",
    repo_path="./example_data/test_repo2/",
    branch="master",
)

In [10]:

data = loader.load()

In [11]:

len(data)

Out [11]:

Filtering files to load

In [12]:

from langchain.document_loaders import GitLoader

# eg. loading only python files
loader = GitLoader(repo_path="./example_data/test_repo1/", file_filter=lambda file_path: file_path.endswith(".py"))

In [ ]:

3.6 KiB Raw Permalink Blame History

Git

Load existing repository from disk

Clone repository from url

Filtering files to load

3.6 KiB

Raw Permalink Blame History