langchain/docs/extras/integrations/document_loaders/conll-u.ipynb

CoNLL-U

CoNLL-U is revised version of the CoNLL-X format. Annotations are encoded in plain text files (UTF-8, normalized to NFC, using only the LF character as line break, including an LF character at the end of file) with three types of lines:

• Word lines containing the annotation of a word/token in 10 fields separated by single tab characters; see below.
• Blank lines marking sentence boundaries.
• Comment lines starting with hash (#).

This is an example of how to load a file in CoNLL-U format. The whole file is treated as one document. The example data (conllu.conllu) is based on one of the standard UD/CoNLL-U examples.

In [1]:

from langchain.document_loaders import CoNLLULoader

In [2]:

loader = CoNLLULoader("example_data/conllu.conllu")

In [3]:

document = loader.load()

In [4]:

document

Out [4]:

[Document(page_content='They buy and sell books.', metadata={'source': 'example_data/conllu.conllu'})]