4 people 9cc85387d1 fix(text-splitters): add validation to prevent infinite loop and prevent empty token splitter (#32205) ...

### Description
1) Add validation to prevent infinite loop condition when
```tokenizer.tokens_per_chunk > tokenizer.chunk_overlap```
2) Avoid empty decoded chunk when splitter appends tokens

---------

Co-authored-by: Eugene Yurtsev <eugene@langchain.dev>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>

2025-09-11 16:55:32 -04:00

..

integration_tests

chore(text-splitters): select ALL rules with exclusions (#32325)

2025-09-08 14:46:09 +00:00

test_data

text-splitters[patch]: fix HTMLSectionSplitter parsing of xslt paths (#22176)

2024-06-03 20:26:59 +00:00

unit_tests

fix(text-splitters): add validation to prevent infinite loop and prevent empty token splitter (#32205)

2025-09-11 16:55:32 -04:00

__init__.py

text-splitters[minor], langchain[minor], community[patch], templates, docs: langchain-text-splitters 0.0.1 (#18346)

2024-02-29 18:33:21 -08:00