4 Commits

Author	SHA1	Message	Date
Christian Risi	e433941405	Added BPE ... TODO: - complete the fit method	2025-09-28 18:04:44 +02:00
Christian Risi	d179e01971	Added Splitter to divide tokens from text	2025-09-28 18:03:16 +02:00
Christian Risi	3e8b5c5579	Added test for chunker	2025-09-26 18:50:32 +02:00
Christian Risi	3f48b5c428	Added text files to test a chunker	2025-09-26 18:48:44 +02:00