155 Commits

Author	SHA1	Message	Date
Christian Risi	8db35732f9	Added Chunker to restrict our domains	2025-09-26 18:50:23 +02:00
Christian Risi	9552d61f8d	Added Excetption for when we don't find a delimiter	2025-09-26 18:49:56 +02:00
Christian Risi	be8a87ce01	Modified the architecture for BPE	2025-09-26 18:49:29 +02:00
Christian Risi	3f48b5c428	Added text files to test a chunker	2025-09-26 18:48:44 +02:00
Christian Risi	9972ab8a51	Added imports	2025-09-26 18:48:23 +02:00