5 Commits

Author	SHA1	Message	Date
Christian Risi	0975c19e69	added nwew method to encode from list of tokens	2025-10-02 08:48:13 +02:00
Christian Risi	b80b4e4112	Fixed returning type hints	2025-10-02 01:29:57 +02:00
Christian Risi	89a0a1f4bb	Fixed bug for utf-8 conversion	2025-09-30 23:58:31 +02:00
Christian Risi	c9032cab09	Added fit method	2025-09-30 13:33:28 +02:00
Christian Risi	e433941405	Added BPE ... TODO: - complete the fit method	2025-09-28 18:04:44 +02:00