Christian Risi 3b274ad807 Added a way to take the default special token list

2025-10-04 19:43:02 +02:00

..

__init__.py

Modified imports for BPE

2025-10-04 19:42:02 +02:00

default_special_tokens.py

Added a way to take the default special token list

2025-10-04 19:43:02 +02:00

json_utils.py

fixed typos

2025-10-03 15:17:44 +02:00

lag_checker_iterator.py

Added utils to make regexps and iterators that check for last element

2025-09-30 13:33:12 +02:00

special_regex_maker.py

fixed typos

2025-10-03 13:27:17 +02:00

vocabulary.py

Added json utils to save and load json files

2025-10-01 12:20:59 +02:00