Christian Risi 1797571bb2 Added test to see if illegal tokens were included in target

2025-10-06 16:17:12 +02:00

..

chunker_files

Added text files to test a chunker

2025-09-26 18:48:44 +02:00

spanner_file

Added testing for spanned masking

2025-10-06 15:55:40 +02:00

trainer_files

little test file

2025-10-04 18:58:20 +02:00

bpe_test.py

test files updated

2025-10-03 01:04:47 +02:00

bpe_trainer_test.py

Logic to test BPE

2025-10-04 18:58:04 +02:00

chunker_test.py

Added test for chunker

2025-09-26 18:50:32 +02:00

spanned_masker_test.py

Added test to see if illegal tokens were included in target

2025-10-06 16:17:12 +02:00

splitter_test.py

Fixed tests to reflect new version of tokenizer

2025-10-03 13:27:38 +02:00

tokenano_test.py

Fixed tests to reflect new version of tokenizer

2025-10-03 13:27:38 +02:00