Christian Risi aac7675b30 Pipeline fix and added a util to decode

2025-10-09 13:24:48 +02:00

..

__init__.py

Modified imports for BPE

2025-10-04 19:42:02 +02:00

Encoder.py

Added Special Encoder

2025-09-28 18:03:47 +02:00

NanoSocratesBPE.py

Pipeline fix and added a util to decode

2025-10-09 13:24:48 +02:00

NanoSocratesChunker.py

Added Chunker

2025-09-28 18:02:06 +02:00

NanoSocratesSpecial.py

Added a way to see vocabulary size

2025-10-04 19:42:29 +02:00

NanoSocratesSplitter.py

Fix of bugs and semantics

2025-10-03 13:26:58 +02:00

NanoSocraTrainer.py

Added a way to recover iteration work

2025-10-01 12:21:42 +02:00

NanoSocraTraineRam.py

Refactored to remove tokens that can't be compressed anymore

2025-10-01 19:42:22 +02:00

NanoSocraTrainerPool.py

Fixed a rare bug over training multiple times

2025-10-04 10:47:39 +02:00

TokeNano.py

implemented token nano for the BPE encoding/decoding

2025-10-03 01:04:06 +02:00

TokeNanoCore.py

WIP training Batching

2025-10-07 17:41:53 +02:00