2.0 KiB
2.0 KiB
BPE
Reasearch Material
- BPE Wikipedia
- BPE Hugging Face
- BPE GeeksForGeeks
- BPE Medium Chetna Khanna
- Stack Overflow "Explain bpe (Byte Pair Encoding) with examples?"
- Implementing a byte pair encoding(BPE) Tokenizer from scratch
- Thoretical Analysis of Byte-Pair Encoding
- A Formal Perspective on Byte-Pair Encoding
- Byte Pair Encoding is Suboptimal for Language Model Pretraining
- Byte pair encoding: a text compression scheme that accelerates pattern matching
- A Formal Perspective on Byte-Pair Encoding
- Controlling byte pair encoding for neural machine translation
- Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
- Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
- Code Completion using Neural Aention and Byte Pair Encoding
- Getting the most out of your tokenizer for pre-training and domain adaptation