NanoSocrates/Project_Model/Libs/BPE/Utils/default_special_tokens.py

5 lines
128 B
Python
Raw Normal View History

from ..Enums import SpecialToken
def default_special_tokens() -> list[str]:
return [token.value for token in SpecialToken]