PoliBa-DeepLearning/NanoSocrates
2
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Added a way to take the default special token list

Browse Source
This commit is contained in:
Christian Risi 2025-10-04 19:43:02 +02:00
parent 8f5e2f2f0d
commit 3b274ad807
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
Project_Model/Libs/BPE/Utils/default_special_tokens.py Normal file
View File

@ -0,0 +1,4 @@
from ..Enums import SpecialToken
def default_special_tokens() -> list[str]:
return [token.value for token in SpecialToken]