PoliBa-DeepLearning/NanoSocrates
2
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

added some stubs on special_regex_maker

Browse Source
This commit is contained in:
GassiGiuseppe 2025-10-03 10:38:35 +02:00
parent 09f7b39512
commit 845d645348
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
Project_Model/Libs/BPE/Utils/special_regex_maker.py
View File

@ -2,6 +2,13 @@ import re
def special_regex_maker(special_tokens: list[str]) -> re.Pattern: def special_regex_maker(special_tokens: list[str]) -> re.Pattern:
""" compile a regex for the special token
Args:
special_tokens (list[str]): the list of special token
Returns:
re.Pattern:
"""
REGEX_STR = "|".join(special_tokens) REGEX_STR = "|".join(special_tokens)