PoliBa-DeepLearning/NanoSocrates
2
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Fixed illegal tokens being added in target output

Browse Source
This commit is contained in:
Christian Risi 2025-10-06 16:16:47 +02:00
parent d3bba9b944
commit e93710af08
1 changed files with 5 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
Project_Model/Libs/Transformer/Classes/SpannedMasker.py
View File

@ -90,6 +90,11 @@ class SpannedMasker:
SPAN_LENGTH = min(CANDIDATE_SPAN, REMAINING_MASK) SPAN_LENGTH = min(CANDIDATE_SPAN, REMAINING_MASK)
for _ in range(0, SPAN_LENGTH): for _ in range(0, SPAN_LENGTH):
INNER_TOKEN = sequence[mask_index]
if self.__is_illegal_token(INNER_TOKEN, forbidden_tokens):
continue
MASK[mask_index] = True MASK[mask_index] = True
mask_index += 1 mask_index += 1