Hotfix for tokenizer modifying spaced ellipses and breaking new text recognition.

This commit is contained in:
KoboldAI Dev 2021-05-13 09:35:11 -04:00
parent c0736a8ec7
commit 0b113a75b4
1 changed files with 6 additions and 1 deletions

View File

@ -701,7 +701,12 @@ def getnewcontent(txt):
else:
delim = vars.actions[-1]
return (txt.split(delim)[-1])
# Fix issue with tokenizer replacing space+period with period
delim = delim.replace(" .", ".")
split = txt.split(delim)
return (split[-1])
#==================================================================#
# Applies chosen formatting options to text submitted to AI