56 Commits

Author	SHA1	Message	Date
Cohee	0844374de5	Remove old GPT-2 tokenizer. Redirect to tiktoken's tokenizer	2023-08-27 22:14:39 +03:00
Cohee	9660aaa2c2	Add NovelAI hypebot plugin	2023-08-27 18:27:34 +03:00
Cohee	c91ab3b5e0	Add Kobold tokenization to best match logic. Fix not being able to stop group chat regeneration	2023-08-24 21:23:35 +03:00
Cohee	ab52af4fb5	Add support for Koboldcpp tokenization endpoint	2023-08-24 20:19:57 +03:00
Cohee	e77da62b85	Add padding to cache key. Fix Safari display issues. Fix 400 on empty translate. Reset bias cache on changing model.	2023-08-23 10:32:48 +03:00
Cohee	bc5fc67906	Put tokenizer functions to a separate file. Cache local models token counts	2023-08-23 02:38:43 +03:00