18 Commits

Author	SHA1	Message	Date
Cohee	865256f5c0	Fix ooba tokenization via API. Fix requiring streaming URL to generate	2023-11-08 03:38:04 +02:00
Cohee	57e845d0d7	Resolve best match tokenizer for itemization. Adjust styles of token counter	2023-11-06 20:25:59 +02:00
Cohee	e8ba328a14	Add text chunks display to token counter	2023-11-06 02:42:51 +02:00
Cohee	f248367ca3	Add Mistral tokenizer	2023-11-06 01:26:13 +02:00
Cohee	f0c0949aa0	Add token ids viewer to tokenizer plugin	2023-11-05 22:45:37 +02:00
Cohee	fedc3b887f	Add llama2 tokenizer for OpenRouter models	2023-11-05 21:54:19 +02:00
Cohee	c2ba3a773a	Delayed tokenizers initialization	2023-10-25 00:32:49 +03:00
Cohee	b167eb9e22	Add raw token ids support to OAI logit bias. Fix token counting for turbo models	2023-10-19 13:37:08 +03:00
Cohee	bfdd071001	Move tokenizer endpoint and functions to separate file	2023-09-16 18:48:06 +03:00
Cohee	853736fa93	Remove legacy NovelAI models	2023-09-06 14:32:06 +03:00
Cohee	267d0eb16f	Fix API tokenizers usage with kcpp	2023-09-01 02:57:35 +03:00
Cohee	3b4e6f0b78	Add debug functions menu	2023-08-27 23:20:43 +03:00
Cohee	0844374de5	Remove old GPT-2 tokenizer. Redirect to tiktoken's tokenizer	2023-08-27 22:14:39 +03:00
Cohee	9660aaa2c2	Add NovelAI hypebot plugin	2023-08-27 18:27:34 +03:00
Cohee	c91ab3b5e0	Add Kobold tokenization to best match logic. Fix not being able to stop group chat regeneration	2023-08-24 21:23:35 +03:00
Cohee	ab52af4fb5	Add support for Koboldcpp tokenization endpoint	2023-08-24 20:19:57 +03:00
Cohee	e77da62b85	Add padding to cache key. Fix Safari display issues. Fix 400 on empty translate. Reset bias cache on changing model.	2023-08-23 10:32:48 +03:00
Cohee	bc5fc67906	Put tokenizer functions to a separate file. Cache local models token counts	2023-08-23 02:38:43 +03:00