97 Commits

Author	SHA1	Message	Date
Cohee	9b631ed048	Support Qwen tokenizer fro Groq	2025-02-22 14:29:04 +02:00
Cohee	6e5db5c41a	Perplexity: Add new models	2025-02-21 23:03:49 +02:00
Cohee	15a3cfcb8a	Perplexity: Add reasoning model, remove deprecated models	2025-01-29 20:57:38 +02:00
Cohee	fc4fecf0fd	Add model hash to TC API tokenizer cache key	2025-01-02 23:49:22 +02:00
Cohee	40b05bb6b1	Add best match tokenizer logic for DeepSeek and Nemo	2025-01-02 23:22:53 +02:00
Succubyss	97dc041777	Revert MANUAL_SELECTION ... This partially reverts commit c89804677a (keeps the turbo-0301 removal).	2024-12-31 11:18:43 -06:00
Cohee	4c7d160d41	DeepSeek ... Closes #3233	2024-12-29 20:38:13 +02:00
Succubyss	c89804677a	tokenizers.js: add MANUAL_SELECTION (& removes turbo-0301 lines)	2024-12-24 16:45:35 -06:00
Cohee	77841dbc21	Add types for SillyTavern.getContext	2024-12-06 16:41:26 +02:00
Cohee	1c4e87b25b	unvendor: Replace localforage	2024-10-16 23:37:54 +03:00
Cohee	24884b3206	TextGenWebUI: remove legacy API mode	2024-10-15 00:32:56 +03:00
Cohee	182756fcb2	Erato (#2896) ... * Erato erato	2024-09-24 03:10:31 +03:00
Cohee	93bf87b035	Shut tokenization errors if not using ooba ... Closes #2849	2024-09-22 17:44:19 +03:00
Cohee	8c5b02d2ea	Silently redirect tokenization to the next matching tokenizer	2024-09-22 17:25:56 +03:00
Cohee	a782dcec33	Add Mistral Nemo downloadable tokenizer	2024-09-19 00:10:22 +03:00
Cohee	0207794a2b	Fix Qwen and Command tokenizers not used for logit bias	2024-09-17 13:01:19 +00:00
Cohee	2478e17164	Remove legacy API for Aphrodite	2024-09-15 08:05:53 +03:00
Cohee	81251b073a	Implement downloadable tokenizers ... Closes #2574, #2754	2024-09-06 16:28:34 +00:00
Cohee	5fc16a2474	New AI21 Jamba + tokenizer	2024-08-26 12:07:36 +03:00
AlpinDale	62fed2ff6a	fix: pass model parameter for aphrodite tokenizer	2024-08-24 22:43:20 +00:00
Wolfsblvt	b68969ee3b	Small update to enum display + consistency	2024-08-19 20:18:44 +02:00
Eugén Cowie	62818055af	Get available tokenizers from UI	2024-08-19 15:51:11 +01:00
Eugén Cowie	5f92b8a09e	Add slash command to set tokenizer	2024-08-19 00:39:15 +01:00
Cohee	e707def7dd	Add tokenizer for Gemma/Gemini	2024-08-16 00:00:43 +03:00
Edward Kim	b9857eb315	Added block entropy endpoints for chat and image	2024-08-08 00:08:03 -04:00
Cohee	6bdbe7299f	Merge branch 'staging' into vllm-tc	2024-07-06 11:18:38 +03:00
Cohee	cf56bfb6a9	Add 01.AI as a chat completion source	2024-06-28 00:51:09 +03:00
sasha0552	6c87dfe10d	Add a token counting endpoint for vLLM	2024-06-26 13:34:02 +00:00
daiaji	66454bb711	Add reverse proxy support to Google MakerSuite to allow some Google MakerSuite URLs to no longer be hardcoded with domain names. (#2307) ... * Add reverse proxy support to Google MakerSuite. * Remove hardcoded URLs for some Google MakerSuite API calls. * Don't send real key to alt.endpoint * Fix for image captioning * Fix key validation * +fix key check for mistral * Fix caption key validation * Fix tokenization endpoint use --------- Co-authored-by: Cohee <18619528+Cohee1207@users.noreply.github.com>	2024-05-24 21:38:29 +03:00
Cohee	181b5aff97	Add Groq as chat completion source	2024-05-05 18:53:12 +03:00
Cohee	7bfd666321	Add llama 3 tokenizer	2024-05-03 23:59:39 +03:00
Cohee	2a2a63c52c	Add Perplexity as Chat Completion source	2024-04-20 00:09:38 +03:00
Cohee	1c4bad35b2	#2085 Implement async token counting	2024-04-13 21:05:31 +03:00
Cohee	2cdfda9d69	Actually use getCurrentDreamGenModelTokenizer	2024-03-08 08:40:03 +02:00
Cohee	f962ad5c02	Add OpenRouter as a text completion source	2024-02-25 22:47:07 +02:00
Cohee	10d78f9a25	Aphrodite tokenizer	2024-02-02 01:09:03 +02:00
khanon	60044c18a4	Implement Token Probabilities UI using logprobs	2024-01-25 18:34:46 -06:00
Cohee	86d715cc16	Improve tokenizer detection	2024-01-05 16:17:06 +02:00
Cohee	a39b6b31f4	Allow API tokenizers for textgen bias/bans	2024-01-05 16:07:05 +02:00
Cohee	5734dbd17c	Add custom endpoint type	2023-12-20 18:29:03 +02:00
Cohee	67dd52c21b	#1309 Ollama text completion backend	2023-12-19 16:38:11 +02:00
Cohee	edd737e8bd	#371 Add llama.cpp inference server support	2023-12-18 22:38:28 +02:00
Cohee	b0d9f14534	Re-add Together as a text completion source	2023-12-17 23:38:03 +02:00
based	7acb61ab68	use appropriate tokenizer with mistral completion source	2023-12-16 08:47:51 +10:00
Cohee	2045e414d1	lint: format fix	2023-12-14 20:57:43 +02:00
based	3e82a7d439	tokenizer changes and fixes. + a toggle	2023-12-14 16:31:08 +10:00
based	e26159c00d	refactor and rework palm request to work with the 'content' format and added an endpoint for googles tokenizer	2023-12-14 15:49:50 +10:00
Cohee	cebd6e9e0f	Add API token ids from KoboldCpp	2023-12-14 01:28:18 +02:00
Cohee	9acef0fae6	Horde doesn't support API tokenizers	2023-12-10 16:21:06 +02:00
Cohee	f54bf99006	Fix token ids not displaying in "API_CURRENT" mode for TextGen	2023-12-10 16:09:00 +02:00