Commit Graph

62 Commits

Author SHA1 Message Date
Cohee 10d78f9a25 Aphrodite tokenizer 2024-02-02 01:09:03 +02:00
khanon 60044c18a4 Implement Token Probabilities UI using logprobs 2024-01-25 18:34:46 -06:00
Cohee 86d715cc16 Improve tokenizer detection 2024-01-05 16:17:06 +02:00
Cohee a39b6b31f4 Allow API tokenizers for textgen bias/bans 2024-01-05 16:07:05 +02:00
Cohee 5734dbd17c Add custom endpoint type 2023-12-20 18:29:03 +02:00
Cohee 67dd52c21b #1309 Ollama text completion backend 2023-12-19 16:38:11 +02:00
Cohee edd737e8bd #371 Add llama.cpp inference server support 2023-12-18 22:38:28 +02:00
Cohee b0d9f14534 Re-add Together as a text completion source 2023-12-17 23:38:03 +02:00
based 7acb61ab68 use appropriate tokenizer with mistral completion source 2023-12-16 08:47:51 +10:00
Cohee 2045e414d1 lint: format fix 2023-12-14 20:57:43 +02:00
based 3e82a7d439 tokenizer changes and fixes. + a toggle 2023-12-14 16:31:08 +10:00
based e26159c00d refactor and rework palm request to work with the 'content' format and added an endpoint for googles tokenizer 2023-12-14 15:49:50 +10:00
Cohee cebd6e9e0f Add API token ids from KoboldCpp 2023-12-14 01:28:18 +02:00
Cohee 9acef0fae6 Horde doesn't support API tokenizers 2023-12-10 16:21:06 +02:00
Cohee f54bf99006 Fix token ids not displaying in "API_CURRENT" mode for TextGen 2023-12-10 16:09:00 +02:00
Cohee 6957d9e7cf Fix display names of Best match tokenizers 2023-12-10 16:03:25 +02:00
Cohee 6e5eea5dba Unbreak previously selected API tokenizer in dropdown 2023-12-10 15:56:38 +02:00
valadaptive 55976e61a3 Fix tokenizer override
I searched for all users of tokenizers.API, but missed that the menu
converts the numerical select values directly to enum values. I've used
the special tokenizer value 98 to represent "the tokenizer API for
whichever backend we're currently using".
2023-12-09 23:57:21 -05:00
valadaptive 014416546c Add padding once in getTokenCount
This means we don't have to pass the "padding" parameter into every
function so they can add the padding themselves--we can do it in just
one place instead.
2023-12-09 20:53:16 -05:00
valadaptive 2f2cd197cc Clean up tokenizer API code
Store the URLs for each tokenizer's action in one place at the top of
the file, instead of in a bunch of switch-cases. The URLs for the
textgen and Kobold APIs don't change and hence don't need to be
function arguments.
2023-12-09 20:48:41 -05:00
valadaptive 09465fbb97 Inline most get(...)TokenizerParams calls
For everything except textgenerationwebui, these params are now simple
enough that it doesn't make sense for them to be in a separate function.
2023-12-09 20:35:11 -05:00
valadaptive 30502ac949 Split up Kobold and textgenerationwebui endpoints
The endpoint was one big if/else statement that did two entirely
different things depending on the value of main_api. It makes more sense
for those to be two separate endpoints.
2023-12-09 20:26:24 -05:00
valadaptive 7486ab3886 Separate textgen and Kobold tokenization APIs
They function differently and have different logic and API parameters,
so it makes sense to count them as two different APIs. Kobold's API
doesn't return tokens, so it can only be used to count them.

There's still a lot of duplicate code which I will clean up in the
following commits.
2023-12-09 20:24:56 -05:00
valadaptive 18177c147d Separate remote and server tokenization code paths
This lets us remove extraneous API params from paths where they aren't
needed.
2023-12-09 20:08:48 -05:00
valadaptive ddd73a204a Remove "remote" language from tokenizer functions
We'll be making a distinction between tokenizing *on* the server itself,
and tokenizing via the server having the AI service do it. It makes more
sense to use the term "remote" for the latter.
2023-12-09 19:49:22 -05:00
valadaptive 8bad059a62 Rename /tokenize_via_api endpoint
No redirect for this since I don't expect any extensions to be calling this directly.
2023-12-09 19:29:24 -05:00
valadaptive 57bc95133e Rename tokenizer routes
They're all under tokenizers/ now, and there are "count", "encode", and
"decode" endpoints. This forms a clearer hierarchy.
2023-12-04 10:17:43 -05:00
valadaptive 9c33ddbafc Make textgen settings type checks more concise 2023-12-03 14:56:01 -05:00
valadaptive 047c897ead Remove is[API] functions
Just use an equality comparison. It's a bit longer, but only because
"textgenerationwebui_settings" is a long identifier.
2023-12-03 14:56:01 -05:00
valadaptive ba54e3dea0 Replaces is_[api] params with api_type param
These were 5 mutually-exclusive booleans, which can be replaced with one
param that takes on 5 values, one for each API type.
2023-12-03 14:56:01 -05:00
Cohee 64a3564892 lint: Comma dangle 2023-12-02 22:06:57 +02:00
Cohee c63cd87cc0 lint: Require semicolons 2023-12-02 21:11:06 +02:00
valadaptive a37f874e38 Require single quotes 2023-12-02 13:04:51 -05:00
Cohee a367285ac2
Merge pull request #1430 from valadaptive/eslint-fixes-2
ESLint fixes, part 2 - bulky changes
2023-12-02 19:43:11 +02:00
Cohee 0477f6a553 Use best match API tokenizers for Text Completion sources 2023-12-02 19:42:15 +02:00
valadaptive 27e63a7a77 Enable no-case-declarations lint 2023-12-02 10:32:26 -05:00
Cohee e6c96553d0 Add text trimming commands 2023-11-26 13:55:22 +02:00
Cohee 1ebfddf07e Use mistral and yi tokenizers for custom token bans 2023-11-21 01:04:27 +02:00
Cohee 9b75e49b54 Add support for Yi tokenizer 2023-11-21 00:21:58 +02:00
Cohee 96caddfd71 Add koboldcpp as Text Completion source 2023-11-19 17:14:53 +02:00
kingbri 4cfa267b1b API Tokenizer: Add support for TabbyAPI
Use Tabby's /v1/token endpoints.

Signed-off-by: kingbri <bdashore3@proton.me>
2023-11-17 01:48:03 -05:00
Cohee 81fe9aa699 Fix updated tokenization via ooba API 2023-11-09 19:39:08 +02:00
Cohee 480099ee97 Mancer will work in legacy API mode. Remove Soft Prompt mentions. 2023-11-08 18:16:47 +02:00
Cohee e76c18c104 Legacy ooba API compatibility shim 2023-11-08 10:13:28 +02:00
Cohee 865256f5c0 Fix ooba tokenization via API. Fix requiring streaming URL to generate 2023-11-08 03:38:04 +02:00
Cohee 57e845d0d7 Resolve best match tokenizer for itemization. Adjust styles of token counter 2023-11-06 20:25:59 +02:00
Cohee e8ba328a14 Add text chunks display to token counter 2023-11-06 02:42:51 +02:00
Cohee f248367ca3 Add Mistral tokenizer 2023-11-06 01:26:13 +02:00
Cohee f0c0949aa0 Add token ids viewer to tokenizer plugin 2023-11-05 22:45:37 +02:00
Cohee fedc3b887f Add llama2 tokenizer for OpenRouter models 2023-11-05 21:54:19 +02:00