valadaptive
ddd73a204a
Remove "remote" language from tokenizer functions
...
We'll be making a distinction between tokenizing *on* the server itself,
and tokenizing via the server having the AI service do it. It makes more
sense to use the term "remote" for the latter.
2023-12-09 19:49:22 -05:00
valadaptive
8bad059a62
Rename /tokenize_via_api endpoint
...
No redirect for this since I don't expect any extensions to be calling this directly.
2023-12-09 19:29:24 -05:00
valadaptive
57bc95133e
Rename tokenizer routes
...
They're all under tokenizers/ now, and there are "count", "encode", and
"decode" endpoints. This forms a clearer hierarchy.
2023-12-04 10:17:43 -05:00
valadaptive
9c33ddbafc
Make textgen settings type checks more concise
2023-12-03 14:56:01 -05:00
valadaptive
047c897ead
Remove is[API] functions
...
Just use an equality comparison. It's a bit longer, but only because
"textgenerationwebui_settings" is a long identifier.
2023-12-03 14:56:01 -05:00
valadaptive
ba54e3dea0
Replaces is_[api] params with api_type param
...
These were 5 mutually-exclusive booleans, which can be replaced with one
param that takes on 5 values, one for each API type.
2023-12-03 14:56:01 -05:00
Cohee
64a3564892
lint: Comma dangle
2023-12-02 22:06:57 +02:00
Cohee
c63cd87cc0
lint: Require semicolons
2023-12-02 21:11:06 +02:00
valadaptive
a37f874e38
Require single quotes
2023-12-02 13:04:51 -05:00
Cohee
a367285ac2
Merge pull request #1430 from valadaptive/eslint-fixes-2
...
ESLint fixes, part 2 - bulky changes
2023-12-02 19:43:11 +02:00
Cohee
0477f6a553
Use best match API tokenizers for Text Completion sources
2023-12-02 19:42:15 +02:00
valadaptive
27e63a7a77
Enable no-case-declarations lint
2023-12-02 10:32:26 -05:00
Cohee
e6c96553d0
Add text trimming commands
2023-11-26 13:55:22 +02:00
Cohee
1ebfddf07e
Use mistral and yi tokenizers for custom token bans
2023-11-21 01:04:27 +02:00
Cohee
9b75e49b54
Add support for Yi tokenizer
2023-11-21 00:21:58 +02:00
Cohee
96caddfd71
Add koboldcpp as Text Completion source
2023-11-19 17:14:53 +02:00
kingbri
4cfa267b1b
API Tokenizer: Add support for TabbyAPI
...
Use Tabby's /v1/token endpoints.
Signed-off-by: kingbri <bdashore3@proton.me>
2023-11-17 01:48:03 -05:00
Cohee
81fe9aa699
Fix updated tokenization via ooba API
2023-11-09 19:39:08 +02:00
Cohee
480099ee97
Mancer will work in legacy API mode. Remove Soft Prompt mentions.
2023-11-08 18:16:47 +02:00
Cohee
e76c18c104
Legacy ooba API compatibility shim
2023-11-08 10:13:28 +02:00
Cohee
865256f5c0
Fix ooba tokenization via API. Fix requiring streaming URL to generate
2023-11-08 03:38:04 +02:00
Cohee
57e845d0d7
Resolve best match tokenizer for itemization. Adjust styles of token counter
2023-11-06 20:25:59 +02:00
Cohee
e8ba328a14
Add text chunks display to token counter
2023-11-06 02:42:51 +02:00
Cohee
f248367ca3
Add Mistral tokenizer
2023-11-06 01:26:13 +02:00
Cohee
f0c0949aa0
Add token ids viewer to tokenizer plugin
2023-11-05 22:45:37 +02:00
Cohee
fedc3b887f
Add llama2 tokenizer for OpenRouter models
2023-11-05 21:54:19 +02:00
Cohee
c2ba3a773a
Delayed tokenizers initialization
2023-10-25 00:32:49 +03:00
Cohee
b167eb9e22
Add raw token ids support to OAI logit bias. Fix token counting for turbo models
2023-10-19 13:37:08 +03:00
Cohee
bfdd071001
Move tokenizer endpoint and functions to separate file
2023-09-16 18:48:06 +03:00
Cohee
853736fa93
Remove legacy NovelAI models
2023-09-06 14:32:06 +03:00
Cohee
267d0eb16f
Fix API tokenizers usage with kcpp
2023-09-01 02:57:35 +03:00
Cohee
3b4e6f0b78
Add debug functions menu
2023-08-27 23:20:43 +03:00
Cohee
0844374de5
Remove old GPT-2 tokenizer. Redirect to tiktoken's tokenizer
2023-08-27 22:14:39 +03:00
Cohee
9660aaa2c2
Add NovelAI hypebot plugin
2023-08-27 18:27:34 +03:00
Cohee
c91ab3b5e0
Add Kobold tokenization to best match logic. Fix not being able to stop group chat regeneration
2023-08-24 21:23:35 +03:00
Cohee
ab52af4fb5
Add support for Koboldcpp tokenization endpoint
2023-08-24 20:19:57 +03:00
Cohee
e77da62b85
Add padding to cache key. Fix Safari display issues. Fix 400 on empty translate. Reset bias cache on changing model.
2023-08-23 10:32:48 +03:00
Cohee
bc5fc67906
Put tokenizer functions to a separate file. Cache local models token counts
2023-08-23 02:38:43 +03:00
