629988ce13
Fix a problem with the Lua regeneration API
...
It was an egregious typo that caused tokens to be rearranged on
regeneration.
2021-12-14 23:04:03 -05:00
6670168a47
Update aiserver.py
2021-12-14 16:26:23 +01:00
6e6e0b2b4d
Allow Lua to stop generation from input modifier
2021-12-13 19:32:01 -05:00
e9ed8602b2
Add a "corescript" setting
2021-12-13 19:28:33 -05:00
e5bb20cc8f
Fix Lua regeneration system
2021-12-13 19:17:18 -05:00
462040ed6f
Restore missing `loadsettings()` call
2021-12-13 18:39:33 -05:00
661cca63e8
Make sure stopping criteria still work with dynamic scan off
2021-12-13 18:10:51 -05:00
338d437ea3
Use eventlet instead of gevent-websocket
2021-12-13 17:19:04 -05:00
34c52a1a23
Remove escape characters from all error messages
2021-12-13 11:47:34 -05:00
11f9866dbe
Enable more of the IO library in Lua sandbox
...
Also changes the Lua warning color to red.
2021-12-13 11:22:58 -05:00
28e86563b8
Change `self.scores` to `scores` in aiserver.py
2021-12-13 11:18:01 -05:00
82e149ee02
Catch Lua errors properly
2021-12-13 02:32:09 -05:00
5f06d20085
Format Lua printed messages and warnings
2021-12-13 01:59:53 -05:00
d2f5544468
Add Userscripts menu into GUI
2021-12-13 01:03:26 -05:00
5d13339a52
Allow the retry button to call the Lua scripts properly
2021-12-12 20:48:10 -05:00
39bfb0862a
Allow user input to be modified from Lua
...
Also adds some handlers in the Lua code for when the game is not started
yet
2021-12-12 20:44:03 -05:00
fbf3e7615b
Add API for generated tokens and output text
2021-12-12 19:27:20 -05:00
ceabd2ef7b
Add Lua API for editing logits during generation
...
TPU backend not supported yet.
2021-12-12 16:18:45 -05:00
e2c3ac041b
Complete the Lua generation halting API
2021-12-12 12:52:03 -05:00
d76dd35791
Add Lua API for reading model information
2021-12-12 12:09:59 -05:00
00eb125ad0
Allow Lua API to toggle dynamic scan
2021-12-12 01:55:46 -05:00
5692a7dfe2
Add Lua API for reading the text the user submitted to the AI
2021-12-12 01:52:42 -05:00
03453c4e27
Change script directory tree
...
Userscripts have been moved from /scripts/userscripts to /userscripts.
Core scripts have been moved from /scripts/corescripts to /cores.
2021-12-11 23:46:30 -05:00
36209bfe69
Add Lua API for story chunks
2021-12-11 23:44:07 -05:00
8e6a62259e
Fix the Lua tokenizer API
2021-12-11 21:24:34 -05:00
67974947b2
Fix numerous problems in the Lua world info API
2021-12-11 19:11:38 -05:00
3327f1b471
Fix Lua settings API
2021-12-11 17:01:41 -05:00
f8aa578f41
Enable generation modifiers for transformers backend only
2021-12-11 16:28:25 -05:00
e289a0d360
Connect bridge.lua to aiserver.py
...
Also enables the use of input modifiers and output modifiers, but not
generation modifiers.
2021-12-11 12:45:45 -05:00
35966b2007
Upload bridge.lua, default.lua and some Lua libs
...
base64
inspect
json.lua
Lua-hashings
Lua-nums
Moses
mt19937ar-lua
Penlight
Serpent
2021-12-10 19:45:57 -05:00
683bcb824f
Merge branch 'united' into world-info
2021-12-05 13:06:32 -05:00
6d8517e224
Fix some minor coding errors
2021-12-05 11:39:59 -05:00
150ce033c9
TPU backend no longer needs to recompile after changing softprompt
2021-12-05 02:49:15 -05:00
b99ac92a52
WI folders and WI drag-and-drop
2021-12-04 23:59:28 -05:00
44d8068bab
Ngrok Support
...
Not recommended for home users due to DDoS risks, but might make Colab tunnels more reliable.
2021-11-29 18:11:14 +01:00
9f51c42dd4
Allow bad words filter to ban <|endoftext|> token
...
The official transformers bad words filter doesn't allow this by
default. Finetune's version does allow this by default, however.
2021-11-27 11:42:06 -05:00
2bc93ba37a
Whitelist 6B in breakmodel
...
Now that we properly support it, allow the menu option to use breakmodel
2021-11-27 10:09:54 +01:00
b56ee07ffa
Fix for CPU mode
...
Recent optimizations caused the CPU version to load in an incompatible format, now we convert it back to the correct format after loading it efficiently first.
2021-11-27 05:34:29 +01:00
e5e2fb088a
Remember to actually import `GPTJModel`
2021-11-26 12:38:52 -05:00
871ed65570
Remove an unnecessary `**maybe_low_cpu_mem_usage()`
2021-11-26 11:42:04 -05:00
a93a76eb01
Load model directly in fp16 if using GPU or breakmodel
2021-11-26 10:55:52 -05:00
32e1d4a7a8
Enable `low_cpu_mem_usage`
2021-11-25 18:09:25 -05:00
25c9be5d02
Breakmodel support for GPTJModel
2021-11-25 18:09:16 -05:00
f8bcc3411b
In breakmodel mode, move layers to GPU as soon as model loads
...
Rather than during the first generation.
2021-11-25 11:44:41 -05:00
cbb6efb656
Move TFS warper code into aiserver.py
2021-11-24 13:36:54 -05:00
a2c82bbcc8
num_layers fixes
...
As requested by VE_FORBRYDERNE (Possibly implemented it on to many places, needs testing but since the other one is already broken I am committing it first so I can more easily test)
2021-11-24 03:44:11 +01:00
d7a2424d2d
No Half on CPU
...
Should fix CPU executions
2021-11-23 17:14:01 +01:00
be0881a8d0
Use model.config.n_layer if model.config.num_layers doesn't exist
2021-11-23 10:09:24 -05:00
9b8bcb5516
Always convert soft prompt to float32 if using TPU backend
...
TPUs do not support float16. Attempting to use a float16 soft prompt
throws an error.
2021-11-21 18:22:10 -05:00
e068aa9f26
Add soft prompt support to TPU backend
2021-11-21 18:08:04 -05:00
df2768b745
Simplify the comment regex
2021-11-21 01:09:19 -05:00
7ab0d96b8a
Change the comment regex again to use fixed-length lookbehind
2021-11-21 01:06:31 -05:00
624cfbd5a4
Use a smarter regex for comments
...
If the beginning of the comment is at the beginning of a line AND the
end of a comment is at the end of a line, an additional newline will now
be ignored so that the AI doesn't see a blank line where the comment
was.
For example, consider the following message:
```
Hello
<|This is
a comment|>
World
```
The AI will now see this:
```
Hello
World
```
instead of this:
```
Hello
World
```
2021-11-21 00:42:57 -05:00
a51f88aeb3
Also apply comment formatting to prompt in `refresh_story()`
2021-11-21 00:26:45 -05:00
1968be82bb
Remove comments from prompt in WI processor and InferKit mode
2021-11-20 22:23:06 -05:00
8ce8e621ce
Fix typo (one of the `comregex_ui` should be `comregex_ai`)
2021-11-20 22:19:12 -05:00
c2ed31de28
Add syntax for comments <|...|>
2021-11-20 01:27:57 -05:00
a65c4de840
Integrate TPU backend
...
This commit puts the TPU backend code directly in to the KoboldAI code
to make it easier to modify.
2021-11-19 18:06:57 -05:00
4a678deaa5
Merge branch 'KoboldAI:main' into united
2021-11-18 06:51:44 +01:00
b25c54cf91
Polishing and Optimizations
...
Multiple things have changed, for now models default to half mode even on the official transformers to make sure its as efficient on the GPU as finetune's. GPU selection is streamlined and cache files are now stored inside the KoboldAI folder (for the most part). A new command line parameter to force the models to run at their full size still needs to be added for the few users that would want a quality bump at the cost of ram.
2021-11-18 00:06:57 +01:00
27ee45b9cc
Merge pull request #31 from VE-FORBRYDERNE/cpu
...
Fix gen_in device logic in generate()
2021-11-17 22:42:31 +01:00
2f0b673b28
Fix gen_in device logic in generate()
2021-11-17 16:37:37 -05:00
e71271933a
Merge pull request #29 from VE-FORBRYDERNE/hidden-size
...
Fix hidden size detection for GPTJForCausalLM
2021-11-17 22:30:24 +01:00
a1bc10246c
Support for multiple gens per action with dynamic scan
2021-11-17 16:17:59 -05:00
98a72e34a4
Replace slashes in model name with underscores
2021-11-17 15:36:36 -05:00
ab1a65f13a
Fix hidden size detection for GPTJForCausalLM
2021-11-15 11:56:02 -05:00
17d07b280a
Correct `gpu_layers` to `gpu_blocks`
2021-11-14 21:08:49 -05:00
805cb0c8b9
Make sure device_config() still works with all layers on CPU
2021-11-14 18:46:00 -05:00
80aee07816
Use old GPU-only generation if all layers are on the same GPU
...
Apparently, this mode uses less RAM than breakmodel does.
2021-11-14 18:42:18 -05:00
b0ab30cec4
Re-enable GPU-only generation option
2021-11-14 18:24:51 -05:00
3e38b462c6
Hidden Size fix for GPT2 Custom
...
Replaced the JS Hidden Size load with the newer function to fix these models
2021-11-14 16:40:04 +01:00
ecea169553
Improved Unix Support
...
Changes the line-endings to the Unix format and sets KoboldAI to launch with Python3 if executed directly.
(cherry picked from commit 5b0977ceb6807c0f80ce6717891ef5e23c8eeb77)
2021-11-13 21:54:32 -05:00
1596a238f7
Breakmodel automation
...
The only changes are a small addition to the breakmodel section where GPU0 is automatically chosen if the CLI options are used without specifying breakmodel. Lineendings have been changed to Linux formatting for compatibility reasons.
2021-11-14 03:13:52 +01:00
8a916116e3
Remove device=0 because of incompatibility
...
Device=0 breaks some of the pytorch implementations, removed to restore hardware compatibility to 0.16 levels.
2021-11-14 02:33:27 +01:00
4bcffc614e
Allow directly running KoboldAI from CLI in Linux
...
Its made for Python3, so we assume python3 is installed in its usual location. If it isn't you can always run it yourself with whatever command you used prior to this change.
2021-11-14 01:57:43 +01:00
21ae45e9ab
Merge branch 'KoboldAI:main' into united
2021-11-11 17:05:39 +01:00
1fadcbe1e3
Send allowsp command on connect instead of on startup
2021-11-11 00:18:46 -05:00
2fe815e092
Don't broadcast emit calls inside do_connect()
...
This prevents the "thinking" animation from appearing on top of the
submit button under certain circumstances:
* When someone connects to the KoboldAI server while the model is
generating (occurs after generation finishes)
* Occasionally, the browser may suddenly disconnect and reconnect from
Flask-SocketIO during generation, which causes the same problem
2021-11-11 00:14:12 -05:00
11b0291bc4
Use model.transformer.embed_dim if model.transformer.hidden_size doesn't exist
2021-11-10 17:47:14 -05:00
752e19a2bb
Fix vars.modeldim not always being set
2021-11-10 17:38:30 -05:00
2679df9664
Merge branch 'main' into united
2021-11-09 21:33:14 -05:00
c2371cf801
Merge pull request #23 from VE-FORBRYDERNE/scan-test
...
Dynamic world info scan
2021-11-10 03:31:42 +01:00
4af0d9dabd
Merge pull request #78 from VE-FORBRYDERNE/patch
...
Allow remote mode to load from client-side story files
2021-11-06 16:58:05 +01:00
02a56945de
Version bump
2021-11-06 11:50:56 -04:00
bc0f9c8032
Allow remote mode to load from client-side story files
...
(cherry picked from commit a1345263df
2021-11-06 11:48:20 -04:00
7c099fe93c
Allow remote mode to load from client-side story files
2021-11-04 19:33:17 -04:00
81bd058caf
Make sure calcsubmitbudget uses the correct reference to vars.actions
2021-11-03 18:57:02 -04:00
a2d7735a51
Dynamic WI scanner should ignore triggers that are already in context
2021-11-03 18:55:53 -04:00
ecfbbdb4a9
Merge branch 'united' into scan-test
2021-11-03 18:23:22 -04:00
0fa47b1249
Fix budget calculation for stories with at least one non-prompt chunk
2021-11-03 18:22:31 -04:00
c11dab894e
Put placeholder variables into calcsubmitbudget
2021-11-03 18:02:19 -04:00
9b18068999
Shallow copy story chunks when generating
2021-11-03 17:53:38 -04:00
b8c3d8c12e
Fix generator output having the wrong length
2021-11-03 16:10:12 -04:00
5b3ce4510f
Make sure that soft_tokens is on the correct device
2021-11-03 16:07:50 -04:00
90fd5a538a
Merge branch 'united' into scan-test
2021-11-03 12:42:18 -04:00
fe2987d894
Fix missing break statement in device_config
2021-11-03 12:42:04 -04:00
bd76ab333c
Set numseqs to 1 if using dynamic world info scan
2021-11-03 12:28:17 -04:00
0a91ea27b3
Make the dynamic world info scan toggleable
2021-11-03 12:18:48 -04:00
de3664e73c
Add an assertion for the value of already_generated
2021-11-03 12:01:45 -04:00
ec8ec55256
Dynamic world info scan
2021-11-03 11:54:48 -04:00
aa998ba5e9
Merge pull request #20 from VE-FORBRYDERNE/sp
...
Soft prompt support for PyTorch models
2021-10-30 00:35:44 +02:00
206c01008e
Fix budget calculation when using soft prompt
2021-10-29 11:44:51 -04:00
c9c370aa17
Merge branch 'KoboldAI:main' into united
2021-10-28 23:29:29 +02:00
bf4e7742ac
Patch GPTJForCausalLM, if it exists, to support soft prompting
2021-10-28 17:18:28 -04:00
40b4631f6c
Clamp input_ids in place
...
Apparently transformers maintains an internal reference to input_ids
(to use for repetition penalty) so we have to clamp the internal
version, too, because otherwise transformers will throw an out-of-bounds
error upon attempting to access token IDs that are not in the
vocabulary.
2021-10-28 16:52:39 -04:00
24d5d63c9f
Use the correct generation min and max when using soft prompt
2021-10-28 16:39:59 -04:00
511817132a
Don't change the shape of transformer.wte
2021-10-28 15:39:59 -04:00
a1ae11630a
Make sure to cast vars.sp to the correct dtype
2021-10-28 13:22:07 -04:00
1556bd32a5
Use torch.where to inject the soft prompt instead of torch.cat
2021-10-28 13:20:14 -04:00
248e0bd24b
Fix soft prompt loading code
2021-10-28 00:29:42 -04:00
4e3cc93020
Merge branch 'united' into sp
2021-10-23 11:45:03 -04:00
7b73d7cfdd
Single Line Mode
...
Adds Single Line mode, optimized for things like chatbot testing and other cases where you want to have control over what happens after a paragraph.
This can also be used as a foundation for a chatbot optimized interface mode.
2021-10-23 17:30:48 +02:00
1f449a9dda
Soft prompt support (6B Colabs not supported yet)
2021-10-22 14:18:10 -04:00
3501f03153
Create settings directory if it doesn't exist when using InferKit/OAI
2021-10-21 23:33:32 -04:00
fa0f8af1d6
Merge branch 'KoboldAI:main' into united
2021-10-15 08:23:06 +02:00
9513240dfb
Version bump
...
Since VE fixed important things in the editor i want users to be able to see this easier
2021-10-15 08:22:32 +02:00
c854a62549
Clarified GPU Layers
...
breakmodel_layers and layers is confusing, changed the new method to breakmodel_gpulayers. The old one should no longer be used by people, but since it works in reverse we leave it in so scripts don't break.
2021-10-06 18:55:01 +02:00
bd063f7590
Merge pull request #19 from VE-FORBRYDERNE/multi-gpu
...
Multiple GPU support
2021-10-06 18:50:58 +02:00
82c7eaffb5
Merge branch 'KoboldAI:main' into united
2021-10-06 00:26:08 +02:00
8893916fef
Don't always submit prompt by default
...
Feedback from users is that its better to not always submit the prompt, this is consistent with the randomly generated stories. You can always toggle it on if you need this for coherency. This change does not override existing user settings.
2021-10-06 00:25:05 +02:00
aa59f8b4b2
Fix CPU layers not displaying correctly when using --layers
2021-10-05 11:29:47 -04:00
91352ea9f1
Change the command line flags for breakmodel
2021-10-05 11:22:09 -04:00
a1e4405aa6
Automatically use breakmodel instead of GPU-only where supported
...
There's really no reason to use GPU-only mode if breakmodel is supported
because breakmodel can run in GPU-only mode too.
2021-10-05 10:36:51 -04:00
fb90a7ed17
Change the help text for breakmodel to be more helpful
2021-10-05 10:31:28 -04:00
231621e7c2
Use AutoModelForCausalLM for custom models with a model_type
2021-10-05 09:45:12 -04:00
a283d34b27
Multiple GPU support
2021-10-05 09:38:57 -04:00
a42b580027
Merge branch 'united' into multi-gpu
2021-10-02 11:44:26 -04:00
dab58d8393
Merge branch 'KoboldAI:main' into united
2021-09-29 17:05:06 +02:00
a179bb2820
Bump version number to 1.16.2
2021-09-28 21:50:33 -04:00
e6cd28243e
Scroll to the bottom of the gamescreen after retrying
2021-09-28 21:34:36 -04:00
bb323152d7
Disable vars.recentedit again
2021-09-28 21:24:08 -04:00
2b89bcb16e
Fix random story generator
2021-09-28 21:04:26 -04:00
af93c96c0f
Submit Action mode action in Story mode if action is empty
2021-09-28 19:50:00 -04:00
9ab1d182ac
Guard against empty prompts
2021-09-28 19:48:43 -04:00
da55ed3b49
Merge branch 'KoboldAI:main' into united
2021-09-28 10:41:01 +02:00
03c1a3ebf9
Put vars.recentedit = True in deleterequest() for consistency
2021-09-28 01:10:20 -04:00
97e1760af5
Prevent retry from popping chunks after edit/delete
2021-09-28 01:07:11 -04:00
231290608d
Do a better job of preventing editing of text when required
2021-09-28 00:48:37 -04:00
13b81c7523
Prevent the user from deleting the prompt
2021-09-27 22:21:14 -04:00
01b30b315f
Merge branch 'KoboldAI:main' into united
2021-09-28 02:31:20 +02:00
e29e7b11ec
Bump version number to 1.16.1
2021-09-27 18:12:15 -04:00
a327eed2c3
Fix editor scrolling issues
2021-09-27 17:44:22 -04:00
01339f0b87
Merge branch 'KoboldAI:main' into united
2021-09-25 17:44:51 +02:00
5893e495b6
Change AutoModel to AutoModelForCausalLM
...
This fixes breakmodel mode for the official models from the model
selection menu.
2021-09-25 11:41:15 -04:00
c9290d02dc
Update aiserver.py
...
Better way of checking for the model type
2021-09-25 16:50:24 +02:00
7d35f825c6
Huggingface GPT-J Support
...
Finetune's fork has unofficial support which we supported, but this is not compatible with models designed for the official version. In this update we let models decide which transformers backend to use, and fall back to Neo if they don't choose any. We also add the 6B to the menu and for the time being switch to the github version of transformers to be ahead of the waiting time. (Hopefully we can switch back to the conda version before merging upstream).
2021-09-25 16:26:17 +02:00
4d9eab3785
K80 test
2021-09-23 20:57:18 -04:00
6520cac75d
Support models that are formatted with CRLF
...
A new model was released that uses a different formatting for its enters, this causes to many enters in the UI. In this change we fix the issue so that when this happens the UI still displays the content as you would expect. Removing the formatting burden from the Model developers.
2021-09-22 00:34:05 +02:00
30a7e945a1
Merge pull request #18 from VE-FORBRYDERNE/doc
...
Correct misindicated model VRAM requirements
2021-09-21 18:54:19 +02:00
dd1c3ab67e
Allow models to set formatting defaults
...
Originally omitted when model settings were forced. Now that models can only define the defaults for KoboldAI its a good idea to give model authors control over what formatting they think works best for their models.
2021-09-21 15:46:54 +02:00
Gnome Ann
henk717