mirror of
https://github.com/KoboldAI/KoboldAI-Client.git
synced 2024-12-12 16:47:28 +01:00
60 lines
3.5 KiB
Plaintext
60 lines
3.5 KiB
Plaintext
Thanks for checking out the KoboldAI Client!
|
|
|
|
[ABOUT]
|
|
|
|
This is a test release of a quickly-assembled front-end for multiple local & remote AI models.
|
|
The purpose is to provide a smoother, web-based UI experience than the various command-line AI apps.
|
|
I'm pushing this out now that the major quality-of-life fearures have been roughed in (generate,
|
|
undo, edit-by-line, memory, save/load, etc), which means there will probably be bugs.
|
|
|
|
This application uses Transformers (https://huggingface.co/transformers/) to interact with the AI models
|
|
via Tensorflow. Tensorflow has CUDA/GPU support for shorter generation times, but I do not have anything
|
|
in this test release to set up CUDA/GPU support on your system. If you have a high-end GPU with
|
|
sufficient VRAM to run your model of choice, see (https://www.tensorflow.org/install/gpu) for
|
|
instructions on enabling GPU support.
|
|
|
|
Transformers/Tensorflow can still be used on CPU if you do not have high-end hardware, but generation
|
|
times will be much longer. Alternatively, KoboldAI also supports InferKit (https://inferkit.com/).
|
|
This will allow you to send requests to a remotely hosted Megatron-11b model for fast generation times
|
|
on any hardware. This is a paid service, but signing up for a free account will let you generate up
|
|
to 40,000 characters, and the free account will work with KoboldAI.
|
|
|
|
[SETUP]
|
|
|
|
1. Install Python. (https://www.python.org/downloads/)
|
|
(Development was done on 3.7, I have not tested newer versions)
|
|
2. When installing Python make sure "pip" is selected under Optional features.
|
|
(If pip isn't working, run the installer again and choose Modify to choose Optional features.)
|
|
3. Run install_requirements.bat.
|
|
(This will install the necessary python packages via pip)
|
|
4. Run play.bat
|
|
5. Select a model from the list. Flask will start and give you a message that it's ready to connect.
|
|
6. Open a web browser and enter http://127.0.0.1:5000/
|
|
|
|
[FOR INFERKIT INTEGRATION]
|
|
|
|
If you would like to use InferKit's Megatron-11b model, sign up for a free account on their website.
|
|
https://inferkit.com/
|
|
After verifying your email address, sign in and click on your profile picture in the top right.
|
|
In the drop down menu, click "API Key".
|
|
On the API Key page, click "Reveal API Key" and copy it. When starting KoboldAI and selecting the
|
|
InferKit API model, you will be asked to paste your API key into the terminal. After entering,
|
|
the API key will be stored in the client.settings file for future use.
|
|
You can see your remaining budget for generated characters on their website under "Billing & Usage".
|
|
|
|
[ENABLE COLORS IN WINDOWS 10 COMMAND LINE]
|
|
|
|
If you see strange numeric tags in the console output, then your console of choice does not have
|
|
color support enabled. On Windows 10, you can enable color support by lanching the registry editor
|
|
and adding the REG_DWORD key VirtualTerminalLevel to Computer\HKEY_CURRENT_USER\Console and setting
|
|
its value to 1.
|
|
|
|
[ENABLE GPU FOR SUPPORTED VIDEO CARDS]
|
|
|
|
1. Install NVidia CUDA toolkit from https://developer.nvidia.com/cuda-10.2-download-archive
|
|
2. Visit PyTorch's website(https://pytorch.org/get-started/locally/) and select Pip under "Package"
|
|
and your version of CUDA under "Compute Platform" (I linked 10.2) to get the pip3 command.
|
|
3. Copy and paste pip3 command into command prompt to install torch with GPU support
|
|
|
|
Be aware that when using GPU mode, inference will be MUCH faster but if your GPU doesn't have enough
|
|
VRAM to load the model it will crash the application. |