rixty/KoboldAI-Client
1
1
Fork 0
mirror of https://github.com/KoboldAI/KoboldAI-Client.git synced 2025-06-05 21:59:24 +02:00
Code Issues Projects Releases Wiki Activity

Added AMD instructions, added formatting

Browse Source 
Added AMD install instructions
Formatted the install/run section for improved readability
This commit is contained in:
nerodiafasciata
2023-04-12 21:37:44 -05:00
committed by 0cc4m
parent ded5542d3a
commit 10c99a853c
1 changed files with 23 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
README.md
View File

@@ -3,6 +3,7 @@
### Install/Use Guide
(This guide is for both Linux and Windows and assumes user has git installed and a basic grasp of command line use)
#### Installation
In the command prompt/command line navigate to where you want the KoboldAI subfolder to be created.
`git clone https://github.com/0cc4m/KoboldAI -b latestgptq --recurse-submodules`
@@ -11,10 +12,28 @@ In the command prompt/command line navigate to where you want the KoboldAI subfo
Next step, (Windows) subfolder mode or B: option doesn't matter choose either
[if on Windows] `install_requirements.bat` if it closes the window when it finishes, reopen a command prompt and navigate back to your KoboldAI directory.
* [if on Windows]
```
install_requirements.bat
```
* if it closes the window when it finishes, reopen a command prompt and navigate back to your KoboldAI directory.
[if on Linux] `install_requirements.sh`
* [if on Linux with Nvidia]
```
./install_requirements.sh
```
* [if on Linux with AMD]
```
./install_requirements.sh rocm
./commandline-rocm.sh
cd repos/gptq
python setup_cuda.py install
```
* If you get error missing hip/hip_runtime_xxx.h you dont have proper rocm & hip pkg installed
* If you get CUDA_HOME envar is not set run in env:
`pip3 install torch --index-url https://download.pytorch.org/whl/rocm5.4.2 --force-reinstall`
#### Setting up models
If you haven't already done so, create a model folder with the same name as your model (or whatever you want to name the folder)
Put your 4bit quantized .pt or .safetensors in that folder with all associated .json files and tokenizer.model (.json files and tokenizer.model should be from the Huggingface model folder of the same model type).
@@ -23,9 +42,10 @@ Then move your model folder to KoboldAI/models, and rename the .pt or .safetenso
So - your .pt's model folder should look like this: "4bit.pt, config.json, generation_config.json, pytorch_model.bin.index.json, special_tokens_map.json, tokenizer.model, tokenizer_config.json" Note: the 4bit.pt file can be in the same folder as the regular HF .bin files it was quantized from, so long as the 4-bit toggle switch is on, it'll load the quantized model (4-bit switch explained below).
#### Running KoboldAI and loading 4bit models
If you haven't done so already, exit the command prompt/leave KAI's conda env. (Close the commandline window on Windows, run `exit` on Linux)
Run `play.bat` [windows] or `play.sh` [linux]
Run `play.bat` [windows], `play.sh` [linux Nvidia], or `play-rocm.sh` [linux AMD]
Switch to UI2, enable Experimental UI under the Interface tab, then load your model and be sure 4-bit toggle is on.