From 10c99a853c207c34d163914042b903d606dad8ee Mon Sep 17 00:00:00 2001 From: nerodiafasciata Date: Wed, 12 Apr 2023 21:37:44 -0500 Subject: [PATCH] Added AMD instructions, added formatting Added AMD install instructions Formatted the install/run section for improved readability --- README.md | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 0657fa0b..170c4f42 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,7 @@ ### Install/Use Guide (This guide is for both Linux and Windows and assumes user has git installed and a basic grasp of command line use) +#### Installation In the command prompt/command line navigate to where you want the KoboldAI subfolder to be created. `git clone https://github.com/0cc4m/KoboldAI -b latestgptq --recurse-submodules` @@ -11,10 +12,28 @@ In the command prompt/command line navigate to where you want the KoboldAI subfo Next step, (Windows) subfolder mode or B: option doesn't matter choose either -[if on Windows] `install_requirements.bat` if it closes the window when it finishes, reopen a command prompt and navigate back to your KoboldAI directory. +* [if on Windows] + ``` + install_requirements.bat + ``` + * if it closes the window when it finishes, reopen a command prompt and navigate back to your KoboldAI directory. -[if on Linux] `install_requirements.sh` +* [if on Linux with Nvidia] + ``` + ./install_requirements.sh + ``` +* [if on Linux with AMD] + ``` + ./install_requirements.sh rocm + ./commandline-rocm.sh + cd repos/gptq + python setup_cuda.py install + ``` + * If you get error missing hip/hip_runtime_xxx.h you dont have proper rocm & hip pkg installed + * If you get CUDA_HOME envar is not set run in env: + `pip3 install torch --index-url https://download.pytorch.org/whl/rocm5.4.2 --force-reinstall` +#### Setting up models If you haven't already done so, create a model folder with the same name as your model (or whatever you want to name the folder) Put your 4bit quantized .pt or .safetensors in that folder with all associated .json files and tokenizer.model (.json files and tokenizer.model should be from the Huggingface model folder of the same model type). @@ -23,9 +42,10 @@ Then move your model folder to KoboldAI/models, and rename the .pt or .safetenso So - your .pt's model folder should look like this: "4bit.pt, config.json, generation_config.json, pytorch_model.bin.index.json, special_tokens_map.json, tokenizer.model, tokenizer_config.json" Note: the 4bit.pt file can be in the same folder as the regular HF .bin files it was quantized from, so long as the 4-bit toggle switch is on, it'll load the quantized model (4-bit switch explained below). +#### Running KoboldAI and loading 4bit models If you haven't done so already, exit the command prompt/leave KAI's conda env. (Close the commandline window on Windows, run `exit` on Linux) -Run `play.bat` [windows] or `play.sh` [linux] +Run `play.bat` [windows], `play.sh` [linux Nvidia], or `play-rocm.sh` [linux AMD] Switch to UI2, enable Experimental UI under the Interface tab, then load your model and be sure 4-bit toggle is on.