rixty/KoboldAI-Client
1
1
Fork 0
mirror of https://github.com/KoboldAI/KoboldAI-Client.git synced 2025-02-17 12:10:49 +01:00
Code Issues Projects Releases Wiki Activity

Fix float16 models

Browse Source 
Forcefully convert float16 models to work on the CPU
This commit is contained in:
henk717 2021-12-16 00:31:51 +01:00
parent 46b0473229
commit 5e3e3f3578
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
aiserver.py
View File

@ -827,8 +827,10 @@ if(not vars.model in ["InferKit", "Colab", "OAI", "ReadOnly", "TPUMeshTransforme
elif(vars.breakmodel): # Use both RAM and VRAM (breakmodel) elif(vars.breakmodel): # Use both RAM and VRAM (breakmodel)
device_config(model) device_config(model)
else: else:
model = model.to('cpu').float()
generator = model.generate generator = model.generate
else: else:
model = model.to('cpu').float()
generator = model.generate generator = model.generate
# If custom GPT2 model was chosen # If custom GPT2 model was chosen
elif(vars.model == "GPT2Custom"): elif(vars.model == "GPT2Custom"):
@ -843,6 +845,7 @@ if(not vars.model in ["InferKit", "Colab", "OAI", "ReadOnly", "TPUMeshTransforme
model = model.half().to(0) model = model.half().to(0)
generator = model.generate generator = model.generate
else: else:
model = model.to('cpu').float()
generator = model.generate generator = model.generate
# If base HuggingFace model was chosen # If base HuggingFace model was chosen
else: else: