4 Commits

Author	SHA1	Message	Date
Cohee	bc5fc67906	Put tokenizer functions to a separate file. Cache local models token counts	2023-08-23 02:38:43 +03:00
Cohee	41cc86af9f	Add example extension for chat variables. Allow registering custom text processing functions for extensions	2023-08-22 17:46:37 +03:00
Cohee	9deaec5877	More typechecks and comments	2023-08-22 14:30:49 +03:00
Cohee	e2bac7ec5f	Add type checking and JSDoc comments to some utils	2023-08-22 13:07:24 +03:00