8 Commits

Author	SHA1	Message	Date
Cohee	57e845d0d7	Resolve best match tokenizer for itemization. Adjust styles of token counter	2023-11-06 20:25:59 +02:00
RossAscends	0effbebae7	tokenizer colorization underpinnings (inactive)	2023-11-06 18:04:47 +09:00
Cohee	e8ba328a14	Add text chunks display to token counter	2023-11-06 02:42:51 +02:00
Cohee	fd9c8b86e9	Token counter layout adjust	2023-11-05 22:55:10 +02:00
Cohee	f0c0949aa0	Add token ids viewer to tokenizer plugin	2023-11-05 22:45:37 +02:00
Cohee	85de505553	Move counter to plugin. Use chat context to get messages	2023-10-21 14:23:56 +03:00
Cohee	bc5fc67906	Put tokenizer functions to a separate file. Cache local models token counts	2023-08-23 02:38:43 +03:00
Cohee	edd41989fd	Initial commit	2023-07-20 20:32:15 +03:00