4 Commits

Author	SHA1	Message	Date
Cohee	5fc16a2474	New AI21 Jamba + tokenizer	2024-08-26 12:07:36 +03:00
Cohee	e707def7dd	Add tokenizer for Gemma/Gemini	2024-08-16 00:00:43 +03:00
Cohee	7bfd666321	Add llama 3 tokenizer	2024-05-03 23:59:39 +03:00
Cohee	212e61d2a1	Lazy initialization of Claude tokenizer. Add JSDoc for tokenizer handlers	2024-04-26 15:17:02 +03:00