5 commits

Author	SHA1	Message	Date
Timothy Carambat	d1ca16f7f8	Add tokenizer improvments via Singleton class and estimation (#3072) ... * Add tokenizer improvments via Singleton class linting * dev build * Estimation fallback when string exceeds a fixed byte size * Add notice to tiktoken on backend	2025-01-30 17:55:03 -08:00
timothycarambat	2185753068	patch text.substring bug from compressor	2024-07-22 12:53:11 -07:00
timothycarambat	3e088f22b1	fix: Patch tiktoken method missing ... resolves #541	2024-01-05 09:39:19 -08:00
Timothy Carambat	92da23e963	Handle special token in TikToken (#528) ... * Handle special token in TikToken resolves #525 * remove duplicate method add clarification comment on implementation	2024-01-04 15:47:00 -08:00
Timothy Carambat	be9d8b0397	Infinite prompt input and compression implementation (#332) ... * WIP on continuous prompt window summary * wip * Move chat out of VDB simplify chat interface normalize LLM model interface have compression abstraction Cleanup compressor TODO: Anthropic stuff * Implement compression for Anythropic Fix lancedb sources * cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources * Resolve Weaviate citation sources not working with schema * comment cleanup	2023-11-06 13:13:53 -08:00