6 commits

Author	SHA1	Message	Date
timothycarambat	64b3210db2	build our own worker fanout and wrapper	2025-02-14 10:31:06 -08:00
timothycarambat	c463710b0f	wip	2025-02-13 21:59:12 -08:00
timothycarambat	2f89bbae74	OCR PDFs as fallback in spawn thread	2025-02-13 16:06:47 -08:00
Timothy Carambat	d1ca16f7f8	Add tokenizer improvments via Singleton class and estimation (#3072) ... * Add tokenizer improvments via Singleton class linting * dev build * Estimation fallback when string exceeds a fixed byte size * Add notice to tiktoken on backend	2025-01-30 17:55:03 -08:00
Sean Hatfield	9b86bbd2b8	[FIX] PDFLoader module bug fix (#1879) ... use pdf.js by importing it from pdf-parse and fix custom PDFLoader module	2024-07-16 13:09:43 -07:00
Sean Hatfield	79656718b2	[FEAT] Create custom pdfloader (#1852) ... * implement custom PDFLoader to remove LC dep * remove unneeded comment * remove pdfjs as dep and fix page splitting using pdf-parse * linting + export rename for desktop compat --------- Co-authored-by: timothycarambat <rambat1010@gmail.com>	2024-07-11 12:26:11 -07:00