Debanjum ecc6fbfeb2 Push Files to Index from Emacs, Obsidian & Desktop Clients using Multi-Part Forms (#499) ... ### Overview - Add ability to push data to index from the Emacs, Obsidian client - Switch to standard mechanism of syncing files via HTTP multi-part/form. Previously we were streaming the data as JSON - Benefits of new mechanism - No manual parsing of files to send or receive on clients or server is required as most have in-built mechanisms to send multi-part/form requests - The whole response is not required to be kept in memory to parse content as JSON. As individual files arrive they're automatically pushed to disk to conserve memory if required - Binary files don't need to be encoded on client and decoded on server ### Code Details ### Major - Use multi-part form to receive files to index on server - Use multi-part form to send files to index on desktop client - Send files to index on server from the khoj.el emacs client - Send content for indexing on server at a regular interval from khoj.el - Send files to index on server from the khoj obsidian client - Update tests to test multi-part/form method of pushing files to index #### Minor - Put indexer API endpoint under /api path segment - Explicitly make GET request to /config/data from khoj.el:khoj-server-configure method - Improve emoji, message on content index updated via logger - Don't call khoj server on khoj.el load, only once khoj invoked explicitly by user - Improve indexing of binary files - Let fs_syncer pass PDF files directly as binary before indexing - Use encoding of each file set in indexer request to read file - Add CORS policy to khoj server. Allow requests from khoj apps, obsidian & localhost - Update indexer API endpoint URL to` index/update` from `indexer/batch` Resolves #471 #243		2023-10-17 06:05:15 -07:00
..
data	Fix plaintext HTML parsing and rendering (#464)	2023-08-27 11:24:30 -07:00
__init__.py	Move tests out to project root. Use absolute import in project	2021-09-30 04:12:14 -07:00
conftest.py	Update chat config schema. Make max_prompt, chat tokenizer configurable	2023-10-15 16:35:11 -07:00
test_cli.py	Remove PySide dependency and deprecate desktop builds (#475)	2023-09-07 11:36:27 -07:00
test_client.py	Mark test_get_configured_types_via_api unit test as flaky	2023-10-17 05:56:00 -07:00
test_conversation_utils.py	Add additional check for chat_messages length in UT	2023-08-01 09:25:52 -07:00
test_date_filter.py	Update date filter test to use compiled rather than raw key	2023-09-18 11:24:56 -07:00
test_file_filter.py	Add method to extract filter terms from query to all filters	2023-08-28 00:55:28 -07:00
test_gpt4all_chat_actors.py	Update Chat Actor test to not incept with prior world knowledge	2023-10-15 17:22:44 -07:00
test_gpt4all_chat_director.py	Enable passing search query filters via chat and test it	2023-08-28 09:24:32 -07:00
test_helpers.py	Use Black to format Khoj server code and tests	2023-02-17 11:55:17 -06:00
test_image_search.py	Update Tests to setup both content_index, search_models before testing	2023-07-14 01:29:48 -07:00
test_jsonl_to_jsonl.py	Prepend entry heading to all compiled org snippets to improve search context	2023-05-03 22:50:31 +08:00
test_markdown_to_jsonl.py	Move to a push-first model for retrieving embeddings from local files (#457)	2023-08-31 12:55:17 -07:00
test_openai_chat_actors.py	Add support for our first Local LLM 🤖🏠 (#330)	2023-07-26 16:27:08 -07:00
test_openai_chat_director.py	Enable passing search query filters via chat and test it	2023-08-28 09:24:32 -07:00
test_org_to_jsonl.py	Move to a push-first model for retrieving embeddings from local files (#457)	2023-08-31 12:55:17 -07:00
test_orgnode.py	Move to a push-first model for retrieving embeddings from local files (#457)	2023-08-31 12:55:17 -07:00
test_pdf_to_jsonl.py	Let fs_syncer pass PDF files directly as binary before indexing	2023-10-17 04:58:13 -07:00
test_plaintext_to_jsonl.py	Move to a push-first model for retrieving embeddings from local files (#457)	2023-08-31 12:55:17 -07:00
test_rawconfig.py	Add front-end Electron application for Khoj local file syncing (#473)	2023-09-06 12:04:18 -07:00
test_text_search.py	Add front-end Electron application for Khoj local file syncing (#473)	2023-09-06 12:04:18 -07:00
test_word_filter.py	Add method to extract filter terms from query to all filters	2023-08-28 00:55:28 -07:00