2311 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Debanjum Singh Solanky	3f24e53b6e	Render URL as link in web interface if file param of result is a web link	2023-06-17 04:26:40 -07:00
Debanjum Singh Solanky	63ec84ad78	Store Github URL of Markdown files on Github in file jsonl param	2023-06-17 04:23:01 -07:00
Debanjum Singh Solanky	0c1c7583b5	Handle pagination, API rate limits. Get all commits from Github repo	2023-06-17 04:21:39 -07:00
Debanjum Singh Solanky	31d17d0b22	Index commits message from repository with the github plugin	2023-06-17 02:59:54 -07:00
Debanjum Singh Solanky	c29c141a7e	Use Github Rest API to index Markdown files in Github Repository ... The Llama_Hub Github plugin is fairly limited. The Github Rest API is well supported and can easily be extended to index commit messages, issues, discussions, PRs etc.	2023-06-17 02:16:13 -07:00
Debanjum	9f00a366ab	Add a Github plugin to index content from a Github repository ... - Use the Github plugin on LlamaHub to read in markdown files from specified Github repository for indexing - Update the desktop GUI application to take in the required parameters to read from Github - Requires a classic PAT token for Github access	2023-06-17 12:28:47 +05:30
Saba	ac96f43b1b	Remove try-catch specific to Github plugin; consolidate GUI logic	2023-06-16 23:46:25 -07:00
Saba	07ade2262a	Set default value of pat_token in conftest.py to be empty string	2023-06-13 17:03:03 -07:00
Saba	751edfefe5	Add separate unit test for github. Will only run of a PAT token is set	2023-06-13 16:55:58 -07:00
Saba	3a61919344	Fix failing unit tests by hard-coding model presence of expected search types	2023-06-13 16:32:47 -07:00
Saba	019d3732de	Rename orgmode_search to org_search	2023-06-13 16:06:54 -07:00
Saba	08d79f5ba4	Unify types used in Github and other text-based configs. Fix typing issues	2023-06-13 15:52:36 -07:00
Saba	a6cd96a6a9	Add a Github plugin which can be used to read from a Github repository	2023-06-13 14:40:06 -07:00
Debanjum	c68cde4803	Log clients calling API endpoints on Khoj server ... - Make API endpoints on Khoj server accept `client` as request parameter - Khoj API endpoints: /chat, /search, /update - Make Khoj clients set `client` request param when calling the API endpoints on the Khoj server - Khoj clients: Emacs, Obsidian and Web - Also log khoj server_version running to telemetry server	2023-06-09 18:36:49 +05:30
sabaimran	59fa48036f	Merge pull request #224 from debanjum/fix/message-exceeds-prompt-size ... Pass truncated message as string in ChatMessage when exceeding max prompt size	2023-06-08 17:32:53 -07:00
Debanjum Singh Solanky	139a3ba060	Update server to log new server version field to telemetry db	2023-06-08 14:14:21 +05:30
Saba	c5666e0404	Move factory dependencies to optional settings	2023-06-06 23:26:24 -07:00
Saba	5d5ebcbf7c	Rename truncate messages method and update unit tests to simplify assertion logic	2023-06-06 23:25:43 -07:00
Saba	7119ed0849	Run pre-commit script	2023-06-05 19:29:23 -07:00
Saba	948ba6ddca	Remove unused logger	2023-06-05 19:01:03 -07:00
Saba	6212d7c2e8	Remove debug line	2023-06-05 19:00:25 -07:00
Saba	f65ff9815d	Move message truncation logic into a separate function. Add unit tests with factory boy.	2023-06-05 18:58:29 -07:00
Debanjum Singh Solanky	eb6175e9b0	Update description field in webmanifest of Khoj, Khoj Chat PWA	2023-06-06 01:53:42 +05:30
Debanjum Singh Solanky	bb2363f324	Set client request param when calling khoj server APIs from Web	2023-06-06 00:05:00 +05:30
Debanjum Singh Solanky	caab55fbdd	Set client request param when calling khoj server APIs from Obsidian	2023-06-06 00:04:46 +05:30
Debanjum Singh Solanky	de2494154f	Set client request param when calling khoj server APIs from Emacs	2023-06-06 00:02:10 +05:30
Debanjum Singh Solanky	168c11cea7	Make server API endpoints accept client as query param ... - The chat, search and update API will accept client as request param. - This will allow logging the client from which these APIs was called.	2023-06-05 23:57:08 +05:30
Debanjum Singh Solanky	8617cf1389	Push telemetry to Posthog to grok Khoj usage	2023-06-05 22:47:49 +05:30
Debanjum Singh Solanky	d13db2e666	Make old telemetry server forward requests to new server	2023-06-05 13:06:45 +05:30
Saba	5f4223efb4	Increase timeout to OpenAI call	2023-06-04 20:49:47 -07:00
Saba	0e63a90377	Fix the mechanism to retrieve the message content	2023-06-04 20:25:37 -07:00
Saba	f0efe0177e	Pass truncated message as string in ChatMessage when exceeding max prompt size	2023-06-04 19:33:46 -07:00
Debanjum	f6ceb22373	Use api_key keyword argument to set the openai_api_key parameter for GPT	2023-06-04 15:05:34 +05:30
Saba	068ee0ac5e	Swap elif with else, as usage of this method does not use openai_api_key	2023-06-04 02:25:08 -07:00
Saba	6508379d7b	Use api_key keyword argument to set the openai_api_key parameter for GPT	2023-06-04 00:57:00 -07:00
Debanjum Singh Solanky	7af8a56434	Remove filename from reference before rendering references in khoj.el ... Fixes bug where actual reference heading in next line jumping out of references footnote section	2023-06-02 10:42:44 +05:30
Debanjum Singh Solanky	ec280067ef	Do not retrieve relevant notes when having a general chat with Khoj ... - This improves latency of @general chat by avoiding unnecessary compute - It also avoids passing references in API response when they haven't been used to generate the chat response. So interfaces don't have to add logic to not render them unnecessarily	2023-06-02 10:42:44 +05:30
Debanjum Singh Solanky	90439a8db1	Update Khoj subtitle to AI personal assistant for your digital brain	2023-06-02 10:42:44 +05:30
Debanjum	e022910f31	Search PDF files with Khoj. Integrate with LangChain ... - **Introduce Khoj to LangChain**: Call GPT with LangChain for Khoj Chat - **Search (and Chat about) PDF files with Khoj** - Create PDF to JSONL Processor: Convert PDF content into standardized JSONL format - Expose PDF search type via Khoj server API - Enable querying PDF files via Obsidian, Emacs and Web interfaces	2023-06-02 10:20:26 +05:30
Debanjum Singh Solanky	e9ed7a19fd	Update search prompt to extract PDF search type. Fix extract_question prompt	2023-06-02 10:06:03 +05:30
Debanjum Singh Solanky	89fbfce20a	Mention PDF are also supported in Khoj Readme	2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky	bbe3bf9733	Render PDF search results in Khoj Obsidian interface ... - Make plugin update khoj server config to index PDF files in vault too - Make Obsidian plugin update index for PDF files in vault too - Show PDF results in Khoj Search modal as well - Ensure combined results are sorted by score across both types - Jump to PDF file when select it PDF search result from modal	2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky	e3892945d4	Render PDF search results in Khoj.el Emacs interface	2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky	85144006a1	Render PDF search results in khoj web interface	2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky	acd14a5e41	Wire up PDF to jsonl processor to Khoj server layer (API, config) ... - Specify PDF content to index via khoj.yml - Index PDF content on app start, reconfigure - Expose PDF as a search type via API	2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky	d63194c3a9	Create tests for PDF to JSONL processor	2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky	286b500f66	Create PDF to JSONL processor using PyPDF and LangChain ... Switch `pydantic' to >= 1.9.1 else `langchain.document_loaders' starts throwing typing error for python 3.8, 3.9	2023-06-01 21:41:49 +05:30
Debanjum Singh Solanky	1b3effd8e6	Fork Markdown to JSONL processor as start template for PDF to Jsonl Processor	2023-06-01 09:13:31 +05:30
Debanjum Singh Solanky	1cd9ecd449	Truncate last message if still over max supported prompt size by model	2023-06-01 08:50:59 +05:30
Debanjum Singh Solanky	ed4d0f9076	Simplify argument names used in khoj openai completion functions ... - Match argument names passed to khoj openai completion funcs with arguments passed to langchain calls to OpenAI - This simplifies the logic in the khoj openai completion funcs	2023-06-01 08:50:59 +05:30