3027 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
sabaimran	6224dce49d	Merge pull request #228 from debanjum/features/pretty-config-page ... Update the config page to be more usable	2023-06-19 18:11:35 -07:00
sabaimran	4722a2c16d	Add Github configuration page and success notifications	2023-06-18 10:06:45 -07:00
sabaimran	668135c763	Merge branch 'master' of github.com:debanjum/khoj into features/pretty-config-page	2023-06-18 08:35:09 -07:00
sabaimran	81183a1fe1	Address misc PR comments and update logo in all clients ... - Rename the new logo to reflect accuracy on size (e.g., 128x128) - Update the icns file for Mac - Update nomenclature in settings pages	2023-06-18 08:34:58 -07:00
Debanjum Singh Solanky	a44cde2865	Show hint to re-index vault if wonky results in Obsidian search modal ... Remove spurious indentation in Obsidian styles.css Resolves #207	2023-06-18 04:53:51 -07:00
Debanjum Singh Solanky	595cc5b0f5	Use printer icon for PDF logs. Only split lines if file at web link in web interface	2023-06-18 02:26:03 -07:00
Debanjum	e06be395f9	Use Github REST API and Index Commit Messages off Github Repository ... - Migrate to Github REST API instead of Llama Hub to index Markdown Docs in Github Repository - Index Commit Messages from Github Repository as well	2023-06-18 14:51:32 +05:30
Debanjum Singh Solanky	e31a540a5e	Get all md files recursively in repository by passing recursive param ... Previously the `get_markdown_files' method was only getting files at root of the repository Fix, improve logger messages in github to jsonl processor	2023-06-18 01:47:15 -07:00
Debanjum Singh Solanky	6fdac24416	Set page size to 100 to reduce requests required to Github API to 1/3 ... - Default is 30. So number of paginated requests required to get all items (commits, files) will reduce by 67% - No need to increase page size for the get tree Github API request from `get_markdown_files' Get tree Github API doesn't support pagination and return 100K items in response. This should be way more than enough for our current use-cases	2023-06-18 01:44:36 -07:00
Debanjum Singh Solanky	87975e589a	Fix passing auth token to Github API to increase rate limits by x85 ... - Previously wasn't prefixing "token" to PAT token in Auth header This resulted in the request being considered unauthenticated - Unauthenticated requests to Github API are limited to 60 requests/hour Authenticated requests to Github API are allowed 5000 requests/hour	2023-06-18 01:19:26 -07:00
Debanjum Singh Solanky	9c70af960c	Extract logic to get file content from Github into a separate method	2023-06-18 01:19:13 -07:00
Debanjum Singh Solanky	10d4c38ce9	Extract Wait for rate limit reset logic into a function for reuse	2023-06-18 01:06:46 -07:00
sabaimran	aad7f825e0	Remove music configuration	2023-06-17 21:23:56 -07:00
sabaimran	5f97afbfac	Ignore type checks from mypy in subindexed fields	2023-06-17 16:53:36 -07:00
sabaimran	c2d46de8bc	Add endpoint for regenerating directly from the config page and add music content-type	2023-06-17 15:47:33 -07:00
sabaimran	ded3100caf	Update the configuration page to make config management easier ... - Add a central configuration management page to make management of config details easier - Add relevant api endpoints both for client and server to update/request data as necessary - Attempt to update the favicon	2023-06-17 15:21:28 -07:00
Debanjum Singh Solanky	3f24e53b6e	Render URL as link in web interface if file param of result is a web link	2023-06-17 04:26:40 -07:00
Debanjum Singh Solanky	63ec84ad78	Store Github URL of Markdown files on Github in file jsonl param	2023-06-17 04:23:01 -07:00
Debanjum Singh Solanky	0c1c7583b5	Handle pagination, API rate limits. Get all commits from Github repo	2023-06-17 04:21:39 -07:00
Debanjum Singh Solanky	31d17d0b22	Index commits message from repository with the github plugin	2023-06-17 02:59:54 -07:00
Debanjum Singh Solanky	c29c141a7e	Use Github Rest API to index Markdown files in Github Repository ... The Llama_Hub Github plugin is fairly limited. The Github Rest API is well supported and can easily be extended to index commit messages, issues, discussions, PRs etc.	2023-06-17 02:16:13 -07:00
Debanjum	9f00a366ab	Add a Github plugin to index content from a Github repository ... - Use the Github plugin on LlamaHub to read in markdown files from specified Github repository for indexing - Update the desktop GUI application to take in the required parameters to read from Github - Requires a classic PAT token for Github access	2023-06-17 12:28:47 +05:30
Saba	ac96f43b1b	Remove try-catch specific to Github plugin; consolidate GUI logic	2023-06-16 23:46:25 -07:00
Saba	07ade2262a	Set default value of pat_token in conftest.py to be empty string	2023-06-13 17:03:03 -07:00
Saba	751edfefe5	Add separate unit test for github. Will only run of a PAT token is set	2023-06-13 16:55:58 -07:00
Saba	3a61919344	Fix failing unit tests by hard-coding model presence of expected search types	2023-06-13 16:32:47 -07:00
Saba	019d3732de	Rename orgmode_search to org_search	2023-06-13 16:06:54 -07:00
Saba	08d79f5ba4	Unify types used in Github and other text-based configs. Fix typing issues	2023-06-13 15:52:36 -07:00
Saba	a6cd96a6a9	Add a Github plugin which can be used to read from a Github repository	2023-06-13 14:40:06 -07:00
Debanjum	c68cde4803	Log clients calling API endpoints on Khoj server ... - Make API endpoints on Khoj server accept `client` as request parameter - Khoj API endpoints: /chat, /search, /update - Make Khoj clients set `client` request param when calling the API endpoints on the Khoj server - Khoj clients: Emacs, Obsidian and Web - Also log khoj server_version running to telemetry server	2023-06-09 18:36:49 +05:30
sabaimran	59fa48036f	Merge pull request #224 from debanjum/fix/message-exceeds-prompt-size ... Pass truncated message as string in ChatMessage when exceeding max prompt size	2023-06-08 17:32:53 -07:00
Debanjum Singh Solanky	139a3ba060	Update server to log new server version field to telemetry db	2023-06-08 14:14:21 +05:30
Saba	c5666e0404	Move factory dependencies to optional settings	2023-06-06 23:26:24 -07:00
Saba	5d5ebcbf7c	Rename truncate messages method and update unit tests to simplify assertion logic	2023-06-06 23:25:43 -07:00
Saba	7119ed0849	Run pre-commit script	2023-06-05 19:29:23 -07:00
Saba	948ba6ddca	Remove unused logger	2023-06-05 19:01:03 -07:00
Saba	6212d7c2e8	Remove debug line	2023-06-05 19:00:25 -07:00
Saba	f65ff9815d	Move message truncation logic into a separate function. Add unit tests with factory boy.	2023-06-05 18:58:29 -07:00
Debanjum Singh Solanky	eb6175e9b0	Update description field in webmanifest of Khoj, Khoj Chat PWA	2023-06-06 01:53:42 +05:30
Debanjum Singh Solanky	bb2363f324	Set client request param when calling khoj server APIs from Web	2023-06-06 00:05:00 +05:30
Debanjum Singh Solanky	caab55fbdd	Set client request param when calling khoj server APIs from Obsidian	2023-06-06 00:04:46 +05:30
Debanjum Singh Solanky	de2494154f	Set client request param when calling khoj server APIs from Emacs	2023-06-06 00:02:10 +05:30
Debanjum Singh Solanky	168c11cea7	Make server API endpoints accept client as query param ... - The chat, search and update API will accept client as request param. - This will allow logging the client from which these APIs was called.	2023-06-05 23:57:08 +05:30
Debanjum Singh Solanky	8617cf1389	Push telemetry to Posthog to grok Khoj usage	2023-06-05 22:47:49 +05:30
Debanjum Singh Solanky	d13db2e666	Make old telemetry server forward requests to new server	2023-06-05 13:06:45 +05:30
Saba	5f4223efb4	Increase timeout to OpenAI call	2023-06-04 20:49:47 -07:00
Saba	0e63a90377	Fix the mechanism to retrieve the message content	2023-06-04 20:25:37 -07:00
Saba	f0efe0177e	Pass truncated message as string in ChatMessage when exceeding max prompt size	2023-06-04 19:33:46 -07:00
Debanjum	f6ceb22373	Use api_key keyword argument to set the openai_api_key parameter for GPT	2023-06-04 15:05:34 +05:30
Saba	068ee0ac5e	Swap elif with else, as usage of this method does not use openai_api_key	2023-06-04 02:25:08 -07:00