Commit graph

1931 commits

Author SHA1 Message Date
sabaimran
a474c31e02 Move the django app into the src/khoj folder for better organization and functionality
- Our pypi package currently does not work because the django app and associated database is not included. To remedy this issue, move the app into the src/khoj folder. This has the added benefit of improved organization of the codebase, as all server related code is now in a single folder
- Update associated file paths and system references
2023-11-21 10:56:04 -08:00
Debanjum
6d8e889917
Improve Self Hosted Khoj Setup (#557)
- c07401cf Fix, Improve chat config via CLI on first run by using defaults
- d61b0dd5 Add Khoj Django app package to sys path to load Django module via pip install
- 4e98acbc Update minimum pydantic version to one with model_validate function
2023-11-20 17:25:53 -08:00
Debanjum Singh Solanky
c07401cf76 Fix, Improve chat config via CLI on first run by using defaults
- Fix setting prompt size for online chat
- generally improve chat config via cli by using default chat model,
  prompt size for online and offline chat
2023-11-20 17:01:20 -08:00
Debanjum Singh Solanky
d61b0dd55c Add Khoj Django app package to sys path to load Django module via pip install 2023-11-20 14:55:00 -08:00
Debanjum Singh Solanky
4e98acbca7 Update minimum pydantic version to one with model_validate function 2023-11-20 14:52:37 -08:00
sabaimran
237195e20e Make all name-related fields nullable within the GoogleUser 2023-11-19 14:22:32 -08:00
sabaimran
4def8cce36
Merge pull request #541 from asim-shrestha/patch-1
Add test separators
2023-11-19 14:14:34 -08:00
Debanjum
71799add0b
Index Parent Headings of Org-Mode Entries to Improve Search Context (#548)
### Overview
The parent hierarchy of org-mode entries can store important context. 
This change updates OrgNode to track parent headings for each org entry and adds the parent outline for each entry to the index

### Details
- Test search uses ancestor headings as context for improved results
- Add ancestor headings of each org-mode entry to their compiled form
- Track ancestor headings for each org-mode entry in org-node parser

Resolves #85
2023-11-19 13:18:19 -08:00
sabaimran
e398a76779 Fix test word filter 2023-11-19 13:14:58 -08:00
sabaimran
33a9304428 Resolve merge conflicts 2023-11-19 12:57:55 -08:00
sabaimran
cfd76b8472 Add open graph links to configure Khoj Docs preview 2023-11-19 12:16:59 -08:00
Debanjum Singh Solanky
c3465d6982 Release Khoj version 1.0.0 2023-11-19 09:50:25 -08:00
Debanjum
736744be3a
Update documentation to reflect new multi-user config scenario (#550)
- Update docs to show how to use Khoj Cloud
- Move self-hosting Khoj to separate section
- Add page to setup Desktop app
- Set default URL to Khoj Cloud URL in Obsidian, Emacs clients
2023-11-18 18:22:46 -08:00
Debanjum Singh Solanky
d0e84385f2 Simplify links in Khoj docs to use page_name.md with no prefixes
This allows jumping to page via VSCode IDE and on docs website
2023-11-18 18:17:46 -08:00
Debanjum Singh Solanky
fc65d8a9fe Add documentation page for the Khoj Desktop client 2023-11-18 18:17:35 -08:00
Debanjum Singh Solanky
35b469e488 Simplify setup, features since Khoj cloud in docs
- No Khoj server setup required to start using Khoj from Obsidian, Emacs
- Use tabs for install, upgrade in Emacs with different package
  managers
- Use default subtitles in Khoj Docs
- Deduplicate query filters, remove backend setup instructions in
  plugin pages
- Remove stale Setup demo on Khoj Obsidian plugin docs
2023-11-18 17:25:52 -08:00
Debanjum Singh Solanky
e1bf1f0e86 Update default Khoj server URL to Khoj cloud on Emacs, Obsidian clients 2023-11-18 16:25:45 -08:00
Debanjum Singh Solanky
8775ce730a Use URL fragments to allow jumping to config page sections on Web app 2023-11-18 16:25:45 -08:00
sabaimran
a5613cb08a
Merge pull request #554 from khoj-ai/fix/issues-with-prod-chat
Fix misc. issues with chat configuration
2023-11-18 14:45:06 -08:00
sabaimran
f792b1e301 Remove already defined identical function 2023-11-18 14:08:50 -08:00
sabaimran
e2fff5dc47 Don't explicitly use value to get the model type value 2023-11-18 14:01:01 -08:00
sabaimran
a8a25ceac2 Honor user's chat settings when running the extract questions phase
- Add marginally better error handling when GPT gives a messed up respones to the extract questions method
- Remove debug log lines
2023-11-18 13:31:51 -08:00
sabaimran
67156e6aec Add new logs for debugging issues with chat references 2023-11-18 12:10:50 -08:00
sabaimran
5de2ab6098 Change parse_obj calls to use model_validate per new pydantic specification 2023-11-18 12:10:36 -08:00
sabaimran
ebdb423d3e
Merge pull request #553 from khoj-ai/features/validation-errors
Update types of base config models for pydantic 2.0
2023-11-18 00:42:56 -08:00
sabaimran
6d249645a6 Fix interpretation of the default search type 2023-11-18 00:04:18 -08:00
sabaimran
f180b2ba94 Resolve mypy errors for various data types 2023-11-17 23:26:15 -08:00
sabaimran
3328a41f08 Update types of base config models for pydantic 2.0 2023-11-17 23:08:52 -08:00
sabaimran
f688529150 Update the default configuration for the AppConfig 2023-11-17 19:26:31 -08:00
Debanjum Singh Solanky
ca87b4ede9 Wrap common API query parameters into shared class to deduplicate code
- Upgrade FastAPI to >= latest version. Required upgrade of FastAPI.
  Earlier version didn't support wrapping common query params in class

- Use per fixture app instead of a global FastAPI app in conftest

- Upgrade minimum required Django version

- Fix no notes chat director test with updated no notes message
  No notes message was updated in commit 118f1143
2023-11-17 18:43:49 -08:00
Debanjum Singh Solanky
33ad9b8e64 Update text search test since indexing ancestor hierarchy added 2023-11-17 15:26:55 -08:00
Debanjum Singh Solanky
55785d50c3 Use title, when present, as root ancestor of entries instead of file path 2023-11-17 15:03:27 -08:00
Debanjum Singh Solanky
68ac1e0193 Automate Desktop app builds on new release or push to master branch 2023-11-16 16:09:03 -08:00
sabaimran
45a42faec8 Make adjectives more positive for api token generation 2023-11-16 15:55:35 -08:00
sabaimran
3934633947 Update references to all documentation to reflect instructions for managed service
- By default assume the audience of this website is people looking to understand the featuer offering of Khoj, and then people who are looking to self-host
2023-11-16 15:26:03 -08:00
sabaimran
7688228b9c Update docs to reflect new setup processes and instructions based on rearchitecture
- Most important updates include the depedency requirement to setup Postgres when running/setting Khoj up locally
- Add instructiosn for Docker
- Shift to recommend desktop client and update instructions for how to configure Khoj for user
2023-11-16 12:56:42 -08:00
sabaimran
118f1143ff When user tries using the notes slash command without having any data indexed 2023-11-16 12:52:39 -08:00
sabaimran
e8a13f0813
Add multi-user support to Khoj and use Postgres for backend storage (#549)
- Adds support for multiple users to be connected to the same Khoj instance using their Google login credentials
- Moves storage solution from in-memory json data to a Postgres db. This stores all relevant information, including accounts, embeddings, chat history, server side chat configuration
- Adds the concept of a Khoj server admin for configuring instance-wide settings regarding search model, and chat configuration
- Miscellaneous updates and fixes to the UX, including chat references, colors, and an updated config page
- Adds billing to allow users to subscribe to the cloud service easily
- Adds a separate GitHub action for building the dockerized production (tag `prod`) and dev (tag `dev`) images, separate from the image used for local building. The production image uses `gunicorn` with multiple workers to run the server.
- Updates all clients (Obsidian, Emacs, Desktop) to follow the client/server architecture. The server no longer reads from the file system at all; it only accepts data via the indexer API. In line with that, removes the functionality to configure org, markdown, plaintext, or other file-specific settings in the server. Only leaves GitHub and Notion for server-side configuration.
- Changes license to GNU AGPLv3

Resolves #467 
Resolves #488 
Resolves #303 
Resolves #345 
Resolves #195 
Resolves #280 
Resolves #461 
Closes #259 
Resolves #351
Resolves #301
Resolves #296
2023-11-16 11:48:01 -08:00
sabaimran
1466aef554 Change license to GNU AGPLv3 from GNU GPLv3
- This enforces that upstream consumers of this code should open source their software for any network-distributed services
2023-11-16 11:14:06 -08:00
sabaimran
36d200580b Use a different name for the production-config containers 2023-11-16 10:28:28 -08:00
sabaimran
ba633c4015 Only build the production docker image when pushing to master 2023-11-16 09:24:57 -08:00
Debanjum Singh Solanky
ddb07def0d Test search uses ancestor headings as context for improved results
- Update test data to add deeper outline hierarchy for testing
  hierarchy as context
- Update collateral tests that need count of entries updated, deleted
  asserts to be updated
2023-11-16 03:05:19 -08:00
Debanjum Singh Solanky
74403e3536 Add ancestor headings of each org-mode entry to their compiled form
Resolves #85
2023-11-16 02:54:41 -08:00
Debanjum Singh Solanky
305c25ae1a Track ancestor headings for each org-mode entry in org-node parser 2023-11-16 02:39:14 -08:00
Debanjum
208ddddc6a
Make Search Model Configurable on Server (#544)
- Make search model configurable on server
- Update migration script to get search model from `khoj.yml` to Postgres
- Update first run message on Khoj Desktop and Web app landing page
- Other miscellaneous bug fixes
2023-11-16 00:11:58 -08:00
Debanjum Singh Solanky
cc05013715 Update first run message on Web app with Chat models setup instructions
- Link to Django admin panel for user to create Chat Models on their
  Khoj server
- This should only get hit when user is not using Khoj cloud, as Khoj
  cloud would already have Chat models configured
2023-11-15 22:44:24 -08:00
Debanjum Singh Solanky
6c1693b8f4 Update first run message on Desktop app with API token setup instructions
- Open Web app settings in the default browser via link click
- Open Desktop app settings via link click
2023-11-15 22:44:11 -08:00
Debanjum Singh Solanky
922983bd53 Set max cos distance to 0.18. Test search API query with max distance 2023-11-15 20:26:21 -08:00
Debanjum Singh Solanky
18dbad5edb Use Sigmoid to normalize cross-encoder score between 0-1
- While sigmoid normalization isn't required for reranking.
  Normalizing score to distance metrics for both encoder and cross
  encoder scores is useful to reason about them
- Softmax wasn't required as don't need probabilities, sigmoid is good
  enough to get distance metric
2023-11-15 19:31:59 -08:00
sabaimran
0da4db4310
Merge pull request #547 from khoj-ai/features/fix-api-token-generator
Update the return type of the API token generator
2023-11-15 19:23:18 -08:00