Commit graph

3200 commits

Author SHA1 Message Date
Debanjum
1d3bdf8fdb
Create Billing integration. Improve Settings pages on Desktop, Web apps (#537)
### Major
- Expose Billing via Stripe on Khoj Web app for Khoj Cloud subscription
  - Expose card on web app config page to manage subscription to Khoj cloud
  - Create API webhook, endpoints for subscription payments using Stripe
- Put Computer files to index into Card under Content section
  - Show file type icons for each indexed file in config card of web app
  - Enable deleting all indexed desktop files from Khoj via Desktop app
  - Create config page on web app to manage computer files indexed by Khoj
- Track data source (computer, github, notion) of each entry
  - Update content by source via API. Make web client use this API for config
  - Store the data source of each entry in database

### Cleanup
- Set content enabled status on update via config buttons on web app
- Delete deprecated content config pages for local files from web client
- Rename Sync button, Force Sync toggle to Save, Save All buttons

### Fixes
- Prevent Desktop app triggering multiple simultaneous syncs to server
- Upgrade langchain version since adding support for OCR-ing PDFs
- Bubble up content indexing errors to notify user on client apps
2023-11-08 19:55:35 -08:00
Debanjum Singh Solanky
a2609973b8 Disable Subscription if Stripe environment not setup
Deduplicate DJANGO_SECRET_KEY and KHOJ_DJANGO_SECRET_KEY to latter
name as prefixed with KHOJ as KHOJ app specific
2023-11-08 19:39:32 -08:00
Debanjum Singh Solanky
09e1235832 Auto update billing card UI on (re/un-)subscribe click on web app
Previously required a page load to see the updated billing state after
clicking resubscribe or unsubscribe buttons
2023-11-08 18:38:12 -08:00
Debanjum Singh Solanky
8b8bb15866 Keep sync state in memory, initialized to false in Desktop app
Prevent deadlock if desktop app killed in middle of syncing
2023-11-08 18:03:08 -08:00
Debanjum Singh Solanky
c043eb54ae Use typed entry source instead of raw str to map source to conf in api.py 2023-11-08 18:03:08 -08:00
Debanjum Singh Solanky
8178004e6d Move Subscription data into separate table in DB. Merge migrations 2023-11-08 18:03:08 -08:00
Debanjum Singh Solanky
3bb10128ef Move subscription API to separate, independent router 2023-11-08 16:20:27 -08:00
Debanjum Singh Solanky
ec1395d072 Clean, merge subscription update events, API and functions
- Reduce webhook triggers for subscription updates
- Merge subscription update API endpoint, functions for (re/un-)subscribe
2023-11-08 15:55:20 -08:00
Debanjum Singh Solanky
ef5c13f968 Keep user subscription state. Update it when user has unsubscribed 2023-11-08 12:08:36 -08:00
Debanjum Singh Solanky
c52affc6d9 Get Khoj Cloud Subscription URL via environment variable 2023-11-08 12:07:53 -08:00
sabaimran
609d358b1a Use sql datetime comparison for detecting validity of subscription renewal date
- Update the unsubscribe endpoint to use query params
- Use subscription id to process unsubscribe endpoint, rather than the customer id
2023-11-07 19:17:36 -08:00
sabaimran
98cf095b65 Fix bug for rendering chat references in LLM response 2023-11-07 16:44:41 -08:00
sabaimran
0e1cdb6536 Add additional error handling for processing unknown Stripe events and fix typo in STRIPE_SIGNING env variable 2023-11-07 16:43:05 -08:00
sabaimran
08c86927cb Merge branch 'features/multi-user-support-khoj' of github.com:khoj-ai/khoj into fix-improve-config-page-on-desktop-and-web-app 2023-11-07 12:46:49 -08:00
sabaimran
cec54e3a8a
Merge pull request #536 from khoj-ai/features/update-chat-ui
Update the chat UI to have richer representation of the references
2023-11-07 12:34:57 -08:00
Debanjum Singh Solanky
f466751f4d Expose card on web app config page to manage subscription to Khoj cloud 2023-11-07 10:21:00 -08:00
Debanjum Singh Solanky
9aaf475c8a Create API webhook, endpoints for subscription payments using Stripe
- Add fields to mark users as subscribed to a specific plan and
  subscription renewal date in DB
- Add ability to unsubscribe a user using their email address
- Expose webhook for stripe to callback confirming payment
2023-11-07 10:20:51 -08:00
Debanjum Singh Solanky
156421d30a Show file type icons for each indexed file in config card of web app 2023-11-07 05:48:44 -08:00
Debanjum Singh Solanky
045c2252d6 Set content enabled status on update via config buttons on web app
Previously hitting configure or disable wouldn't update the state of
the content cards. It needed page refresh to see if the content was
synced correctly.

Now cards automatically get set to new state on hitting disable button
on card or global configure buttons
2023-11-07 05:28:13 -08:00
Debanjum Singh Solanky
7c424e0d5f Enable deleting all indexed desktop files from Khoj via Desktop app 2023-11-07 05:28:13 -08:00
Debanjum Singh Solanky
779fa531a5 Prevent Desktop app triggering multiple simultaneous syncs to server
Lock syncing to server if a sync is already in progress.

While the sync save button gets disabled while sync is in progress,
the background sync job can still trigger a sync in parallel. This
sync lock prevents that
2023-11-07 05:28:13 -08:00
Debanjum Singh Solanky
404d47f1a1 Bubble up content indexing errors to notify user on client apps 2023-11-07 05:28:13 -08:00
Debanjum Singh Solanky
6e957584ac Create config page on web app to manage computer files indexed by Khoj
Remove the table of all files indexed by Khoj. This seems overkill and
doesn't match the UI semantics of the other data sources like Github,
Notion.

Create instead a data source card for computer files with the same
update, disable semantics of the Github and Notion data source cards

Users can disable each data source from its card on the main config page.

They can see/delete individual files indexed from the computer data source
once they click into the computer files data source card on the config page
2023-11-07 04:42:53 -08:00
Debanjum Singh Solanky
d527b644f4 Update content by source via API. Make web client use this API for config 2023-11-07 03:41:19 -08:00
Debanjum Singh Solanky
9ab327a2b6 Store the data source of each entry in database
This will be useful for updating, deleting entries by their data
source. Data source can be one of Computer, Github or Notion for now

Store each file/entries source in database
2023-11-07 02:18:48 -08:00
Debanjum Singh Solanky
c82cd0862a Delete deprecated content config pages for local files from web client
The desktop app now manages syncing local computer files to index
The server only manages "cloud" data source like github and notion.
2023-11-06 23:55:37 -08:00
Debanjum Singh Solanky
9f47fc8e34 Upgrade langchain version since adding support for OCR-ing PDFs 2023-11-06 21:58:33 -08:00
Debanjum Singh Solanky
97cf8339aa Rename Sync button, Force Sync toggle to Save, Save All buttons 2023-11-06 21:57:37 -08:00
Debanjum Singh Solanky
a08b152358 Improve log messages in text_entries and memory leak unit test 2023-11-06 19:27:31 -08:00
sabaimran
6c8689e4ae Update corresponding chat UX in the desktop client as well 2023-11-06 16:18:41 -08:00
sabaimran
e01ecf1419 /s/references/reference to fix bug of jumping references 2023-11-06 16:12:25 -08:00
Debanjum
38f24a037d
Improve Indexing Text Entries (#535)
Major
- Ensure search results logic consistent across migration to DB, multi-user
- Manually verified search results for sample queries look the same across migration
 - Flatten indexing code for better indexing progress tracking and code readability

Minor
- a4f407f Test memory leak on MPS device when generating vector embeddings
- ef24485 Improve Khoj with DB setup instructions in the Django app readme (for now)
- f212cc7 Arrange remaining text search tests in arrange, act, assert order
- 022017d Fix text search tests to test updated indexing log messages
2023-11-06 16:01:53 -08:00
sabaimran
270f7b3eb3 Update the chat UI to have richer representation of the references 2023-11-05 15:46:43 -08:00
sabaimran
81a615d7dd
Merge pull request #534 from khoj-ai/features/code-config-cleanup
Small fixes and update config UI to manage indexed data
2023-11-05 15:45:45 -08:00
sabaimran
8ebb12820c Add OCR runtime dependencies to prod Dockerfile as well 2023-11-05 15:40:05 -08:00
sabaimran
d697d752c2
Use repeat rather than manually specify auto in grid-template-rows
Co-authored-by: Debanjum <debanjum@gmail.com>
2023-11-05 15:23:42 -08:00
sabaimran
3d6e8d53fe Try adding dependencies for libgl in order to run OCR in github action unit tests 2023-11-05 15:09:40 -08:00
sabaimran
5f1e37fff0 Adjust indentation for css property 2023-11-05 14:33:23 -08:00
sabaimran
fdd727712f Rename test files from x_to_jsonl to x_to_entries 2023-11-05 14:33:07 -08:00
Debanjum Singh Solanky
a4f407f595 Test memory leak on MPS device when generating vector embeddings
Slope threshold of 2.0 determined qualitatively on local Mac device
Minor unused import and clean-up
2023-11-05 03:48:54 -08:00
Debanjum Singh Solanky
ef24485ada Improve Khoj with DB setup instructions in the Django app readme (for now) 2023-11-05 02:04:52 -08:00
Debanjum Singh Solanky
f212cc7174 Arrange remaining text search tests in arrange, act, assert order 2023-11-05 02:04:52 -08:00
Debanjum Singh Solanky
022017dd0f Fix text search tests to test updated indexing log messages 2023-11-05 02:04:52 -08:00
sabaimran
084a8becc5 Fix but to prevent default in chat trigger 2023-11-04 20:13:33 -07:00
Debanjum Singh Solanky
5489e98b9c Do not index org heading entries by default
This is to maintain the previous default behavior
2023-11-04 20:09:25 -07:00
Debanjum Singh Solanky
34b5a86d1d Use SentenceTransformer to disable progress bar when encoding query
The Langchain HuggingFaceEmbeddings wrapper doesn't support disabling
progressbar, not especially for only query but not documents.

This makes the logs noisy with encoding progressbar for each
incremental queries

No features of the Langchain wrapper for SentenceTransformer was
currently being used anyway for now, and we can always switch back to
it if required
2023-11-04 20:09:25 -07:00
Debanjum Singh Solanky
dc9946fc03 Flatten nested loops, improve progress reporting in text_to_jsonl indexer
Flatten the nested loops to improve visibilty into indexing progress

Reduce spurious logs, report the logs at aggregated level and update
the logging description text to improve indexing progress reporting
2023-11-04 20:09:25 -07:00
sabaimran
88eeee3f4b Move try/catch for import one line later 2023-11-04 19:46:47 -07:00
sabaimran
dbaa892665 Flip catching modulenotfound to import error exception 2023-11-04 19:34:10 -07:00
sabaimran
8c3d5a49da Add try/except around image extraction step 2023-11-04 19:27:18 -07:00