### Major
- Expose Billing via Stripe on Khoj Web app for Khoj Cloud subscription
- Expose card on web app config page to manage subscription to Khoj cloud
- Create API webhook, endpoints for subscription payments using Stripe
- Put Computer files to index into Card under Content section
- Show file type icons for each indexed file in config card of web app
- Enable deleting all indexed desktop files from Khoj via Desktop app
- Create config page on web app to manage computer files indexed by Khoj
- Track data source (computer, github, notion) of each entry
- Update content by source via API. Make web client use this API for config
- Store the data source of each entry in database
### Cleanup
- Set content enabled status on update via config buttons on web app
- Delete deprecated content config pages for local files from web client
- Rename Sync button, Force Sync toggle to Save, Save All buttons
### Fixes
- Prevent Desktop app triggering multiple simultaneous syncs to server
- Upgrade langchain version since adding support for OCR-ing PDFs
- Bubble up content indexing errors to notify user on client apps
- Add fields to mark users as subscribed to a specific plan and
subscription renewal date in DB
- Add ability to unsubscribe a user using their email address
- Expose webhook for stripe to callback confirming payment
Previously hitting configure or disable wouldn't update the state of
the content cards. It needed page refresh to see if the content was
synced correctly.
Now cards automatically get set to new state on hitting disable button
on card or global configure buttons
Lock syncing to server if a sync is already in progress.
While the sync save button gets disabled while sync is in progress,
the background sync job can still trigger a sync in parallel. This
sync lock prevents that
Remove the table of all files indexed by Khoj. This seems overkill and
doesn't match the UI semantics of the other data sources like Github,
Notion.
Create instead a data source card for computer files with the same
update, disable semantics of the Github and Notion data source cards
Users can disable each data source from its card on the main config page.
They can see/delete individual files indexed from the computer data source
once they click into the computer files data source card on the config page
This will be useful for updating, deleting entries by their data
source. Data source can be one of Computer, Github or Notion for now
Store each file/entries source in database
Major
- Ensure search results logic consistent across migration to DB, multi-user
- Manually verified search results for sample queries look the same across migration
- Flatten indexing code for better indexing progress tracking and code readability
Minor
- a4f407f Test memory leak on MPS device when generating vector embeddings
- ef24485 Improve Khoj with DB setup instructions in the Django app readme (for now)
- f212cc7 Arrange remaining text search tests in arrange, act, assert order
- 022017d Fix text search tests to test updated indexing log messages
The Langchain HuggingFaceEmbeddings wrapper doesn't support disabling
progressbar, not especially for only query but not documents.
This makes the logs noisy with encoding progressbar for each
incremental queries
No features of the Langchain wrapper for SentenceTransformer was
currently being used anyway for now, and we can always switch back to
it if required
Flatten the nested loops to improve visibilty into indexing progress
Reduce spurious logs, report the logs at aggregated level and update
the logging description text to improve indexing progress reporting
