Commit graph

272 commits

Author SHA1 Message Date
Debanjum Singh Solanky
a19cbde2d7 Add About page for Khoj to Desktop app. Expose it via system tray
- Pass current khoj version from package.json to about page via
  electron IPC between backend js and frontend page
- Update Khoj information in default About screen as well, in case
  it's exposed anywhere else
2023-11-03 23:59:21 -07:00
Debanjum Singh Solanky
a327294ee9 Rename khoj.js to utils.js in web and desktop client apps 2023-11-03 18:13:37 -07:00
Debanjum Singh Solanky
db57eeaefe Console log a welcome message on loading Desktop client 2023-11-03 05:15:41 -07:00
Debanjum Singh Solanky
4cd76311ad Slow down spinning at end of splash sequence. Make animation bigger 2023-11-03 04:28:17 -07:00
Debanjum Singh Solanky
34661c33a2 Show splash screen on starting desktop app 2023-11-03 03:19:08 -07:00
Debanjum Singh Solanky
126d3f4563 Render each file, folder to index row with icon in desktop app
Make the file, folders to index look less like an editable field
2023-11-03 02:48:42 -07:00
Debanjum Singh Solanky
80ae132cad Update Desktop, Obsidian client color theme to lighter yellow
- Update background color to a different shade of white
- Make primary and primary hover colors less intense and more aligned
  with lantern flame shade
- Add water, leaf, flower color variables
2023-11-03 02:48:42 -07:00
Debanjum Singh Solanky
041074ccd6 Make chat the landing page for the desktop app
Chat, unlike search, doesn't knowledge base indexing setup.
So you can get started with chat much faster.
2023-11-02 20:42:21 -07:00
Debanjum Singh Solanky
9fc6c97139 Use Khoj standard font family, weight in web client settings page 2023-11-02 20:42:21 -07:00
sabaimran
54a387326c
[Multi-User Part 6]: Address small bugs and upstream PR comments ()
- 08654163cb: Add better parsing for XML files
- f3acfac7fb: Add a try/catch around the dateparser in order to avoid internal server errors in app
- 7d43cd62c0: Chunk embeddings generation in order to avoid large memory load
- e02d751eb3: Addresses comments from PR  
- a3f393edb4: Addresses comments from PR  
- 66eb078286: Addresses comments from PR  
- Address various items in https://github.com/khoj-ai/khoj/issues/527
2023-10-31 17:59:53 -07:00
Debanjum
9acc722f7f
[Multi-User Part 4]: Authenticate using API Tokens ()
###  New
- Use API keys to authenticate from Desktop, Obsidian, Emacs clients
- Create API, UI on web app config page to CRUD API Keys
- Create user API keys table and functions to CRUD them in Database

### 🧪 Improve
- Default to better search model, [gte-small](https://huggingface.co/thenlper/gte-small), to improve search quality
- Only load chat model to GPU if enough space, throw error on load failure
- Show encoding progress, truncate headings to max chars supported
- Add instruction to create db in Django DB setup Readme

### ⚙️ Fix
- Fix error handling when configure offline chat via Web UI
- Do not warn in anon mode about Google OAuth env vars not being set
- Fix path to load static files when server started from project root
2023-10-26 12:33:03 -07:00
Debanjum Singh Solanky
8346e1193c Release Khoj version 0.13.0 2023-10-18 03:43:54 -07:00
Debanjum Singh Solanky
53abd1a506 Mark sync completed on desktop client, even when no files to send
Previously Sync spinner on desktop config screen would hang when no
files to send to server & the Sync button had been manually triggered
2023-10-18 01:30:56 -07:00
sabaimran
ba60c869c9 Fix encoding binary files like PDFs for sync from Desktop client
Use readFileSync, Buffer to pass appropriately formatted binary data
2023-10-17 15:08:23 -07:00
Debanjum
ecc6fbfeb2
Push Files to Index from Emacs, Obsidian & Desktop Clients using Multi-Part Forms ()
### Overview
- Add ability to push data to index from the Emacs, Obsidian client
- Switch to standard mechanism of syncing files via HTTP multi-part/form. Previously we were streaming the data as JSON
  - Benefits of new mechanism
    - No manual parsing of files to send or receive on clients or server is required as most have in-built mechanisms to send multi-part/form requests
    - The whole response is not required to be kept in memory to parse content as JSON. As individual files arrive they're automatically pushed to disk to conserve memory if required
    - Binary files don't need to be encoded on client and decoded on server

### Code Details
### Major
- Use multi-part form to receive files to index on server
- Use multi-part form to send files to index on desktop client
- Send files to index on server from the khoj.el emacs client
  - Send content for indexing on server at a regular interval from khoj.el
- Send files to index on server from the khoj obsidian client
- Update tests to test multi-part/form method of pushing files to index

#### Minor
- Put indexer API endpoint under /api path segment
- Explicitly make GET request to /config/data from khoj.el:khoj-server-configure method
- Improve emoji, message on content index updated via logger
- Don't call khoj server on khoj.el load, only once khoj invoked explicitly by user
- Improve indexing of binary files
  - Let fs_syncer pass PDF files directly as binary before indexing
  - Use encoding of each file set in indexer request to read file 
- Add CORS policy to khoj server. Allow requests from khoj apps, obsidian & localhost
- Update indexer API endpoint URL to` index/update` from `indexer/batch`

Resolves  
2023-10-17 06:05:15 -07:00
Debanjum Singh Solanky
5efae1ad55 Update indexer API endpoint query params for force, content type
New URL query params, `force' and `t' match name of query parameter in
existing Khoj API endpoints

Update Desktop, Obsidian and Emacs client to call using these new API
query params. Set `client' query param from each client for telemetry
visibility
2023-10-17 04:58:13 -07:00
Debanjum Singh Solanky
84654ffc5d Update indexer API endpoint URL to index/update from indexer/batch
New URL follows action oriented endpoint naming convention used for
other Khoj API endpoints

Update desktop, obsidian and emacs client to call this new API
endpoint
2023-10-17 04:58:13 -07:00
Debanjum Singh Solanky
d27dc71dfe Use encoding of each file set in indexer request to read file
Get encoding type from multi-part/form-request body for each file
Read text files as utf-8 and pdfs, images as binary
2023-10-17 04:58:12 -07:00
Debanjum Singh Solanky
96c0b21285 Sync desktop app package.json with other Khoj clients metadata
- Make `bump_version.sh' script set version for the Khoj desktop app too
- Sync Khoj desktop app authors, license, description and version with
  the other interfaces and server
- Update description in packages metadata to match project subtitle on Github
2023-10-13 20:43:55 -07:00
sabaimran
80fb56b8a5 Sync deksktop app package version with the other releases 2023-10-13 19:23:00 -07:00
Debanjum Singh Solanky
68018ef397 Use multi-part form to send files to index on desktop client
- Add typing for variables in for loop and other minor formatting clean-up
- Assume utf8 encoding for text files and binary for image, pdf files
2023-10-12 20:58:49 -07:00
Debanjum Singh Solanky
6aa69da3ef Put indexer API endpoint under /api path segment
Update FastAPI app router, desktop app and to use new url path to
batch indexer API endpoint

All api endpoints should exist under /api path segment
2023-10-09 21:35:58 -07:00
sabaimran
4a5ed7f06c
Update Khoj package version for Electron, Desktop app ()
* Address package upgrade for Electron application
* Update package version for Electron desktop application
2023-10-03 12:21:32 -07:00
sabaimran
3f962a55c3
Fix Linux Desktop Application ()
* Use separate functions for adding files and folders to configuration for indexing
* Add a loading bar while data is syncing
* Bump the minor version for the application
2023-10-03 11:43:19 -07:00
sabaimran
38f0df3d53 Remove unused icons from electron app folder 2023-09-26 07:56:29 -07:00
sabaimran
16874e1953 Provide force fallback for regeneration 2023-09-12 16:35:07 -07:00
sabaimran
76562f4250
Add front-end Electron application for Khoj local file syncing ()
* Initial version - setup a file-push architecture for generating embeddings with Khoj
* Use state.host and state.port for configuring the URL for the indexer
* Fix parsing of PDF files
* Read markdown files from streamed data and update unit tests
* On application startup, load in embeddings from configurations files, rather than regenerating the corpus based on file system
* Init: refactor indexer/batch endpoint to support a generic file ingestion format
* Add features to better support indexing from files sent by the desktop client
* Initial commit with Electron application
- Adds electron app
* Add import for pymupdf, remove import for pypdf
* Allow user to configure khoj host URL
* Remove search type configuration from index.html
* Use v1 path for current indexer routes
2023-09-06 12:04:18 -07:00
Debanjum Singh Solanky
25a749ca1d Use the src/ layout to fix packaging Khoj for PyPi
- Why
  The khoj pypi packages should be installed in `khoj' directory.
  Previously it was being installed into `src' directory, which is a
  generic top level directory name that is discouraged from being used

- Changes
 - move src/* to src/khoj/*
 - update `setup.py' to `find_packages' in `src' instead of project root
 - rename imports to form `from khoj.*' in complete project
 - update `constants.web_directory' path to use `khoj' directory
 - rename root logger to `khoj' in `main.py'
 - fix image_search tests to use the newly rename `khoj' logger
 - update config, docs, workflows to reference new path `src/khoj'
2023-02-14 15:19:06 -06:00
Debanjum Singh Solanky
8f57a62675 Remove unused imports. Fix typing and indentation
- Typing issues discovered using `mypy'. Fixed manually
- Unused imports discovered and fixed using `autoflake'
- Fix indentation in `org_to_jsonl' manually
2022-09-14 04:56:52 +03:00
Debanjum Singh Solanky
588f598949 Pass empty list of `input_files' to FileBrowser on first run
- Default config has `input_files' set to None
- This was being passed to `FileBrowser' on Initialization
- But `FileBrowser' expects `content_files' of list type, not None
- This resulted in an unexpected NoneType failure
2022-09-09 07:26:40 +03:00
Debanjum Singh Solanky
5e6625ac68 Fix file browser to not add empty line when no file/dir selected
- When no file selected in file browser an empty line/entry gets added
  to input entries list
- Bug got introduced due to insufficient update on change to add
  instead of insert
- Update is_none_or_empty helper method to also check for empty string
2022-08-21 02:03:28 +03:00
Debanjum Singh Solanky
8098b8c3a8 Bring Configure Window to Top when Opened from System Tray
- Previously the window could get hidden behind other app windows when
  user clicked configure from the system tray
2022-08-20 23:38:43 +03:00
Debanjum Singh Solanky
1c122a8a91 Place window near top so buttons are not hidden by OS bottom bar 2022-08-20 22:38:06 +03:00
Debanjum Singh Solanky
dfe2546c04 Set Khoj Icon on Main Desktop Window 2022-08-20 20:36:15 +03:00
Debanjum Singh Solanky
7c4417126c Append files, directories selected by user to config in Desktop GUI
- Allows adding multiple image directories via GUI
- Allow adding multiple files in different directories via GUI
- Previously users couldn't add multiple directories via GUI
  They'd have to manually append to input field if multiple files, directories
- To clear/overwrite is much easier.
  The user can just select text to delete in input area
2022-08-19 19:16:10 +03:00
Debanjum Singh Solanky
0079c13bf7 Set input-directories in config for image search type on Desktop GUI
- Issue
  Fix configuring image search from Desktop GUI. It was broken before.
  The Desktop GUI was updating input-files field under content-type > image.
  This field is not used for image search. So image search couldn't be
  configured from the Desktop GUI

- Fix
  - Set input-directories when field of search type image is set from GUI
  - Otherwise set input-files field in config
2022-08-18 18:29:55 +03:00
Debanjum Singh Solanky
9bc4fd539e Set Web Interface URL from loaded state in Desktop GUIs. Not hard-coded 2022-08-16 00:37:16 +03:00
Debanjum Singh Solanky
7f479b0104 Improve Displaying Error to User on Khoj window in Desktop GUI
- Show a helpful error message in the GUI to the user, instead of the
  crashing if loading config fails, for e.g if file wasn't found
- Collate GUI errors into an ErrorType enum class
- Remove previous error messages before showing the new one
2022-08-16 00:37:16 +03:00
Debanjum Singh Solanky
873bb9dd97 Do not force the Khoj window to always be on top. It's needlessly annoying 2022-08-16 00:37:16 +03:00
Debanjum Singh Solanky
67ab40bb01 Regenerate embeddings everytime user clicks configure in Desktop GUI
Previously if the embeddings were already there only the khoj.yml
config file would get updated. The embeddings would remain old.

1. This results in a stale app state where the config doesn't
   match the embeddings

2. Currently the user cannot update their config from the config
   screen. They'd have to use a combination of config screen and web
   interface>regenerate button to trigger it or delete their ~/.khoj dir

This commit should resolve the above issues
2022-08-16 00:37:16 +03:00
Debanjum Singh Solanky
a0759dd923 Convert Configure Screen into the Main Application Window
- What
  - Convert the config screen into the main application window
    with configuration as just one of the functionality it provides
  - Rename config screen to main window to match new designation

- Why
  - System Tray isn't available everywhere (e.g Linux)
  - This requires moving functionality into a normal window for cross-compat
2022-08-13 02:05:52 +03:00
Debanjum Singh Solanky
c2815c5d09 Open Search from Khoj Configure Screen
- Start evolving configure screen away from just being a configure screen
  - Update Window Title to just say Khoj
- Allow Opening Web Interface to Search from Khoj configure screen
- Rename "Start" Button to more accurate "Configure"
- Disable Search button on first run and while configuring app
2022-08-13 00:43:49 +03:00
Debanjum Singh Solanky
28a91ad1fd Deep copy the default_config constant to prevent it being overwritten
- Issue
  - In the previous form, updates to self.current_config would update
    default_config as python does a shallow copy
  - So self.current_config is just referencing the values of default_config
  - Hence updates to current_config updates the default_config values too
  - This is not what we want

- Fix
  - Deep copy the default_config values. Now updates to
    self.current_config wouldn't affect the default_config
2022-08-12 23:54:16 +03:00
Debanjum Singh Solanky
62ac41ce3b Reload settings in a separate thread to not freeze Config Screen
- Generating embeddings takes time
- If user enables a content type and clicks start.
  The app starts to generate embeddings when loading the new settings
- Run this function in a separate thread to keep config screen responsive
- But disable start button to prevent re-entrant threads
- Also show a minimal visual indication that the app is saving state
2022-08-12 23:34:00 +03:00
Debanjum Singh Solanky
927547d0af Update Title of Configure Screen to follow "<Screen> - App" pattern 2022-08-12 22:53:10 +03:00
Debanjum Singh Solanky
43301d488a Increase Width of Configure Screen 2022-08-12 18:34:47 +03:00
Debanjum Singh Solanky
9baea9c9fd Let Input Fields Wrap. Adjust Height based on Text in Field
- Convert Input Fields into PlainTextEdit
- Display Each Selected File on a Separate Line in Field
- Set Height of FileBrowser Input Field based on Number of Lines/Files
2022-08-12 18:33:56 +03:00
Debanjum Singh Solanky
b7b96110e9 Rename FileBrowser Button Text to "Select" instead of "Add" 2022-08-12 17:08:40 +03:00
Debanjum Singh Solanky
a1c58a9470 Create, Use a Labelled Text Field for the Conversation Input Field
- This fixes the field expanding when configure screen is expanded
- Allows for reusability of the labelled text field
- Simplifies the logic to save settings for conversation processor
2022-08-12 16:59:15 +03:00
Debanjum Singh Solanky
110e3df0b7 Set default config in the constant module. Use from there to configure app
- Avoid having to pass the khoj_sample.yml data file into pip, native apps
- Packaging data files into python packages is annoying.
  - There's `MANIFEST.in`, `data_files` and `package_data` in setup.py
  - Bdist, wheel, generated source tarball use different set of these fields
    and put the data files in different locations
  - Rather just code the default config into a constant. Avoid
    pointless file reads as well this way
2022-08-12 02:18:46 +03:00
Debanjum Singh Solanky
44fe70513a Handle situation where default config directory or file does not exist
- Include khoj_sample.yml in pip package to load default config from
- Create khoj config directory if it doesn't exist
- Load config from khoj_sample.yml if khoj.yml config doesn't exist
2022-08-12 01:17:34 +03:00
Debanjum Singh Solanky
41520e1608 Improve Docstring for Configure Screen and System Tray class, funcs 2022-08-11 23:36:02 +03:00
Debanjum Singh Solanky
b74ca1def6 Wrap error message instead of expanding screen to show message 2022-08-11 20:51:56 +03:00
Debanjum Singh Solanky
2646fa825b Get Files from File input line to match user expectation
- If a user manually edits the input file lines, clicking start should
  use that. Currently it just looks at the files selected last via file
  browser
- We want to allow users to manually enter file paths in field. Which
  is why the field hasn't been set to read-only
2022-08-11 20:48:45 +03:00
Debanjum Singh Solanky
dad9133598 Split save_settings method into smaller methods for modularization 2022-08-11 20:00:52 +03:00
Debanjum Singh Solanky
56ba91fec8 Remove unused methods in file browser widget. Improve name of existing 2022-08-11 19:46:09 +03:00
Debanjum Singh Solanky
fd4e41495c Use appropriate label for directory input types to minimize confusion 2022-08-11 19:45:19 +03:00
Debanjum Singh Solanky
c1e1466fb1 Validate new config before write. Show error if new config invalid 2022-08-11 19:18:22 +03:00
Debanjum Singh Solanky
1ff049599f Show current config on config screen. Load default config if config unset
- Track current (saved/loaded) config separate from the new config (to
  be written) when user clicks Start

- Fallback to using default config when no config for the specific
  content type or processor is specified in khoj.yml
  - Earlier were only loading default config on first run, not after

- Create Child CheckBox, LineEdit classes for Processor Widgets
  - Create ProcessorType, similar to SearchType
  - Track ProcessorType the widgets are associated with
  - Simplify update, save, load of config based on type
2022-08-11 19:11:25 +03:00
Debanjum Singh Solanky
678fb6a3c7 Add Settings Panel for Conversation Settings to Config Screen 2022-08-11 04:52:40 +03:00
Debanjum Singh Solanky
c1fcf44405 Initialize Settings on Config Screen with Existing Settings from File 2022-08-11 04:51:33 +03:00
Debanjum Singh Solanky
3cec6229ad Hot swap backend config via config screen start button click
- Update configuration to use by the backend, while app is running
- Trigger after user hits start button with their config.
  The config gets written to khoj.yml file first, then the updated
  config is loaded onto memory
2022-08-11 00:32:11 +03:00
Debanjum Singh Solanky
cc6ef0f450 Save configure screen settings to app config yaml on clicking Start 2022-08-10 23:10:39 +03:00
Debanjum Singh Solanky
dae65c5b6b Create child class of Qt CheckBox to track search type it enables/disables 2022-08-10 22:44:37 +03:00
Debanjum Singh Solanky
f42f54019b Type parent_layout passed as arguments to ConfigureScreen methods 2022-08-10 22:43:20 +03:00
Debanjum Singh Solanky
f63f11186f Pass config file for app to configure screen 2022-08-10 22:42:32 +03:00
Debanjum Singh Solanky
daef276fd1 Add files for each search type. Extract config on clicking start
- Only allow adding files with appropriate file extension for each search type
  - e.g .org for org-mode search, directory for image search

- Extract file paths added to config and enablement state of each search type
  - This extracted state will be used to populate the khoj.yml config file
2022-08-10 03:27:22 +03:00
Debanjum Singh Solanky
d74134e6cc Reuse Single Method to Create Setting Panels for each Search Type 2022-08-09 23:50:43 +03:00
Debanjum Singh Solanky
509d52e2cd Toggle Editability instead of Visibility of Per Search Type Settings
- Simplifies the configure screen layout and allows it to be of constant width

- It was buggy, the configure screen would dynamically expand but not
  restore back to original size on disabling search type after enable
2022-08-09 23:34:54 +03:00
Debanjum Singh Solanky
3c788f1d29 Rename configure window to more generic configure screen 2022-08-09 22:44:05 +03:00
Debanjum Singh Solanky
c50ab7c3ad Split config settings GUI into functions. Convert Config Window to Dialog 2022-08-09 22:36:41 +03:00
Debanjum Singh Solanky
664713b24e Extract Qt GUI code from main.py into separate interface/desktop dir 2022-08-09 22:12:29 +03:00