Commit graph

3998 commits

Author SHA1 Message Date
Debanjum Singh Solanky
4839f2901a Open external links in Desktop app with default app for url on OS
- Open external links using the default link handler registered on OS
  for the link type, e.g http:// -> firefox, mailto: thunderbird etc
- Confirm before opening non-http URL using an external app
2024-02-25 13:21:52 +05:30
Debanjum
170bce2c02
Fix, Improve rendering images in Obsidian, Desktop, Web clients (#659)
- Improve render of inferred query in image chat messages in Web, Desktop apps
- Add inferred queries to image chat responses in Obsidian client
- Fix rendering images from Khoj response in Obsidian client
2024-02-25 00:56:26 +05:30
Debanjum Singh Solanky
f84606325c Improve render of inferred query in image chat messages in Web, Desktop apps 2024-02-25 00:47:06 +05:30
Debanjum Singh Solanky
a2e53d5e41 Add inferred queries to image chat responses in Obsidian client 2024-02-25 00:24:58 +05:30
Debanjum Singh Solanky
9b61f0b5f7 Fix rendering images from Khoj response in Obsidian client 2024-02-25 00:11:11 +05:30
sabaimran
b9d0533d92
Misc. fixes to prompting, admin, and others (#658)
* Simplify and clarify prompt for selecting toolset dynamically

* Add error handling around call to OLOSTEP api

* Fix conversation admin page

* Skip adding none or empty entries in the chunking method
2024-02-24 10:25:42 -08:00
Debanjum Singh Solanky
0e0e751ef7 Improve docstring of entrypoint function to the emacs client 2024-02-24 21:09:41 +05:30
Debanjum
8855529637
Improve Syncing Obsidian Vault, Invalidate Static Assets in Browser Cache in Web Client (#657)
- Improve
  - Only send files modified since their last sync for indexing on server from the Obsidian client
- Fix 
  - Invalidate static asset browser cache in Web client when Khoj version changes
2024-02-24 20:20:30 +05:30
Debanjum Singh Solanky
a46f70c4b0 Remove deprecated lastSyncedFiles settings field from Obsidian client 2024-02-24 20:18:22 +05:30
Debanjum Singh Solanky
03a6b491b2 Warn when can't identify mimeType of files in Desktop, Obsidian clients 2024-02-24 19:59:03 +05:30
Debanjum Singh Solanky
3675ab4864 Only sync modified files from the Obsidian client
Previously we'd send all files in vault and let the server
deduplicate.

This changes takes inspiration from the desktop app, and only pushes
files which were modified after their previous sync with the server.

This should reduce the processing load on the server
2024-02-24 07:48:40 +05:30
Debanjum Singh Solanky
ddfbf31bc8 Append version query param to web asset URLs to bypass browser cache
Ensure latest assets are loaded when khoj version is updated
2024-02-24 06:49:25 +05:30
sabaimran
42773e808c
Retrieve, create, and save conversations differently for ClientApplications (#656)
* Retrieve, create, and save conversations differently if they're coming from a client application

- Not all of our client apps will necessarily maintain state over the conversation IDs available to a user. For some (single-threaded conversations), it should just use a single conversation. Fix the code to do so

* Simplify conversation retrieval logic

* Keep 0 padding below chat response

* Add order_by sorting to retrieving the conversation without id
2024-02-23 11:32:00 -08:00
Debanjum
9afb2a14ef
Fix and Improve Chat UI in Web, Desktop apps (#655)
### Improvements to Chat UI on Web, Desktop apps
- Improve styling of chat session side panel
- Improve styling of chat message bubble in Desktop, Web app
- Add frosted, minimal chat UI to background of Login screen
- Improve PWA install experience of Khoj

### Fixes to Chat UI on Web, Desktop apps
- Fix creating new chat sessions from the Desktop app
- Only show 3 starter questions even when consecutive chat sessions created

### Other Improvements
- Update Khoj cloud trial period to a fortnight instead of a week
- Document using venv to handle dependency conflict on khoj pip install

Resolves #276
2024-02-23 19:27:02 +05:30
Debanjum Singh Solanky
c70ca78cdc Improve PWA install experience for Khoj on Desktop, Mobile
- Resolve PWA issues thrown by Chrome/Edge
  - Add screenshot samples showcasing remember, browse and draw features
    - This can provide a richer app store like experience when
      installing Khoj PWA on Mobile or Desktop
    - Add wide and narrow screenshots to show Mobile vs Desktop UX
  - Add higher resolution favicon for PWA
- Use single web manifest instead of separate ones for Chat, Search
- Update manifest description with more details about Khoj features
2024-02-23 18:59:52 +05:30
Debanjum Singh Solanky
e10b260988 Update web login screen to show frosted minimal chat UI in background 2024-02-23 18:59:52 +05:30
Debanjum Singh Solanky
1b0318564e Log when conversation turn is saved to DB 2024-02-23 18:59:52 +05:30
Debanjum Singh Solanky
4c39960917 Make number of conversation starters to get from DB configurable 2024-02-23 18:59:52 +05:30
Debanjum Singh Solanky
50617594fd Only show 3 starter questions even when consecutive chat sessions created
Reset starter question suggestions before appending in web, desktop app

Otherwise previously it'd keep adding to existing starter question
suggestions on each new session creation if multiple consecutive new
chat sessions created.

This would result in more than the 3 expected starter questions being
displayed at a time
2024-02-23 18:59:52 +05:30
Debanjum Singh Solanky
102f5c3f53 Improve styling of chat session side panel
- Make collapse, expand toggle arrow point in the direction the action
  will expand the side panel in
- Make the collapsed side panel reduce to a 1px sliver
2024-02-23 18:59:52 +05:30
Debanjum Singh Solanky
6283d9fe83 Update Khoj cloud trial period to a fortnight instead of a week
- Improve rate limit error message wording
- Make the "too many requests" error message more robust. Should throw
  that exception fix self.request >= self.subscribed_requests because
  upgrading wouldn't fix this rate limiting
2024-02-23 18:33:56 +05:30
Debanjum Singh Solanky
05c1903784 Fix creating new chat sessions from the Desktop app
Code wasn't passing the authorization header in the POST request to
create new chat session
2024-02-23 18:33:56 +05:30
Debanjum Singh Solanky
8a219b6e9c Improve styling of chat message bubble in Desktop, Web app
- Respect newline with pre-line but not for bullets to improve
  formatting of responses by Khoj
- Respect bold font by loading tajawal font with other weights
- Reduce bottom margin in chat message bubble, its taking too much space
2024-02-23 18:33:56 +05:30
sabaimran
b4902090e7
Misc. chat and application improvements (#652)
* Document original query when subqueries can't be generated
* Only add messages to the chat message log if it's non-empty
* When changing the search model, alert the user that all underlying data will be deleted
* Adding more clarification to the prompt input for username, location
* Check if has_more is in the notion results before getting next_cursor
* Update prompt template for user name/location, update confirmation message when changing search model
2024-02-22 19:09:22 -08:00
Debanjum Singh Solanky
57dce91c91 Document using venv to handle dependency conflict on khoj pip install
Resolves #276
2024-02-23 02:07:08 +05:30
Debanjum Singh Solanky
7271164256 Set chat session title to textContent of the chat session HTML element
We don't expect/want the user to use HTML titles for chat session
2024-02-23 02:07:08 +05:30
sabaimran
f8ec6b4464 Remove backslash for default route in api_chat 2024-02-20 20:09:44 -08:00
sabaimran
699545366b Set gunicorn config to use 4 workers 2024-02-20 15:06:20 -08:00
sabaimran
b1c86fee3b Release Khoj version 1.6.0 2024-02-20 14:12:24 -08:00
sabaimran
45c5a2598d Temp - change gunicorn config to use a single worker 2024-02-20 13:56:51 -08:00
sabaimran
44f8f20ea7
Miscellaneous bugs and fixes for chat sessions (#646)
* Display given_name field only if it is not None

* Add default slugs in the migration script

* Ensure that updated_at is saved appropriately, make sure most recent chat is returned for default history

* Remove the bin button from the chat interface, given deletion is handled in the drop-down menus

* Refresh the side panel when a new chat is created

* Improveme tool retrieval prompt, don't let /online fail, and improve parsing of extract questions

* Fix ending chat response by offline chat on hitting a stop phrase

Previously the whole phrase wouldn't be in the same response chunk, so
chat response wouldn't stop on hitting a stop phrase

Now use a queue to keep track of last 3 chunks, and to stop responding
when hit a stop phrase

* Make chat on Obsidian backward compatible post chat session API updates

- Make chat on Obsidian get chat history from
  `responseJson.response.chat' when available (i.e when using new api)
- Else fallback to loading chat history from
  responseJson.response (i.e when using old api)

* Fix detecting success of indexing update in khoj.el

When khoj.el attempts to index on a Khoj server served behind an https
endpoint, the success reponse status contains plist with certs. This
doesn't mean the update failed.

Look for :errors key in status instead to determine if indexing API
call failed. This fixes detecting indexing API call success on the
Khoj Emacs client, even for Khoj servers running behind SSL/HTTPS

* Fix the mechanism for populating notes references in the conversation primer for both offline and online chat

* Return conversation.default when empty list for dynamic prompt selection, send all cmds in telemetry

* Fix making chat on Obsidian backward compatible post chat session API updates

New API always has conversation_id set, not `chat' which can be unset
when chat session is empty.

So use conversation_id to decide whether to get chat logs from
`responseJson.response.chat' or `responseJson.response' instead

---------

Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
2024-02-20 13:55:35 -08:00
sabaimran
138f5223bd
Fix process for generating embeddings for Notion entries (#648)
* Fix process for generating embeddings for Notion entries
* If no title field found, just log a warning and set the title to
2024-02-20 13:46:56 -08:00
Debanjum
43013c4fd4
Make Production Dependencies for Khoj Cloud Optional to Install (#647)
- Remove unused git dependency from Docker images
- Move python packages used for test into dev dependency group
- Only enable API token, Whatsapp cards on Web UI when Stripe, Twilio setup
- Move production dependencies to prod python packages group
- Fix docs links in Khoj welcome chat message
2024-02-16 17:42:23 +05:30
Debanjum Singh Solanky
4696577636 Upgrade python dependencies 2024-02-16 17:41:09 +05:30
Debanjum Singh Solanky
4007c871ae Remove unused git dependency from Docker images 2024-02-16 17:41:09 +05:30
Debanjum Singh Solanky
e21a8530f3 Move used python packages for test into dev dependency group
The test dependency group was being used independently
2024-02-16 17:41:09 +05:30
Debanjum Singh Solanky
4722da9642 Only enable API token, Whatsapp cards on Web UI when Stripe, Twilio setup 2024-02-16 17:41:09 +05:30
Debanjum Singh Solanky
cf4a524988 Move production dependencies to prod python packages group
This will reduce khoj dependencies to install for self-hosting users

- Move auth production dependencies to prod python packages group
  - Only enable authentication API router if not in anonymous mode
  - Improve error with requirements to enable authentication when not in
    anonymous mode
2024-02-16 17:41:08 +05:30
Debanjum Singh Solanky
d7dbb715ef Fix docs links in khoj introductory chat message 2024-02-13 22:38:03 +05:30
sabaimran
32ec54172e
Add additional personalization in Chat via Location, Username (#644)
* Add location metadata to chat history
* Add support for custom configuration of the user name
* Add region, country, city in the desktop app's URL for context in chat
* Update prompts to specify user location, rather than just location.
* Add location data to Obsidian chat query
* Use first word for first name, last word for last name when setting profile name
2024-02-13 17:05:13 +05:30
sabaimran
a3eb17b7d4
Have Khoj dynamically select conversation command(s) in chat (#641)
* Have Khoj dynamically select which conversation command(s) are to be used in the chat flow
- Intercept the commands if in default mode, and have Khoj dynamically guess which tools would be the most relevant for answering the user's query
* Remove conditional for default to enter online search mode
* Add multiple-tool examples in the prompt, make prompt for tools more specific to info collection
2024-02-11 17:11:32 +05:30
sabaimran
69344a6aa6
Add support for multiple chat sessions in the desktop application (#639)
* Add chat sessions to the desktop application
* Increase width of the main chat body to 90vw
* Update the version of electron
* Render the default message if chat history fails to load
* Merge conversation migrations and fix slug setting
* Update the welcome message, use the hostURL, and update background color for chat actions
* Only update the window's web contents if the page is config
2024-02-11 16:05:28 +05:30
sabaimran
1412ed6a00
Support multiple chat sessions within the web UI (#638)
* Enable support for multiple chat sessions within the web client

- Allow users to create multiple chat sessions and manage them
- Give chat session slugs based on the most recent message
- Update web UI to have a collapsible menu with active chats
- Move chat routes into a separate file

* Make the collapsible side panel more graceful, improve some styling elements of the new layout

* Support modification of the conversation title

- Add a new field to the conversation object
- Update UI to add a threedotmenu to each conversation

* Get the default conversation if a matching one is not found by id
2024-02-11 15:48:28 +05:30
sabaimran
208ccc83ec Fix version of gpt4all to 2.1.0 as it's not backwards compatible 2024-02-10 09:32:04 +05:30
Debanjum Singh Solanky
70f74cde68 Fix timestamps to separate each logline. Info log response start time 2024-02-07 20:45:16 +05:30
Debanjum Singh Solanky
667b975400 Free space on Github workflow VM to build Khoj docker images 2024-02-06 23:37:51 +05:30
Debanjum Singh Solanky
8e5db72140 Release Khoj version 1.5.1 2024-02-06 23:09:33 +05:30
Debanjum
fc1b8f6fb6
Fix Khoj Obsidian plugin on Obsidian Mobile (#635)
- Removed node-fetch dependency to work on mobile. 
- Fix CORS issue for Khoj (streaming) chat on Obsidian mobile
- Verified Khoj plugin, search, chat work on Obsidian mobile.

## Details
### Major
- Allow calls to Khoj server from Obsidian mobile app to fix CORS issue
- Chat stream using default `fetch' not `node-fetch' in obsidian plugin

### Minor
- Load chat history after other elements in chat modal on Obsidian are rendered
- Scroll to bottom of chat modal on Obsidian across mobile & desktop
2024-02-06 22:03:51 +05:30
Debanjum
c6fa98ce3e
Make Offline Chat Date Aware (#636)
- Provide more context and instructions to offline chat on Khoj
- Upgrade offilne chat quality tests to support more use-cases

### Details
- Improve offline chat system prompt to think step by step
- Make offline chat model current date aware. Improve system prompts
- Fix actor, director tests using freeze time by ignoring transformers package
2024-02-06 21:32:34 +05:30
Debanjum Singh Solanky
fd238ff792 Load chat history after other elements in chat modal on Obsidian rendered
This reduces laggy feeling due to latency of loading chat history from
server
2024-02-06 21:25:43 +05:30