Commit graph

3198 commits

Author SHA1 Message Date
sabaimran
c9665fb20b Convert conversationId int to string before making api request to bulk update file filters 2024-09-16 15:45:23 -07:00
sabaimran
3466f04992 Fix handling for new conversation in agents page 2024-09-16 15:04:49 -07:00
sabaimran
ece2ec2d90
Add a unique_id field for identifiying conversations (#914)
* Add a unique_id field to the conversation object

- This helps us keep track of the unique identity of the conversation without expose the internal id
- Create three staged migrations in order to first add the field, then add unique values to pre-fill, and then set the unique constraint. Without this, it tries to initialize all the existing conversations with the same ID.

* Parse and utilize the unique_id field in the query parameters of the front-end view

- Handle the unique_id field when creating a new conversation from the home page
- Parse the id field with a lightweight parameter called v in the chat page
- Share page should not be affected, as it uses the public slug

* Fix suggested card category
2024-09-16 12:19:16 -07:00
sabaimran
e6bc7a2ba2 Fix links to log in email templates 2024-09-15 19:14:19 -07:00
Debanjum Singh Solanky
79980feb7b Release Khoj version 1.23.2 2024-09-15 03:07:26 -07:00
Debanjum Singh Solanky
575ff103cf Frame chat response error on web app in a more conversational form
Also indicate hitting dislike on the message should be enough to
convey the issue to the developers.
2024-09-15 03:00:49 -07:00
Debanjum Singh Solanky
893ae60a6a Improve handling of harmful categorized responses by Gemini
Previously Khoj would stop in the middle of response generation when
the safety filters got triggered at default thresholds. This was
confusing as it felt like a service error, not expected behavior.

Going forward Khoj will
- Only block responding to high confidence harmful content detected by
  Gemini's safety filters instead of using the default safety settings
- Show an explanatory, conversational response (w/ harm category)
  when response is terminated due to Gemini's safety filters
2024-09-15 02:17:54 -07:00
sabaimran
ec1f87a896 Release Khoj version 1.23.1 2024-09-12 22:46:39 -07:00
sabaimran
2a4416d223 Use prefetch_related for the openai_config when retrieving all chatmodeloptions async 2024-09-12 22:45:43 -07:00
sabaimran
253ca92203 Release Khoj version 1.23.0 2024-09-12 20:25:29 -07:00
Debanjum Singh Solanky
178b78f87b Show debug log, not warning when use default tokenizer for context stuffing 2024-09-12 20:21:01 -07:00
Debanjum Singh Solanky
75d3b34452 Extract image generation code into new image processor for modularity 2024-09-12 20:01:32 -07:00
Debanjum Singh Solanky
84051d7d89 Make generate better image prompt chat actor add composition details 2024-09-12 19:58:57 -07:00
Debanjum Singh Solanky
ed12f45a26 Generate vivid images with DALLE-3
It's apparently the default setting in chatgpt app according to the
openai cookbook at https://cookbook.openai.com/articles/what_is_new_with_dalle_3#examples-and-prompts
2024-09-12 19:58:57 -07:00
Debanjum Singh Solanky
1b82aea753 Support using image generation models like Flux via Replicate
Enables using any image generation model on Replicate's Predictions
API endpoints.

The server admin just needs to add text-to-image model on the
server/admin panel in organization/model_name format and input their
Replicate API key with it

Create db migration (including merge)
2024-09-12 19:58:56 -07:00
Brian Kanya
1d512b4986
Use environment variable to set sender email of auth link emails (#907)
Set sender email using `RESEND_EMAIL` environment variable for magic link sent via Resend API for authentication . It was previously hard-coded. This prevented hosting Khoj on other domains. 

Resolves #908
2024-09-12 18:48:11 -07:00
Debanjum Singh Solanky
0685a79748 Remove any markdown json codeblock in chat actors expecting json responses
Strip any json md codeblock wrapper if exists before processing
response by output mode, extract questions chat actor. This is similar
to what is already being done by other chat actors

Useful for succesfully interpreting json output in chat actors when
using non (json) schema enforceable models like o1 and gemma-2

Use conversation helper function to centralize the json md codeblock
removal code
2024-09-12 18:26:15 -07:00
Debanjum Singh Solanky
6e660d11c9 Override block display styling of links by Katex in chat messages
This happens sometimes when LLM respons contains [\[1\]] kind of links
as reference. Both markdown-it and katex apply styling.

Katex's span uses display: block which makes the rendering of these
references take up a whole line by themselves.

Override block styling of spans within an `a' element to prevent such
chat message styling issues
2024-09-12 18:22:46 -07:00
Debanjum Singh Solanky
272eae5d66 Add support for the newly released OpenAI O1 model series for preview
The O1 series doesn't seem to support streaming, stop words or
temperature, response_format currently.
2024-09-12 18:22:46 -07:00
Alexander Matyasko
9570933506
Support Google's Gemini model series (#902)
* Add functions to chat with Google's gemini model series
  * Gracefully close thread when there's an exception in the gemini llm thread
* Use enums for verifying the chat model option type
* Add a migration to add the gemini chat model type to the db model
* Fix chat model selection verification and math prompt tuning
* Fix extract questions method with gemini. Enforce json response in extract questions.
* Add standard stop sequence for Gemini chat response generation

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
2024-09-12 18:17:55 -07:00
Debanjum Singh Solanky
42b727e926 Revert additional logging enabled to debug automation failures in prod
Additional logging was enabled to debug automation failures in
production since migration chat API to use POST request method (from
earlier GET).

Redirect from http to https was default to use GET instead of POST
method to call /api/chat on redirect. This has been resolved now
2024-09-12 17:56:54 -07:00
sabaimran
14a495cbb5 Release Khoj version 1.22.3 2024-09-12 12:39:04 -07:00
sabaimran
91cee2eaa8 Handle redirects when scheduling chats from automations 2024-09-12 11:36:47 -07:00
sabaimran
4555969d38 Add additional log lines 2024-09-12 10:50:36 -07:00
Debanjum Singh Solanky
2cc4a0769e Release Khoj version 1.22.2 2024-09-11 18:39:24 -07:00
Debanjum Singh Solanky
7f186be742 Fix json payload passed by automations to the new POST chat API 2024-09-11 18:35:31 -07:00
sabaimran
5038d15574 Route to config_page, not to deprecated notion_config_page, on notion callback API 2024-09-11 18:30:23 -07:00
Debanjum Singh Solanky
b61d825cbc Sanitize user attached image in chat message input pane of web app 2024-09-11 18:02:33 -07:00
Debanjum Singh Solanky
de60ad7da6 Update automations to call new POST chat API endpoint 2024-09-11 17:28:40 -07:00
Debanjum Singh Solanky
055ead550c Update desktop shortcut, web app factchecker to use new POST chat API 2024-09-11 17:28:32 -07:00
Debanjum Singh Solanky
3f51af9a96 Keep the GET chat API endpoint for a bit before deprecating it
This is to avoid breaking non-updated Khoj clients
2024-09-11 16:50:10 -07:00
Debanjum Singh Solanky
03befc9b12 Use consistent user attached image placeholder text for chat actors
Get information sources and get output mode don't actually see the
images. They just get placeholder text to indicate that the user
attached an image to their message for context
2024-09-11 16:16:55 -07:00
Debanjum Singh Solanky
04363a504c Prompt Whisper to know "Khoj" term for speech to text transcription 2024-09-11 16:16:55 -07:00
Debanjum Singh Solanky
3dcc8695b2 Improve vertical alignment of lists in chat messages on web app
- Make train of thought icons to be top aligned, next to the
  their intermediate step heading
- Add margin bottom to ordered, unordered lists in chat message,
  similar to how it is already added for paragraphs
2024-09-11 16:16:55 -07:00
Debanjum Singh Solanky
179357b28a Default to gpt-4o-mini as online chat model 2024-09-11 16:16:55 -07:00
sabaimran
ae74c6ca55 Release Khoj version 1.22.1 2024-09-11 13:03:53 -07:00
sabaimran
cd5db277f3 Fix sync to async issue when getting all valid vision configs 2024-09-11 12:57:54 -07:00
sabaimran
9b12290c17 Release Khoj version 1.22.0 2024-09-11 11:21:02 -07:00
sabaimran
2932d305b0 Simplify redundant logic for constructing structured messages with image url 2024-09-10 21:09:43 -07:00
sabaimran
07e2c49a7a Set default temperature to 0.7 in the extract_questions method 2024-09-10 21:09:21 -07:00
sabaimran
8d40fc0aef Limit vision_enabled image formatting to OpenAI APIs and send vision to extract_questions query 2024-09-10 20:08:14 -07:00
Debanjum Singh Solanky
aa31d041f3 Style list html elements by default on web app to improve readability
Previously list styling was turned off for some reason in Next.js
2024-09-10 17:45:04 -07:00
Debanjum Singh Solanky
596db603e0 Pass query params to chat API in POST body instead of URL query string
Closes #899, #678
2024-09-10 13:57:03 -07:00
Debanjum Singh Solanky
fc6345e246 Simplify setImagePath for upload from chat input area of web app 2024-09-10 09:18:54 -07:00
Raghav Tirumale
549686a7a4
Add Vision Support (#889)
# Summary of Changes
* New UI to show preview of image uploads
* ChatML message changes to support gpt-4o vision based responses on images
* AWS S3 image uploads for persistent image context in conversations
* Database changes to have `vision_enabled` option in server admin panel while configuring models
* Render previously uploaded images in the chat history, show uploaded images for pending msgs
* Pass the uploaded_image_url through to subqueries
* Allow image to render upon first message from the homepage
* Add rendering support for images to shared chat as well
* Fix some UI/functionality bugs in the share page
* Convert user attached images for chat to webp format before upload
* Use placeholder to attached image for data source, response mode actors
* Update all clients to call /api/chat as a POST instead of GET request
* Fix copying chat messages with images to clipboard

TLDR; Add vision support for openai models on Khoj via the web UI!

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
2024-09-09 15:22:18 -07:00
Debanjum Singh Solanky
b553bba1d8 Release Khoj version 1.21.6 2024-09-09 14:55:36 -07:00
sabaimran
223d310ea2 CTA in welcome email 2024-09-09 14:33:27 -07:00
Debanjum Singh Solanky
7941b12d50 Toggle speak, send buttons based on chat input text entered on Desktop 2024-09-09 10:40:53 -07:00
Debanjum Singh Solanky
b5f6550de2 Move link to source code from Nav pane to About page on Desktop app 2024-09-09 10:40:53 -07:00
Debanjum Singh Solanky
77b44f6db0 Update Desktop app dependencies 2024-09-09 10:40:53 -07:00
Debanjum Singh Solanky
303d8ed64e Update Obsidian plugin package dependencies 2024-09-09 10:40:53 -07:00
sabaimran
8e6b9afeb7 Add an automation for research paper summaries 2024-09-08 11:50:49 -07:00
Debanjum
05c169bb37
Set File Types to Sync from Obsidian via Khoj Plugin Settings Page (#904)
Limit file types to sync with Khoj from Obsidian to:
- Avoid hitting per user index-able data limits, especially for folks on the Khoj cloud free tier. E.g by excluding images in Obsidian vault from being synced
- Improve context used by Khoj to generate responses
2024-09-05 22:40:30 -07:00
Husain007
4e8ead66a8
Fix URL to web, desktop settings pages on Desktop application (#903)
Update web and desktop settings URLs on desktop application from previous 'config' path to new 'settings' path
2024-09-05 14:47:43 -07:00
Debanjum Singh Solanky
bc26cf8b2f Only show updated index notice on success in Obsidian plugin
Previously it'd show indexing success notice on error and success
2024-09-04 17:52:32 -07:00
Debanjum Singh Solanky
cb425a073d Use rich text error to better guide when exceed data sync limits in Obsidian
When user exceeds data sync limits. Show error notice with
- Link to web app settings page to upgrade subscription
- Link to Khoj plugin settings in Obsidian to configure file types to
  sync from vault to Khoj
2024-09-04 17:52:32 -07:00
Debanjum Singh Solanky
19efc83455 Set File Types to Sync from Obsidian via Khoj Plugin Settings Page
Useful to limit file types to sync with Khoj. Avoids hitting indexed
data limits, especially for users on the Khoj cloud free tier

Closes #893
2024-09-04 16:09:56 -07:00
sabaimran
7216a06f5f Release Khoj version 1.21.5 2024-09-03 21:58:00 -07:00
sabaimran
895f1c8e9e Gracefully close thread when there's an exception in the anthropic llm thread. Include full stack traces. 2024-09-03 13:16:51 -07:00
sabaimran
17901406aa Gracefully close thread when there's an exception in the openai llm thread. Closes #894. 2024-09-03 13:16:51 -07:00
sabaimran
912cc0074a Use nonlocal for conversation_id when running the event_generator 2024-09-03 11:55:06 -07:00
sabaimran
591f5a522c Release Khoj version 1.21.4 2024-09-02 17:45:39 -07:00
sabaimran
9306a0bb2c Prefetch the settings and openai_config of a texttoimagemodelconfig 2024-09-02 17:35:21 -07:00
sabaimran
6eb06e8626 Downgrade rate limit to 200MB 2024-08-25 15:26:27 -07:00
sabaimran
439a2680fd Increase rate limits for data indexing 2024-08-25 15:09:30 -07:00
sabaimran
4b77325f63 Default to infinite distance when using the search API 2024-08-24 19:57:49 -07:00
sabaimran
e919d28f1c Add support for custom search model-specific thresholds 2024-08-24 19:28:26 -07:00
sabaimran
fa4d808a5f Encode uri components when sending automations data to the server 2024-08-24 18:45:50 -07:00
sabaimran
387b7c7887 Release Khoj version 1.21.3 2024-08-23 11:15:15 -07:00
Debanjum Singh Solanky
5927ca8032 Properly close chat stream iterator even if response generation fails
Previously chat stream iterator wasn't closed when response streaming
for offline chat model threw an exception.

This would require restarting the application. Now application doesn't
hang even if current response generation fails with exception
2024-08-23 02:06:26 -07:00
Debanjum Singh Solanky
238bc11a50 Fix, improve openai chat actor, director tests & online search prompt 2024-08-22 19:09:33 -07:00
Debanjum Singh Solanky
9986c183ea Default to gpt-4o-mini instead of gpt-3.5-turbo in tests, func args
GPT-4o-mini is cheaper, smarter and can hold more context than
GPT-3.5-turbo. In production, we also default to gpt-4o-mini, so makes
sense to upgrade defaults and tests to work with it
2024-08-22 19:04:49 -07:00
Debanjum Singh Solanky
8a4c20d59a Enforce json response by offline models when requested by chat actors
- Background
  Llama.cpp allows enforcing response as json object similar to OpenAI
  API. Pass expected response format to offline chat models as well.

- Overview
  Enforce json output to improve intermediate step performance by
  offline chat models. This is especially helpful when working with
  smaller models like Phi-3.5-mini and Gemma-2 2B, that do not
  consistently respond with structured output, even when requested

- Details
  Enforce json response by extract questions, infer output offline
  chat actors
  - Convert prompts to output json objects when offline chat models
    extract document search questions or infer output mode
  - Make llama.cpp enforce response as json object

- Result
  - Improve all intermediate steps by offline chat actors via json
    response enforcement
  - Avoid the manual, ad-hoc and flaky output schema enforcement and
    simplify the code
2024-08-22 18:07:44 -07:00
Debanjum Singh Solanky
ab7fb5117c Release Khoj version 1.21.2 2024-08-20 12:38:54 -07:00
Debanjum Singh Solanky
de24ffcf0d Upgrade Axios, a desktop app dependency, to version 1.7.4 2024-08-20 12:32:36 -07:00
sabaimran
1ac8de6c3a Release Khoj version 1.21.1 2024-08-20 11:55:34 -07:00
sabaimran
f6ce2fd432 Handle end of chunk logic in openai stream processor 2024-08-20 10:50:09 -07:00
sabaimran
029775420c Release Khoj version 1.21.0 2024-08-20 10:01:56 -07:00
sabaimran
4808ce778a
Merge pull request #892 from khoj-ai/upgrade-offline-chat-models-support
Upgrade offline chat model support. Default to Llama 3.1
2024-08-20 11:51:20 -05:00
Debanjum Singh Solanky
58c8068079 Upgrade default offline chat model to llama 3.1 2024-08-20 09:28:56 -07:00
sabaimran
2d9dd81e76 Re-add authenticated decorator to api_chat.py /chat endpoint 2024-08-19 05:37:18 -05:00
sabaimran
2c5350329a Remove the hashes from titles in found relevant notes 2024-08-18 22:31:15 -05:00
Debanjum Singh Solanky
acdc3f9470 Unwrap any json in md code block, when parsing chat actor responses
This is a more robust way to extract json output requested from
gemma-2 (2B, 9B) models which tend to return json in md codeblocks.

Other models should remain unaffected by this change.

Also removed request to not wrap json in codeblocks from prompts. As
code is doing the unwrapping automatically now, when present
2024-08-16 14:16:29 -05:00
Debanjum Singh Solanky
ca45fce8ac Break long links in train of thought to stay within chat page width 2024-08-16 14:16:29 -05:00
sabaimran
c0316a6b5d
Enable free tier users to have unlimited chats with the default chat model (#886)
- Allow free tier users to have unlimited chats with default chat model. It'll only be rate-limited and at the same rate as subscribed users
- In the server chat settings, replace the concept of default/summarizer models with default/advanced chat models. Use the advanced models as a default for subscribed users.
- For each `ChatModelOption' configuration, allow the admin to specify a separate value of `max_tokens' for subscribed users. This allows server admins to configure different max token limits for unsubscribed and subscribed users
- Show error message in web app when hit rate limit or other server errors
2024-08-16 12:14:44 -07:00
Debanjum
8dad9362e7
Improve search model config display for admin (#887) from aam-at/feature/improve_search_model_config_admin
Currently, the search model config display for admins only shows the id of the search model config, which is not very informative. 

The changes enhances the admin console by displaying the name of the search model config (name), as well as the bi-encoder model (bi_encoder) and cross-encoder model (cross_encoder) along the id.
2024-08-16 07:33:55 -07:00
Debanjum
2b1482d2b4
Fix indexing content from Emacs #883 from aam-at/bugfix/fix_emacs_if
Previously `force' was passed as a query param to the single indexing API. After the recent API updates, it is meant to select the API method to use (PATCH vs PATCH). Converting `force' argument to a bool fixes implementing this new behavior
2024-08-16 07:32:46 -07:00
Debanjum
0b568e204e
Add model_config for cross-encoder model (#885) from aam-at/feature/crossencoder_model_config
Add `model_config' for the cross-encoder model, so the server admin can use models which require the `trust_remote_code' argument to run locally
2024-08-16 07:32:19 -07:00
Debanjum
39e566ba91
Improve Document, Online Search to Answer Vague or Meta Questions (#870)
- Major
  - Improve doc search actor performance on vague, random or meta questions
  - Pass user's name to document and online search actors prompts

- Minor
  - Fix and improve openai chat actor tests
  - Remove unused max tokns arg to extract qs func of doc search actor
2024-08-16 06:46:13 -07:00
Debanjum Singh Solanky
27ad9b1302 Remove unused max tokns arg to extract qs func of doc search actor 2024-08-13 12:53:39 +05:30
Debanjum Singh Solanky
f75606d7f5 Improve doc search actor performance on vague, random or meta questions
- Issue
  Previously the doc search actor wouldn't extract good search queries
  to run on user's documents for broad, vague questions.

- Fix
  The updated extract questions prompt shows and tells the doc search
  actor on how to deal with such questions

  The doc search actor's temperature was also increased to support more
  creative/random questions. The previous temp of 0 was meant to
  encourage structured json output. But now with json mode, a low temp is
  not necessary to get json output
2024-08-13 12:53:39 +05:30
Debanjum Singh Solanky
3675938df6 Support passing temperature to offline chat model chat actors
- Use temperature of 0 by default for extract questions offline chat
  actor
- Use temperature of 0.2 for send_message_to_model_offline (this is
  the default temperature set by llama.cpp)
2024-08-13 12:53:00 +05:30
Shantanu Sakpal
b5bcce7f85
Cycle through chat history in chat input on Obsidian (#861)
* Add ability to cycle through the chat history in the chat input on Obsidian (similar to terminal history navigation)
* Add mod key shortcut to cycle through chat history in chat input
* Add shortcut help text in chat input placeholder

---------

Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
2024-08-12 23:55:25 -07:00
srikary12
05c0aa3882
Support exclusion file filters (#826)
### Overview
Support exclude file filter in user search queries

### Details
- All of the exclude file filter terms need to be satisfied
- Any one of the include file filter terms should be satisfied

### Example
- **Search Query**: *what happened yesterday? -file:"tasks.org" -file:"work.md" file:"diary.org" file:"journal.org*
- **Behavior**: Query will try find relevant notes in any of `journal.org` or `diary.org` and not in `tasks.org` and not in `work.md`

### Details
* Add support for exclusion file filters
* Translate file filter to valid Django DB entry filter regex
* Exclude all files when multiple exclude file filter in query

Previously we were applying an "Or" filter, which would exclude any
file mentioned in a query with multiple exclude file filter.

This is not what we naturally mean when we ask excluding a file in a query

* Rename, rearrange, deduplicate and add file filter tests

Closes #728
---------

Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
2024-08-12 05:41:54 -07:00
Alexander Matyasko
2d9bf14ecb Improve search model config display for admin 2024-08-11 19:13:25 +08:00
Debanjum Singh Solanky
7815e02dd4 Release Khoj version 1.20.4 2024-08-11 16:00:13 +05:30
Debanjum Singh Solanky
d951e36945 Update khoj.el package description, it had gone stale 2024-08-11 15:52:46 +05:30
Debanjum Singh Solanky
16b31c3e35 Refresh automation data shown by edit automation card after update
Previously required the automation page to be refreshed to see updates
to the automation in the edit automation card. This would be seen when
user tries to edit an automation multiple times (without a page refresh)
2024-08-11 15:52:46 +05:30
Debanjum Singh Solanky
f2f37ae444 Fix creating, editing automations that start weekly on Sunday 2024-08-11 15:52:46 +05:30
Debanjum Singh Solanky
ec9add9a51 Fix automation edit cards height. Scroll when card longer than screen 2024-08-11 15:52:46 +05:30
sabaimran
d99f03e4f3 If the list of choices in a chunk is empty, continue in openai response 2024-08-11 15:30:09 +05:30
Alexander Matyasko
f16b0f628b Fix true/false evaluation in Emacs to prevent unintended index re-indexing
Previously, the code incorrectly treated all non-nil values as true, leading to
the index being re-indexed with the force flag whenever the user selected to
update the index.
2024-08-11 17:24:11 +08:00
Alexander Matyasko
0e9e9648e6 Fix emacs if syntax 2024-08-11 17:24:11 +08:00
sabaimran
6f94a076f7 Add conversation_id parameter to the create_automation method 2024-08-11 10:45:13 +05:30
sabaimran
acb825f4f5 Bug fixes for automations
- Pass the new conversation id as kwarg for the scheduled_chat function
- For edit automations, re-use the original conversation id
- Parse images correctly for image automations
2024-08-11 10:41:43 +05:30
Debanjum Singh Solanky
5075d13902 Give visual feedback when interact with chat message feedback buttons
- Use color to provide visual feedback when hover, click on feedback
  buttons
- Use color to provide visual feedback when hover on speech, copy
  buttons click
- Add cooldown period before being able to send feedback on that message again.
  Avoids inadvertent multiple consecutive clicks on feedback buttons
2024-08-10 20:09:52 +05:30
Debanjum Singh Solanky
b3c6c8c84b Add OpenGraph metadata to web app pages for improve social share links 2024-08-10 18:14:05 +05:30
Debanjum Singh Solanky
fc411091c8 Add apple favicon, load favicons for each web app page from assets folder 2024-08-10 18:14:05 +05:30
Debanjum Singh Solanky
a7623e64fa Move Khoj webmanifest, assets to new web app public directory 2024-08-10 18:14:04 +05:30
sabaimran
af1d4b9ba4 Remove the premium requirement for speech for now 2024-08-10 14:10:12 +05:30
sabaimran
1d581464e6 Filter out any undefined agents when rendering the home page 2024-08-10 13:33:55 +05:30
sabaimran
acf1c14122 Release Khoj version 1.20.3 2024-08-09 18:11:11 +05:30
sabaimran
7d3a25f8c0 Handle processing case for the schedule leader process lock when it's empty 2024-08-09 16:37:06 +05:30
sabaimran
faf3584acd Fix automations edit button 2024-08-09 14:21:11 +05:30
sabaimran
5ef198a5b2 Improve default background color styling for inputs 2024-08-08 18:08:05 +05:30
sabaimran
64b2073e63 In the time-based job for managing the schedule leader, and logic to create a new lock when the current one is expired. 2024-08-08 12:42:59 +05:30
sabaimran
7ee0d9067d Fix apostrophe issue in copy text when commandempty in settings page 2024-08-08 11:41:10 +05:30
sabaimran
f28693c8c7 create a useismobilewidth method for standardized mobile view detection. 2024-08-07 21:04:44 +05:30
sabaimran
2943bed5d4 Update category colors 2024-08-07 18:51:31 +05:30
sabaimran
37afa3411f Improve the file upload experience in the settings page 2024-08-07 18:51:20 +05:30
sabaimran
1ee21f5150 Add support for showing files outside of conversation view and linking people to manage files in settings 2024-08-07 18:50:53 +05:30
sabaimran
93f4ceabc1 Add drag/drop file upload support to the chat input area 2024-08-07 18:50:19 +05:30
sabaimran
370ebdee24 Standardized the mobile width calculation 2024-08-07 18:49:06 +05:30
sabaimran
52fed6023f Overlay the side panel on top of other content 2024-08-07 18:46:06 +05:30
Alexander Matyasko
823f8d58bb Add model_config for crossencoder model
Add model_config for crossencoder model, so the user can use models
which require trust_remote_code.
2024-08-07 18:00:12 +08:00
sabaimran
09b71846be Remove favicon.ico as it's interfering with favicon rendering in the home page 2024-08-07 11:53:25 +05:30
Debanjum Singh Solanky
167ef000f4 Fix chat API for non-streaming mode json response 2024-08-06 19:27:54 +05:30
sabaimran
00ee4c2697 Release Khoj version 1.20.2 2024-08-06 16:16:33 +05:30
sabaimran
c4be3b43e5 Add the compiled folder to the list of directories to look through for static templates 2024-08-06 14:50:44 +05:30
sabaimran
265d2a79be Remove duplicate assets from being included in the pypi output 2024-08-06 13:51:37 +05:30
sabaimran
81c75e1024 Fix static file folder path for the pypi build
- Since the .gitignore will ignore any of the assets in the src/ folder when building the package wheel, we need to output the static assets to another folder just for the python pypi package. Use /compiled for this.
2024-08-06 13:24:26 +05:30
sabaimran
7607abc726 Release Khoj version 1.20.1 2024-08-06 10:05:41 +05:30
Debanjum
c23688e2de
Fixes and Improvements Post Spring UX Release (#880)
- Auto focus on email input on login screen for smoother login experience
- Use file icon associated with search page results. Improve search bar
- Show logged in user's email in nav menu for context
- Use previous icons with eyes for search, agents and automations items in nav menu
2024-08-05 14:32:31 -07:00
Debanjum Singh Solanky
a4388c5e65 Use custom Khoj Icons for Search, Agents & Automation in Nav Menu
- Update agents, automations, search svg icons
2024-08-06 02:55:29 +05:30
sabaimran
e9d6899fc2 Change the way the export is created for the pypi package in order to transfer static files out of the tmp shell 2024-08-05 22:46:54 +05:30
sabaimran
b17577c138 Fix configuration for default voice model settings 2024-08-05 19:57:21 +05:30
Debanjum Singh Solanky
ec106d743d Use file icon associated with search page results. Improve search bar 2024-08-05 19:24:39 +05:30
Debanjum Singh Solanky
4258392fc7 Auto focus on email input on login screen for smoother login experience 2024-08-05 19:24:16 +05:30
Debanjum Singh Solanky
020a956c89 Show user email address on settings menu for logged in account context 2024-08-05 19:19:47 +05:30
sabaimran
998d08f155 Fix logic for deletion to automatically re-render the side pane 2024-08-05 18:07:20 +05:30
sabaimran
20d95dc45e Add the favicon.ico file to the public directory of app.khoj.dev 2024-08-05 18:04:03 +05:30
sabaimran
1eab6c8590 Add additional icons for agents, pencil line and chalkboard 2024-08-05 17:23:29 +05:30
sabaimran
bafda233e2 Add standlone khoj_domain for allowed_hosts 2024-08-05 17:11:37 +05:30
Debanjum Singh Solanky
e412ed3bcb Release Khoj version 1.20.0 2024-08-05 16:25:21 +05:30
Debanjum Singh Solanky
9f785dbafe Format web app package.json using prettier 2024-08-05 16:23:31 +05:30
sabaimran
b7ed32f455 Merge branch 'master' of github.com:khoj-ai/khoj into features/migrate-to-spring-ui 2024-08-05 16:12:46 +05:30
sabaimran
34d54c75f7 Lint new changes again 2024-08-05 15:54:50 +05:30
sabaimran
91047d1619 Use a png for the windows desktop icon 2024-08-05 15:29:30 +05:30
sabaimran
1151d14466 Add a separate windows object in the todesktop configuration 2024-08-05 14:27:56 +05:30
sabaimran
c56072aa7b Update todesktop runtime and use the icns file for the todesktop configuration 2024-08-05 14:19:38 +05:30
sabaimran
484b0aa96b Use the newer, simpler favicon across desktop and documentation. Update the macos icon set 2024-08-05 14:06:04 +05:30
sabaimran
1b35a3b16e Fix link to login in the nav menu 2024-08-05 12:32:19 +05:30
sabaimran
5a5bbe3852 Remove deprecate views, assets 2024-08-05 12:31:47 +05:30
sabaimran
c61b289bd1 Migrate all existing pages except login to the new spring ui 2024-08-05 12:17:56 +05:30
sabaimran
f835e330b8 Fix selection of icons, colors, add examples for personal finance 2024-08-05 12:08:18 +05:30
sabaimran
af6a70c9fb Fix fuschia spelling in the colorutils file as well 2024-08-05 11:51:45 +05:30
sabaimran
e0775446c9 fix spelling of fuschia :( 2024-08-05 11:50:11 +05:30
sabaimran
de1cd8c264 Clean up some of the suggestions code, improve randomness of cards' 2024-08-05 11:19:50 +05:30
sabaimran
37e261ff93 Show connected icon when files or notion is indexed 2024-08-05 10:33:18 +05:30
sabaimran
22cfedcaff In the chat history side panel, order conversations by updated time 2024-08-05 09:48:00 +05:30
sabaimran
8220dc6115 Include the updated_at datetime when returning a conversation session 2024-08-05 09:47:13 +05:30
Debanjum Singh Solanky
e296d387e1 Clean duplicate title shown in reference snippets of hierarchical docs
Hierarchical documents like org-mode, markdown have their ancestry
shown in first line. Remove it to show cleaner, deduplicated reference
text from org-mode, markdown files
2024-08-05 04:59:06 +05:30
Debanjum Singh Solanky
95c2a52775 Show file icons in references for first party supported document types
Add org, markdown, pdf, word, icon and default file icons to simplify
identifying file type used as reference for generating chat response
2024-08-05 04:59:06 +05:30
Debanjum Singh Solanky
18a973b666 Fix name of Khoj logo component file in web app 2024-08-05 04:59:06 +05:30
Debanjum Singh Solanky
842036688d Format next.js web app with prettier 2024-08-05 04:59:06 +05:30
Debanjum Singh Solanky
41bdd6d6d9 Throw warning on prettier formatting issues in web app 2024-08-05 03:58:20 +05:30
Debanjum Singh Solanky
1cdfa8087c Update Khoj tagline to "Your Second Brain" 2024-08-05 02:27:05 +05:30
Debanjum Singh Solanky
46f928165c Fix deep linking to settings page cards from docs 2024-08-05 02:27:05 +05:30
sabaimran
f7840782a4 Fix broken rendering of math equations via katex 2024-08-05 00:20:43 +05:30
Debanjum Singh Solanky
b803ed19d3 Add simplified, cleaner khoj logo images to web app static dir 2024-08-04 23:40:21 +05:30
Debanjum Singh Solanky
51e56e17ee Align padding of agent pills to home screen chat input on small screens 2024-08-04 21:57:54 +05:30
Debanjum Singh Solanky
b744dffefd Align voice message button with send chat message button style 2024-08-04 21:04:38 +05:30
Debanjum Singh Solanky
70f670dcf7 Show send button when text in chat input else voice message button
Utilize chat footer space more efficiently. This is especially useful
on small screens

- Send button is anyway only enabled when there is text in chat input
- Otherwise voice message button is better to show by default
2024-08-04 19:25:49 +05:30
Debanjum Singh Solanky
c627527a6f Reorder automation card actions buttons. Put Delete action last 2024-08-04 19:01:11 +05:30
Debanjum Singh Solanky
c7b67a978e Align agents and automation page structure, widths and spacings
- Remove invalid call to styles.main
- Remove unnecessary top padding above side pane to keep side pane at
  consistent position across web app
- Use same pageLayout styles and styling structure on agent like
  automation
- Vertically center automation section and page title on it's row
- Fix applying flex vs grid with tailwind
2024-08-04 19:01:11 +05:30
Debanjum Singh Solanky
60af173c4a Improve responsive spacing of chat page footer buttons
- Remove x axis footer padding on small screens to preserve space,
  keep equal spacing between footer items
- Add 1rem margin to buttons to not have overlap in boundary
- Add 1rem y-axis padding to chat footer to not have focus boundary
  leave the footer boundary on smaller screens
2024-08-04 19:01:10 +05:30
sabaimran
4f2fcc82f0 Make the input area only rounded on the top corners when in mobile view
- Create better styling for the input area buttons, resizing in mobile and creating more even height with a more minimal send button
2024-08-04 18:28:33 +05:30
sabaimran
322fb34d4b Add top padding to the automations header to align it with the agents page 2024-08-04 12:27:37 +05:30
sabaimran
3e1e4a1857 Move the clients section back to the bottom 2024-08-04 11:32:22 +05:30
Debanjum Singh Solanky
caf5c3d74c Link to Khoj manifest in home page metadata to support PWA install
Installing Khoj as PWA was supported in previous web UX as well. This
just adds link to the existing webmanifest to continue support for
installing Khoj as PWA with new web UX
2024-08-04 05:06:38 +05:30
Debanjum Singh Solanky
692058bbdd Fix time of day calculation logic
Previously between 00:00 - 04:00 it'd trigger afternoon insteead of
evening
2024-08-04 04:53:50 +05:30
Debanjum Singh Solanky
015c155582 Simplify structure of chat page to match other pages 2024-08-04 04:43:55 +05:30
Debanjum Singh Solanky
bf71e472c4 Load static assets from Khoj server in dev environment 2024-08-04 04:25:48 +05:30
Debanjum Singh Solanky
f38c072f07 Update chat session title in side pane to new title after rename
Previously the rename wasn't updating the chat session title. We'd
have to refresh the page or side pane to get latest chat session names
after rename action.
2024-08-04 04:25:48 +05:30
Debanjum Singh Solanky
2f7a8698a0 Fix width and equalize spacing between buttons in chat footer
Previously the footer's right border wasn't visible on small screens
due to usage of w-full

Use mr-1 on send button instead of px-1 on chat input parent to
eualize chat footer buttons spacing
2024-08-04 04:25:48 +05:30
Debanjum Singh Solanky
5541bc09c8 Prefix Khoj page breadcrumbs to chat page title for orientation
Allows tab search by looking at standard prefix. Still allows
page title based identification of different Khoj chat sessions
2024-08-04 04:25:48 +05:30
Debanjum Singh Solanky
6a9865ace7 Only show API keys card in non anon mode
- Show informative toast messages on copy, delete of API keys
- Onle show API keys card in non anonymous mode. API keys aren't
  required (and is disabled on server side) in anon mode. Not showing
  card at all in anon mode reduces chance of unnecessary confusion
2024-08-04 04:25:45 +05:30
Debanjum Singh Solanky
f28208d35b Only show chat sessions uptil last month in side pane
- Reduce chat title size
2024-08-04 01:52:08 +05:30
sabaimran
75559a55aa only show search if logged in. update agents icon 2024-08-03 23:23:03 +05:30
sabaimran
185dcb61f7 Update the settings page to better match the design 2024-08-03 20:49:19 +05:30
sabaimran
3e74d383fe Strip quotes from the response mode llm response 2024-08-03 17:33:20 +05:30
sabaimran
87e97e40f4 Resolve various warnings during export 2024-08-03 17:33:04 +05:30
sabaimran
5a75f2c00f Use filled icons when side panel is open 2024-08-03 15:42:49 +05:30
sabaimran
e6260a7bb6 Improve oadding for h9me page chat iput area and inc margin on api keys 2024-08-03 15:33:33 +05:30
Debanjum Singh Solanky
7a8a9fc807 Auto focus cursor on search input when open search page 2024-08-03 13:52:36 +05:30
Debanjum Singh Solanky
30304ccc56 Fix session drawer to fit title, action triple-dot in width on mobile 2024-08-03 13:52:36 +05:30
Debanjum Singh Solanky
5b17fa5dda Set home, chat page height so footer, header visible w/o scroll on phone
Set dynamic view height of page to 100%
2024-08-03 13:52:36 +05:30
sabaimran
687a881ad2 Remove the agents header in the loading view 2024-08-03 13:44:56 +05:30
sabaimran
0db630a123 image cards should be /image, not /paint 2024-08-03 13:44:31 +05:30
sabaimran
261f62e353 Fix automations mobile view by using a wrapper component that chooses a dialog or a drawer 2024-08-03 13:44:17 +05:30
sabaimran
4ce17acd00 Set greeting message to longer text in default view. Only show two agents in mobile 2024-08-03 12:14:58 +05:30
sabaimran
6c35ee4960 Revert height of the side panel on the home page 2024-08-03 11:59:07 +05:30
Debanjum Singh Solanky
e66adf60c5 Have the home and chat page take full height, reduce greeting top space 2024-08-03 11:54:12 +05:30
Debanjum Singh Solanky
cf8745ef78 Improve structure of chat footer on mobile to put agents above input 2024-08-03 11:31:57 +05:30
Debanjum Singh Solanky
529ffdb7e3 Make Title, Chat Footer Icons larger to ease click, tap on Mobile 2024-08-03 11:23:29 +05:30
Debanjum Singh Solanky
8d1c5226ec Remove unnecessary debug logs 2024-08-03 09:55:31 +05:30
sabaimran
f136214290 Improve the nav menu in the not logged in experience 2024-08-03 09:44:04 +05:30
Debanjum Singh Solanky
d8fe677933 Prevent overflow on Search page by search results 2024-08-03 07:07:35 +05:30
Debanjum Singh Solanky
f3765a20b9 Improve content alignment on automation page for small screens
- Left align email, location, timezone pills on small screens
- Indent user enabled automations to improve delineation between
  sections
2024-08-03 07:05:15 +05:30
Debanjum Singh Solanky
a6e1b2c7cb Style nav menu button and expand nav menu item click area to full-width
Style profile pircture button on nav menu
 - Use primary colored ring around subscribed user profile on nav menu
 - Use gray colored ring around non-subscribed user profile on nav menu
 - Use upper case initial as profile pic for user with no profile pic

- Click anywhere on nav menu item to trigger action
  Previously the actual clickable area was smaller than the width of
  the nav menu item
2024-08-03 05:43:24 +05:30
Debanjum Singh Solanky
eed9e401a2 Improve alignment of title bar elements 2024-08-03 04:11:58 +05:30
sabaimran
f188396395 Prompt to login when authenticated, click on suggestion card
- Improve styling for the side panel when not logged in
2024-08-03 01:42:32 +05:30
sabaimran
9c5ff1699a Use new nav menu alignment in the settings page 2024-08-03 01:42:32 +05:30
sabaimran
b1d3979ed9 Fix navmenu in settings, share/chat pages 2024-08-03 01:42:21 +05:30
sabaimran
5f8b76c8f2 Fix layout/styling of the factchecker app 2024-08-03 01:07:59 +05:30
sabaimran
1bb746aaed Adjust spacing when side panel is opened 2024-08-03 01:07:59 +05:30
sabaimran
07b3bdf181 Update nav menu styling to include everything in one header
- Move the nav menu into the chat history side panel component, so that they both show up on one line
- Update all pages to use it with the new formatting
- in mobile, present the sidebar button, home button, and profile button evenly centered in the middle
2024-08-03 01:07:55 +05:30
Debanjum Singh Solanky
e62888659f Only show greeting once userConfig is fetched from server
- Pass userConfig from Home as prop to chatBodyData component with
  loading state
- Pass loading state of userConfig to allow components to handle
  rendering dependent elements once it is loaded
2024-08-02 20:25:09 +05:30
Debanjum Singh Solanky
0adee07d40 Update home page greetings to use user name, when available 2024-08-02 20:25:09 +05:30
sabaimran
5a8ea884a9
Use new HTTP stream format within the new UX
Use updated format for HTTP streamed responses from the Khoj server in the new chat UX
Remove references to the websocket connected field, as websocket use has been deprecated
2024-08-02 02:35:10 -07:00
Debanjum Singh Solanky
02b46a1784 Render references after chat response is streamed for smoother render
Otherwise the Khoj's chat response is filling up in between the
streamed message and already rendered references section at the bottom
of the message

Define OnlineContext type to simplify typing online context param
across other interfaces and functions
2024-08-02 14:11:34 +05:30
Debanjum Singh Solanky
a733e5c1d4 Remove unused handleCompiledReferences chat functions 2024-08-02 13:18:55 +05:30
Debanjum Singh Solanky
7858aff2e2 Trigger welcomeConsole only once on chat, shared chat page load 2024-08-02 13:18:01 +05:30
Debanjum Singh Solanky
cab0957fd3 Just show Khoj logo on title bar on small screens
Continue to show logo + text on larger screens
2024-08-02 13:18:01 +05:30
Debanjum Singh Solanky
3f607b3978 Add icons, improve description of home, chat & search page metadata 2024-08-02 13:18:01 +05:30
Debanjum Singh Solanky
4f783b911c Update DOMPurify imports correctly to resolve compilation warnings 2024-08-02 13:18:01 +05:30
sabaimran
4492017b96 Move processmessagechunk file into a common chat function 2024-08-02 12:31:43 +05:30
sabaimran
13dee7d89e Remove status update for understanding query 2024-08-01 19:22:21 +05:30
sabaimran
1b2cad2a2c Use af in the default state and configure the phone number input styling 2024-08-01 17:04:57 +05:30
sabaimran
723b37955a Disable input for phone number only if its pending verification 2024-08-01 16:45:38 +05:30
sabaimran
84dd1b57fe Use an intl phone input number field and fix the whole verification flow
- There were some state mismatches in configuring a whatsapp number. This commit fixes those issues and uses an external library for phone number validation
2024-08-01 16:44:17 +05:30
sabaimran
ed16914ac3 Remove deprecated fields and fix erroneous export in settings page 2024-08-01 14:45:54 +05:30
sabaimran
7941f4d54d Remove references to deprecated setupwebsocket function 2024-08-01 14:43:17 +05:30
sabaimran
db93ac5d4b Merge branch 'features/big-upgrade-chat-ux' of github.com:khoj-ai/khoj into features/use-new-sse-in-new-chat-ux 2024-08-01 14:41:50 +05:30
sabaimran
fd0e0405af Fix logic for setting and sending the initial chat message from the home page
- Load agents only once when the page loads, rather than triggering constant re-renders
2024-08-01 13:53:16 +05:30
sabaimran
9a43622cef Remove usages of the websocketconnected variable 2024-08-01 13:14:23 +05:30
sabaimran
bfeb64b48f Migrate the shared chat page to also use the new SSE streaming format 2024-08-01 13:14:09 +05:30
sabaimran
833553c3a3 Move conversation commands selection earlier to include in telemetry collected 2024-08-01 12:52:41 +05:30
sabaimran
dbbcf2564f Remove the usage of emojis in the incremental status updates 2024-08-01 12:52:05 +05:30
sabaimran
cd85a51980 Ingest new format for server sent events within the HTTP streamed response
- Note that the SSR for next doesn't support rendering on the client-side, so it'll only update it one big chunk
- Fix unique key error in the chatmessage history for incoming messages
- Remove websocket value usage in the chat history side panel
- Remove other websocket code from the chat page
2024-08-01 12:50:43 +05:30
Debanjum Singh Solanky
32ce564b7c Remove unused Files Connect button and setup Github content card 2024-07-30 18:55:14 +05:30
Debanjum Singh Solanky
ecb873c488 Only allow search model to be updated without being subscribed
Do not make fetch request to server if user is not subscribed
2024-07-30 18:50:57 +05:30
Debanjum Singh Solanky
f58cff5bcc Increase rate limit in the api/content vs deprecated indexer API 2024-07-30 16:09:26 +05:30
Debanjum Singh Solanky
f0bb6883f8 Improve Delete experience on Files Card in Settings Page
Improve placeholder text for notion API key and Whatsapp
number (mention country code required)
2024-07-30 15:25:14 +05:30
sabaimran
b1eb564706 remove the optional pydantic typing from the files param 2024-07-30 15:25:14 +05:30
sabaimran
4a7efdc552 Use patch in place of put in the indexer API call, ensure that files are not being required in the indexer path 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
ffbf57292c Create synced files management modal on the settings page
Use a Command Dialog to allow easier filtering of files to view
without having to leave the settings page
2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
ccc46a09b5 Add new API to batch delete a list of files by filename
- Rearrange DELETE content API definitions order to go from more
  specific to more general
- Create batched file deletion DB adapter
2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
9d86cb57ac Build UX to Connect and Manage Notion Integration via Settings Page 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
7ee179ee1f Return user's Notion token in API call for detailed user settings 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
00a908ae12 Move subscription card to Profile settings section. Remove Billing section
- Why
  Profile section and billing section looked too empty (1 card each).
  Combining them makes the setting page look more complete. Shows
  subscription options early on
- Details
  - Made Futurist text orange
  - Made Unsubscribe a down button instead of cloud slash
  - Updated toast title to subscription
  - Improve Futurist trial title and description
2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
058c902dc7 Delete unused npm package-lock.json as Web app uses yarn.lock instead 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
b8c9b3ffa3 Reduce padding height of input area on new home page 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
8a447107dd Set user name on clicking Save button on settings page 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
44e0b20202 Align Content, Client & Billing settings sections with new designs 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
51e83bcc26 Improve responsive behavior of settings cards 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
efcad4996d Add phone number verification for Whatsapp to new settings page 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
48548684c0 Add card to connect Whatsapp to Khoj on settings page 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
8ec90f194f Add title icons for each content section card on settings page 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
60cdf61737 Create billing section for managing subscription on settings page
- Replicate behavior on current settings.html page
- Improve text for each subscription state to make it more informative, fun
2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
2e165a0e0a Create client API keys section on settings page
- Add table shadcn component to use in API keys settings section
- In dev mode, route requests to auth to khoj server at localhost:42110
2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
00fa4fa0fa Save model on selecting model in dropdown. No extra save action reqd
- Remove now unnecessary button to Save in Card with dropdown
- Use toast to show success, failure (not working)
- Rename language to search, Move it to features section. Add icon to
  the card
2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
13292fc4ca Add icons to card headings 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
a5a06da3fc Use Dropdown component for model options. Make cards more responsive
- Ensure model name doesn't stretch or shrink dropdown width from
  parent card width
- Ensure buttons flex wrap on smaller displays
2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
ade2f6f5d1 Rename selected voice model in get config API response for consistency
- Update references in new and old web client settings
- Arrange new client settings props and add header comments similar to
- config response for code readability
2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
b3253562a5 Dynamically set Content cards buttons based on already setup or not 2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
7e8e80f29e Create config page using tailwind, shadcn components, styling
- Include side pane but with only the account info in it
- Replicate styling of the old config page
2024-07-30 15:25:14 +05:30
Debanjum Singh Solanky
88007d7552 Get user config in the new web client from the new user config APIs 2024-07-30 15:25:14 +05:30
sabaimran
a6339bb973 Add mroe card suggestions and simplify color selection for suggestion cards 2024-07-29 19:11:39 +05:30
sabaimran
551630f0f1 Code clean-up and some fit and finish
- Add a lot more suggestions cards, improve mobile rendering of suggestion cards, improve alignment of chat input, shift message when starts recording voice, remove dead code
2024-07-28 15:19:36 +05:30
sabaimran
1a1d9c7257 Merge branch 'master' of github.com:khoj-ai/khoj into features/big-upgrade-chat-ux 2024-07-27 14:18:05 +05:30
Raghav Tirumale
1685c60e3c
Nav Menu Upgrades and Minor UX Improvements (#869)
* Converted navigation menu into a dropdown menu
* Moved collapsed side panel menu icons into top row
* Auto refresh when conversation is deleted to update side panel and route back to main page if deletion is on current conversation
* Highlight the current conversation in the side panel
* Dynamic homepage messages with current day and time of day.
* `colorutils` upgraded to have more expansive tailwind color options and dynamic class name generation.
* Converted create agent button alert into shadcn `ToolTip`
* Colored lines and icons for agents in chat window
* Cleaned up border styling in dark mode
* fixed three dot menu in side panel to be more easier to click
* Add the KhojLogo import in the nav menu and use a default user profile icon when not authenticated
* Get rid of custom --box-shadow CSS variable
* Pass the agent metadat through the chat body data in order to style the send button
* Add login to the unauthenticated login view, redirecto to home if conversation history not loaded
* Set a max height for the input text area
* Simplify tailwind class names

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-07-27 14:12:00 +05:30
Debanjum
8503d7a07b
Split Configure API into Content, Model API paths (#857)
## Major: Breaking Changes
- Move API endpoints under /configure/<type>/model to /api/model/<type>
- Move API endpoints under /api/configure/content/ to /api/content/
- Accept file deletion requests by clients during sync
- Split /api/v1/index/update into /api/content PUT, PATCH API endpoints

## Minor: Create New API Endpoint
- Create API endpoints to get user content configurations

Related: #852
2024-07-26 23:48:41 -07:00
Debanjum Singh Solanky
a47a54f207 Pass user name to document and online search actors prompts
This should improve the quality of personal information extraction
from document and online sources. The user name is only used when it
is set
2024-07-26 23:53:17 +05:30
sabaimran
e86143dbb0
Merge pull request #867 from khoj-ai/features/search-page-v2
Update the search page
2024-07-26 08:08:04 -07:00
sabaimran
eb5af38f33 Release Khoj version 1.17.0 2024-07-26 20:14:45 +05:30
Raghav Tirumale
5dcac18ba5
New Agents Page User Interface (#866)
Changes for new agents page
- Modernized agent cards
- Responsive design to support mobile users
- Button for users to create their own agents (coming soon)
- Optimized to use tailwind and icon utils
- Side panel added for quick access to conversations
2024-07-26 20:12:31 +05:30
Debanjum Singh Solanky
3daef910c0 Remove stale comment from api content 2024-07-26 20:05:35 +05:30
sabaimran
377f7668c5
Merge pull request #858 from khoj-ai/use-sse-instead-of-websocket
Use Single HTTP API for Robust, Generalizable Chat Streaming
2024-07-26 07:11:54 -07:00
sabaimran
6607e666dc Increase rate limit for data upload packet size in indexer.py 2024-07-26 19:35:32 +05:30
Debanjum Singh Solanky
778c571288 Use enum to track chat stream event types in chat api router 2024-07-26 00:19:43 +05:30
sabaimran
7482797605 Add some better default states for no files found, prompt to search. Add link to search in the file search compnoent in side panel 2024-07-25 13:00:28 +05:30
sabaimran
662dffea3b Press enter to search 2024-07-24 19:28:38 +05:30
sabaimran
19cd607c96 Style the see content button correctly 2024-07-24 18:28:23 +05:30
sabaimran
75a370cc06 Implement focus mode to click into full text of the note 2024-07-24 18:00:33 +05:30
sabaimran
5adbfe14ab Add a search page that just renders truncated results when you click search 2024-07-24 17:43:19 +05:30
sabaimran
52db15706d Remove unused styling 2024-07-24 17:42:36 +05:30
sabaimran
cfe7a1068e Update the navmenu title if prop is updated and undefined 2024-07-24 17:41:31 +05:30
Debanjum Singh Solanky
ebe92ef16d Do not send references twice in streamed image response
Remove unused image content to reduce response payload size.
References are collated, sent separately
2024-07-24 17:18:14 +05:30
Debanjum Singh Solanky
37b8fc5577 Extract events even when http chunk contains partial or mutiple events
Previous logic was more brittle to break with simple unbalanced
'{' or '}' string present in the event data. This method of trying to
identify valid json obj was fairly brittle. It only allowed json
objects or processed event as raw strings.

Now we buffer chunk until we see our unicode magic delimiter and only
then process it.

This is much less likely to break based on event data and the
delimiter is more tunable if we want to reduce rendering breakage
likelihood further
2024-07-24 17:17:39 +05:30
sabaimran
4d30e5b158 Fix indexing error for notion, expecting image and docx in dict 2024-07-24 16:58:31 +05:30
sabaimran
694bedc25b
Add support for text to speech and speech to text (#863)
- Add support for text to speech, speech to text. Add loading and responsive indicators to reflect state.
- When streaming for speech to text, show incremental transcription in the message input field
- When streaming text to speech, and a pause button in the chat message to allow user to stop playback
2024-07-24 14:36:40 +05:30
Raghav Tirumale
3e4325edab
Upgrade: New Home Screen for Khoj (#860)
* V1 of the new automations page
Implemented:
- Shareable
- Editable
- Suggested Cards
- Create new cards
- added side panel new conversation button
- Implement mobile-friendly view for homepage
- Fix issue of new conversations being created when selected agent is changed
- Improve center of the homepage experience
- Fix showing agent during first chat experience
- dark mode gradient updates

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-07-24 13:16:19 +05:30
Debanjum Singh Solanky
70201e8db8 Log total, ttft chat response time on start, end llm_response events
- Deduplicate code to collect chat telemetry by relying on
  end_llm_response event
- Log time to first token and total chat response time for latency
  analysis of Khoj as an agent. Not just the latency of the LLM
- Remove duplicate timer in the image generation path
2024-07-23 23:21:12 +05:30
Debanjum Singh Solanky
b36a7833a6 Remove the old mechanism of streaming compiled references
Do not need response generator to stuff compiled references in chat
stream using "### compiled references:" separator.

References are now sent to clients as structured json while streaming
2024-07-23 19:53:51 +05:30
Debanjum Singh Solanky
eb4e12d3c5 s/online_context/onlineContext chat API response field for consistency
This will align the name of the online context field returned by
current chat message and chat history
2024-07-23 19:50:43 +05:30
Debanjum
498fe2458c
Support Gemma 2 Model Family for Offline Chat (#855)
## Overview
- Gemma 2 is a new open model family by Google. They've released a 9B, 29B param model. A 2B model is also expected.
- It performs really well on the Chatbot arena and shows good performance when testing within Khoj as well.
- Llama.cpp support for Gemma 2 architecture seems to have stabilized
- If Gemma 2 performs well in further testing, it can be made the default offline chat model for Khoj
  - Once the 2B param model is released, the model size to download can be automatically chosen based on (V)RAM available

## Major
- Support Gemma 2 for Offline Chat
- Improve and fix chat model prompts for better, consistent context

## Minor
- Fix and improve offline chat actor, director tests
- Improve offline chat truncation to consider chat message delimiter tokens
2024-07-23 06:57:02 -07:00
Debanjum Singh Solanky
0277d16daf Share desktop chat streaming utility funcs across chat, shortcut views
Null check menu, menuContainer to avoid errors on Khoj mini
2024-07-23 19:16:33 +05:30
Debanjum Singh Solanky
e439a6ddac Use async/await in web client chat stream instead of promises
Align streaming logic across web, desktop and obsidian clients
2024-07-23 18:17:47 +05:30
Debanjum Singh Solanky
fafc467173 Put loading spinner at bottom of chat message in web client 2024-07-23 18:17:47 +05:30
Debanjum Singh Solanky
fc33162ec6 Use new chat streaming API to show Khoj train of thought in Desktop app
Show loading spinner at end of current message
2024-07-23 18:17:47 +05:30
Debanjum Singh Solanky
c5ad172616 Keep loading animation at message end & reduce lists padding in Obsidian
Previously loading animation would be at top of message. Moving it to
bottom is more intuitve and easier to track.

Remove white-space: pre from list elements. It was adding too much y
axis padding to chat messages (and train of thought)
2024-07-23 17:56:03 +05:30
Debanjum Singh Solanky
54b4203683 Update chat API client tests to mix testing of batch and streaming mode 2024-07-23 17:56:03 +05:30
Debanjum Singh Solanky
3f5f418d0e Use new chat streaming API to show Khoj train of thought in Obsidian client 2024-07-23 17:56:03 +05:30
Debanjum Singh Solanky
8303b09129 Convert snake case to camel case in chat view of obsidian plugin 2024-07-23 15:29:12 +05:30
Debanjum Singh Solanky
b224d7ffad Simplify get_conversation_by_user DB adapter code 2024-07-23 14:51:11 +05:30
Debanjum Singh Solanky
daec439d52 Replace old chat router with new chat router with advanced streaming
- Details
  Only return notes refs, online refs, inferred queries and generated
  response in non-streaming mode. Do not return train of throught and
  other status messages

  Incorporate missing logic from old chat API router into new one.

- Motivation
  So we can halve chat API code by getting rid of the duplicate logic
  for the websocket router

  The deduplicated code:
  - Avoids inadvertant logic drift between the 2 routers
  - Improves dev velocity
2024-07-23 14:51:11 +05:30
Debanjum Singh Solanky
2d4b284218 Simplify streaming chat function in web client 2024-07-23 14:38:55 +05:30
Debanjum Singh Solanky
6b9550238f Simplify advanced streaming chat API, align params with normal chat API 2024-07-22 22:51:24 +05:30
Debanjum Singh Solanky
b8d3e3669a Stream Status Messages via Streaming Response from server to web client
- Overview
Use simpler HTTP Streaming Response to send status messages, alongside
response and references from server to clients via API.

Update web client to use the streamed response to show train of thought,
stream response and render references.

- Motivation
This should allow other Khoj clients to pass auth headers and recieve
Khoj's train of thought messages from server over simple HTTP
streaming API.

It'll also eventually deduplicate chat logic across /websocket and
/chat API endpoints and help maintainability and dev velocity

- Details
  - Pass references as a separate streaming message type for simpler
    parsing. Remove passing "### compiled references" altogether once
    the original /api/chat API is deprecated/merged with the new one
    and clients have been updated to consume the references using this
    new mechanism
  - Save message to conversation even if client disconnects. This is
    done by not breaking out of the async iterator that is sending the
    llm response. As the save conversation is called at the end of the
    iteration
  - Handle parsing chunked json responses as a valid json on client.
    This requires additional logic on client side but makes the client
    more robust to server chunking json response such that each chunk
    isn't itself necessarily a valid json.
2024-07-22 15:41:21 +05:30
Debanjum Singh Solanky
91fe41106e Convert Websocket into Server Side Event (SSE) API endpoint
- Convert functions in SSE API path into async generators using yields
- Validate image generation, online, notes lookup and general paths of
  chat request are handled fine by the web client and server API
2024-07-21 14:20:22 +05:30
sabaimran
9cf52bb7e4
Update automations UX for more consistency (#856)
* Update the automations UI to be a more suitable color distribution based on new designs

* Use accented colors for the metadata, update dark mode colors

* Update form to use icons as well and render more pretty inline form labels
2024-07-21 12:22:23 +05:30
sabaimran
7426a4f819 Prefetch related agent when retrieving the conversation for performance improvements 2024-07-19 14:43:30 +05:30
Debanjum Singh Solanky
07f36fa95a Update new web interface with update calls to /content, /model APIs 2024-07-19 12:23:22 +05:30
Debanjum Singh Solanky
f03525f431 Add back /api/configure as /api/settings API endpoint
It had been removed during the /api/configure/content to /api/content
API migration before
2024-07-19 05:40:34 +05:30
Debanjum Singh Solanky
3832ef0236 Move API endpoints under /api/configure/phone/ to /api/phone/
Pull out /api/configure/phone API endpoints into /api/phone for
more concise and sufficiently explanatory API path

Refactor Flow
1. Rename /api/configure/phone -> /api/phone
2024-07-19 05:40:34 +05:30
Debanjum Singh Solanky
1197266912 Move API endpoints under /configure/<type>/model to /api/model/<type>
Now the API to configure all the AI models is under /api/models.
This provides better organization and API hierarchy. The /configure
url segment was redundant.

- Rename POST /api/phone to PATCH /api/phone
- Rename GET /api/configure to GET /api/settings

Refactor Flow
1. Move out POST /user/name to main api.py
2. Rename /api/configure/<type>/model -> /api/model/<type>
3. Rename @api_configure to @api_mode
4. Rename file api_config.py to api_model.py
2024-07-19 05:40:34 +05:30
Debanjum Singh Solanky
469a1cb6a2 Move API endpoints under /api/configure/content/ to /api/content/
Pull out /api/configure/content API endpoints into /api/content to
allow for more logical organization of API path hierarchy

This should make the url more succinct and API request intent more
understandable by using existing HTTP method semantics along with the
path.

The /configure URL path segment was either
- redundant (e.g POST /configure/notion) or
- incorrect (e.g GET /configure/files)

Some example of naming improvements:
- GET /configure/types -> GET /content/types
- GET /configure/files -> GET /content/files
- DELETE /configure/files -> DELETE /content/files

This should also align, merge better the the content indexing API
triggered via PUT, PATCH /content

Refactor Flow
1. Rename /api/configure/types -> /api/content/types
2. Rename /api/configure -> /api
3. Move /api/content to api_content from under api_config
2024-07-19 05:40:34 +05:30
Debanjum Singh Solanky
bba4e0b529 Accept file deletion requests by clients during sync
- Remove unused full_corpus boolean. The full_corpus=False code path
  wasn't being used (accept for in a test)
- The full_corpus=True code path used was ignoring file deletion
  requests sent by clients during sync. Unclear why this was done
- Added unit test to prevent regression and show file deletion by
  clients during sync not ignored now
2024-07-19 04:53:01 +05:30
Debanjum Singh Solanky
5923b6d89e Split /api/v1/index/update into /api/content PUT, PATCH API endpoints
- This utilizes PUT, PATCH HTTP method semantics to remove need for
  the "regenerate" query param and "/update" url suffix
- This should make the url more succinct and API request intent more
  understandable by using existing HTTP method semantics
2024-07-19 01:45:53 +05:30
Debanjum Singh Solanky
b0ee78586c Improve offline chat truncation to consider message separator tokens 2024-07-18 03:43:09 +05:30
Debanjum Singh Solanky
6f46e6afc6 Improve and fix chat model prompts for better, consistent context
- Add day of week to system prompt of openai, anthropic, offline chat models
- Pass more context to offline chat system prompt to
  - ask follow-up questions
  - know where to find information about khoj (itself)
- Fix output mode selection prompt. Log error if model does not select
  valid option from list of valid output modes provided
- Use consistent names for question, answers passed to
  extract_questions_offline prompt

- Log which model extracts question, what the offline chat model sees
  as context. Similar to debug log shown for openai models
2024-07-18 03:43:09 +05:30
Debanjum Singh Solanky
53eabe0c06 Support Gemma 2 for Offline Chat
- Pass system message as the first user chat message as Gemma 2
  doesn't support system messages
- Use gemma-2 chat format
- Pass chat model name to generic, extract questions chat actors
  Used to figure out chat template to use for model
  For generic chat actor argument was anyway available but not being
  passed, which is confusing
2024-07-18 03:09:38 +05:30
Debanjum Singh Solanky
65dade4838 Create API endpoints to get user content configurations
This is to be used by the new Next.js web client
2024-07-17 13:41:14 +05:30
Debanjum Singh Solanky
a1c362a4f7 Expose web, desktop settings page under /settings, not /configure
- Update references to the settings page to use new url across docs
  and code
- Rename desktop and web settings page to settigns.html instead of
  config[ure].html
2024-07-17 13:17:29 +05:30
Debanjum Singh Solanky
b015b0e83d Arrange config API detailed response fields to improve readability
There are a lot of fields being returned. Group returned fields and
add comment header to each Group for readability
2024-07-17 13:17:28 +05:30
Debanjum Singh Solanky
71ebf31a54 Make config API detailed response fields more intuitive, consistent
- Use name, id for every [search|chat|voice|pain]_model_option
- Rename current_model_state field to more intuitive enabled_content_source
- Update references to the update fields in config.html
2024-07-17 12:41:01 +05:30
Debanjum Singh Solanky
583fa3c188 Migrate the pypi package to khoj project name. Update references
- Deprecate khoj-assistant pypi package. Use more accurate and
  succinct pypi project name, khoj
- Update references to sye khoj pypi package in docs and code instead
  of the legacy khoj-assistant pypi package
- Update pypi workflow to publish to both khoj, khoj-assistant for now
- Update stale python 3.9 support mentioned in our pyproject. Can't
  support python 3.9 as depend on latest django which support >=3.10
2024-07-17 10:41:16 +05:30
Debanjum Singh Solanky
7316e6b9d3 Pass isMobileWidth prop to SidePanel via chat share interface 2024-07-16 16:13:27 +05:30
Debanjum Singh Solanky
4759c4ac96 Turn prettier off instead of throwing error for now
Until web interface code is reformatted with prettier
2024-07-16 16:13:27 +05:30
Debanjum Singh Solanky
466ef3f8f1 Do no explicitly add line-clamp plugin as it's in Tailwind by default 2024-07-16 16:13:27 +05:30
Debanjum Singh Solanky
59000a47cb Move Desktop config page to /configure from /config url path
Update references to point to page at /configure instead of /config
2024-07-16 16:13:27 +05:30
Debanjum Singh Solanky
a5c16ad600 Move Web client config page to /configure from /config url path
Update docs, clients and error messages to point to /configure
instead of /config
2024-07-16 16:13:27 +05:30
Debanjum Singh Solanky
de15a7a3fc Rename API path /api/config to /api/configure
- Update clients calling /api/config to call /api/configure instead
2024-07-16 16:13:27 +05:30
Debanjum Singh Solanky
dd31936746 Make config api endpoint urls consistent
- Consistently use /content/ for data. Remove content-source from path
- Remove unnecessary /data/ prefix for API endpoints under /config
2024-07-16 16:13:27 +05:30
Debanjum Singh Solanky
e8176b41ef Reuse get config data logic across config pages on web client
- Put logic to get config data, detailed or basic into router helpers module
- Use the get config func across the config pages on web clients

- Put configure content and get_notion_auth_url funcs in router helper
  module to avoid circular import
2024-07-16 16:13:27 +05:30
sabaimran
1a5405e24c Fix interpretation of day of week in automation form 2024-07-16 10:12:30 +05:30
sabaimran
c837f3779e
Update the agents page with new UX (#850)
- Use icons/colors for setting the styling of agents
- Update automations page to use the shadcn cards: https://github.com/shadcn-ui/ui
2024-07-16 10:10:55 +05:30
sabaimran
1c6ed9bc6d
Migrate the existing automations page to use React (#849)
Migrates the Automations page to React, mostly keeping the overall design consistent with organization. Use component library, with some changes in color. Add easier management with straightforward form and editing experience.
Use system preference for determining dark mode if not explicitly set.
2024-07-15 21:42:33 +05:30
Debanjum Singh Solanky
6c630bc6c3 Constrain chat messages to stay in view port across screen sizes
- Constrain chat messages max width to view port across screen sizes
- Wrap references on smaller screens, use tailwind, not js to apply styling
2024-07-15 21:00:50 +05:30
sabaimran
9a5bf4c701 Fix rendering of teaser reference panel in mobile width 2024-07-15 19:40:55 +05:30
sabaimran
2e9275c0f3 Remove side panel padding in desktop view. Fix width in mobile view 2024-07-15 19:33:12 +05:30
Debanjum Singh Solanky
ba0ba6b59f Merge branch 'features/big-upgrade-chat-ux' of github.com:khoj-ai/khoj into document-styling-on-chat-ux 2024-07-15 10:42:56 +05:30
Debanjum
23f61d49e0
Support syncing, searching images from Obsidian plugin (#847)
- Sync images from Obsidian vault with Khoj server now that Khoj can OCR images
- Support rendering images returned by Khoj search modal
2024-07-14 20:41:39 -07:00
Debanjum Singh Solanky
6f8f846086 Standardize code format for web interface with prettier
Use husky, lint-staged to run prettier pre-commit
2024-07-15 00:34:54 +05:30
sabaimran
06dce4729b
Make most major changes for an updated chat UI (#843)
- Updated references panel
- Use subtle coloring for chat cards
- Chat streaming with train of thought
- Side panel with limited sessions, expandable
- Manage conversation file filters easily from the side panel
- Updated nav menu, easily go to agents/automations/profile
- Upload data from the chat UI (on click attachment icon)
- Slash command pop-up menu, scrollable and selectable
- Dark mode-enabled
- Mostly mobile friendly
2024-07-14 23:18:06 +05:30
Debanjum Singh Solanky
6dd90931e8 Fix closing side panel on mobile when click open a chat session 2024-07-14 22:54:49 +05:30
Debanjum Singh Solanky
47b754c07b Only open side panel as drawer in mobile view 2024-07-14 14:08:41 +05:30
Debanjum Singh Solanky
b47f30ad77 Make Khoj icon appropriately sized and side panel arrow bold 2024-07-14 14:06:36 +05:30
Debanjum Singh Solanky
e6b21144e2 Conversations list should resize to take max space on side panel 2024-07-14 13:49:36 +05:30
Debanjum Singh Solanky
c2bf405489 Make loading message, styling configurable. Do not show agent when no data
- Pass Loading message, class name via props to both inline and normal
  loading spinners
- Pass loading conversation message to loading spinner when chat
  history is being fetched
2024-07-14 13:00:36 +05:30
Debanjum Singh Solanky
63719747cb Show agent profile card with description when hover on agent in chat view
- Create profile card componennt. Use it for agent profile card
- Pass agent persona from khoj server via API
- Put link to agent profile page in the hover card to make it 2 clicks
  away. Othewise inadvertent clicks on agent in chat view lead away to
  agent page
- Use tailwind line-clamp extension to clamp card to first two lines
2024-07-14 12:20:11 +05:30
Debanjum Singh Solanky
dbbd4b9777 Show action bar on last chat message without hover 2024-07-14 10:32:31 +05:30
Debanjum Singh Solanky
a0f38e079f Improve Train of Thought icons spacing and loading circle 2024-07-14 09:35:15 +05:30
Debanjum Singh Solanky
e9567741eb Improve mobile friendly styling of chat session side panel 2024-07-14 00:57:08 +05:30
Debanjum Singh Solanky
b26a6e25d1 Show chat message action buttons without hover on mobile interfaces
This is because hover maybe hard to do on mobile devices
2024-07-14 00:54:23 +05:30
Debanjum Singh Solanky
f69f9e3523 Close chat sessions side panel on click open a chat session 2024-07-14 00:53:16 +05:30
Debanjum Singh Solanky
d51011314f Improve styling of chat input, references UI across screen sizes
Use tailwind screen breakpoints shorthand instead of js to apply
different styling for different screen sizes
2024-07-13 20:45:34 +05:30
Debanjum Singh Solanky
2668e42e7f Center cursor in chat input. See upto 2 lines for multi-line context
- Reuse class name when get slash command icons
- Previous chat input styling didn't have the cursor centered in the
  chat input text area. But it did allow seeing multi line chat inputs
  for context
2024-07-13 02:51:29 +05:30
Debanjum Singh Solanky
aeaebfb515 Show chat message timestamp on hover in chat view 2024-07-13 02:51:19 +05:30
Debanjum Singh Solanky
e00c6b486e Add hover text descriptions of action buttons on chat message in web view 2024-07-12 15:40:51 +05:30
Debanjum Singh Solanky
5fccccfdff Do not render references with broken links in chat view 2024-07-12 15:14:11 +05:30
Debanjum Singh Solanky
b98a0cfe1b Render inline png, webp images generated by Khoj in chat view
Add spacing between chat message paragraphs
2024-07-12 15:13:19 +05:30
sabaimran
3e7e73ddd6 Switch from using dynamic routes to static routes and extracting slug from URL manually. See https://github.com/vercel/next.js/discussions/64660 for limitations with static export / dynamic routes 2024-07-11 23:06:27 +05:30
sabaimran
bea0aa5445 Improve the logged out share experience 2024-07-11 20:11:21 +05:30
Debanjum Singh Solanky
cbae8b68fb Add DB migration from making bi_encode configs optional in #834 2024-07-11 16:33:31 +05:30
Debanjum Singh Solanky
3a75838196 Add Keyboard shortcuts to navigate in Khoj Desktop 2024-07-11 16:29:53 +05:30
Debanjum Singh Solanky
6c1861b319 Improve the prompt to generate images with DALLE3 and SD3
- Major
  - Ask for prompt in prose
  - Remove seed from SD3 image generation to improve diversity of output
    for a given prompt
    Otherwise for conversations with similar sounding
    prompts, the images would be almost exactly the same. This maybe
    another indicator of SD3's inability to capture detailed
    instructions
  - Consistently use "prompt" wording instead of "query" in improved
    image generation prompts.
    Previously a mix of those terms were being used, which could confuse
    the chat model

- Minor
  - Add day of week to prompt
  - Remove 2-5 sentence limit on instructions to SD3. It seems to be
    able to follow longer instructions just with less fidelity than
    DALLE. And the 2-5 sentence instruction limit wasn't being adhered to
  - Improve ability to edit, improve the image based on follow-up
    instructions by the user
  - Align prompts for DALLE and SD3. Only difference is to wrap text to
    be rendered in quotes for SD3. This improves it's ability to render
    requested text. DALLE cannot render text as well or consistently
2024-07-11 16:29:53 +05:30
Debanjum Singh Solanky
21fe1a917b Support syncing, searching images from Obsidian plugin 2024-07-11 16:22:31 +05:30
sabaimran
6f1d799759 Modularize code and implemenet share experience 2024-07-10 23:08:16 +05:30
sabaimran
1b4a51f4a2 Remove print statement for debugging timestamps 2024-07-10 14:54:22 +05:30
sabaimran
0369eb6e0e Fix timestamp bug for pending message and expand CSP for thumbnails 2024-07-10 14:53:31 +05:30
sabaimran
375685530f Add content security policy to the chat page 2024-07-10 11:18:41 +05:30
sabaimran
c5cfd0f2cf Remove unused slash command-related useeffect hook 2024-07-10 10:03:58 +05:30
sabaimran
e1a5c17775 Add DOMPurify for rendering md text. Add a easter egg in the console 2024-07-10 10:03:08 +05:30
sabaimran
e358723baa Fix image rendering and unique key for pending message? 2024-07-09 21:55:54 +05:30
sabaimran
c8c5d50b1a Improve command bar slash experience 2024-07-09 21:39:13 +05:30
sabaimran
c25bf97831 Update hover styling for see all button 2024-07-09 20:55:54 +05:30
sabaimran
23b71b0dff Remove shadow from the slash command bar 2024-07-09 20:52:38 +05:30
sabaimran
998e2aec30 Update dark mode, fix chat message time stamp, fix rendering for new message 2024-07-09 20:50:20 +05:30
sabaimran
0c6b6de09e Revert web client route chat page rendering logic 2024-07-09 19:47:04 +05:30
sabaimran
cc22e1b013 Add pop-up module for the slash commands 2024-07-09 19:46:17 +05:30
sabaimran
5b69252337 Add hover effects for chat messages 2024-07-09 14:56:57 +05:30
sabaimran
a0e9530fa4 Merge branch 'master' of github.com:khoj-ai/khoj into features/chat-ui-updates-big 2024-07-09 12:57:50 +05:30
sabaimran
4471c1e37f Apply mitigations for piling up open connections
- Because we're using a FastAPI api framework with a Django ORM, we're running into some interesting conditions around connection pooling and clean-up. We're ending up with a large pile-up of open, stale connections to the DB recurringly when the server has been running for a while. To mitigate this problem, given starlette and django run in different python threads, add a middleware that will go and call the connection clean up method in each of the threads.
2024-07-09 12:22:58 +05:30
sabaimran
609e7ee19c Fix width of side panel 2024-07-09 12:02:01 +05:30
sabaimran
bf4c2f219e Merge branch 'master' of github.com:khoj-ai/khoj into features/chat-ui-updates-big 2024-07-08 17:00:42 +05:30
Debanjum Singh Solanky
9e31ebff93 Release Khoj version 1.16.0 2024-07-07 18:26:10 +05:30
Debanjum Singh Solanky
54132efd67 Fix Khoj Obsidian plugin build 2024-07-07 18:26:10 +05:30
Debanjum Singh Solanky
510d9b3a29 Add short keys to open chat menu, new chat, search from Obsidian pane 2024-07-07 17:57:17 +05:30
Debanjum Singh Solanky
3e0c882e27 Transcribe only when keyboard shortcut or button pressed in Obsidian
- Transcribe on holding Ctrl+s keyboard shortcut
- Transcribe on holding the transcribe button pressed via mouse too
- Make the transcribe button robust to inadvertent touches by using timeout
- Do not transcribe, trigger auto-send on silences. Silence detection
  is super rudimentary, just blocks standard emanations by whisper
  when no speech
2024-07-07 17:57:17 +05:30
sabaimran
0eb000c3ea Add health checks for the django ORM 2024-07-07 16:11:28 +05:30
sabaimran
6f8a65c529 References, mobile friendly chat sessions and file filter 2024-07-07 15:42:29 +05:30
Debanjum Singh Solanky
a31cd0dec1 Fix async batch delete of indexed entries 2024-07-06 22:45:26 +05:30
Debanjum
08b379c2ab
Fix, Improve Indexing, Deleting Files (#840)
### Fix
- Fix degrade in speed when indexing large files
- Resolve org-mode indexing bug by splitting current section only once by heading
- Improve summarization by fixing formatting of text in indexed files

### Improve
- Improve scaling user, admin flows to delete all entries for a user
2024-07-06 19:52:42 +05:30
Debanjum Singh Solanky
d693baccbc Make it optional to set the encoder, cross-encoder configs via admin UI 2024-07-06 19:35:59 +05:30
Debanjum Singh Solanky
1baebb8d0e Identify markdown headings by any whitespace character after ^#+
Previously only markdown headings with space characters after # would
be considered a heading. So ^##\t wouldn't be considered a valid heading
2024-07-06 19:35:59 +05:30
Debanjum Singh Solanky
010486fb36 Split current section once by heading to resolve org-mode indexing bug
- Split once by heading (=first_non_empty) to extract current section body
  Otherwise child headings with same prefix as current heading will
  cause the section split to go into infinite loop
- Also add check to prevent getting into recursive loop while trying
  to split entry into sub sections
2024-07-06 19:35:59 +05:30
Debanjum Singh Solanky
6a135b1ed7 Fix degrade in speed of indexing large files. Improve summarization
Adding files to the DB for summarization was slow, buggy in two ways:
- We were updating same text of modified files in DB = no of chunks
  per file times

- The `" ".join(file_content)' code was breaking each character in the
  file content by a space. This formats the original file content
  incorrectly before storing in the DB

Because this code ran in the main file indexing path, it was slowing down
file indexing. Knowledge bases with larger files were impacted more strongly
2024-07-06 19:35:59 +05:30
Debanjum Singh Solanky
e6ffb6b52c Improve scaling user flow to delete all entries
- Delete entries by batch to improve efficiency of query at scale
- Share code to delete all user entries between it's async, sync methods
- Add indicator to show when files being deleted on web config page
2024-07-06 19:35:59 +05:30
Debanjum Singh Solanky
1ab59865b5 Improve scaling admin flow to delete all entries for user 2024-07-06 19:35:59 +05:30
Debanjum Singh Solanky
9bdb48807b Ratelimit text to speech model. Validate share chat url domain
- Do not log auth error message on server when Resend setup as Magic
  links for sign-in are now supported
2024-07-06 12:53:19 +05:30
Debanjum Singh Solanky
b334db0fca Add CSP to web config pages. Load phone no validation js, css from S3 2024-07-06 12:48:28 +05:30
Debanjum Singh Solanky
2f034f807a Construct config page elements on Web via DOM scripting.
Minimize isage of innerHTML to prevent DOM clobbering and unintended
escape by user Input
2024-07-06 12:48:28 +05:30
Debanjum Singh Solanky
69c9e8cc08 Disable CSP in Khoj Obsidian as it interferes with Obsidian functionality
The Khoj CSP interferes with other Obsidian features and plugins as
CSP is applied page wide.

For now chat message sanitization via Dompurify should suffice.

Enable CSP when can scope it to only the Khoj Obsidian plugin.
2024-07-05 16:10:08 +05:30
Debanjum Singh Solanky
a353d883a0 Make it optional to set the encoder, cross-encoder configs via admin UI
Upgrade sentence-transformer, add einops dependency for some sentence
transformer models like nomic
2024-07-05 16:09:30 +05:30
Debanjum Singh Solanky
6d59ad7fc9 Add listening circle animation to speak button in Obsidian plugin
Use icon active focus as color of animation button
2024-07-05 14:00:53 +05:30
sabaimran
aec44a0b89 Add dark mode toggle! And improve experience for train of thought 2024-07-04 18:29:21 +05:30
Debanjum Singh Solanky
516af86575 Fix add, remove of the text to speech loader element in Obsidian 2024-07-04 17:38:45 +05:30
sabaimran
465ef0b772 Add a loading experience when waiting for khoj response 2024-07-04 13:49:51 +05:30
Debanjum Singh Solanky
814aca6d69 Skip summarize when not triggered via slash cmd and can't summarize
Maybe better to fallback to non-summarize behavior if summarize intent
is just inferred but we can't actually summarize because the single
file added to conversation isn't satisfied
2024-07-04 13:31:00 +05:30
Debanjum
4446de00d3
Enable Voice, Keyboard Shortcuts in Khoj Obsidian Plugin (#837)
- Simplify quick jump between Khoj side pane and main editor view using keyboard shortcuts
- Enable voice chat in Obsidian to make interactions with Khoj more seamless
2024-07-04 13:28:29 +05:30
sabaimran
5ea8b16f84 Fix missing method error 2024-07-04 12:08:22 +05:30
sabaimran
d61bddf56c Fix retrieving image model by prefetching the openai config in the async method 2024-07-04 11:58:33 +05:30
sabaimran
a129b017b9 Fix image generation on server -- use default config when not set by user 2024-07-04 09:13:23 +05:30
sabaimran
34118078bf kill the emojis 2024-07-04 00:30:21 +05:30
sabaimran
d5ba916978 Working example of streaming, intersection observer, other UI updates 2024-07-04 00:30:01 +05:30
sabaimran
78d1a29bc1 Finish up filte filter side panel menu 2024-07-02 23:32:36 +05:30
sabaimran
6fa2dbc042 Do not use the custom configured max prompt size to send message to anthropic 2024-07-02 21:59:06 +05:30
sabaimran
8a6722ba97 Add basic implementation for chat side panel components 2024-07-02 21:56:43 +05:30
Debanjum Singh Solanky
afcfc60637 Merge DB migrations post merge of SD3 via API support PR 2024-07-02 17:54:58 +05:30
Debanjum
c015eeb5dd
Improve Online Search: Parallelize Search, Use Jina Reader API by default (#832)
- Overview
  Khoj wil be able to do online search out of the box, even for self-hosted users
  - Default to Jina search, reader API when no Serper.dev, Olostep API keys
  - Run online searches in parallel to process multiple queries faster

- Details
  - Jina provides a [reader API](https://github.com/jina-ai/reader) for online search and web page reading
     It requires no API key. This provides a good default to enable 
     online search for self-hosted readers requiring no additional setup. 

  - Jina search API also returns webpage contents with the results, so
     just use those directly when Jina Search API used instead of
     trying to read webpages separately. The extract relevant content from
     webpage step using a chat model is still used from the
    `read_webpage_and_extract_content' func in this case.

  - Parse search results from Jina search API into same format as
     Serper.dev for accurate rendering of online references by clients

  - Run online searches in parallel with AsyncIO to process multiple queries faster
2024-07-02 17:44:51 +05:30
Debanjum
826c3dc9cc
Enable using Stable Diffusion 3 for Image Generation via API (#830)
- Support Stable Diffusion 3 via API
  Server Admin needs to setup model similar to DALLE-3 via Django Admin Panel
- Use shorter prompt generator to prompt SD3 to create better images
- Allow users to set paint model to use from web client config page
2024-07-02 17:28:50 +05:30
Debanjum Singh Solanky
553beae848 No need to set OpenAI API key from environment variable explicitly
It is unnecessary as the OpenAI client automatically tries to use API
key from OPENAI_API_KEY env var when the api_key field is unset
2024-07-02 17:19:09 +05:30
Debanjum Singh Solanky
a038e4911b Default to Jina search, reader API when no Serper.dev, Olostep API keys
Jina AI provides a search and webpage reader API that doesn't require
an API key. This provides a good default to enable online search for
self-hosted readers requiring no additional setup.

Jina search API also returns webpage contents with the results, so
just use those directly when Jina Search API used instead of
trying to read webpages separately. The extract relvant content from
webpage step using a chat model is still used from the
`read_webpage_and_extract_content' func in this case.

Parse search results from Jina search API into same format as
Serper.dev for accurate rendering of online references by clients
2024-07-02 17:19:08 +05:30
Debanjum Singh Solanky
ff44734774 Run online searches in parallel to process multiple queries faster 2024-07-02 17:19:08 +05:30
sabaimran
0ee7cc8c47 Change overall architecure of how information is flowing for better statefulness 2024-07-02 12:39:54 +05:30
sabaimran
541ce04ebc Checkpoint: Updated sidebar panel with new components
- Add non-functional UI elements for chat, references, feedback buttons, rename/share session, mic, attachment, websocket connection
2024-07-02 11:18:50 +05:30
Raghav Tirumale
8eccd8a5e4
Support Indexing Images via OCR (#823)
- Added support for uploading .jpeg, .jpg, and .png files to Khoj from Web, Desktop app
- Updating indexer to generate raw text and entries using RapidOCR
- Details
  * added support for indexing images via ocr
  * fixed pyproject.toml
  * Update src/khoj/processor/content/images/image_to_entries.py
     Co-authored-by: Debanjum <debanjum@gmail.com>
  * Update src/khoj/processor/content/images/image_to_entries.py
     Co-authored-by: Debanjum <debanjum@gmail.com>
  * removed redudant try except blocks
  * updated desktop js file to support image formats
  * added tests for jpg and png
  * Fix processing for image to entries files
  * Update unit tests with working image indexer
  * Change png test from version verificaition to open-cv verification

---------

Co-authored-by: Debanjum <debanjum@gmail.com>
Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-07-01 06:00:00 -07:00
Debanjum Singh Solanky
cffc14a46a Trigger voice chat via keyboard shortcut in Khoj side pane
Quickly trigger voice chat from Khoj side pane using Keyboard shortcuts
2024-07-01 18:06:09 +05:30
Debanjum Singh Solanky
3723904512 Toggle jump between Khoj side pane & previous editor via cmd, kbd shortcut
Improve quick navigation to, from Khoj side pane using Keyboard
shortcut or Obsidian command
2024-07-01 18:05:59 +05:30
Debanjum Singh Solanky
fbb95ca342 Put cursor on chat input when focus on chat view in Obsidian
This should improve fluidity of keyboard interactions with Khoj on
Obsidian.

Open Khoj chat view via keybinding or command pallete and ask
question using only the keyboard, with no mouse clicks required
2024-07-01 18:05:55 +05:30
Debanjum Singh Solanky
093e276908 Enable Voice chat in Khoj Obsidian plugin
- Automatically carry out voice chats with Khoj from within Obsidian
  When send voice message, Khoj will auto respond with voice as well
- Listen to past Khoj messages as speech
- Add circular loading spinner to use while message is being converted
  to speech
2024-07-01 18:02:28 +05:30
sabaimran
c83b8f2768
Allow just one worker to be the background schedule leader (#836)
* Add a leader election mechanism to circumvent runtime issues for multiple schedulers

- Reduce the load on the DB and risk of issues on the service side by limiting the execution environment to one elected leader at a given time. This one is responsible for managing all of the execution of the jobs, though all workers are capable of adding and removing jobs

* Set a max duration for the schedule leader position (12 hrs), add some error if automation not added successfully
2024-06-28 13:13:25 +05:30
sabaimran
80fe5ce182 Fix user not authenticated interpretation error 2024-06-27 21:13:54 +05:30
Raghav Tirumale
24a0d8b073
Add OS Level Shortcut Window for Quick Access to Khoj Desktop (#815)
* rough sketch of desktop shortcuts. many bugs to fix still

* working MVP of desktop shortcut khoj

* UI fixes

* UI improvements for editable shortcut message

* major rendering fix to prevent clipboard text from getting lost

* UI improvements and bug fixes

* UI upgrades: custom top bar, edit sent message and color matching

* removed debug javascript file

* font reverted to Noto Sans

* cleaning up the code and removing diffs

* UX fixes

* cleaning up unused methods from html

* front end for button to send user back to main window to continue conversation

* UX fix for window and continue conversation support added

* migrated common js functions into chatutils.js

* Fix window closing issue in macos by

1. Use a helper function to determine if the window is open by seeing if there's a browser window with shortcut.html loaded
2. Use the  event listener on the window to handle teardown

* removed extra comment and renamed continue convo button

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-06-27 07:20:13 -07:00
sabaimran
870d9ecdbf
Add a fact checker feature with updated styling (#835)
- Add an experimental feature used for fact-checking falsifiable statements with customizable models. See attached screenshot for example. Once you input a statement that needs to be fact-checked, Khoj goes on a research spree to verify or refute it.
- Integrate frontend libraries for [Tailwind](https://tailwindcss.com/) and [ShadCN](https://ui.shadcn.com/) for easier UI development. Update corresponding styling for some existing UI components. 
- Add component for model selection 
- Add backend support for sharing arbitrary packets of data that will be consumed by specific front-end views in shareable scenarios
2024-06-27 18:45:38 +05:30
sabaimran
3b7a9358c3
Add our first view via Next.js for Agents (#817)
Initialize our migration to use Next.js for front-end views via Agents. This includes setup for getting authenticated users, reading in available agents, setting up a pop-up modal when you're clicking on an agent, and allowing users to start new conversations with agents.

Best attempt at an in-place migration, though there are some noticeable differences.

Also adds view for chat that are not being used, but in experimental phase.
2024-06-27 13:56:16 +05:30
Debanjum Singh Solanky
afbeee9e82 Rename copy-button to more general chat-action-button in Obsidian client
- Use 4 space indent of activateView function in pane_view component
2024-06-26 18:09:23 +05:30
sabaimran
8c12a69570 Fix issue in anthropic chat when khoj message becomes top message
This is because Anthropic requires the first message in the chat history to be from the user.
2024-06-26 12:59:34 +05:30
Debanjum Singh Solanky
4f89319b40 Release Khoj version 1.15.0 2024-06-26 10:38:16 +05:30
Debanjum Singh Solanky
c793d8a69e Add Validation logic to save PaintModel. Use API key from Paint Model
Rename Paint Model, Adapters to TextToImage for consistency
2024-06-26 10:16:26 +05:30
Debanjum Singh Solanky
1acf969c6e Do not require OpenAI to generate image as local chat + sd3 works now
Previously the text_to_image helper would only trigger the image
generation flow if OpenAI client was setup. This is not required
anymore as offline chat model + sd3 API works. So remove that check
2024-06-26 10:16:26 +05:30
Debanjum Singh Solanky
2c4bf91a61 Allow user to set paint model to use from web client config page 2024-06-26 10:16:26 +05:30
Debanjum Singh Solanky
eb09aba747 Remove quotes wrapping the prompt from being passed to image gen model 2024-06-26 10:16:26 +05:30
Debanjum Singh Solanky
fdd4c02461 Use shorter prompt generator to prompt SD3 to create better images 2024-06-26 10:16:26 +05:30
Debanjum Singh Solanky
eda33e092f Enable using Stable Diffusion 3 for Image Generation via API 2024-06-26 10:16:26 +05:30
Debanjum Singh Solanky
cfe46fd9f5 Add Border Color instead of BG Color for Chat Message in Obsidian 2024-06-26 08:11:04 +05:30
sabaimran
fb818ead60 Use active bg instead of code background for khoj response 2024-06-26 08:05:13 +05:30
sabaimran
a4b2552540 Update conversation session selection menu to use Obsidian theme colors as well 2024-06-26 08:05:13 +05:30
sabaimran
da5b07e913 Remove custom styling on the reference buttons 2024-06-26 08:05:13 +05:30
sabaimran
c4a1ae9375 Make the Khoj Obsidian plugin more user theme friendly
Use the CSS variables from the theme for the Khoj UI components
2024-06-26 08:04:17 +05:30
Debanjum Singh Solanky
d6fe5d9a63 Pass current component as arg to markdown renderer in chat view
This doesn't work on search modal, but hopefully will get resolved
once we migrate search into a view from a modal
2024-06-24 16:12:20 +05:30
Debanjum Singh Solanky
732332a3c5 Spell fix s/e.g/e.g./ across code, tests and docs 2024-06-24 15:24:45 +05:30
Debanjum Singh Solanky
8fc7f980aa Revert KHOJ_DOMAIN to only support single domain.
Multiple domain support didn't generalize to other portions where it
is used
2024-06-24 15:24:45 +05:30
sabaimran
939811e9b5 Fix conversation look up logic 2024-06-24 09:10:03 +05:30
Debanjum Singh Solanky
a4d88612c1 Just use yarn for package version locking. Remove npm package lock 2024-06-23 16:06:20 +05:30
Debanjum Singh Solanky
55be90cdd2 Sanitize user input fields on Automations page of web client
Use Dompurify to sanitize user input
2024-06-23 14:14:47 +05:30
Debanjum Singh Solanky
1c7a562880 Generate automation cards via DOM scripting 2024-06-23 13:22:38 +05:30
Debanjum Singh Solanky
c7c32a7467 Improve online chat reference extraction in Khoj.el Emacs package
- Handle online references with no title
- Improve handling references which are arrays instead of lists
2024-06-23 08:13:36 +05:30
Debanjum Singh Solanky
9d33d8c0fa Upgrade typescript eslint dev dependency of Khoj Obsidian plugin 2024-06-23 07:36:49 +05:30
Debanjum
a94062469a
Automatically Find Similar Notes on Emacs in Background (#827)
Khoj will find and display notes similar to the current entry in the side pane when
1. find similar is open in side pane and
2. cursor has moved to a new entry

### Major
- Find similar notes to current note at cursor automatically in background
- Only show headings of search result and increase default results count

### Minor
- Pass absolute path of file to index from khoj.el emacs client
- Update help message to only show the smaller set of new keybindings
- Fix edge cases in loading some chat sessions
2024-06-23 07:36:11 +05:30
sabaimran
a53178cab9
Add developer support for using next.js to serve generated static files (#814)
To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road.

Dev instructions for using the next.js setup are in the added README.

This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up.

This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
2024-06-22 20:12:41 +05:30
Debanjum Singh Solanky
abd6f58aee Upgrade Desktop app package dependencies 2024-06-22 17:38:52 +05:30
Debanjum Singh Solanky
f413dc62cd Upgrade Obsidian plugin dependencies. Add package lock file for it
Add it to bump_version script as well.
2024-06-22 17:38:52 +05:30
Debanjum Singh Solanky
7e277e9381 Fix getting file-toggle-button element in chat of web app 2024-06-21 15:54:38 +05:30
Debanjum Singh Solanky
fa7b40ab86 Automatically respond with Voice if subscribed user sent Voice message 2024-06-21 15:53:01 +05:30
Debanjum Singh Solanky
5e5fe4b7af Improve font size, spacing of conversation session on desktop app 2024-06-21 12:25:35 +05:30
sabaimran
d3c0111121 Include base URL when using openai api config in extract questions. Close #831 2024-06-21 12:18:50 +05:30
sabaimran
b9966eb3d4
Add support for text to speech in chat responses (#821)
* Enable speech to text responses in khoj chat

- Current issue: reads out all the markdown formatting, plus waits for the whole result to be streamed before playing it

* Extract content from markdown-formatted text

* Add a loader for while you're waiting for Khoj's response

* Add user configuration option for chat model options, allow server side configuration for option list

* Join up APIs, views, admin pages to allow configuring custom voice models
2024-06-21 11:30:28 +05:30
Debanjum Singh Solanky
427575e958 Improve khoj chat new, delete session flows
When create new conversation session, automatically request query. As
that is expected next action after creating new session

Pass session-id to khoj-chat to allow reuse from
create-new-conversation func

When delete conversation session, do not call load chat session.
Unnecessary action.

Use thread-last to improve code flow in new, delete conversation funcs
2024-06-21 10:54:59 +05:30
Debanjum Singh Solanky
59032a06d5 Improve defaults when extracting fields from online reference in khoj.el 2024-06-21 10:54:59 +05:30
Debanjum Singh Solanky
9262aea7a5 Fix comments, func calls based on melpazoid, checkdoc, package-lint 2024-06-21 10:54:59 +05:30
sabaimran
ff26b19d2b Add a migration for allowing the docx field in the entries file type 2024-06-21 09:47:49 +05:30
sabaimran
3cfe5aabe5
Add support for magic link email sign-in (#820)
* Add magic link email sign-in option

* Adding backend routes and model changes to keep state of email verification code and status

* Test and fix end to end email verification flow

* Add documentation for how to use the magic link sign-in when self-hosting Khoj

* Add magic link sign in to public conversation page
2024-06-20 13:32:58 +05:30
Debanjum Singh Solanky
0afe66ac39 Restore cursor to original window after opening Khoj side pane
Previously the cursor would move to the Khoj side pane on opening it.
This would break user's flow, especially when find similar triggers
automatically

New behavior maintains smoother update of auto find similar without
disrupting user browsing
2024-06-20 12:50:13 +05:30
Debanjum Singh Solanky
afe91a2633 Only show headings of search result and increase total count returned
Previously it would show complete result body this would make the
result width variable and hard to track all the returned results

Showing just heading makes it easier to track
2024-06-20 12:50:13 +05:30
Debanjum Singh Solanky
2b12a5514e Find similar notes to current note at cursor automatically in background
- Call find similar on current element if point has moved to new
  element
- Delete the first result from find-similar search results as that'll
  be the current note (which is trivially most similar to itself)
- Determine find-similar based text formating at the rendering layer
  rather than at the top level find-similar func
2024-06-20 12:50:13 +05:30
Raghav Tirumale
bd3b590153
Support Indexing Docx Files (#801)
* Add support for indexing docx files and associated unit tests

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-06-20 11:18:01 +05:30
Debanjum Singh Solanky
d042e073cc Pass absolute path of file to index from khoj.el emacs client 2024-06-20 00:26:18 +05:30
Debanjum Singh Solanky
d23f2849d4 Update help message to only show the smaller set of new keybindings 2024-06-20 00:26:18 +05:30
Raghav Tirumale
d4e5c95711
Add Ability to Summarize Documents (#800)
* Uses entire file text and summarizer model to generate document summary.
* Uses the contents of the user's query to create a tailored summary.
* Integrates with File Filters #788 for a better UX.
2024-06-18 19:31:07 +05:30
Debanjum Singh Solanky
677d49d438 Release Khoj version 1.14.0 2024-06-18 17:13:46 +05:30
Debanjum Singh Solanky
2930b57c78 Use hashed value to improve deduplication of search results on server 2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
6814dadd21 Fix opening Web, Desktop setup links on first run from Desktop app
Previous version failed to open the setup links
2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
632f55a9e8 Do not default to rerank if device has GPU 2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
f1120f24a1 Use solarized light css styling to highlight code in chat messages 2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
d8a5a01cea Pass multiple allowed Khoj domains via KHOJ_DOMAIN env var
To add multiple allowed Khoj domains pass them as a comma separated
list of domains via the KHOJ_DOMAIN environment variable

Resolve comment in issue #662
2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
4daf16e5f9 Only redirect to next url relative to current domain 2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
86a3505d89 Remove image HTML elements from non whitelisted sources in Obsidian chat
Given img src enforcement via CSP required loosening. Soft enforce it
via a regex replace of img HTML elements if the src isn't from the
whitelisted set of source prefixes.

Currently allowed source prefixes are
- app: for local images
- data: for inline generated images
- https://generated.khoj.dev: for cloud generated images
2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
c7d825bddb Sanitize markdown in Obsidian after conversion to HTML too
- Create and use a function to convert markdown to sanitized html
- Remove unused Latex delimiter handling as Katex isn't used in
  Khoj chat on Obsidian
2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
08c3aa496d Loosen CSP in Obsidian to load images, sync and allow Obsidian domain 2024-06-18 17:04:25 +05:30
sabaimran
ba0187798a Get converastion id before retrieving relevant notes in non-socket code 2024-06-17 14:26:06 +05:30
Debanjum
d2d9f4888e
Upgrade Khoj Emacs UX (#812)
- Open Khoj in Emacs Side pane
   Open Khoj chat, search in right pane to allow for ambient engagement
- Improve Khoj Chat
  - Show online references used for chat
  - Make chat API call async to not block user interactions
  - Fix loading chat history, references in khoj.el chat buffer
- Improve Khoj Search, Find Similar functions
   - Make calls to Khoj search API async to not block user interactions
- Support Conversation Sessions
  - Create transient menu to open, create, delete conversation sessions from the Khoj Emacs client
2024-06-16 10:39:48 +05:30
Debanjum Singh Solanky
fe36adb7b9 Remove short keys to switch content type during search to avoid conflict
- C-x o to switch to search org content conflicts with switch buffer shortkey
  This is more apparent in the async search scenario as it prevents
  perform other actions while async search is in progress

- Also switching content type wouldn't scale to all the content types
  Khoj will support without causing more conflicting keybinding
2024-06-15 17:31:19 +05:30
Debanjum Singh Solanky
2a84524d19 Make khoj.el search, similar API calls async to not block user interactions 2024-06-15 17:30:58 +05:30
Debanjum Singh Solanky
c6b95f8776 Handle rendering messages using the old reference schema in khoj.el
Previously references were a list instead of a map
2024-06-15 17:30:58 +05:30
Debanjum Singh Solanky
db056c896d Delete old conversation sessions from the chat menu in Khoj Emacs 2024-06-15 17:30:58 +05:30
Debanjum Singh Solanky
e3d995a74f Extract select conversation session logic into func for reusability 2024-06-15 17:30:38 +05:30
Debanjum Singh Solanky
e15dc23bbe Improve logic to create vs reuse window for khoj side pane logic
Khoj side pane occupies a vertically split bottom right side pane.
If the bottom right window is not a vertical split, create a new
vertical split pane for khoj, otherwise reuse the existing window
2024-06-15 16:37:41 +05:30
Debanjum Singh Solanky
055e5e8d26 Create new conversation from the chat menu in Khoj Emacs 2024-06-15 16:37:41 +05:30
Debanjum Singh Solanky
c33954cd93 Fix loading an empty chat session in Emacs 2024-06-15 16:37:41 +05:30
Debanjum Singh Solanky
e21c0648ae Create, use reusable function to call Khoj API from elisp 2024-06-15 16:37:41 +05:30
Debanjum Singh Solanky
7bcb49b6e7 Support conversation sessions in the Khoj Emacs client
Add option in khoj main transient menu option to open menu to
- switch between existing conversations
2024-06-15 13:13:20 +05:30
Debanjum Singh Solanky
df9c5ff263 Show online references used for chat response as footnotes in Emacs
Previously online references used weren't being shown
2024-06-15 13:13:19 +05:30
sabaimran
25d8cdd9cd Misc fixes:
- Fix getting file filters for not found conversations
- Allow iamge rendering in automation emails
- Fix nearest 15th minute calculation in automations creation
2024-06-14 16:20:22 +05:30
Raghav Tirumale
35715096f4
UX Improvement: Keyboard Shortcuts for Recent Messages (#804)
* added keyboard shortcuts to access old queries
2024-06-14 12:45:09 +05:30
sabaimran
2dcfb3c2f0 Fix bug for drag and drop single file 2024-06-14 12:01:10 +05:30
sabaimran
7e4a61f2ac Disable rate limiting if billing is not enabled 2024-06-12 21:39:02 +05:30
Debanjum Singh Solanky
385057f09e Make khoj.el chat API call async to not block user interactions 2024-06-12 21:04:48 +05:30
sabaimran
45e725ac9c Use the summarizer model for generating improved image prompts 2024-06-12 17:41:12 +05:30
Raghav Tirumale
673d0d367c
Fix: Adding Support for Uploading Multiple Files (#803)
* added support for uploading multiple files at a time.

* optimized multiple file upload to use a batch upload

* allowing files to upload even if there is one unsupported file
2024-06-12 15:51:35 +05:30
Debanjum Singh Solanky
906ebee075 Open Khoj chat, search in right pane to allow for ambient engagement
See the currently active window in context while doing chat, search
or find similar operations in a side pane.

This is similar to how we've moved Khoj on Obsidian into the side pane
as well
2024-06-09 23:32:34 +05:30
Debanjum Singh Solanky
cd4baa3fa5 Fix loading chat history, references in khoj.el chat buffer 2024-06-09 18:34:00 +05:30
Debanjum
6afbd8032e
Improve Intermediate Steps in Formulating Chat Response (#799)
# Major
- Disambiguate Text output mode to disambiguate from Default data source lookup
- Fix showing headings in intermediate step in generating chat response
- Remove "Path" prefix from org ancestor heading in compiled entry

# Minor
- Fix OpenAI chat actor, director unit tests
2024-06-09 07:55:01 +05:30
sabaimran
2e209ab28b Handle case where conversation does not (yet) exist 2024-06-08 16:22:12 +05:30
sabaimran
849c38c0a4 Add support for managing audiences for new users 2024-06-08 15:51:17 +05:30
sabaimran
06a47ee457
Add language-specific syntax highlighting via highlight.js (#802)
* Add language-specific syntax highlighting via highlight.js

- Add highlight.js to our assets CDN for fast load and compliance with the CSP
- See other stylesheets options here: https://cdnjs.com/libraries/highlight.js

* Bonus: set min-height to prevent increasing length of the sessions pane

* Fix references rendering and add highlight.js in public conversation
2024-06-08 15:17:09 +05:30
sabaimran
dbb06466bf Minor fit/finish updates to the file filter experience 2024-06-07 15:05:00 +05:30
sabaimran
58a02f06ea
Fix multilingual font rendering (#797)
* Fix multilingual font rendering; fallback to an Arabic language font which contains more Asian characters. Close #756

* Tune font-sizes and styling to accomodate new fonts with old sizing

- Move connection-status styling out from inline html into css block
- Remove start typing chat-input height jitter
- align new-conversation button, text
- use relative font sizes instead of absolute font sizes in most places

---------

Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
2024-06-07 11:53:47 +05:30
Raghav Tirumale
ba16afd3c2
New Feature: Adding File Filtering to Conversations (#788)
* UI update for file filtered conversations
* Interactive file menu #UI to add/remove files on each conversation as references.
* Backend changes implemented to load selected file filters from a conversation into the querying process.
---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-06-07 10:53:37 +05:30
Debanjum Singh Solanky
f91cdf8e18 Fix showing headings in intermediate step in generating chat response 2024-06-06 16:52:23 +05:30
Debanjum Singh Solanky
18f7e6e7ed Remove "Path" prefix from org ancestor heading in compiled entry 2024-06-06 16:51:26 +05:30
sabaimran
8d701ebe22
Add fedCM to accommodate google migration (#798)
- See migration guidelines here: https://developers.google.com/identity/gsi/web/guides/fedcm-migration#fedcm_flag
2024-06-06 14:23:16 +05:30
Debanjum Singh Solanky
dd2225b1aa Use Text output mode to disambiguate from Default data source lookup
Previously if default output was selected by Khoj, we'd end up doing
an documents search as well, even when Khoj selected internet or
general data source to lookup.

This update disambiguates the default information mode from the text
output mode. To avoid doing documents search when not deemed necessary
by Khoj
2024-06-06 11:56:48 +05:30
Debanjum Singh Solanky
a1e4f4bde7 Gracefully skip indexing when empty list of docs provided
Improve error message when fail to index content
2024-06-05 19:39:15 +05:30
Debanjum Singh Solanky
21987f60c7 Use `-difference' to get files to delete. Make batch size defcustom
Improve docstrings to align with `checkdoc' requirement for all args
being mentioned
2024-06-05 19:39:15 +05:30
Debanjum
bfacd65971
Batch upload files for indexing from the Emacs client (#735) from yuzhou721/master
Encode filenames and batch file uploads to improve sending content to index from the Emacs client
2024-06-05 19:31:06 +05:30
sabaimran
a9c383e62c Use an ASGI application, rather than WSGI
- ASGI should be the preferred application, as our codebase runs a lot of async code
2024-06-05 09:25:08 +05:30
sabaimran
0816cec4bc Manually close old db connections periodically 2024-06-04 22:19:47 +05:30
sabaimran
acfdc8da77 Explicitly set the connection age to 0 in the django settings. Seems to be some strange behavior with async gunicorn + django db 2024-06-04 20:31:51 +05:30
Debanjum Singh Solanky
85a343363b Release Khoj version 1.13.0 2024-06-04 11:57:44 +05:30
Debanjum Singh Solanky
b757ba664f Sanitize chat messages to render in Obsidian, Desktop, Web apps
Use DOMPurify to escape any unsafe HTML in chat message before adding
it to DOM via innerHTML updates to a HTML element
2024-06-04 10:53:30 +05:30
Debanjum Singh Solanky
9f80c2ab76 Enforce Content-Security-Policy (CSP) in Obsidian, Desktop, Web apps
Prevent XSS attacks by enforcing Content-Security-Policy (CSP) in apps.
Do not allow loading images, other assets from untrusted domains.

- Only allow loading assets from trusted domains
  like 'self', khoj.dev, ipapi for geolocation, google (fonts, img)
  - images from khoj domain, google (for profile pic)
  - assets from khoj domain
  - Do not allow iframe src
  - Allow unsafe-inline script and styles for now as markdown-it escapes html
    in user, khoj chat

- Add hostURL to CSP of the Desktop, Obsidian apps
  Given web client is served by khoj server, it doesn't need to
  explicitly allow for khoj.dev domain. So if user self-hosting, it'll
  automatically allow the domain in the CSP (via 'self')

  Whereas the Obsidian, Desktop clients allow configure the server URL.
  Note *switching server URL breaks CSP until app is reloaded*
2024-06-04 10:53:30 +05:30
Debanjum Singh Solanky
bbcdb8413d Add null checks, fix build errors in Khoj plugin on newer Obsidian 2024-06-03 18:03:11 +05:30
Debanjum Singh Solanky
d8ace4d34c Highlight the agents, automation tab when active on the web app 2024-06-03 16:57:03 +05:30
sabaimran
4679f07336 Clean up some of the design of agents, inspired by dicussion #792 2024-06-03 12:52:07 +05:30
Debanjum Singh Solanky
8cdab5f31a Update slash command UX in chat UI of desktop app to match web app
Make commands in popup menu on typing slash in chat input selectable
2024-06-02 17:27:37 +05:30
Debanjum Singh Solanky
7828bd6f2e Hide command popup & focus on chatInput on selecting command in web app
Style command popup cursor and add highlight to indicate using slash
command
2024-06-02 17:27:37 +05:30
Debanjum
cf8c9c2a3d
Serve image assets from Khoj domain, not directly from S3 bucket (#734)
- Serve generated images from Khoj domain instead of directly from AWS S3
- Rename assets URL from Khoj S3 bucket to assets.khoj.dev
2024-06-02 17:24:35 +05:30
sabaimran
5bb3689562 Do not stream responses in the scheduled_chat response 2024-06-02 11:31:15 +05:30
sabaimran
5132b01ab1 Remove intent_type from telemetry update in api_chat 2024-06-02 10:21:38 +05:30
Raghav Tirumale
a3934b3aaa
Improved Command Menu and Help Command (#774)
* The command menu (triggered by "/") now has a clickable list of possible commands, that automatically fill into the chat when pressed.
* The `/help` command now searches `khoj.dev` pages to provide useful assistance to the user.

---------

Co-authored-by: raghavt3 <raghavt3@illinois.edu>
Co-authored-by: sabaimran <65192171+sabaimran@users.noreply.github.com>
2024-06-01 22:33:31 +05:30
sabaimran
89178bcebd Fix formatting issues for task email in mobile 2024-06-01 14:19:12 +05:30
Debanjum
b499b3fe2a
Upgrade Khoj Obsidian: Chat from Side Pane, Stream Intermediate Steps, Copy Message to Clipboard (#736)
### Details
- **Chat with Khoj from right pane on Obsidian**
  - Modal was too ephemeral, couldn't have it open for reference, quick jump to Khoj chat
- **Stream intermediate steps taken by Khoj** for generating response to the chat pane
  Gives more transparency into Khoj 'thinking' process, e.g internet, notes searches performed, documents read etc. 
  The feedback allows us to tune our messages to elicit better responses by Khoj
- Add ability to **copy message to clipboard, paste chat messages directly into current file**
- Jump to **Search**, **Find Similar** functions from navigation bar on the Khoj Obsidian side pane
- Improve spacing, use consistent colors in chat message references and buttons

Resolves #789, #754
2024-06-01 13:29:21 +05:30
sabaimran
8b9c26c468 Remove unused method 2024-06-01 12:54:43 +05:30
sabaimran
5ec641837a
Allow automations to be shareable (#790)
* Updating the API / UI to support sharing of automations
* Allow people to see the automations even when not logged in, and add an overlay effect
* Handle unauthenticated users taking actions
* Support showing pre-filled automation details on the config automations page
* Redirect user to login if they try to add an automation while unauthenticated
2024-06-01 12:44:49 +05:30
Debanjum Singh Solanky
7d7d4cf5c3 Make new chat message text selectable in Obsidian side pane
Resolves #789
2024-06-01 11:01:39 +05:30
Debanjum Singh Solanky
7fb7f200b3 Fix rendering text in chat messages with bulleted lists
Improves #789
2024-06-01 10:51:22 +05:30
Debanjum Singh Solanky
7a93599fe8 Merge branch 'master' into upgrade-khoj-on-obsidian
- Conflicts:
  - src/khoj/interface/web/chat.html
    Use our changes with feedback button changes from master
2024-06-01 10:07:43 +05:30
Debanjum Singh Solanky
92bab9fa61 Get Conversation session action buttons out from under the three dot menu 2024-05-31 20:11:00 +05:30
Debanjum Singh Solanky
7fa42daf89 Render action buttons for new Khoj chat responses in Obsidian
- Dedupe the code to add action buttons to chat messages
- Update the renderIncrementalMessage function to also add the action
  buttons to newly generated chat messages by Khoj
2024-05-31 20:11:00 +05:30
Debanjum Singh Solanky
2d010db83f Toggle chat session view on clicking the Obsidian chat sessions button 2024-05-31 20:11:00 +05:30
Debanjum Singh Solanky
275d4877a6 Fix loading spinner visibility by using contrasting background color
Fix code formating of Khoj chat view in Obsidian
2024-05-31 20:09:24 +05:30
sabaimran
2667ef4544 Refresh the conversation from the db in the websocket flow 2024-05-31 16:15:56 +05:30
sabaimran
fd07abbfc8 Decrease the life of one connection 2024-05-31 15:39:15 +05:30
Debanjum
3090b84252
Disable Minutely Recurrence for Automations (#781)
* Disable automation recurrence at minute level frequency

* Set a max lifetime for django's connections to the db

* Disable any automation that has a non-numeric first digit (i.e., recuring on the minute level)

* Re-enable automations

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-05-31 12:50:19 +05:30
sabaimran
5dca48d9fc Fix setting of conn_max_age variable 2024-05-31 11:07:13 +05:30
sabaimran
76f941f4e5 Revert email from from to sender again in resend API. keeps switching? 2024-05-31 10:30:18 +05:30
sabaimran
b27f59b12b Remove all unused code related to websockets 2024-05-30 11:39:04 +05:30
sabaimran
4b3d3fe7ea /s/sender/from in resend calls 2024-05-30 08:43:46 +05:30
sabaimran
2076543e32 Disable AP Scheduler while performing maintenance 2024-05-30 08:02:59 +05:30
Debanjum Singh Solanky
7823ef09dc Simplify conditional code. Improve logs to track conversion progress 2024-05-29 17:50:07 +05:30
Debanjum Singh Solanky
215db8cab3 Reduce log level of noisy process lock logs 2024-05-29 13:14:44 +05:30
Debanjum Singh Solanky
7b18919564 Tag external links to open in a separate window on the Desktop app
Previously clicking inline links would open the URL directly in the
Desktop app. This was strange and it didn't provide any way to go back
to Khoj desktop app UI from the opened link
2024-05-29 10:12:50 +05:30
Debanjum Singh Solanky
c957a6cb43 Delete unused base_processor_integration html file from web interface 2024-05-29 08:30:13 +05:30
sabaimran
cb33fb67fe Remove the automations-related dead code in the web config 2024-05-29 04:22:45 +05:30
Debanjum Singh Solanky
7594401461 Fix expand chat reference animation in web, desktop, obsidian clients 2024-05-28 20:56:26 +05:30
Debanjum Singh Solanky
1ea7675fc9 View, switch chat sessions from Obsidian chat pane 2024-05-28 20:33:39 +05:30
Debanjum Singh Solanky
e86899eec4 Click on referenced notes by Khoj chat to open it in Obsidian vault
Allow opening Khoj chat references in Obsidian vault if the reference
is a heading or file in the current Obsidian vault
2024-05-28 10:16:40 +05:30
Desmond
70fea6c6b6 fix: delete file request 2024-05-27 14:46:26 +08:00
sabaimran
607534021b Add a link to github in the settings menu, improve styling 2024-05-27 11:39:30 +05:30
Desmond
3f49b5a4ab fix: emacs tests 2024-05-27 10:42:09 +08:00
sabaimran
b97ca9d19d Skip using max_tokens as input to the extract questions step, as that's not used for max_output 2024-05-27 01:23:54 +05:30
sabaimran
9ebf3a4d80 Improve the admin experience, add more metadata to the list_display
- Don't propagate max_tokens to the openai chat completion method. the max for the newer models is fixed at 4096 max output. The token limit is just used for input
2024-05-27 00:49:20 +05:30
sabaimran
01cdc54ad0
Add support for Anthropic models (#760)
* Add support for chatting with Anthropic's suite of models

- Had to use a custom class because there was enough nuance with how the anthropic SDK works that it would be better to simply separate out the logic. The extract questions flow needed modification of the system prompt in order to work as intended with the haiku model
2024-05-26 22:50:34 +05:30
Debanjum Singh Solanky
0f796a79ec Extract function to get link to entry in Obsidian vault for reuse 2024-05-26 18:03:15 +05:30
Debanjum Singh Solanky
e24ca9ec28 Pass file path of each doc reference in references returned by API
- Pass file path of reference along with the compiled reference in
  list of references returned by chat API converts
- Update the structure of references from list of strings to list of
  dictionary (containing 'compiled' and 'file' keys)
- Pull out the compiled reference from the new references data struct
  wherever it was is being used
2024-05-26 18:02:11 +05:30
Debanjum Singh Solanky
ba330712f8 Fix to always pass online results in chat API response 2024-05-26 13:56:55 +05:30
Debanjum Singh Solanky
38d8d2bb56 Show online references used to generate response in Obsidian chat view 2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
f495d338eb Modularize render message with references func in web based clients
Simplify, reuse, standardize code to render messages with references
in the obsidian, web and desktop clients. Specifically:

- Reuse function to create reference section, dedupe code
- Create reusable function to generate image markdown
- Simplify logic to render message with references
2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
14a2006c76 Stream steps taken to generate response in Obsidian chat pane
- Setup websocket using Khoj web app as reference.
- Moved the geolocating code to chat view out from the general pane
  view
- Use loading spinner from web instead of the thinking emoji
2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
afcd22d30c Improve spacing, colors of chat message references and buttons
Works better with dark modes. References have more spacing and adhere
to background color of the chat message itself
2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
bd4931e70b Add ability to paste chat messages directly into current file
It'll replace any highlighted text with the chat message or if not
text is highlighted, it'll insert the chat message at the last cursor
position in the active file
2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
032ad3b521 Add ability to copy messages to clipboard from Obsidian Khoj chat 2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
57f1c53214 Create Nav bar for Obsidian pane. Use abstract View class for reuse
- Jump to chat, show similar actions from nav menu of Khoj side pane
  - Add chat, search icons from web, desktop app
  - Use lucide icon for find similar (for now)
  - Match proportions of find similar icon to khoj other icons via css, js

- Use KhojPaneView abstract class to allow reuse of common functionality like
  - Creating the nav bar header in side pane views
  - Loading geo-location data for chat context
  This should make creating new views easier
2024-05-26 13:55:22 +05:30
sabaimran
e23c803cee Release Khoj version 1.12.1 2024-05-24 21:42:03 +05:30
sabaimran
0308699849 Use links from assets.khoj.dev to render images in the automations page 2024-05-24 20:18:02 +05:30
sabaimran
3f9c20a399
Make it easier to manage server-level chat settings (#729)
* Add support for server-wide model settings fix web page reading results returning logic
2024-05-24 20:15:18 +05:30
sabaimran
cbbbe2da9a
Add a schedule picker and automations preview func (#747)
* Update suggested automations
* add a schedule picker when creating an automation
* Create a new conversation in flow of the automation scheduling in order to send a preview and deliver more consistent results
* Start adding in scaffolding to manually trigger a test job for an automation
* Add support for manually triggering automations for testing
* Schedule automation asynchronously
* Update styling of the preview button
* Improve admin lookup experience and prevent jobs from being scheduled to run every minute of everyday
* Ignore mypy issues on job info short description
2024-05-24 19:42:47 +05:30
sabaimran
4511c6ae7c Fix bug in chat feedback flow - user message not included during live chat 2024-05-21 14:55:39 -05:00
Desmond
a3c6045328 Merge remote-tracking branch 'origin/master' 2024-05-21 21:55:53 +08:00
Desmond
b0630c1a98 Simplify partition 2024-05-21 21:52:01 +08:00
Raghav Tirumale
d57772f9e7
Add Feedback Buttons on Chat (#721)
### Description and Rationale for Changes
This feature includes thumbs up and thumbs down buttons on Khoj's chat responses that provide automated feedback. When a thumbs up/down button is clicked, the code sends an email to team@khoj.dev with the following:
* user query
* khoj's response
* whether the sentiment of the user was good or bad. 

This is critical in improving Khoj's nondeterministic LLM model for a better user experience.
### List of Changes
* new endpoint in `api_chat.py` (/feedback) that can be used to trigger mail sending).
* thumbs up and thumbs down buttons implemented in `chat.html`
* new function in `routers/email.py` to handle feedback email sending via resend
* `feedback.html` template for a formatted email with the feedback.

---------

Co-authored-by: mythicalcow <mythicalcow@linux.myguest.virtualbox.org>
Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-05-20 16:29:08 -05:00
sabaimran
7feaf34702 Fix capitalization, update suggeted prompt 2024-05-10 02:36:13 -07:00
sabaimran
b545aceb47 Use a simpler example for the sample automation and put schedule on top of instructions 2024-05-09 13:53:19 -07:00
sabaimran
7ae00832bd Rname from parameter to sender in resend call 2024-05-09 13:29:39 -07:00
sabaimran
fbd76f8ebe
Improve the UX of automations (#737)
* Improve the automations UX

- Add suggested jobs to elimiinate some of the cold start problem
- Make each of the tasks cards that are clickable/editable

* Hide suggested automations that have already been added

* Add a footer and reapply styling when a save action is taken on a card
2024-05-09 01:29:48 -07:00
sabaimran
70d0ee4310 Only remove the process lock from a process that created it 2024-05-08 10:14:52 -07:00
Desmond Deng
20303feb3a
Merge branch 'khoj-ai:master' into master 2024-05-08 13:46:34 +08:00
Desmond
150cd18bf3 Update batch-size 2024-05-08 13:44:22 +08:00
Desmond
192cd53003 Batch send of index files 2024-05-08 13:38:40 +08:00
sabaimran
a50deb2762 Add better handling for empty responses 2024-05-07 11:49:33 -07:00
sabaimran
4aed6bd274 Add an admin view for subscriptions 2024-05-07 11:48:52 -07:00
sabaimran
77626d28d1 Include stack trace when automation is not successfully craeted 2024-05-07 06:52:41 -07:00
sabaimran
0c8c565ab0 Don't include the whole stack trace for an integrity error 2024-05-07 06:48:18 -07:00
Debanjum Singh Solanky
0a1a6cd041 Get detailed user info in Obsidian from the new v1/user API
Previously we were just getting user email from the /health API
Instead store the retrieved user info in the user settings
2024-05-07 04:37:26 +08:00
Debanjum Singh Solanky
f8f9d066db Focus on input field, scroll to latest message on opening chat pane
Previously scroll and chat input focus weren't applied as view hadn't
been rendered yet
2024-05-07 04:37:26 +08:00
Debanjum Singh Solanky
9f65e8de98 Open Khoj Chat as a Pane instead of a Modal
- Allows having it open on the side as you traverse your Obsidian notes
- Allow faster time to response, having responses visible for context
- Enables ambient interactions
2024-05-07 04:37:26 +08:00
sabaimran
9ae828cf11 Use asssets.khoj.dev for loading math katex rendering 2024-05-07 01:43:46 +08:00
sabaimran
cf0b7628d0 Add the url scheme to the public share url 2024-05-06 21:37:49 +08:00
sabaimran
f6aaecb04f Fix construction method for public share conversation URL 2024-05-06 08:32:51 +05:30
sabaimran
14c9bea663
Make conversations optionally shareable (#712)
* Make conversations optionally shareable

- Shared conversations are viewable by anyone, without a login wall
- Can share a conversation from the three dot menu
- Add a new model for Public Conversation
- The rationale for a separate model is that public and private conversations have different assumptions. Separating them reduces some of the code specificity on our server-side code and allows us for easier interpretation and stricter security. Separating the data model makes it harder to accidentally view something that was meant to be private
- Add a new, read-only view for public conversations
2024-05-05 23:16:04 +05:30
Debanjum Singh Solanky
80cbaca935 Serve generated images from Khoj domain instead of directly from S3
Use CNAME to forward requests from the khoj subdomain to the
equivalent S3 bucket
2024-05-04 20:07:10 +05:30
Debanjum Singh Solanky
425496844b Rename assets URL from Khoj S3 bucket to assets.khoj.dev
Server khoj assets from khoj domain
2024-05-04 20:07:10 +05:30
sabaimran
88daa841fd Rename process lock migration and add a reverse migration step 2024-05-04 20:05:00 +05:30
sabaimran
509a8a412c Throw an error if trying to create a process lock that already exists. Names should be unique 2024-05-04 19:03:53 +05:30
sabaimran
7100614de5
Add support for rendering math equations in the web view (#733)
- Add parsing logic for LaTeX-format math equations in the web chat
- Add placeholder delimiters when converting the markdown to HTML in order to avoid removing the escaped characters
- Add the `<!DOCTYPE html>` specification to the page
2024-05-04 15:59:17 +05:30
Debanjum Singh Solanky
d9b3482b1a Show error when required fields to create automation are not set 2024-05-04 11:17:30 +05:30
Debanjum Singh Solanky
91a5643c5c Use Preview label for Automate feature. Prefix mailto: link to contact 2024-05-04 10:59:17 +05:30
Debanjum Singh Solanky
fd2328ab40 Do not hard code base url of path to automation icon in chat message 2024-05-04 10:59:07 +05:30
sabaimran
a38f3227e2 Revert domain in task task send emails 2024-05-03 15:27:27 +05:30
sabaimran
a1263951e9 Use mail to in email contact link 2024-05-03 12:16:56 +05:30
sabaimran
7c9847fe48 Increase jitter to 60 2024-05-03 11:38:22 +05:30
sabaimran
737ebfd521 Make improvements to online search prompts and use a custom domain for automations emails 2024-05-03 10:47:42 +05:30
sabaimran
42e9504ba8 Use a different function for getting last run time, avoid async/sync issues 2024-05-02 12:13:45 +05:30
sabaimran
9e8491b814 Add experimental disclaimers to the automations 2024-05-02 11:40:37 +05:30
sabaimran
c418449311 Add additional robustness in verifying job execution parameters at run time 2024-05-02 11:13:04 +05:30
sabaimran
690e9d8ed3 Collapse the reminders after they're successfully scheduled 2024-05-02 09:55:04 +05:30
sabaimran
6b648ee3ad Add experimental disclaimer in the automation page 2024-05-02 09:21:27 +05:30
sabaimran
f4fbc91515 Remove the exclamation point from the email 2024-05-01 19:01:51 +05:30
sabaimran
bddd1d0fcb Quip, smart reminders 2024-05-01 16:39:07 +05:30
sabaimran
bc8b92a77d Release Khoj version 1.12.0 2024-05-01 16:30:48 +05:30
sabaimran
b499851097 Use the cleaned query as the reference query in the email notification 2024-05-01 15:33:11 +05:30
sabaimran
f24495e0e6 Fix time zone used in query history. Closes #694 2024-05-01 15:31:48 +05:30
sabaimran
7fd57d737e Adjustments to improve overall styling of config page, email template 2024-05-01 14:19:47 +05:30
sabaimran
28578310d1 Add log line when sending a task-related email 2024-05-01 13:56:02 +05:30
sabaimran
a86f95117e Add the subject generation prompt and helper method 2024-05-01 13:55:32 +05:30
sabaimran
c30ba2e551 Set subject dynamically when creating new tasks, and make some minor improvments to the automations UI 2024-05-01 13:54:59 +05:30
sabaimran
d1b2037676 Shutdown the scheduler when the application is exiting 2024-05-01 13:53:34 +05:30
Debanjum Singh Solanky
89a8dbb81a Fix edit job API. Use user timezone, pass all reqd. params to automation
- Pass user and calling_url to the scheduled chat too when modifying
  params of automation
- Update to use user timezone even when update job via API
- Move timezone string to timezone object calculation into the
  schedule automation method
2024-05-01 10:29:49 +05:30
Debanjum Singh Solanky
19c5af3ebc Handle natural language to cron translation error on web client 2024-05-01 09:10:18 +05:30
Debanjum Singh Solanky
70ee9ddf91 Merge migrations from main with feature branch 2024-05-01 09:10:18 +05:30
Debanjum Singh Solanky
8f28f6cc1e Remove now unused location data from being passed to automation funcs 2024-05-01 08:48:16 +05:30
Debanjum Singh Solanky
815966cb25 Unify, modularize DB adapters to get automation metadata by user further 2024-05-01 08:47:50 +05:30
Debanjum Singh Solanky
21bdf45d6f Add link to Automate page in nav pane of the web app 2024-05-01 08:47:50 +05:30
Debanjum Singh Solanky
bd5008136a Move automations into independent page. Allow direct automation
- Previously it was a section in the settings page. Move it to
  independent, top-level page to improve visibility of feature

- Calculate crontime from natural language on web client before
  sending it to to server for saving new/updated schedule to disk.
  - Avoids round-trip of call to chat model

- Convert POST /api/automation API endpoint into a direct request for
  automation with query_to_run, subject and schedule provided via the
  automation page. This allows more granular control to create automation
  - Make the POST automations endpoint more robust; runs validation
    checks, normalizes parameters
2024-05-01 08:47:48 +05:30
Debanjum Singh Solanky
cbc8a02179 Make, use func for constructing the automation created response
- Dedupe logic across http, ws chat API endpoints
- Reduces size of already too long http, ws chat API endpoint funcs
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
c52ed333fa Make content, cards on config pages occupy the whole middle column
- Make the config page content use  the same top level 3-column layout
  as the khoj-header-wrapper
  This ensures the content is aligned with heading pane width
- Let cards and other settings sections scale to the width of their
  grid element. This utilizes more of the screen space and does it
  consistently across the different settings pages
2024-05-01 08:30:10 +05:30
sabaimran
ad4145e48c Fix unique has for job id 2024-05-01 08:30:10 +05:30