Commit graph

2076 commits

Author SHA1 Message Date
Desmond
3f49b5a4ab fix: emacs tests 2024-05-27 10:42:09 +08:00
sabaimran
b97ca9d19d Skip using max_tokens as input to the extract questions step, as that's not used for max_output 2024-05-27 01:23:54 +05:30
sabaimran
9ebf3a4d80 Improve the admin experience, add more metadata to the list_display
- Don't propagate max_tokens to the openai chat completion method. the max for the newer models is fixed at 4096 max output. The token limit is just used for input
2024-05-27 00:49:20 +05:30
sabaimran
01cdc54ad0
Add support for Anthropic models (#760)
* Add support for chatting with Anthropic's suite of models

- Had to use a custom class because there was enough nuance with how the anthropic SDK works that it would be better to simply separate out the logic. The extract questions flow needed modification of the system prompt in order to work as intended with the haiku model
2024-05-26 22:50:34 +05:30
Debanjum Singh Solanky
0f796a79ec Extract function to get link to entry in Obsidian vault for reuse 2024-05-26 18:03:15 +05:30
Debanjum Singh Solanky
e24ca9ec28 Pass file path of each doc reference in references returned by API
- Pass file path of reference along with the compiled reference in
  list of references returned by chat API converts
- Update the structure of references from list of strings to list of
  dictionary (containing 'compiled' and 'file' keys)
- Pull out the compiled reference from the new references data struct
  wherever it was is being used
2024-05-26 18:02:11 +05:30
Debanjum Singh Solanky
ba330712f8 Fix to always pass online results in chat API response 2024-05-26 13:56:55 +05:30
Debanjum Singh Solanky
38d8d2bb56 Show online references used to generate response in Obsidian chat view 2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
f495d338eb Modularize render message with references func in web based clients
Simplify, reuse, standardize code to render messages with references
in the obsidian, web and desktop clients. Specifically:

- Reuse function to create reference section, dedupe code
- Create reusable function to generate image markdown
- Simplify logic to render message with references
2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
14a2006c76 Stream steps taken to generate response in Obsidian chat pane
- Setup websocket using Khoj web app as reference.
- Moved the geolocating code to chat view out from the general pane
  view
- Use loading spinner from web instead of the thinking emoji
2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
afcd22d30c Improve spacing, colors of chat message references and buttons
Works better with dark modes. References have more spacing and adhere
to background color of the chat message itself
2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
bd4931e70b Add ability to paste chat messages directly into current file
It'll replace any highlighted text with the chat message or if not
text is highlighted, it'll insert the chat message at the last cursor
position in the active file
2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
032ad3b521 Add ability to copy messages to clipboard from Obsidian Khoj chat 2024-05-26 13:55:22 +05:30
Debanjum Singh Solanky
57f1c53214 Create Nav bar for Obsidian pane. Use abstract View class for reuse
- Jump to chat, show similar actions from nav menu of Khoj side pane
  - Add chat, search icons from web, desktop app
  - Use lucide icon for find similar (for now)
  - Match proportions of find similar icon to khoj other icons via css, js

- Use KhojPaneView abstract class to allow reuse of common functionality like
  - Creating the nav bar header in side pane views
  - Loading geo-location data for chat context
  This should make creating new views easier
2024-05-26 13:55:22 +05:30
sabaimran
e23c803cee Release Khoj version 1.12.1 2024-05-24 21:42:03 +05:30
sabaimran
0308699849 Use links from assets.khoj.dev to render images in the automations page 2024-05-24 20:18:02 +05:30
sabaimran
3f9c20a399
Make it easier to manage server-level chat settings (#729)
* Add support for server-wide model settings fix web page reading results returning logic
2024-05-24 20:15:18 +05:30
sabaimran
cbbbe2da9a
Add a schedule picker and automations preview func (#747)
* Update suggested automations
* add a schedule picker when creating an automation
* Create a new conversation in flow of the automation scheduling in order to send a preview and deliver more consistent results
* Start adding in scaffolding to manually trigger a test job for an automation
* Add support for manually triggering automations for testing
* Schedule automation asynchronously
* Update styling of the preview button
* Improve admin lookup experience and prevent jobs from being scheduled to run every minute of everyday
* Ignore mypy issues on job info short description
2024-05-24 19:42:47 +05:30
sabaimran
4511c6ae7c Fix bug in chat feedback flow - user message not included during live chat 2024-05-21 14:55:39 -05:00
Desmond
a3c6045328 Merge remote-tracking branch 'origin/master' 2024-05-21 21:55:53 +08:00
Desmond
b0630c1a98 Simplify partition 2024-05-21 21:52:01 +08:00
Raghav Tirumale
d57772f9e7
Add Feedback Buttons on Chat (#721)
### Description and Rationale for Changes
This feature includes thumbs up and thumbs down buttons on Khoj's chat responses that provide automated feedback. When a thumbs up/down button is clicked, the code sends an email to team@khoj.dev with the following:
* user query
* khoj's response
* whether the sentiment of the user was good or bad. 

This is critical in improving Khoj's nondeterministic LLM model for a better user experience.
### List of Changes
* new endpoint in `api_chat.py` (/feedback) that can be used to trigger mail sending).
* thumbs up and thumbs down buttons implemented in `chat.html`
* new function in `routers/email.py` to handle feedback email sending via resend
* `feedback.html` template for a formatted email with the feedback.

---------

Co-authored-by: mythicalcow <mythicalcow@linux.myguest.virtualbox.org>
Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-05-20 16:29:08 -05:00
sabaimran
7feaf34702 Fix capitalization, update suggeted prompt 2024-05-10 02:36:13 -07:00
sabaimran
b545aceb47 Use a simpler example for the sample automation and put schedule on top of instructions 2024-05-09 13:53:19 -07:00
sabaimran
7ae00832bd Rname from parameter to sender in resend call 2024-05-09 13:29:39 -07:00
sabaimran
fbd76f8ebe
Improve the UX of automations (#737)
* Improve the automations UX

- Add suggested jobs to elimiinate some of the cold start problem
- Make each of the tasks cards that are clickable/editable

* Hide suggested automations that have already been added

* Add a footer and reapply styling when a save action is taken on a card
2024-05-09 01:29:48 -07:00
sabaimran
70d0ee4310 Only remove the process lock from a process that created it 2024-05-08 10:14:52 -07:00
Desmond Deng
20303feb3a
Merge branch 'khoj-ai:master' into master 2024-05-08 13:46:34 +08:00
Desmond
150cd18bf3 Update batch-size 2024-05-08 13:44:22 +08:00
Desmond
192cd53003 Batch send of index files 2024-05-08 13:38:40 +08:00
sabaimran
a50deb2762 Add better handling for empty responses 2024-05-07 11:49:33 -07:00
sabaimran
4aed6bd274 Add an admin view for subscriptions 2024-05-07 11:48:52 -07:00
sabaimran
77626d28d1 Include stack trace when automation is not successfully craeted 2024-05-07 06:52:41 -07:00
sabaimran
0c8c565ab0 Don't include the whole stack trace for an integrity error 2024-05-07 06:48:18 -07:00
Debanjum Singh Solanky
0a1a6cd041 Get detailed user info in Obsidian from the new v1/user API
Previously we were just getting user email from the /health API
Instead store the retrieved user info in the user settings
2024-05-07 04:37:26 +08:00
Debanjum Singh Solanky
f8f9d066db Focus on input field, scroll to latest message on opening chat pane
Previously scroll and chat input focus weren't applied as view hadn't
been rendered yet
2024-05-07 04:37:26 +08:00
Debanjum Singh Solanky
9f65e8de98 Open Khoj Chat as a Pane instead of a Modal
- Allows having it open on the side as you traverse your Obsidian notes
- Allow faster time to response, having responses visible for context
- Enables ambient interactions
2024-05-07 04:37:26 +08:00
sabaimran
9ae828cf11 Use asssets.khoj.dev for loading math katex rendering 2024-05-07 01:43:46 +08:00
sabaimran
cf0b7628d0 Add the url scheme to the public share url 2024-05-06 21:37:49 +08:00
sabaimran
f6aaecb04f Fix construction method for public share conversation URL 2024-05-06 08:32:51 +05:30
sabaimran
14c9bea663
Make conversations optionally shareable (#712)
* Make conversations optionally shareable

- Shared conversations are viewable by anyone, without a login wall
- Can share a conversation from the three dot menu
- Add a new model for Public Conversation
- The rationale for a separate model is that public and private conversations have different assumptions. Separating them reduces some of the code specificity on our server-side code and allows us for easier interpretation and stricter security. Separating the data model makes it harder to accidentally view something that was meant to be private
- Add a new, read-only view for public conversations
2024-05-05 23:16:04 +05:30
Debanjum Singh Solanky
80cbaca935 Serve generated images from Khoj domain instead of directly from S3
Use CNAME to forward requests from the khoj subdomain to the
equivalent S3 bucket
2024-05-04 20:07:10 +05:30
Debanjum Singh Solanky
425496844b Rename assets URL from Khoj S3 bucket to assets.khoj.dev
Server khoj assets from khoj domain
2024-05-04 20:07:10 +05:30
sabaimran
88daa841fd Rename process lock migration and add a reverse migration step 2024-05-04 20:05:00 +05:30
sabaimran
509a8a412c Throw an error if trying to create a process lock that already exists. Names should be unique 2024-05-04 19:03:53 +05:30
sabaimran
7100614de5
Add support for rendering math equations in the web view (#733)
- Add parsing logic for LaTeX-format math equations in the web chat
- Add placeholder delimiters when converting the markdown to HTML in order to avoid removing the escaped characters
- Add the `<!DOCTYPE html>` specification to the page
2024-05-04 15:59:17 +05:30
Debanjum Singh Solanky
d9b3482b1a Show error when required fields to create automation are not set 2024-05-04 11:17:30 +05:30
Debanjum Singh Solanky
91a5643c5c Use Preview label for Automate feature. Prefix mailto: link to contact 2024-05-04 10:59:17 +05:30
Debanjum Singh Solanky
fd2328ab40 Do not hard code base url of path to automation icon in chat message 2024-05-04 10:59:07 +05:30
sabaimran
a38f3227e2 Revert domain in task task send emails 2024-05-03 15:27:27 +05:30
sabaimran
a1263951e9 Use mail to in email contact link 2024-05-03 12:16:56 +05:30
sabaimran
7c9847fe48 Increase jitter to 60 2024-05-03 11:38:22 +05:30
sabaimran
737ebfd521 Make improvements to online search prompts and use a custom domain for automations emails 2024-05-03 10:47:42 +05:30
sabaimran
42e9504ba8 Use a different function for getting last run time, avoid async/sync issues 2024-05-02 12:13:45 +05:30
sabaimran
9e8491b814 Add experimental disclaimers to the automations 2024-05-02 11:40:37 +05:30
sabaimran
c418449311 Add additional robustness in verifying job execution parameters at run time 2024-05-02 11:13:04 +05:30
sabaimran
690e9d8ed3 Collapse the reminders after they're successfully scheduled 2024-05-02 09:55:04 +05:30
sabaimran
6b648ee3ad Add experimental disclaimer in the automation page 2024-05-02 09:21:27 +05:30
sabaimran
f4fbc91515 Remove the exclamation point from the email 2024-05-01 19:01:51 +05:30
sabaimran
bddd1d0fcb Quip, smart reminders 2024-05-01 16:39:07 +05:30
sabaimran
bc8b92a77d Release Khoj version 1.12.0 2024-05-01 16:30:48 +05:30
sabaimran
b499851097 Use the cleaned query as the reference query in the email notification 2024-05-01 15:33:11 +05:30
sabaimran
f24495e0e6 Fix time zone used in query history. Closes #694 2024-05-01 15:31:48 +05:30
sabaimran
7fd57d737e Adjustments to improve overall styling of config page, email template 2024-05-01 14:19:47 +05:30
sabaimran
28578310d1 Add log line when sending a task-related email 2024-05-01 13:56:02 +05:30
sabaimran
a86f95117e Add the subject generation prompt and helper method 2024-05-01 13:55:32 +05:30
sabaimran
c30ba2e551 Set subject dynamically when creating new tasks, and make some minor improvments to the automations UI 2024-05-01 13:54:59 +05:30
sabaimran
d1b2037676 Shutdown the scheduler when the application is exiting 2024-05-01 13:53:34 +05:30
Debanjum Singh Solanky
89a8dbb81a Fix edit job API. Use user timezone, pass all reqd. params to automation
- Pass user and calling_url to the scheduled chat too when modifying
  params of automation
- Update to use user timezone even when update job via API
- Move timezone string to timezone object calculation into the
  schedule automation method
2024-05-01 10:29:49 +05:30
Debanjum Singh Solanky
19c5af3ebc Handle natural language to cron translation error on web client 2024-05-01 09:10:18 +05:30
Debanjum Singh Solanky
70ee9ddf91 Merge migrations from main with feature branch 2024-05-01 09:10:18 +05:30
Debanjum Singh Solanky
8f28f6cc1e Remove now unused location data from being passed to automation funcs 2024-05-01 08:48:16 +05:30
Debanjum Singh Solanky
815966cb25 Unify, modularize DB adapters to get automation metadata by user further 2024-05-01 08:47:50 +05:30
Debanjum Singh Solanky
21bdf45d6f Add link to Automate page in nav pane of the web app 2024-05-01 08:47:50 +05:30
Debanjum Singh Solanky
bd5008136a Move automations into independent page. Allow direct automation
- Previously it was a section in the settings page. Move it to
  independent, top-level page to improve visibility of feature

- Calculate crontime from natural language on web client before
  sending it to to server for saving new/updated schedule to disk.
  - Avoids round-trip of call to chat model

- Convert POST /api/automation API endpoint into a direct request for
  automation with query_to_run, subject and schedule provided via the
  automation page. This allows more granular control to create automation
  - Make the POST automations endpoint more robust; runs validation
    checks, normalizes parameters
2024-05-01 08:47:48 +05:30
Debanjum Singh Solanky
cbc8a02179 Make, use func for constructing the automation created response
- Dedupe logic across http, ws chat API endpoints
- Reduces size of already too long http, ws chat API endpoint funcs
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
c52ed333fa Make content, cards on config pages occupy the whole middle column
- Make the config page content use  the same top level 3-column layout
  as the khoj-header-wrapper
  This ensures the content is aligned with heading pane width
- Let cards and other settings sections scale to the width of their
  grid element. This utilizes more of the screen space and does it
  consistently across the different settings pages
2024-05-01 08:30:10 +05:30
sabaimran
ad4145e48c Fix unique has for job id 2024-05-01 08:30:10 +05:30
sabaimran
311d58e1ed Ensure the automated_task command is removed from the prepended query 2024-05-01 08:30:10 +05:30
sabaimran
eb65532386 Use Django ap scheduler in place of the sqlalchemy one 2024-05-01 08:30:10 +05:30
sabaimran
06213ea814 Fix token retrieval when executing the job and name async job approriately 2024-05-01 08:30:10 +05:30
sabaimran
ca8a7d8368 Revert sync -> aync in send welcome email method 2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
6936875a82 Use DB adapter to unify logic to get, delete automation by auth user
To use place with logic to get, view, delete (and edit soon) automations
by (authenticated) user, instead of scattered across code
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
1238cadd31 Allow editting query-to-run from the automation config section 2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
cb2b1dccc5 Add icon for Automation feature. Replace old icons for delete, new 2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
23f2057868 Allow creating automations from automation settings section in web ui
- Create new POST API endpoint to create automations
- Use it in the settings page on the web interface to create
  new automations

This simplified managing automations from the setting page by allowing
both delete and create from the same page
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
2f9241b5a3 Rename scheduled task to automations across code and UX
- Fix query, subject parameters passed to email template
- Show 12 hour scheduled time in automation created chat message
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
230d160602 Improve rendering task scheduled settings view and message
- Render crontime string in natural language in message & settings UI
- Show more fields in tasks web config UI
- Add link to the tasks settings page in task scheduled chat response
- Improve task variables names
  Rename executing_query to query_to_run. scheduling_query to
  scheduling_request
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
d341b1efe8 Store, retrieve task metadata from the job name field 2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
ae10ff4a5f Create create_scheduled_task func to dedupe logic across ws, http APIs
Previously, both the websocket and http endpoint were implementing
the same logic. This was becoming too unwieldy
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
8dfa0bf047 Simplify task scheduler prompt. No timezone conversion. Infer subject
- Make timezone aware scheduling programmatic, instead of asking the
  chat model to do the conversion. This removes the need for
  scratchpad and may let smaller models handle the task as well
- Make chat model infer subject for email. This should make the
  notification email more readable
- Improve email by using subject in email subject, task heading. Move
  query to email final paragraph, which is where task metadata  should
  go
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
2c563ad280 Use hash of query in process lock id to standardize id format
- Using inferred_query directly was brittle (like previous job id)
- And process lock id had a limited size, so wouldn't work for larger
  inferred query strings
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
3ce06a938c Render scheduled task response as html to improve readability in email 2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
c17dbbeb92 Render next run time in user timezone in config, chat UIs
- Pass timezone string from ipapi to khoj via clients
  - Pass this data from web, desktop and obsidian clients to server
- Use user tz to render next run time of scheduled task in user tz
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
6736551ba3 Improve scheduled task text rendered in UI 2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
0e01362469 Merge DB migrations from master with those from scheduled task feature 2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
a5ed4f2af2 Send email to share results of scheduled task 2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
69775b6d6e Add /task command. Use it to disable scheduling tasks from tasks
This takes the load of the task scheduling chat actor / prompt from
having to artifically differentiate query to create scheduled task
from a scheduled task run.
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
22289a0002 Improve task scheduling by using json mode and agent scratchpad
- The task scheduling actor was having trouble calculating the
  timezone. Giving the actor a scratchpad to improve correctness by
  thinking step by step
- Add more examples to reduce chances of the inferred query looping to
  create another reminder instead of running the query and sharing
  results with user
- Improve task scheduling chat actor test with more tests and
  by ensuring unexpected words not present in response
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
7f5981594c Only notify when scheduled task results satisfy user's requirements
There's a difference between running a scheduled task and notifying
the user about the results of running the scheduled task.

Decide to notify the user only when the results of running the
scheduled task satisfy the user's requirements.

Use sync version of send_message_to_model_wrapper for scheduled tasks
2024-05-01 08:30:10 +05:30