Commit graph

3274 commits

Author SHA1 Message Date
Debanjum Singh Solanky
f440ddbe1d Fix openai chat actor, director tests
- Update test ChatModelOptions setup since update to it's schema
- Fix stale function calls using their updated signatures
2024-06-09 07:24:47 +05:30
sabaimran
2e209ab28b Handle case where conversation does not (yet) exist 2024-06-08 16:22:12 +05:30
sabaimran
849c38c0a4 Add support for managing audiences for new users 2024-06-08 15:51:17 +05:30
sabaimran
06a47ee457
Add language-specific syntax highlighting via highlight.js (#802)
* Add language-specific syntax highlighting via highlight.js

- Add highlight.js to our assets CDN for fast load and compliance with the CSP
- See other stylesheets options here: https://cdnjs.com/libraries/highlight.js

* Bonus: set min-height to prevent increasing length of the sessions pane

* Fix references rendering and add highlight.js in public conversation
2024-06-08 15:17:09 +05:30
Debanjum Singh Solanky
5f2442450c Update truncation test to reduce flakyness in cloud tests
Removed dependency on faker, factory for the truncation tests as that
seems to be the point of flakiness
2024-06-07 19:42:48 +05:30
sabaimran
dbb06466bf Minor fit/finish updates to the file filter experience 2024-06-07 15:05:00 +05:30
sabaimran
58a02f06ea
Fix multilingual font rendering (#797)
* Fix multilingual font rendering; fallback to an Arabic language font which contains more Asian characters. Close #756

* Tune font-sizes and styling to accomodate new fonts with old sizing

- Move connection-status styling out from inline html into css block
- Remove start typing chat-input height jitter
- align new-conversation button, text
- use relative font sizes instead of absolute font sizes in most places

---------

Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
2024-06-07 11:53:47 +05:30
Raghav Tirumale
ba16afd3c2
New Feature: Adding File Filtering to Conversations (#788)
* UI update for file filtered conversations
* Interactive file menu #UI to add/remove files on each conversation as references.
* Backend changes implemented to load selected file filters from a conversation into the querying process.
---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-06-07 10:53:37 +05:30
Debanjum Singh Solanky
f91cdf8e18 Fix showing headings in intermediate step in generating chat response 2024-06-06 16:52:23 +05:30
Debanjum Singh Solanky
18f7e6e7ed Remove "Path" prefix from org ancestor heading in compiled entry 2024-06-06 16:51:26 +05:30
sabaimran
8d701ebe22
Add fedCM to accommodate google migration (#798)
- See migration guidelines here: https://developers.google.com/identity/gsi/web/guides/fedcm-migration#fedcm_flag
2024-06-06 14:23:16 +05:30
Debanjum Singh Solanky
dd2225b1aa Use Text output mode to disambiguate from Default data source lookup
Previously if default output was selected by Khoj, we'd end up doing
an documents search as well, even when Khoj selected internet or
general data source to lookup.

This update disambiguates the default information mode from the text
output mode. To avoid doing documents search when not deemed necessary
by Khoj
2024-06-06 11:56:48 +05:30
Debanjum Singh Solanky
a1e4f4bde7 Gracefully skip indexing when empty list of docs provided
Improve error message when fail to index content
2024-06-05 19:39:15 +05:30
Debanjum Singh Solanky
21987f60c7 Use `-difference' to get files to delete. Make batch size defcustom
Improve docstrings to align with `checkdoc' requirement for all args
being mentioned
2024-06-05 19:39:15 +05:30
Debanjum
bfacd65971
Batch upload files for indexing from the Emacs client (#735) from yuzhou721/master
Encode filenames and batch file uploads to improve sending content to index from the Emacs client
2024-06-05 19:31:06 +05:30
sabaimran
a9c383e62c Use an ASGI application, rather than WSGI
- ASGI should be the preferred application, as our codebase runs a lot of async code
2024-06-05 09:25:08 +05:30
sabaimran
0816cec4bc Manually close old db connections periodically 2024-06-04 22:19:47 +05:30
sabaimran
acfdc8da77 Explicitly set the connection age to 0 in the django settings. Seems to be some strange behavior with async gunicorn + django db 2024-06-04 20:31:51 +05:30
Debanjum Singh Solanky
85a343363b Release Khoj version 1.13.0 2024-06-04 11:57:44 +05:30
Debanjum
1dfd6d7391
Merge pull request from GHSA-h2q2-vch3-72qm
Add CSP and sanitize chat messages in Obsidian, Desktop, Web apps
2024-06-04 11:29:21 +05:30
Debanjum Singh Solanky
b757ba664f Sanitize chat messages to render in Obsidian, Desktop, Web apps
Use DOMPurify to escape any unsafe HTML in chat message before adding
it to DOM via innerHTML updates to a HTML element
2024-06-04 10:53:30 +05:30
Debanjum Singh Solanky
9f80c2ab76 Enforce Content-Security-Policy (CSP) in Obsidian, Desktop, Web apps
Prevent XSS attacks by enforcing Content-Security-Policy (CSP) in apps.
Do not allow loading images, other assets from untrusted domains.

- Only allow loading assets from trusted domains
  like 'self', khoj.dev, ipapi for geolocation, google (fonts, img)
  - images from khoj domain, google (for profile pic)
  - assets from khoj domain
  - Do not allow iframe src
  - Allow unsafe-inline script and styles for now as markdown-it escapes html
    in user, khoj chat

- Add hostURL to CSP of the Desktop, Obsidian apps
  Given web client is served by khoj server, it doesn't need to
  explicitly allow for khoj.dev domain. So if user self-hosting, it'll
  automatically allow the domain in the CSP (via 'self')

  Whereas the Obsidian, Desktop clients allow configure the server URL.
  Note *switching server URL breaks CSP until app is reloaded*
2024-06-04 10:53:30 +05:30
Debanjum Singh Solanky
179c70dba8 Upgrade Khoj llama-cpp, django and jinja dependencies 2024-06-04 09:05:53 +05:30
Debanjum Singh Solanky
bbcdb8413d Add null checks, fix build errors in Khoj plugin on newer Obsidian 2024-06-03 18:03:11 +05:30
Debanjum Singh Solanky
d8ace4d34c Highlight the agents, automation tab when active on the web app 2024-06-03 16:57:03 +05:30
sabaimran
4679f07336 Clean up some of the design of agents, inspired by dicussion #792 2024-06-03 12:52:07 +05:30
Debanjum Singh Solanky
8cdab5f31a Update slash command UX in chat UI of desktop app to match web app
Make commands in popup menu on typing slash in chat input selectable
2024-06-02 17:27:37 +05:30
Debanjum Singh Solanky
7828bd6f2e Hide command popup & focus on chatInput on selecting command in web app
Style command popup cursor and add highlight to indicate using slash
command
2024-06-02 17:27:37 +05:30
Debanjum
cf8c9c2a3d
Serve image assets from Khoj domain, not directly from S3 bucket (#734)
- Serve generated images from Khoj domain instead of directly from AWS S3
- Rename assets URL from Khoj S3 bucket to assets.khoj.dev
2024-06-02 17:24:35 +05:30
sabaimran
5bb3689562 Do not stream responses in the scheduled_chat response 2024-06-02 11:31:15 +05:30
sabaimran
5132b01ab1 Remove intent_type from telemetry update in api_chat 2024-06-02 10:21:38 +05:30
Raghav Tirumale
a3934b3aaa
Improved Command Menu and Help Command (#774)
* The command menu (triggered by "/") now has a clickable list of possible commands, that automatically fill into the chat when pressed.
* The `/help` command now searches `khoj.dev` pages to provide useful assistance to the user.

---------

Co-authored-by: raghavt3 <raghavt3@illinois.edu>
Co-authored-by: sabaimran <65192171+sabaimran@users.noreply.github.com>
2024-06-01 22:33:31 +05:30
sabaimran
6d10f98498 Add additional lines for KHOJ_NO_HTTPS and KHOJ_DOMAIN in the docker-compose 2024-06-01 21:48:43 +05:30
sabaimran
841cbff249 Add documentation for setting up google auth in self-hosted khoj. Closes #771 2024-06-01 21:38:21 +05:30
sabaimran
89178bcebd Fix formatting issues for task email in mobile 2024-06-01 14:19:12 +05:30
Debanjum
b499b3fe2a
Upgrade Khoj Obsidian: Chat from Side Pane, Stream Intermediate Steps, Copy Message to Clipboard (#736)
### Details
- **Chat with Khoj from right pane on Obsidian**
  - Modal was too ephemeral, couldn't have it open for reference, quick jump to Khoj chat
- **Stream intermediate steps taken by Khoj** for generating response to the chat pane
  Gives more transparency into Khoj 'thinking' process, e.g internet, notes searches performed, documents read etc. 
  The feedback allows us to tune our messages to elicit better responses by Khoj
- Add ability to **copy message to clipboard, paste chat messages directly into current file**
- Jump to **Search**, **Find Similar** functions from navigation bar on the Khoj Obsidian side pane
- Improve spacing, use consistent colors in chat message references and buttons

Resolves #789, #754
2024-06-01 13:29:21 +05:30
sabaimran
8b9c26c468 Remove unused method 2024-06-01 12:54:43 +05:30
sabaimran
5ec641837a
Allow automations to be shareable (#790)
* Updating the API / UI to support sharing of automations
* Allow people to see the automations even when not logged in, and add an overlay effect
* Handle unauthenticated users taking actions
* Support showing pre-filled automation details on the config automations page
* Redirect user to login if they try to add an automation while unauthenticated
2024-06-01 12:44:49 +05:30
Debanjum Singh Solanky
7d7d4cf5c3 Make new chat message text selectable in Obsidian side pane
Resolves #789
2024-06-01 11:01:39 +05:30
Debanjum Singh Solanky
7fb7f200b3 Fix rendering text in chat messages with bulleted lists
Improves #789
2024-06-01 10:51:22 +05:30
Debanjum Singh Solanky
7a93599fe8 Merge branch 'master' into upgrade-khoj-on-obsidian
- Conflicts:
  - src/khoj/interface/web/chat.html
    Use our changes with feedback button changes from master
2024-06-01 10:07:43 +05:30
Debanjum Singh Solanky
92bab9fa61 Get Conversation session action buttons out from under the three dot menu 2024-05-31 20:11:00 +05:30
Debanjum Singh Solanky
7fa42daf89 Render action buttons for new Khoj chat responses in Obsidian
- Dedupe the code to add action buttons to chat messages
- Update the renderIncrementalMessage function to also add the action
  buttons to newly generated chat messages by Khoj
2024-05-31 20:11:00 +05:30
Debanjum Singh Solanky
2d010db83f Toggle chat session view on clicking the Obsidian chat sessions button 2024-05-31 20:11:00 +05:30
Debanjum Singh Solanky
275d4877a6 Fix loading spinner visibility by using contrasting background color
Fix code formating of Khoj chat view in Obsidian
2024-05-31 20:09:24 +05:30
sabaimran
2667ef4544 Refresh the conversation from the db in the websocket flow 2024-05-31 16:15:56 +05:30
sabaimran
fd07abbfc8 Decrease the life of one connection 2024-05-31 15:39:15 +05:30
Debanjum
3090b84252
Disable Minutely Recurrence for Automations (#781)
* Disable automation recurrence at minute level frequency

* Set a max lifetime for django's connections to the db

* Disable any automation that has a non-numeric first digit (i.e., recuring on the minute level)

* Re-enable automations

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-05-31 12:50:19 +05:30
sabaimran
5dca48d9fc Fix setting of conn_max_age variable 2024-05-31 11:07:13 +05:30
sabaimran
76f941f4e5 Revert email from from to sender again in resend API. keeps switching? 2024-05-31 10:30:18 +05:30