Commit graph

3550 commits

Author SHA1 Message Date
Debanjum Singh Solanky
8fc7f980aa Revert KHOJ_DOMAIN to only support single domain.
Multiple domain support didn't generalize to other portions where it
is used
2024-06-24 15:24:45 +05:30
sabaimran
4110e71e84 Add info in the documentation about text to speech 2024-06-24 12:46:33 +05:30
sabaimran
939811e9b5 Fix conversation look up logic 2024-06-24 09:10:03 +05:30
Debanjum Singh Solanky
a4d88612c1 Just use yarn for package version locking. Remove npm package lock 2024-06-23 16:06:20 +05:30
Debanjum Singh Solanky
55be90cdd2 Sanitize user input fields on Automations page of web client
Use Dompurify to sanitize user input
2024-06-23 14:14:47 +05:30
Debanjum Singh Solanky
1c7a562880 Generate automation cards via DOM scripting 2024-06-23 13:22:38 +05:30
Debanjum Singh Solanky
57a36967bf Run Obsidian version script in bump_version.sh to write to versions.json
This handles updates from manifest.json minAppVersion field to the
versions.json file.

The minAppVersion field is for the minimum Obsidian app version
supported by a Khoj plugin version
2024-06-23 08:18:55 +05:30
Debanjum Singh Solanky
c7c32a7467 Improve online chat reference extraction in Khoj.el Emacs package
- Handle online references with no title
- Improve handling references which are arrays instead of lists
2024-06-23 08:13:36 +05:30
Debanjum Singh Solanky
9d33d8c0fa Upgrade typescript eslint dev dependency of Khoj Obsidian plugin 2024-06-23 07:36:49 +05:30
Debanjum
a94062469a
Automatically Find Similar Notes on Emacs in Background (#827)
Khoj will find and display notes similar to the current entry in the side pane when
1. find similar is open in side pane and
2. cursor has moved to a new entry

### Major
- Find similar notes to current note at cursor automatically in background
- Only show headings of search result and increase default results count

### Minor
- Pass absolute path of file to index from khoj.el emacs client
- Update help message to only show the smaller set of new keybindings
- Fix edge cases in loading some chat sessions
2024-06-23 07:36:11 +05:30
sabaimran
38090b2553 In dockerize.yml file, revert the added configuration 2024-06-22 21:11:25 +05:30
sabaimran
a53178cab9
Add developer support for using next.js to serve generated static files (#814)
To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road.

Dev instructions for using the next.js setup are in the added README.

This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up.

This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
2024-06-22 20:12:41 +05:30
Debanjum Singh Solanky
59edb99f04 Simplify, improve bump version development script
- Just use in-built `npm version' command to update desktop, obsidian version
- Upgrade by major, minor or patch version using new -t flag in script
  E.g bump_version -t minor
2024-06-22 18:19:38 +05:30
Debanjum Singh Solanky
abd6f58aee Upgrade Desktop app package dependencies 2024-06-22 17:38:52 +05:30
Debanjum Singh Solanky
f413dc62cd Upgrade Obsidian plugin dependencies. Add package lock file for it
Add it to bump_version script as well.
2024-06-22 17:38:52 +05:30
Debanjum Singh Solanky
1d7d51a7ab Upgrade Documentation packages 2024-06-22 17:38:48 +05:30
Debanjum Singh Solanky
22f6db0a6b Upgrade RapidOCR and enable for Python 3.12. Fix PDF OCR test 2024-06-22 16:01:55 +05:30
Debanjum Singh Solanky
55a23eae25 Upgrade pillow to fix pytest workflow failure 2024-06-22 15:17:43 +05:30
Debanjum Singh Solanky
7e277e9381 Fix getting file-toggle-button element in chat of web app 2024-06-21 15:54:38 +05:30
Debanjum Singh Solanky
fa7b40ab86 Automatically respond with Voice if subscribed user sent Voice message 2024-06-21 15:53:01 +05:30
Debanjum Singh Solanky
5e5fe4b7af Improve font size, spacing of conversation session on desktop app 2024-06-21 12:25:35 +05:30
sabaimran
d3c0111121 Include base URL when using openai api config in extract questions. Close #831 2024-06-21 12:18:50 +05:30
sabaimran
b9966eb3d4
Add support for text to speech in chat responses (#821)
* Enable speech to text responses in khoj chat

- Current issue: reads out all the markdown formatting, plus waits for the whole result to be streamed before playing it

* Extract content from markdown-formatted text

* Add a loader for while you're waiting for Khoj's response

* Add user configuration option for chat model options, allow server side configuration for option list

* Join up APIs, views, admin pages to allow configuring custom voice models
2024-06-21 11:30:28 +05:30
Debanjum Singh Solanky
427575e958 Improve khoj chat new, delete session flows
When create new conversation session, automatically request query. As
that is expected next action after creating new session

Pass session-id to khoj-chat to allow reuse from
create-new-conversation func

When delete conversation session, do not call load chat session.
Unnecessary action.

Use thread-last to improve code flow in new, delete conversation funcs
2024-06-21 10:54:59 +05:30
Debanjum Singh Solanky
59032a06d5 Improve defaults when extracting fields from online reference in khoj.el 2024-06-21 10:54:59 +05:30
Debanjum Singh Solanky
9262aea7a5 Fix comments, func calls based on melpazoid, checkdoc, package-lint 2024-06-21 10:54:59 +05:30
sabaimran
ff26b19d2b Add a migration for allowing the docx field in the entries file type 2024-06-21 09:47:49 +05:30
sabaimran
3cfe5aabe5
Add support for magic link email sign-in (#820)
* Add magic link email sign-in option

* Adding backend routes and model changes to keep state of email verification code and status

* Test and fix end to end email verification flow

* Add documentation for how to use the magic link sign-in when self-hosting Khoj

* Add magic link sign in to public conversation page
2024-06-20 13:32:58 +05:30
Debanjum Singh Solanky
0afe66ac39 Restore cursor to original window after opening Khoj side pane
Previously the cursor would move to the Khoj side pane on opening it.
This would break user's flow, especially when find similar triggers
automatically

New behavior maintains smoother update of auto find similar without
disrupting user browsing
2024-06-20 12:50:13 +05:30
Debanjum Singh Solanky
afe91a2633 Only show headings of search result and increase total count returned
Previously it would show complete result body this would make the
result width variable and hard to track all the returned results

Showing just heading makes it easier to track
2024-06-20 12:50:13 +05:30
Debanjum Singh Solanky
2b12a5514e Find similar notes to current note at cursor automatically in background
- Call find similar on current element if point has moved to new
  element
- Delete the first result from find-similar search results as that'll
  be the current note (which is trivially most similar to itself)
- Determine find-similar based text formating at the rendering layer
  rather than at the top level find-similar func
2024-06-20 12:50:13 +05:30
Raghav Tirumale
093eb473cb
Add Documentation for the /summarize Command (#822)
* added documentation for the /summarize command

* Add a hint for natural language usage

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-06-20 12:08:01 +05:30
Raghav Tirumale
bd3b590153
Support Indexing Docx Files (#801)
* Add support for indexing docx files and associated unit tests

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-06-20 11:18:01 +05:30
Debanjum Singh Solanky
d042e073cc Pass absolute path of file to index from khoj.el emacs client 2024-06-20 00:26:18 +05:30
Debanjum Singh Solanky
d23f2849d4 Update help message to only show the smaller set of new keybindings 2024-06-20 00:26:18 +05:30
Raghav Tirumale
d4e5c95711
Add Ability to Summarize Documents (#800)
* Uses entire file text and summarizer model to generate document summary.
* Uses the contents of the user's query to create a tailored summary.
* Integrates with File Filters #788 for a better UX.
2024-06-18 19:31:07 +05:30
Debanjum Singh Solanky
677d49d438 Release Khoj version 1.14.0 2024-06-18 17:13:46 +05:30
Debanjum Singh Solanky
2930b57c78 Use hashed value to improve deduplication of search results on server 2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
6814dadd21 Fix opening Web, Desktop setup links on first run from Desktop app
Previous version failed to open the setup links
2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
632f55a9e8 Do not default to rerank if device has GPU 2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
f1120f24a1 Use solarized light css styling to highlight code in chat messages 2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
d8a5a01cea Pass multiple allowed Khoj domains via KHOJ_DOMAIN env var
To add multiple allowed Khoj domains pass them as a comma separated
list of domains via the KHOJ_DOMAIN environment variable

Resolve comment in issue #662
2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
4daf16e5f9 Only redirect to next url relative to current domain 2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
86a3505d89 Remove image HTML elements from non whitelisted sources in Obsidian chat
Given img src enforcement via CSP required loosening. Soft enforce it
via a regex replace of img HTML elements if the src isn't from the
whitelisted set of source prefixes.

Currently allowed source prefixes are
- app: for local images
- data: for inline generated images
- https://generated.khoj.dev: for cloud generated images
2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
c7d825bddb Sanitize markdown in Obsidian after conversion to HTML too
- Create and use a function to convert markdown to sanitized html
- Remove unused Latex delimiter handling as Katex isn't used in
  Khoj chat on Obsidian
2024-06-18 17:04:25 +05:30
Debanjum Singh Solanky
08c3aa496d Loosen CSP in Obsidian to load images, sync and allow Obsidian domain 2024-06-18 17:04:25 +05:30
sabaimran
327045be43 Make some basic updates to the chat documentation. Inc. conversation file filters, new screenshot 2024-06-18 12:14:59 +05:30
sabaimran
76e1bed8f9 Update Obsidian documentation 2024-06-18 08:22:10 +05:30
sabaimran
a57e1e7a14 Fix langchain, tenacity versions 2024-06-17 14:52:11 +05:30
sabaimran
ce9c14f894 Fix more packages related to langchain in the pyproject.toml 2024-06-17 14:38:05 +05:30