Khoj will find and display notes similar to the current entry in the side pane when
1. find similar is open in side pane and
2. cursor has moved to a new entry
### Major
- Find similar notes to current note at cursor automatically in background
- Only show headings of search result and increase default results count
### Minor
- Pass absolute path of file to index from khoj.el emacs client
- Update help message to only show the smaller set of new keybindings
- Fix edge cases in loading some chat sessions
To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road.
Dev instructions for using the next.js setup are in the added README.
This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up.
This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
When create new conversation session, automatically request query. As
that is expected next action after creating new session
Pass session-id to khoj-chat to allow reuse from
create-new-conversation func
When delete conversation session, do not call load chat session.
Unnecessary action.
Use thread-last to improve code flow in new, delete conversation funcs
Previously the cursor would move to the Khoj side pane on opening it.
This would break user's flow, especially when find similar triggers
automatically
New behavior maintains smoother update of auto find similar without
disrupting user browsing
Previously it would show complete result body this would make the
result width variable and hard to track all the returned results
Showing just heading makes it easier to track
- Call find similar on current element if point has moved to new
element
- Delete the first result from find-similar search results as that'll
be the current note (which is trivially most similar to itself)
- Determine find-similar based text formating at the rendering layer
rather than at the top level find-similar func
Given img src enforcement via CSP required loosening. Soft enforce it
via a regex replace of img HTML elements if the src isn't from the
whitelisted set of source prefixes.
Currently allowed source prefixes are
- app: for local images
- data: for inline generated images
- https://generated.khoj.dev: for cloud generated images
- Create and use a function to convert markdown to sanitized html
- Remove unused Latex delimiter handling as Katex isn't used in
Khoj chat on Obsidian
- C-x o to switch to search org content conflicts with switch buffer shortkey
This is more apparent in the async search scenario as it prevents
perform other actions while async search is in progress
- Also switching content type wouldn't scale to all the content types
Khoj will support without causing more conflicting keybinding
Khoj side pane occupies a vertically split bottom right side pane.
If the bottom right window is not a vertical split, create a new
vertical split pane for khoj, otherwise reuse the existing window
See the currently active window in context while doing chat, search
or find similar operations in a side pane.
This is similar to how we've moved Khoj on Obsidian into the side pane
as well
* Add language-specific syntax highlighting via highlight.js
- Add highlight.js to our assets CDN for fast load and compliance with the CSP
- See other stylesheets options here: https://cdnjs.com/libraries/highlight.js
* Bonus: set min-height to prevent increasing length of the sessions pane
* Fix references rendering and add highlight.js in public conversation
* Fix multilingual font rendering; fallback to an Arabic language font which contains more Asian characters. Close#756
* Tune font-sizes and styling to accomodate new fonts with old sizing
- Move connection-status styling out from inline html into css block
- Remove start typing chat-input height jitter
- align new-conversation button, text
- use relative font sizes instead of absolute font sizes in most places
---------
Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
* UI update for file filtered conversations
* Interactive file menu #UI to add/remove files on each conversation as references.
* Backend changes implemented to load selected file filters from a conversation into the querying process.
---------
Co-authored-by: sabaimran <narmiabas@gmail.com>
Prevent XSS attacks by enforcing Content-Security-Policy (CSP) in apps.
Do not allow loading images, other assets from untrusted domains.
- Only allow loading assets from trusted domains
like 'self', khoj.dev, ipapi for geolocation, google (fonts, img)
- images from khoj domain, google (for profile pic)
- assets from khoj domain
- Do not allow iframe src
- Allow unsafe-inline script and styles for now as markdown-it escapes html
in user, khoj chat
- Add hostURL to CSP of the Desktop, Obsidian apps
Given web client is served by khoj server, it doesn't need to
explicitly allow for khoj.dev domain. So if user self-hosting, it'll
automatically allow the domain in the CSP (via 'self')
Whereas the Obsidian, Desktop clients allow configure the server URL.
Note *switching server URL breaks CSP until app is reloaded*
- Dedupe the code to add action buttons to chat messages
- Update the renderIncrementalMessage function to also add the action
buttons to newly generated chat messages by Khoj
Previously clicking inline links would open the URL directly in the
Desktop app. This was strange and it didn't provide any way to go back
to Khoj desktop app UI from the opened link
- Pass file path of reference along with the compiled reference in
list of references returned by chat API converts
- Update the structure of references from list of strings to list of
dictionary (containing 'compiled' and 'file' keys)
- Pull out the compiled reference from the new references data struct
wherever it was is being used
Simplify, reuse, standardize code to render messages with references
in the obsidian, web and desktop clients. Specifically:
- Reuse function to create reference section, dedupe code
- Create reusable function to generate image markdown
- Simplify logic to render message with references
- Setup websocket using Khoj web app as reference.
- Moved the geolocating code to chat view out from the general pane
view
- Use loading spinner from web instead of the thinking emoji
It'll replace any highlighted text with the chat message or if not
text is highlighted, it'll insert the chat message at the last cursor
position in the active file
- Jump to chat, show similar actions from nav menu of Khoj side pane
- Add chat, search icons from web, desktop app
- Use lucide icon for find similar (for now)
- Match proportions of find similar icon to khoj other icons via css, js
- Use KhojPaneView abstract class to allow reuse of common functionality like
- Creating the nav bar header in side pane views
- Loading geo-location data for chat context
This should make creating new views easier
- Allows having it open on the side as you traverse your Obsidian notes
- Allow faster time to response, having responses visible for context
- Enables ambient interactions
* Make conversations optionally shareable
- Shared conversations are viewable by anyone, without a login wall
- Can share a conversation from the three dot menu
- Add a new model for Public Conversation
- The rationale for a separate model is that public and private conversations have different assumptions. Separating them reduces some of the code specificity on our server-side code and allows us for easier interpretation and stricter security. Separating the data model makes it harder to accidentally view something that was meant to be private
- Add a new, read-only view for public conversations
- Pass timezone string from ipapi to khoj via clients
- Pass this data from web, desktop and obsidian clients to server
- Use user tz to render next run time of scheduled task in user tz
* Changed the styling of the link that takes a user to the settings page into a button
* added an indicator that shows if a user is connected to the server or not
* made a class name more descriptive and also made the text in first run message more intuitive
* changed the command to install dependencies in the README.md
* changed the class name of the first run message text to be more descriptive
* added icons in the desktop UI that shows if a file is synced successfully or not
* made the link class name in the homepage more descriptive
* fixed the hover issue on status box in the chat header pane
* fixed hovering issue on status box on macOS
- Reduces time to first render when loading long chat sessions
- Limits size of first page load, when loading long chat sessions
These performance improvements are maximally felt for large chat
sessions with lots of images generated by Khoj
Updated web and desktop app to support these changes for now
- Magika on Desktop app was too bloated (100Mb to 250Mb) and broke
install for some reason. Not sure why it was causing the app install
to fail but do not have time to currently investigate
- Just use file extensions whitelist it's good enough for now. Let
server handle the deeper identification of file type
### Index more text file types
- Index all text, code files in Github repos. Not just md, org files
- Send more text file types from Desktop app and improve indexing them
- Identify file type by content & allow server to index all text files
### Deprecate Github Indexing Features
- Stop indexing commits, issues and issue comments in a Github repo
- Skip indexing Github repo on hitting Github API rate limit
### Fixes and Improvements
- **Fix indexing files in sub-folders from Desktop app**
- Standardize structure of text to entries to match other entry processors
- `file-type' doesn't handle mis-labelled files or files without
extensions well
- Only show supported file types in file selector dialog on Desktop app
Use Magika to get list of text file extensions. Combine with other
supported extensions to get complete list of supported file extensions.
Use it to limit selectable files in the File Open dialog.
Note: Folder selector will index text files with no extensions as well
- Move new conversation button to right of "Conversation" title
- Reduce size of chat message loading ellipsis animation
- Add loading animation for chat session
- `fs.readdir' func in node version 18.18.2 has buggy `recursive' option
See nodejs/node#48640, effect-ts/effect#1801 for details
- We were recursing down a folder in two ways on the Desktop app.
Remove `recursive: True' option to the `fs.readdirSync' method call
to recurse down via app code only
- Allow syncing more file types from desktop app to index on server
- Use `file-type' package to identify valid text file types on Desktop app
- Split plaintext entries into smaller logical units than a whole file
Since the text splitting upgrades in #645, compiled chunks have more
logical splits like paragraph, sentence.
Show those (potentially) smaller snippets to the user as references
- Tangential Fix:
Initialize unbound currentTime variable for error log timestamp
### Overview
Khoj can now read website directly without needing to go through the search step first
### Details
- Parallelize simple webpage read and extractor
- Rename extract_content online results field to web pages
- Tweak prompts to extract information from webpages, online results
- Test select webpage as data source and extract web urls chat actors
- Render webpage read in chat response references on Web, Desktop apps
- Pass multiple webpages with their urls in online results context
- Support webpage command in chat API
- Add webpage chat command for read web pages requested by user
- Create chat actor for directly reading webpages based on user message
- Ask for Confirmation before deleting chat session in Desktop, Web app
- Save chat session rename on hitting enter in title edit input box
- No need to flash previous conversation cleared status message
- Move chat session delete button after rename button in Desktop app
Previously only the notes references would get rendered post response
streaming when when both online and notes references were used to
respond to the user's message
- Trigger
SentenceTransformer Cross Encoder models now run fast on GPU enabled machines, including Mac ARM devices since UKPLab/sentence-transformers#2463
- Details
- Use cross-encoder to rerank search results by default on GPU machines and when using an inference server
- Only call search API when pause in typing search query on web, desktop apps
Wait for 300ms since stop typing before calling search API.
This smooths out UI jitter when rendering search results, especially
now that we're reranking for every search query on GPU enabled devices
Emacs already has 300ms debounce time. More convoluted to add
debounce time to Obsidian search modal, so not updating that yet
- Fix
`get_conversation_by_user' shouldn't return new conversation if
conversation with requested id not found.
It should only return new conversation if no specific conversation
is requested and no conversations found for user at all
- Repro
- Delete a new chat, this calls loadChat via window.onload which
calls server /chat/history API endpoint with conversationId set to
that of just deleted conversation sporadically
The call to GET chat/history API with conversationId set occurs
when window.onload triggers before the conversationId is deleted
by the delete button after the DELETE /chat/history API call (via race)
- In such a scenario, get_conversation_by_user called by
chat/history API with conversationId of deleted conversation
returns a new conversation
- Miscellaneous
- Chat history load should be logged as call to that chat_history api,
not the "chat" api
- Show status updates of clearing conversation history in chat input
- Simplify web, desktop client code by removing unnecessary new variables
* Upload generated images to s3, if AWS credentials and bucket is available.
- In clients, render the images via the URL if it's returned with a text-to-image2 intent type
* Make the loading screen more intuitve, less jerky and update the programmatic copy button
* Update the loading icon when waiting for a chat response
* Make major improvements to the image generation flow
- Include user context from online references and personal notes for generating images
- Dynamically select the modality that the LLM should respond with
- Retun the inferred context in the query response for the dekstop, web chat views to read
* Add unit tests for retrieving response modes via LLM
* Move output mode unit tests to the actor suite, rather than director
* Only show the references button if there is at least one available
* Rename aget_relevant_modes to aget_relevant_output_modes
* Use a shared method for generating reference sections, simplify some of the prompting logic
* Make out of space errors in the desktop client more obvious
- Open external links using the default link handler registered on OS
for the link type, e.g http:// -> firefox, mailto: thunderbird etc
- Confirm before opening non-http URL using an external app
- Improve
- Only send files modified since their last sync for indexing on server from the Obsidian client
- Fix
- Invalidate static asset browser cache in Web client when Khoj version changes
Previously we'd send all files in vault and let the server
deduplicate.
This changes takes inspiration from the desktop app, and only pushes
files which were modified after their previous sync with the server.
This should reduce the processing load on the server
* Retrieve, create, and save conversations differently if they're coming from a client application
- Not all of our client apps will necessarily maintain state over the conversation IDs available to a user. For some (single-threaded conversations), it should just use a single conversation. Fix the code to do so
* Simplify conversation retrieval logic
* Keep 0 padding below chat response
* Add order_by sorting to retrieving the conversation without id
- Resolve PWA issues thrown by Chrome/Edge
- Add screenshot samples showcasing remember, browse and draw features
- This can provide a richer app store like experience when
installing Khoj PWA on Mobile or Desktop
- Add wide and narrow screenshots to show Mobile vs Desktop UX
- Add higher resolution favicon for PWA
- Use single web manifest instead of separate ones for Chat, Search
- Update manifest description with more details about Khoj features
Reset starter question suggestions before appending in web, desktop app
Otherwise previously it'd keep adding to existing starter question
suggestions on each new session creation if multiple consecutive new
chat sessions created.
This would result in more than the 3 expected starter questions being
displayed at a time
- Make collapse, expand toggle arrow point in the direction the action
will expand the side panel in
- Make the collapsed side panel reduce to a 1px sliver
- Respect newline with pre-line but not for bullets to improve
formatting of responses by Khoj
- Respect bold font by loading tajawal font with other weights
- Reduce bottom margin in chat message bubble, its taking too much space
* Display given_name field only if it is not None
* Add default slugs in the migration script
* Ensure that updated_at is saved appropriately, make sure most recent chat is returned for default history
* Remove the bin button from the chat interface, given deletion is handled in the drop-down menus
* Refresh the side panel when a new chat is created
* Improveme tool retrieval prompt, don't let /online fail, and improve parsing of extract questions
* Fix ending chat response by offline chat on hitting a stop phrase
Previously the whole phrase wouldn't be in the same response chunk, so
chat response wouldn't stop on hitting a stop phrase
Now use a queue to keep track of last 3 chunks, and to stop responding
when hit a stop phrase
* Make chat on Obsidian backward compatible post chat session API updates
- Make chat on Obsidian get chat history from
`responseJson.response.chat' when available (i.e when using new api)
- Else fallback to loading chat history from
responseJson.response (i.e when using old api)
* Fix detecting success of indexing update in khoj.el
When khoj.el attempts to index on a Khoj server served behind an https
endpoint, the success reponse status contains plist with certs. This
doesn't mean the update failed.
Look for :errors key in status instead to determine if indexing API
call failed. This fixes detecting indexing API call success on the
Khoj Emacs client, even for Khoj servers running behind SSL/HTTPS
* Fix the mechanism for populating notes references in the conversation primer for both offline and online chat
* Return conversation.default when empty list for dynamic prompt selection, send all cmds in telemetry
* Fix making chat on Obsidian backward compatible post chat session API updates
New API always has conversation_id set, not `chat' which can be unset
when chat session is empty.
So use conversation_id to decide whether to get chat logs from
`responseJson.response.chat' or `responseJson.response' instead
---------
Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
* Add location metadata to chat history
* Add support for custom configuration of the user name
* Add region, country, city in the desktop app's URL for context in chat
* Update prompts to specify user location, rather than just location.
* Add location data to Obsidian chat query
* Use first word for first name, last word for last name when setting profile name
* Add chat sessions to the desktop application
* Increase width of the main chat body to 90vw
* Update the version of electron
* Render the default message if chat history fails to load
* Merge conversation migrations and fix slug setting
* Update the welcome message, use the hostURL, and update background color for chat actions
* Only update the window's web contents if the page is config
* Enable support for multiple chat sessions within the web client
- Allow users to create multiple chat sessions and manage them
- Give chat session slugs based on the most recent message
- Update web UI to have a collapsible menu with active chats
- Move chat routes into a separate file
* Make the collapsible side panel more graceful, improve some styling elements of the new layout
* Support modification of the conversation title
- Add a new field to the conversation object
- Update UI to add a threedotmenu to each conversation
* Get the default conversation if a matching one is not found by id
- Obsidian mobile uses capacitor js. Requests from it have origin as
http://localhost on Android and capacitor://localhost on iOS
- Allow those Obsidian mobile origins in CORS middleware of server
* Fix license in pyproject.toml. Remove unused utils.state import
* Use single debug mode check function. Disable telemetry in debug mode
- Use single logic to check if khoj is running in debug mode.
Previously there were 3 different variants of the check
- Do not log telemetry if KHOJ_DEBUG is set to true. Previously didn't
log telemetry even if KHOJ_DEBUG set to false
* Respect line breaks in user, khoj chat messages to improve formatting
* Disable Whatsapp config section on web client if Twilio not configured
Simplify Whatsapp configuration status checking js by standardizing
external input to lower case
* Disable Phone API when Twilio not setup and rate limit calls to it
- Move phone api to separate router and only enable it if Twilio enabled
- Add rate-limiting to OTP and verification calls
* Add slugs for phone rate limiting
---------
Co-authored-by: sabaimran <narmiabas@gmail.com>
- Capabillity
New default UX has 1 long-press to send transcribed audio message
- Removes the previous default of 3 clicks required to send audio message
- The record > stop > send process to send audio messages was unclear
- Still allows stopping message from being sent, if users want to make
correction to transcribed audio
- Removes inadvertent long audio transcriptions if user forgets to
press stop when recording
- Changes
- Record audio while microphone button pressed
- Show auto-send 3s countdown timer UI for audio chat message
Provide a visual cue around send button for how long before audio
message is automatically sent to Khoj for response
- Auto-send msg in 3s unless stop send message button clicked
- Capabillity
New default UX has 1 long-press to send transcribed audio message
- Removes the previous default of 3 clicks required to send audio message
- The record > stop > send process to send audio messages was unclear
- Still allows stopping message from being sent, if users want to make
correction to transcribed audio
- Removes inadvertent long audio transcriptions if user forgets to
press stop when recording
- Changes
- Record audio while microphone button pressed
- Show auto-send 3s countdown timer UI for audio chat message
Provide a visual cue around send button for how long before audio
message is automatically sent to Khoj for response
- Auto-send msg in 3s unless stop send message button clicked
- Move delete icon to left of chat input. This makes it harder to
inadvertently click
- Add send button to chat footer. Enter being the only way to send
messages is not intuitive, outside standard modern UI patterns
- Color chat message send button to make it primary CTA on web client
- Make chat footer shorter. Use no or round border on action buttons
- Use SVG icons in chat footer on web
- Move delete icon to left of chat input. This makes it harder to
inadvertently click
- Add send button to chat footer. Enter being the only way to send
messages is not intuitive, outside standard modern UI patterns
- Color chat message send button to make it primary CTA on web client
- Make chat footer shorter. Use no or round border on action buttons
- Issue
Users with Dataview plugin would have error as its markdown
post-processor expects the sourcePath to be a string
This prevents Khoj from responding to chat messages in the Obsidian
chat modal. Search via Obsidian still works but it throws the same
dataview plugin error
- Fix
Pass a string as sourcePath to markdownRenderer to fix failing chat response
and stop throwing dataview errors on search
Resolves#614, Resolves#606
Server connection check can be a little flaky in Obsidian. Don't gate
the commands behind it to improve usability of Khoj.
Previously the commands would get disabled when server connection
check failed, even though server was actually accessible
- Update server connection status on every edit of khoj url, api key in
settings instead of only on plugin load
The error message was stale if connection fixed after changes in
Khoj plugin settings to URL or API key, like on plugin install
- Show better welcome message on first plugin install.
Include API key setup instruction
- Show logged in user email on Khoj settings page
- Issue: Users with Dataview plugin would have error as its markdown
post-processor expects the sourcePath to be a string
This prevents Khoj from responding to chat messages in the Obsidian
chat modal. Search via Obsidian still works but it throws the same
dataview error
- Fix: Pass a string as sourcePath to markdownRenderer to fix
failing chat response
Resolves#614, Resolves#606
### Major
- Short-circuit API rate limiter for unauthenticated user
Calls by unauthenticated users were failing at API rate limiter as it
failed to access user info object. This is a bug.
API rate limiter should short-circuit for unauthenicated users so a
proper Forbidden response can be returned by API
Add regression test to verify that unauthenticated users get 403
response when calling the /chat API endpoint
### Minor
- Remove trailing slash to normalize khoj url in obsidian plugin settings
- Move used /api/config API controllers into separate module
- Delete unused /api/beta API endpoint
- Fix error message rendering in khoj.el, khoj obsidian chat
- Handle deprecation warnings for subscribe renew date, langchain, pydantic & logger.warn
- Use /api/health for server up check instead of api/config/default
- Remove unused `khoj--post-new-config' method
- Remove the now unused /config/data GET, POST API endpoints
* Support using hosted Huggingface inference endpoint for embeddings generation
* Since the huggingface inference endpoint is model-specific, make the URL an optional property of the search model config
* Handle ECONNREFUSED error in desktop app
* Drive API key via the search model config model and use more generic names
FastAPI API endpoints only support uploading 1000 files at a time.
So split all files to index into groups of 1000 for upload to
index/update API endpoint
FastAPI API endpoints only support uploading 1000 files at a time.
So split all files to index into groups of 1000 for upload to
index/update API endpoint
Previously you could only index org-mode files and directories from
khoj.el
Mark the `khoj-org-directories', `khoj-org-files' variables for
deprecation, since `khoj-index-directories', `khoj-index-files'
replace them as more appropriate names for the more general case
Resolves#597
- Improve the prompt before sending it for image generation
- Update the help message to include online, image functionality
- Improve styling for the voice, trash buttons
The source URL returned by OpenAI would expire soon. This would make
the chat sessions contain non-accessible images/messages if using
OpenaI image URL
Get base64 encoded image from OpenAI and store directly in
conversation logs. This resolves the image link expiring issue
- Use new style references for Khoj chat modal in Obsidian
- Khoj Chat responses in Obsidian had regressed to not show references
for new questions after modal has been opened. Now even those are
rendered, and use new references style
- Render chat response as markdown while it's being streamed
- Add transcription button with mic icon
- Collect audio recording on pressing mic
- Process and send audio recording to server for transcription
- Extract the functionality to flash status in chat input for reuse
- Extract flashing status message in chat input placeholder into
reusable function
- Use emoji prefixes for status messages
- Improve alt text of transcribe button to indicate what the button does
- Conflicts:
- src/interface/desktop/chat.html
Combine and use common class names for speak component
- src/khoj/database/adapters/__init__.py
Combine imports
- src/khoj/interface/web/chat.html
Combine and use common class names for speak component
- src/khoj/routers/api.py
Combine imports
- Ignore errors in deleting network requests to khoj server
- Also delete open network connection to khoj server on auto reindex
Otherwise when server is unreachable a bunch of failed network
connections accrue in the processes list
- Make auto-update of content index user configurable from khoj.el
- Handle server unavailable error on auto-index schedule job in khoj.el
Resolves#567
