- Use Rich to render uvicorn, fastAPI logs as well
The previous CustomFormatter only worked on khoj logs
- Improve rendering stacktrace on errors using Rich
- What
- The Emacs and Obsidian interfaces stay in their original
directories under src/
- src/khoj now only contains code meant for pypi packaging
- Benefits
- This avoids having to update khoj MELPA, Obsidian plugin config as
the Emacs, Obsidian code is under their original directories
- It separates the code in src/khoj meant for python packaging from
code for external interfaces like Emacs and Obsidian
- Why
The khoj pypi packages should be installed in `khoj' directory.
Previously it was being installed into `src' directory, which is a
generic top level directory name that is discouraged from being used
- Changes
- move src/* to src/khoj/*
- update `setup.py' to `find_packages' in `src' instead of project root
- rename imports to form `from khoj.*' in complete project
- update `constants.web_directory' path to use `khoj' directory
- rename root logger to `khoj' in `main.py'
- fix image_search tests to use the newly rename `khoj' logger
- update config, docs, workflows to reference new path `src/khoj'
- By default the obsidian plugin automatically configures the khoj
backend to index the current vault
- For more complex scenarios, users can manage their ~/.khoj/khoj.yml
manually by toggling the auto-configure setting off in the khoj
plugin settings
Resolves#156
- Background
1. Obsidian stores markdown notes as utf8[1]
2. By default, the python `open' command uses the OS locale encoding[2]
This was causing the `UnicodeDecodeError: <locale_encoding> codec can't decode byte' error
- Fix
- Read markdown files as utf8
The Obsidian plugin is the main use-case for markdown files in
khoj currently and that stores md files as utf8.
Do not assume utf8 for other content types like org-mode, beancount for now.
- Fail if error in reading file as utf8, instead of ignoring errors.
Would rather have user realize that their files are not going to
get indexed correctly.
[1]: https://forum.obsidian.md/t/better-handle-md-files-not-stored-in-utf8-format/13524/3
[2]: https://docs.python.org/3/library/functions.html#open
Khoj plugin page from within Obsidian isn't recognized. Seems like it
needs an uppercase readme file only. So it doesn't show the Khoj
readme from within Obsidian itself.
- Update khoj.el test to reflect updated rendering logic
- Move ledger render function before image rendered to group functions
with similar logic closer
Split find file, jump to file code to make onChooseSuggestion more readable
- Use find, instead of using return in forEach to get first match
- Move the jump to file+heading code out from forEach
Do not reference global app object from child objects and funcs
directly.
It is only available for debugging purposes and access to it maybe
dropped in the future.
Previously no query syntax helpers, like the "file:" prefix, were used
before checking if query contains file path.
This made query to image search brittle to misinterpretation and
pointless checking
Add test to verify search by image at file works as expected
- Support querying with text surrounding point in any text buffer
Previously could only find items similar to org entry at point
- Find similar items of specified content type indexed on khoj
Previously only looked for similar org entries indexed on khoj
Now uses the content-type configured in khoj transient menu to find
items of the specified content type
- Details
- Generalize the get-current-org-entry-text func to get text for any
outline section
- Replace leading whitespaces from query text as well
- Create method to get current paragraph text from non-outline mode
buffers
- Update transient, find-similar funcs to pass, use content-type
configured in khoj transient menu
- Generalize query title creation logic to remove markdown headings
prefix (#) apart from org heading prefix (*) as well
- Update last used khoj content-type and results from the
find-similar and update funcs for later reuse
- Jump to top of results buffer after results rendered
Enable searching for notes similar to the current note being viewed
## Main Changes
- 39a18e2 Extend search modal to search for similar notes
- Hide input field on init, Trigger search on opening modal when in similar notes mode
- Set input to contents of current markdown file and get notes similar to it
- Re-rank, by default, when searching for similar notes
- Filter out current note from similar note search results
- 0bed410 Only show `Find Similar Note' command in Editor
- Hide input field on init, Trigger search on opening modal in similar notes mode
- Set input to current markdown file and get similar notes to it
- Enable rerank when searching for similar notes
- Filter out current note from similar note search results
- Screenshot querying "Setup Editor" on test vault with Khoj Readmes
- New features showcase:
- information keybindings, rerank keybinding at bottom of modal
- fixed top level headings in search results
- search results snipped if greater than N words
- Previously top level headings would have get stripped of the
space between heading text and the prefix # symbols. That is,
`# Top Level Heading' would get converted to `#Top Level Heading'
- This would mess up their rendering as a heading in search results
- Add unit tests to text_to_jsonl processors to prevent regression
Provides a more consistent rendering of results in modal.
Makes it easier to see more results in modal.
To see complete entry, user can always just jump to entry from modal
### Overview
- Provide a chat interface to engage with and inquire your notes
- Simplify interacting with the beta `chat` and `summarize` APIs
### Use
- Open `<khoj-url>/chat`, by default at http://localhost:8000/chat?type=summarize
- Type your queries, see summarized response by Khoj from your notes
**Note**:
- **You will need to add an API key from OpenAI to your khoj.yml**
- **Your query and top note from search result will be sent to OpenAI for processing**
## Details
- 177756b Show chat history on loading chat page on web interface
- d8ee0f0 Save chat history to disk for persistence, seeing chat logs
- 5294693 Style chat messages as speech bubbles
- d170747 Add khoj web interface and chat styling to new chat page on khoj web
- de6c146 Implement functional, unstyled chat page for khoj web interface
- The previous mechanism to trigger saving on shutdown event did not work
- Use scheduler to persist chat sessions to disk at a 5 minute interval
- This improve time granularity, fixed interval of saving chat logs
- It may lose ~5 minutes of chat history until mechanism to also
write on shutdown found/resolved
- Create conversation directory if it doesn't exist before attempting write
- Reset chat_session after writing it to disk
- Wrap messages into speech bubbles
- Color messages by khoj blue, sender grey
- Add those standard protrusions to the speech bubbles for fun
- Align bubbles left or right based on sender
- messages by khoj are left aligned, message by self are right aligned
- Put message metadata like sender and time under speech bubble
- use data-* attribute and ::after css pseudo-selector for this
- Update renderMessage func to accept time param, remove unused type_ param
Not all notes are in the past. Notes can be about stuff in the future.
Casting them to past tense gives the impression that they've already
happened / been done.
- Changes
- Use blue color for khoj heading font
- This fixes the title color issue
- Update background to lighter shade
- This fixes the body text color issue
- Update colors for todo, done, miscellaneous todo state, tag color
- This does not fix the color contrast issue but seems like an acceptable solution
- Using white text rather than black text on blue background
better even though the black text on blue background passes the
WCAG acceptable contrast score
- For details see blog post:
https://uxmovement.com/buttons/the-myths-of-color-contrast-accessibility/
- Add border to tags to give them tag pills look and differntiate
from todo states
- Buttons and inputs
- Change background color of input fields like type dropdown,
update button and results count counter, to match background
color of page
- Add shadow on hover over button, dropdowns
Resolves#111
- Ensure message input box sticks to bottom of screen
- Ensure chat logs div is scrollable when logs become longer than screen
Do not make the whole page scroll, just the chat logs body div
Uses longest file path match to find markdown file in vault
corresponding to file of search result returned by Khoj
Allow jumping to search result from khoj plugin modal on Android too
Previous mechanism of manually triggering getSuggestions,
renderSuggestions flow was corrupting traversing and opening
reranked search results in KhojModal
Emulate event that would anyway trigger the get & render of results in
modal. This lets obsidian core handle the flow without digging too
deep into obsidian cores handling of the flow. Lowers the chance of
breakage
We need the index file paths to make sense on the khoj backend server
Having path of index on backend relative to current vault directory
on frontend ignores the fact that the frontend maybe on a different
machine than the khoj backend server
Using unique index name per vault allows switching vaults without
overwriting indices of other vaults created on khoj backend when khoj
obsidian plugin is loaded on opening a different vault
- Overview
Limits using Khoj with a single vault at a time. This is
automatically configured to the most recently opened vault.
Once directory filters are supported on backend, the plugin will be
updated to index multiple vault but search only current vault from
current vaults khoj obsidian plugin
- Code Details
- Remove setting to configure Vault directory from Khoj Obsidian plugin
- Automatically configure Khoj to index only current Vault.
- Overwrites any previous vaults that were intended to be indexed by
Khoj backend
- Force update of index after configuring vault
- Why
It's not helpful for now and can lead to more problems, confusion.
Once directory filters
- Previously the backend was just throwing backend error.
The frontend calling the /update API wasn't getting notified
- Now the frontend can react appropriately and make the issue
visible to the user
- Only show notification on plugin load and failure.
- In settings page, set current backend status at top of pane instead
of showing notification
Notices bubbles cluttered the UI while typing updates to settings
- Show notification once index updated via settings pane button click
There was no notification on index updated, which usually takes time
on the backend
