Commit graph

870 commits

Author SHA1 Message Date
Debanjum Singh Solanky
d5a7cc5b0f Compact code to map results from search API into SearchResult objects
Make code compact for readability
Remove unneeded temporary variables and return statements
2023-01-17 18:04:11 -03:00
Debanjum Singh Solanky
7b4f78776c Fix extracting Markdown Entries with Top Level Headings
- Previously top level headings would have get stripped of the
  space between heading text and the prefix # symbols. That is,
  `# Top Level Heading' would get converted to `#Top Level Heading'
- This would mess up their rendering as a heading in search results

- Add unit tests to text_to_jsonl processors to prevent regression
2023-01-17 13:06:28 -03:00
Debanjum Singh Solanky
1a296518c5 Limit total words for each Search Result rendered in search modal
Provides a more consistent rendering of results in modal.
Makes it easier to see more results in modal.
To see complete entry, user can always just jump to entry from modal
2023-01-17 13:06:14 -03:00
Debanjum Singh Solanky
e7b89f7fd0 Return compiled entry in additional details of /api/search response
This can be used to highlight portion of raw entry to highlight and
for passing to summarizer to stay with max_tokens limit supported by
GPT models
2023-01-16 22:56:06 -03:00
Debanjum Singh Solanky
7071d081e9 Increase max_tokens returned by GPT summarizer. Remove default params 2023-01-16 22:55:36 -03:00
Debanjum Singh Solanky
3d9cdadbbb Add codebase visualization of Khoj Obsidian to Khoj Obsidian Readme 2023-01-15 14:09:21 -03:00
Debanjum Singh Solanky
d02ba325aa Handle empty chat history returned by API to chat.html on web interface 2023-01-15 13:51:16 -03:00
Debanjum Singh Solanky
721bbbe15c Update Readme. Add Chat with Notes Section to Advanced Usage
- Add Setup OpenAI API key in Khoj Section to Miscellaneous
  Refer all mentions of setting up your OpenAI API key to that section
- Add Demo Screenshot of Chat with Notes
- Put existing Miscellaneous Section under Beta API sub heading
- Fix to make Access Khoj on Mobile a Subsection of Advanceed Usage

- Trigger refresh of github image cache by adding ? at end of image paths
2023-01-14 00:39:15 -03:00
Debanjum Singh Solanky
42f8230b37 Update Troubleshooting Section in Main Readme
- Convert Troubleshooting Issues into Headings instead of Bullets
  Allows them to be linked to more easily. E.g when pointing folks to
  it in github issues etc

- Add index corruption issue and fix to the Troubleshooting section
2023-01-13 23:03:15 -03:00
Debanjum
3f2ea039a7
Add Chat page to the Khoj Web Interface
### Overview
- Provide a chat interface to engage with and inquire your notes
- Simplify interacting with the beta `chat` and `summarize` APIs

### Use
- Open `<khoj-url>/chat`, by default at http://localhost:8000/chat?type=summarize
- Type your queries, see summarized response by Khoj from your notes

**Note**:
 - **You will need to add an API key from OpenAI to your khoj.yml**
 - **Your query and top note from search result will be sent to OpenAI for processing**

## Details
- 177756b Show chat history on loading chat page on web interface
- d8ee0f0 Save chat history to disk for persistence, seeing chat logs
- 5294693 Style chat messages as speech bubbles
- d170747 Add khoj web interface and chat styling to new chat page on khoj web
- de6c146 Implement functional, unstyled chat page for khoj web interface
2023-01-13 23:02:19 -03:00
Debanjum Singh Solanky
16d4560ff8 Comment css styling of chat page for later reference 2023-01-13 22:40:01 -03:00
Debanjum Singh Solanky
cfef346d03 Do not update query field to ever chat message
It doesn't work as well with chat, unlike for search page
Use more appropriate thinking face emoji for you instead of surprise face
2023-01-13 22:24:26 -03:00
Debanjum Singh Solanky
177756be7e Fetch chat history from backend and render it on chat page load 2023-01-13 22:01:57 -03:00
Debanjum Singh Solanky
330febaa1a Update conversation logs from /beta/summary API endpoint too 2023-01-13 22:01:57 -03:00
Debanjum Singh Solanky
cb6f0b53c9 Make user_message_metadata arg to message_to_log in gpt.py optional
- Use a default user_message_metadata if arg not set
- Update conversation to use `by' as `you' and `khoj'
2023-01-13 22:01:57 -03:00
Debanjum Singh Solanky
cc2456e411 Update /beta/chat API to return chat history if no query param passed 2023-01-13 22:01:57 -03:00
Debanjum Singh Solanky
d8ee0f0e9a Use scheduler to save chat history to disk every 5 minutes
- The previous mechanism to trigger saving on shutdown event did not work
- Use scheduler to persist chat sessions to disk at a 5 minute interval
  - This improve time granularity, fixed interval of saving chat logs
  - It may lose ~5 minutes of chat history until mechanism to also
    write on shutdown found/resolved
- Create conversation directory if it doesn't exist before attempting write
- Reset chat_session after writing it to disk
2023-01-13 22:01:57 -03:00
Debanjum Singh Solanky
5294693e97 Style message as speech bubbles on chat page of web interface
- Wrap messages into speech bubbles
  - Color messages by khoj blue, sender grey
  - Add those standard protrusions to the speech bubbles for fun

- Align bubbles left or right based on sender
  - messages by khoj are left aligned, message by self are right aligned

- Put message metadata like sender and time under speech bubble
  - use data-* attribute and ::after css pseudo-selector for this

- Update renderMessage func to accept time param, remove unused type_ param
2023-01-13 22:01:57 -03:00
Debanjum Singh Solanky
7723d656dc Do not force GPT to summarize note using past tense
Not all notes are in the past. Notes can be about stuff in the future.
Casting them to past tense gives the impression that they've already
happened / been done.
2023-01-13 13:10:35 -03:00
Debanjum Singh Solanky
2842e3a035 Automatically scroll to bottom of chat body on new messages 2023-01-13 13:09:51 -03:00
Debanjum Singh Solanky
34014635d0 Improve colors, fix contrast for accessability on web interface
- Changes
  - Use blue color for khoj heading font
    - This fixes the title color issue

  - Update background to lighter shade
    - This fixes the body text color issue

  - Update colors for todo, done, miscellaneous todo state, tag color
    - This does not fix the color contrast issue but seems like an acceptable solution
    - Using white text rather than black text on blue background
      better even though the black text on blue background passes the
      WCAG acceptable contrast score
    - For details see blog post:
      https://uxmovement.com/buttons/the-myths-of-color-contrast-accessibility/

  - Add border to tags to give them tag pills look and differntiate
    from todo states

  - Buttons and inputs
    - Change background color of input fields like type dropdown,
      update button and results count counter, to match background
      color of page
    - Add shadow on hover over button, dropdowns

Resolves #111
2023-01-12 21:59:50 -03:00
Debanjum Singh Solanky
d170747ec2 Add khoj web interface & chat styling to new chat page on khoj web
- Ensure message input box sticks to bottom of screen
- Ensure chat logs div is scrollable when logs become longer than screen
  Do not make the whole page scroll, just the chat logs body div
2023-01-12 21:58:46 -03:00
Debanjum Singh Solanky
de6c146290 Implement functional, unstyled chat page for khoj web interface
Expose it at /chat URL
2023-01-12 21:53:25 -03:00
Debanjum Singh Solanky
f0213d0a82 Fix links to install khoj.el readme from main readme 2023-01-12 02:25:00 -03:00
Debanjum Singh Solanky
e6793816f9 Upgrade Khoj.el Readme. Add TOC, Screenshot, Features Sections
- Update Query filter details
2023-01-12 02:14:02 -03:00
Debanjum Singh Solanky
2fe21f3a78 Update Advanced Usage section in main Readme
- Update Khoj PWA image to show Khoj open as PWA on Android
- Add section to show configuring Khoj to use OpenAI models for search
2023-01-12 01:49:12 -03:00
Debanjum Singh Solanky
26f791e9ad Update Obsidian Plugin Readme. Add Khoj icon to Khoj Modal Placeholder text
- Fold Query Filter, Demo Description
- Add Limitations to Readme
- Add *Update* index bullet to Troubleshooting Options
2023-01-12 01:48:52 -03:00
Debanjum Singh Solanky
3e63af5c94 Constrain grid rows to fix layout of Khoj web interface on Chrome 2023-01-12 01:48:52 -03:00
Debanjum Singh Solanky
a31002bf38 Revert obsidian plugin manifest, versions at project root to 0.2.1 2023-01-11 20:54:12 -03:00
Debanjum Singh Solanky
50c797962c Jump to Search Result from Khoj Modal even on Obsidian Android
Uses longest file path match to find markdown file in vault
corresponding to file of search result returned by Khoj

Allow jumping to search result from khoj plugin modal on Android too
2023-01-11 19:44:11 -03:00
Debanjum Singh Solanky
51ea6d9c9b Do not force index update when configure backend on plugin load
- Backend can handle incremental updates
- Avoid khoj usability delay by avoiding recomputed everytime vault opened
2023-01-11 17:17:08 -03:00
Debanjum Singh Solanky
3fe5ce2721 Merge branch 'master' of github.com:debanjum/khoj 2023-01-11 17:02:30 -03:00
Debanjum
e28af68cbd
Fix, Improve Configuring Khoj from Obsidian Plugin
### Details
- 1c813a6 Convert *Results Count* setting to `Slider` from `Text` in plugin settings pane
- 4e1abd1 Disable `Update` button in plugin settings while indexing vault
- 513c86c Set index file paths relative to current or default path on Khoj backend
- 4407e23 Only index current vault on Khoj. Remove `ObsidianVaultPath` setting from plugin
- 86a1e43 Return HTTP Exception on */api/update* API call failure
- 5af2b68 Update plugin notifications for errors. Remove notification for success
2023-01-11 17:01:33 -03:00
Debanjum Singh Solanky
123b077c68 Use apt update before apt install in test workflow on Github 2023-01-11 16:51:16 -03:00
Debanjum Singh Solanky
5996d47d7c Trigger input event to Get, Render Reranked results from Khoj backend
Previous mechanism of manually triggering getSuggestions,
renderSuggestions flow was corrupting traversing and opening
reranked search results in KhojModal

Emulate event that would anyway trigger the get & render of results in
modal. This lets obsidian core handle the flow without digging too
deep into obsidian cores handling of the flow. Lowers the chance of
breakage
2023-01-11 16:39:23 -03:00
Debanjum Singh Solanky
1c813a6884 Convert results count setting to slider in plugin settings pane 2023-01-11 16:39:23 -03:00
Debanjum Singh Solanky
4e1abd1b72 Disable update button while indexing vault in plugin settings 2023-01-11 16:39:23 -03:00
Debanjum Singh Solanky
513c86c6a1 Set index file paths relative to current or default path on khoj backend
We need the index file paths to make sense on the khoj backend server

Having path of index on backend relative to current vault directory
on frontend ignores the fact that the frontend maybe on a different
machine than the khoj backend server

Using unique index name per vault allows switching vaults without
overwriting indices of other vaults created on khoj backend when khoj
obsidian plugin is loaded on opening a different vault
2023-01-11 16:39:23 -03:00
Debanjum Singh Solanky
4407e23c19 Only index current vault on Khoj. Remove plugin setting to configure it
- Overview
  Limits using Khoj with a single vault at a time. This is
  automatically configured to the most recently opened vault.

  Once directory filters are supported on backend, the plugin will be
  updated to index multiple vault but search only current vault from
  current vaults khoj obsidian plugin

- Code Details
 - Remove setting to configure Vault directory from Khoj Obsidian plugin
 - Automatically configure Khoj to index only current Vault.
 - Overwrites any previous vaults that were intended to be indexed by
   Khoj backend
 - Force update of index after configuring vault

- Why
  It's not helpful for now and can lead to more problems, confusion.
  Once directory filters
2023-01-11 16:39:23 -03:00
Debanjum Singh Solanky
86a1e43605 Return HTTP Exception on /api/update API call failure
- Previously the backend was just throwing backend error.
  The frontend calling the /update API wasn't getting notified
- Now the frontend can react appropriately and make the issue
  visible to the user
2023-01-11 16:39:23 -03:00
Debanjum Singh Solanky
5af2b68e2b Update plugin notifications for errors and success
- Only show notification on plugin load and failure.
- In settings page, set current backend status at top of pane instead
  of showing notification
  Notices bubbles cluttered the UI while typing updates to settings
- Show notification once index updated via settings pane button click
  There was no notification on index updated, which usually takes time
  on the backend
2023-01-11 16:39:23 -03:00
Debanjum Singh Solanky
853192932a setCTA on Khoj Obsidian plugin button. Minor cleanup of space, tabs 2023-01-10 23:36:02 -03:00
Debanjum
531d423715
Enhance Search Modal, Error State Handling in Khoj Obsidian Plugin
### Search Modal Enhancements
  - b52cd85 Allow Reranking results using Keybinding from Khoj Search Modal
  - 580f4ac Add hints to Modal for available Keybindings
  - da49ea2 Add placeholder text to modal in Khoj Obsidian plugin
  
### Handle Failure to Connect to Khoj Backend
Load plugin but warn on failure to connect to Khoj backend

- f046a95 Track connectedToBackend as a setting. Use it across obsidian plugin to:
  - Disable command if not connected to backend
  - Trigger warning notice on clicking Khoj ribbon if not connected to backend
  - Show warning at top of Khoj Obsidian plugin settings pane
- 768e874 Load obsidian plugin even if fail to connect to backend but show warning
  - Allows user to see reason for failure to try resolve it
  - Allows user to update Khoj URL settings to point to URL of Khoj server
  
### Miscellaneous
- 7991ab7 Add button in Obsidian plugin settings to force re-indexing your vault
  - Useful if index gets corrupted
2023-01-10 23:20:32 -03:00
Debanjum Singh Solanky
da49ea272c Add placeholder text to modal in Khoj Obsidian plugin 2023-01-10 22:50:11 -03:00
Debanjum Singh Solanky
580f4aca23 Add hints to Modal for available Keybindings 2023-01-10 22:03:47 -03:00
Debanjum Singh Solanky
b52cd85c76 Allow Reranking Results using Keybinding from Khoj Search Modal 2023-01-10 21:59:38 -03:00
Debanjum Singh Solanky
7991ab7a86 Add button in Obsidian plugin settings to force re-indexing your vault 2023-01-10 19:49:12 -03:00
Debanjum Singh Solanky
f046a95f3d Track connectedToBackend as a setting. Use it across obsidian plugin
- Display warning at top of khoj obsidian plugin settings
- Make search command available only if connected to backend
- Show warning notice on clicking khoj search ribbon button

- Call saveData after configureKhojBackend to ensure
  connnectedToBackend setting saved after being (potentially) updated
  in configureKhojBackend function
2023-01-10 17:28:47 -03:00
Debanjum Singh Solanky
768e874185 Load obsidian plugin even if fail to connect to backend but show warning
- Previously the plugin would not load if cannot connect to Khoj backend
  - Silently failing to load with no reason provided is not helpful
- Load plugin to allow user to fix the Khoj URL in their plugin setting
- Show reason for khoj plugin not working. More helpful than failing silently
2023-01-10 17:20:02 -03:00
Debanjum Singh Solanky
aa22d83172 Create and use a context manager to time code
Use the timer context manager in all places where code was being timed

- Benefits
  - Deduplicate timing code scattered across codebase.
  - Provides single place to manage perf timing code
  - Use consistent timing log patterns
2023-01-09 19:48:16 -03:00