sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-20 03:17:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
khoj/src/interface/emacs/khoj.el

221 lines
8.8 KiB
EmacsLisp
Raw Normal View History

Use khoj-incremental as the main khoj func. Rename khoj to khoj-simple - Update khoj-simple to work cross-encoder re-ranked results like before - Increment major version as incremental search considered a breaking change and a major update to search capability
2022-07-27 14:18:17 +00:00
;;; khoj.el --- Natural, Incremental Search via Emacs
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
;; Copyright (C) 2021-2022 Debanjum Singh Solanky
;; Author: Debanjum Singh Solanky <debanjum@gmail.com>
Use khoj-incremental as the main khoj func. Rename khoj to khoj-simple - Update khoj-simple to work cross-encoder re-ranked results like before - Increment major version as incremental search considered a breaking change and a major update to search capability
2022-07-27 14:18:17 +00:00
;; Version: 2.0
Support Natural Search on Markdown Files - Reason: Allow natural search on markdown based notes, documentation, websites etc - Details: - Create markdown processor to extract Markdown entries (identified by Heading) into standard jsonl format required by text_search - Update API, Configs to support interfacing with new markdown type - Update Emacs, Web clients to support interfacing with new markdown type via API - Update Readme to mentiond markdown is also supported Closes #35
2022-07-21 16:22:24 +00:00
;; Keywords: search, org-mode, outlines, markdown, image
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
;; URL: http://github.com/debanjum/khoj/interface/emacs
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
;; This file is NOT part of GNU Emacs.
;;; License:
;; This program is free software; you can redistribute it and/or
;; modify it under the terms of the GNU General Public License
;; as published by the Free Software Foundation; either version 3
;; of the License, or (at your option) any later version.
;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;; GNU General Public License for more details.
;; You should have received a copy of the GNU General Public License
;; along with this program. If not, see <http://www.gnu.org/licenses/>.
;;; Commentary:
Use khoj-incremental as the main khoj func. Rename khoj to khoj-simple - Update khoj-simple to work cross-encoder re-ranked results like before - Increment major version as incremental search considered a breaking change and a major update to search capability
2022-07-27 14:18:17 +00:00
;; This package provides a natural, incremental search interface to your
;; org-mode notes, markdown files, beancount transactions and images.
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
;; It is a wrapper that interfaces with transformer based ML models.
Use khoj-incremental as the main khoj func. Rename khoj to khoj-simple - Update khoj-simple to work cross-encoder re-ranked results like before - Increment major version as incremental search considered a breaking change and a major update to search capability
2022-07-27 14:18:17 +00:00
;; The models search capabilities are exposed via the Khoj HTTP API.
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
;;; Code:
(require 'url)
(require 'json)
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
(defcustom khoj--server-url "http://localhost:8000"
"Location of Khoj API server."
:group 'khoj
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
:type 'string)
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
(defcustom khoj--image-width 156
"Width of rendered images returned by Khoj"
:group 'khoj
Allow semantic search on images from Emacs Images are rendered inline a temporary org-mode buffer
2021-09-10 08:01:23 +00:00
:type 'integer)
Re-rank incremental search results using cross-encoder if user idle This provides a relatively smooth mechanism - to improve relevance of results on idle - while providing the rapid, incremental results while typing
2022-07-27 02:58:36 +00:00
(defcustom khoj--rerank-after-idle-time 1.0
"Idle time (in seconds) to trigger cross-encoder to rerank incremental search results"
:group 'khoj
:type 'float)
(defvar khoj--rerank-timer nil
"Idle timer to make cross-encoder re-rank incremental search results if user idle.")
Reuse code to query api, render results. Formalize method, arg names
2022-07-27 00:14:14 +00:00
(defconst khoj--query-prompt "Khoj: "
"Query prompt shown to user in the minibuffer.")
Support querying all text search types in incremental search - Before incremental search was hard-coded to only query org
2022-07-27 02:13:04 +00:00
(defvar khoj--search-type "org"
"The type of content to perform search on.")
Support Natural Search on Markdown Files - Reason: Allow natural search on markdown based notes, documentation, websites etc - Details: - Create markdown processor to extract Markdown entries (identified by Heading) into standard jsonl format required by text_search - Update API, Configs to support interfacing with new markdown type - Update Emacs, Web clients to support interfacing with new markdown type via API - Update Readme to mentiond markdown is also supported Closes #35
2022-07-21 16:22:24 +00:00
(defun khoj--extract-entries-as-markdown (json-response query)
"Convert json response from API to markdown entries"
;; remove leading (, ) or SPC from extracted entries string
(replace-regexp-in-string
"^[\(\) ]" ""
;; extract entries from response as single string and convert to entries
(format "# %s\n%s"
query
(mapcar
(lambda (args) (format "%s" (cdr (assoc 'entry args))))
json-response))))
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
(defun khoj--extract-entries-as-org (json-response query)
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
"Convert json response from API to org-mode entries"
;; remove leading (, ) or SPC from extracted entries string
(replace-regexp-in-string
"^[\(\) ]" ""
;; extract entries from response as single string and convert to entries
Set query as heading of notes search results in Emacs Org buffer
2021-09-29 20:30:15 +00:00
(format "* %s\n%s"
query
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
(mapcar
Consistently use "entry", "score" in json response for all search types - Had already made some progress on this earlier by updating the image search responses. But needed to update the text search responses to use lowercase entry and score - Update khoj.el to consume the updated json response keys for text search
2022-07-20 16:33:27 +00:00
(lambda (args) (format "%s" (cdr (assoc 'entry args))))
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
json-response))))
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
(defun khoj--extract-entries-as-images (json-response query)
Support Natural Search on Markdown Files - Reason: Allow natural search on markdown based notes, documentation, websites etc - Details: - Create markdown processor to extract Markdown entries (identified by Heading) into standard jsonl format required by text_search - Update API, Configs to support interfacing with new markdown type - Update Emacs, Web clients to support interfacing with new markdown type via API - Update Readme to mentiond markdown is also supported Closes #35
2022-07-21 16:22:24 +00:00
"Convert json response from API to html with images"
Allow semantic search on images from Emacs Images are rendered inline a temporary org-mode buffer
2021-09-10 08:01:23 +00:00
;; remove leading (, ) or SPC from extracted entries string
(replace-regexp-in-string
Improve Rendering of Image Search Results in Emacs - Use shr to render image response from html in result buffer Earlier was using org-mode. But rendering HTML with shr seems cleaner - Use Headings to Add highlights - Use Random to Force fetch of Image. Similar to what was done for Web interface - Remove trailing elisp brackets from response - Show query match scores by image model for each image in results
2022-07-16 15:31:49 +00:00
"[\(\) ]$" ""
;; remove leading (, ) or SPC from extracted entries string
(replace-regexp-in-string
"^[\(\) ]" ""
;; extract entries from response as single string and convert to entries
(format "<html>\n<body>\n<h1>%s</h1>%s\n\n</body>\n</html>"
query
(mapcar
(lambda (args) (format
"\n\n<h2>Score: %s Meta: %s Image: %s</h2>\n\n<a href=\"%s%s\">\n<img src=\"%s%s?%s\" width=100 height=100>\n</a>"
(cdr (assoc 'score args))
(cdr (assoc 'metadata_score args))
(cdr (assoc 'image_score args))
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
khoj--server-url
Improve Rendering of Image Search Results in Emacs - Use shr to render image response from html in result buffer Earlier was using org-mode. But rendering HTML with shr seems cleaner - Use Headings to Add highlights - Use Random to Force fetch of Image. Similar to what was done for Web interface - Remove trailing elisp brackets from response - Show query match scores by image model for each image in results
2022-07-16 15:31:49 +00:00
(cdr (assoc 'entry args))
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
khoj--server-url
Improve Rendering of Image Search Results in Emacs - Use shr to render image response from html in result buffer Earlier was using org-mode. But rendering HTML with shr seems cleaner - Use Headings to Add highlights - Use Random to Force fetch of Image. Similar to what was done for Web interface - Remove trailing elisp brackets from response - Show query match scores by image model for each image in results
2022-07-16 15:31:49 +00:00
(cdr (assoc 'entry args))
(random 10000)))
json-response)))))
Allow semantic search on images from Emacs Images are rendered inline a temporary org-mode buffer
2021-09-10 08:01:23 +00:00
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
(defun khoj--extract-entries-as-ledger (json-response query)
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
"Convert json response from API to ledger entries"
;; remove leading (, ) or SPC from extracted entries string
(replace-regexp-in-string
Improve Results Rendered on Emacs from Semantic Search on Ledger - Add search query to top of buffer as Beancount comment - Remove trailing ) from response - Separate entries by empty line - Load beancount-mode in semantic search on ledger buffer
2022-02-26 22:33:10 +00:00
"[\(\) ]$" ""
(replace-regexp-in-string
"^[\(\) ]" ""
;; extract entries from response as single string and convert to entries
(format ";; %s\n\n%s\n"
query
(mapcar
(lambda (args)
Consistently use "entry", "score" in json response for all search types - Had already made some progress on this earlier by updating the image search responses. But needed to update the text search responses to use lowercase entry and score - Update khoj.el to consume the updated json response keys for text search
2022-07-20 16:33:27 +00:00
(format "%s\n\n" (cdr (assoc 'entry args))))
Improve Results Rendered on Emacs from Semantic Search on Ledger - Add search query to top of buffer as Beancount comment - Remove trailing ) from response - Separate entries by empty line - Load beancount-mode in semantic search on ledger buffer
2022-02-26 22:33:10 +00:00
json-response)))))
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
(defun khoj--buffer-name-to-search-type (buffer-name)
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
(let ((file-extension (file-name-extension buffer-name)))
(cond
Enable semantic search for songs in org-music Org-Music: https://github.com/debanjum/org-music
2021-08-29 10:07:36 +00:00
((equal buffer-name "Music.org") "music")
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
((equal file-extension "bean") "ledger")
Rename notes search type to org search, now that markdown notes supported
2022-07-21 17:57:57 +00:00
((equal file-extension "org") "org")
Support Natural Search on Markdown Files - Reason: Allow natural search on markdown based notes, documentation, websites etc - Details: - Create markdown processor to extract Markdown entries (identified by Heading) into standard jsonl format required by text_search - Update API, Configs to support interfacing with new markdown type - Update Emacs, Web clients to support interfacing with new markdown type via API - Update Readme to mentiond markdown is also supported Closes #35
2022-07-21 16:22:24 +00:00
((or (equal file-extension "markdown") (equal file-extension "md")) "markdown")
Rename notes search type to org search, now that markdown notes supported
2022-07-21 17:57:57 +00:00
(t "org"))))
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
Re-rank incremental search results using cross-encoder if user idle This provides a relatively smooth mechanism - to improve relevance of results on idle - while providing the rapid, incremental results while typing
2022-07-27 02:58:36 +00:00
(defun khoj--construct-api-query (query search-type &optional rerank)
(let ((rerank (or rerank "false"))
(encoded-query (url-hexify-string query)))
(format "%s/search?q=%s&t=%s&r=%s" khoj--server-url encoded-query search-type rerank)))
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
Reuse code to query api, render results. Formalize method, arg names
2022-07-27 00:14:14 +00:00
(defun khoj--query-api-and-render-results (query search-type query-url buffer-name)
;; get json response from api
(with-current-buffer buffer-name
(let ((inhibit-read-only t))
(erase-buffer)
(url-insert-file-contents query-url)))
;; render json response into formatted entries
(with-current-buffer buffer-name
(let ((inhibit-read-only t)
(json-response (json-parse-buffer :object-type 'alist)))
(erase-buffer)
(insert
(cond ((or (equal search-type "org") (equal search-type "music")) (khoj--extract-entries-as-org json-response query))
((equal search-type "markdown") (khoj--extract-entries-as-markdown json-response query))
((equal search-type "ledger") (khoj--extract-entries-as-ledger json-response query))
((equal search-type "image") (khoj--extract-entries-as-images json-response query))
(t (format "%s" json-response))))
Implement Basic Incremental Search Interface in Emacs for Org Mode Notes
2022-07-26 23:05:00 +00:00
(cond ((equal search-type "org") (org-mode))
((equal search-type "markdown") (markdown-mode))
((equal search-type "ledger") (beancount-mode))
((equal search-type "music") (progn (org-mode)
(org-music-mode)))
((equal search-type "image") (progn (shr-render-region (point-min) (point-max))
(goto-char (point-min))))
(t (fundamental-mode))))
Reuse code to query api, render results. Formalize method, arg names
2022-07-27 00:14:14 +00:00
(read-only-mode t)))
;; Incremental Search on Khoj
Re-rank incremental search results using cross-encoder if user idle This provides a relatively smooth mechanism - to improve relevance of results on idle - while providing the rapid, incremental results while typing
2022-07-27 02:58:36 +00:00
(defun khoj--incremental-query (&optional rerank)
(let* ((rerank (cond (rerank "true") (t "false")))
(search-type khoj--search-type)
Reuse code to query api, render results. Formalize method, arg names
2022-07-27 00:14:14 +00:00
(buffer-name (get-buffer-create (format "*Khoj (t:%s)*" search-type)))
(query (minibuffer-contents-no-properties))
Re-rank incremental search results using cross-encoder if user idle This provides a relatively smooth mechanism - to improve relevance of results on idle - while providing the rapid, incremental results while typing
2022-07-27 02:58:36 +00:00
(query-url (khoj--construct-api-query query search-type rerank)))
Reuse code to query api, render results. Formalize method, arg names
2022-07-27 00:14:14 +00:00
(khoj--query-api-and-render-results
Use post-command-hook to only query on user input - Hooking into after-change-functions results in system logs triggering query
2022-07-27 01:41:22 +00:00
query
search-type
query-url
buffer-name)))
Reuse code to query api, render results. Formalize method, arg names
2022-07-27 00:14:14 +00:00
(defun khoj--remove-incremental-query ()
Re-rank incremental search results using cross-encoder if user idle This provides a relatively smooth mechanism - to improve relevance of results on idle - while providing the rapid, incremental results while typing
2022-07-27 02:58:36 +00:00
(khoj--incremental-query t)
(cancel-timer khoj--rerank-timer)
Use post-command-hook to only query on user input - Hooking into after-change-functions results in system logs triggering query
2022-07-27 01:41:22 +00:00
(remove-hook 'post-command-hook #'khoj--incremental-query)
Reuse code to query api, render results. Formalize method, arg names
2022-07-27 00:14:14 +00:00
(remove-hook 'minibuffer-exit-hook #'khoj--remove-incremental-query))
;;;###autoload
Use khoj-incremental as the main khoj func. Rename khoj to khoj-simple - Update khoj-simple to work cross-encoder re-ranked results like before - Increment major version as incremental search considered a breaking change and a major update to search capability
2022-07-27 14:18:17 +00:00
(defun khoj ()
Reuse code to query api, render results. Formalize method, arg names
2022-07-27 00:14:14 +00:00
"Natural, Incremental Search for your personal notes, transactions and music using Khoj"
(interactive)
(let* ((default-type (khoj--buffer-name-to-search-type (buffer-name)))
(search-type (completing-read "Type: " '("org" "markdown" "ledger" "music") nil t default-type))
(buffer-name (get-buffer-create (format "*Khoj (t:%s)*" search-type))))
Support querying all text search types in incremental search - Before incremental search was hard-coded to only query org
2022-07-27 02:13:04 +00:00
(setq khoj--search-type search-type)
Re-rank incremental search results using cross-encoder if user idle This provides a relatively smooth mechanism - to improve relevance of results on idle - while providing the rapid, incremental results while typing
2022-07-27 02:58:36 +00:00
(setq khoj--rerank-timer (run-with-idle-timer khoj--rerank-after-idle-time t 'khoj--incremental-query t))
Reuse code to query api, render results. Formalize method, arg names
2022-07-27 00:14:14 +00:00
(switch-to-buffer buffer-name)
(minibuffer-with-setup-hook
(lambda ()
Use post-command-hook to only query on user input - Hooking into after-change-functions results in system logs triggering query
2022-07-27 01:41:22 +00:00
(add-hook 'post-command-hook #'khoj--incremental-query nil 'local)
(add-hook 'minibuffer-exit-hook #'khoj--remove-incremental-query nil 'local))
Reuse code to query api, render results. Formalize method, arg names
2022-07-27 00:14:14 +00:00
(read-string khoj--query-prompt))))
Skeleton to allow incremental search on Khoj via Emacs
2022-07-26 22:48:27 +00:00
Autoload main semantic-search function
2021-09-10 05:10:37 +00:00
;;;###autoload
Use khoj-incremental as the main khoj func. Rename khoj to khoj-simple - Update khoj-simple to work cross-encoder re-ranked results like before - Increment major version as incremental search considered a breaking change and a major update to search capability
2022-07-27 14:18:17 +00:00
(defun khoj-simple (query)
"Natural Search for your personal notes, transactions, music and images using Khoj"
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
(interactive "sQuery: ")
Use khoj-incremental as the main khoj func. Rename khoj to khoj-simple - Update khoj-simple to work cross-encoder re-ranked results like before - Increment major version as incremental search considered a breaking change and a major update to search capability
2022-07-27 14:18:17 +00:00
(let* ((rerank "true")
(default-type (khoj--buffer-name-to-search-type (buffer-name)))
Rename notes search type to org search, now that markdown notes supported
2022-07-21 17:57:57 +00:00
(search-type (completing-read "Type: " '("org" "markdown" "ledger" "music" "image") nil t default-type))
Use khoj-incremental as the main khoj func. Rename khoj to khoj-simple - Update khoj-simple to work cross-encoder re-ranked results like before - Increment major version as incremental search considered a breaking change and a major update to search capability
2022-07-27 14:18:17 +00:00
(query-url (khoj--construct-api-query query search-type rerank))
Reuse code to query api, render results. Formalize method, arg names
2022-07-27 00:14:14 +00:00
(buffer-name (get-buffer-create (format "*Khoj (q:%s t:%s)*" query search-type))))
(khoj--query-api-and-render-results
query
search-type
query-url
buffer-name)
(switch-to-buffer buffer-name)))
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
(provide 'khoj)
Add Emacs interface to semantic-search directly to main repository Too much overhead to maintain multiple repositories, especially when the Emacs library for semantic-search is a single file. Import Readme from the emacs-semantic-search repository too
2021-08-16 08:27:46 +00:00
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 14:26:16 +00:00
;;; khoj.el ends here
Reference in a new issue Copy permalink