Commit graph

391 commits

Author SHA1 Message Date
Debanjum Singh Solanky
26b50471b7 Upgrade pillow to fix dependabot security advisory 2022-08-03 00:33:29 +03:00
Debanjum
1295ba90c3
Publish Khoj on PyPi
## Details

- Add `setup.py` to support pip install of khoj package
- Fixup Application to work with pip install
- Update Readme with instructions to install package via pip
- Publish Khoj as `khoj-assistant` on [PyPi](https://pypi.org/project/khoj-assistant/)
2022-08-03 00:29:49 +03:00
Debanjum Singh Solanky
b9e6273644 Include interfaces in pip package. Fix paths to web interface in app 2022-08-03 00:02:39 +03:00
Debanjum Singh Solanky
765e10bc2c Prepare Khoj for PyPi. Include Readme in dist, Fix metadata in setup.py 2022-08-02 22:53:02 +03:00
Debanjum Singh Solanky
d57bd41019 Add Instruction to Install, Run Khoj via Pip to Readme #Development 2022-08-02 21:12:27 +03:00
Debanjum Singh Solanky
1b55462fb0 Convert search_filter, conversation dir to proper modules
Add __init__.py files to their directories
2022-08-02 20:23:42 +03:00
Debanjum Singh Solanky
006a2aef57 Create setup.py to setup Khoj python application 2022-08-02 20:23:05 +03:00
Debanjum Singh Solanky
5108d45951 Wrap application startup steps into a method 2022-08-02 20:13:14 +03:00
Debanjum Singh Solanky
0ebfbb43ce Nest org, md results at level 2 on Emacs interface. Improve readability
- Makes it easier to fold/unfold, traverse and read results
- This 2 level nesting is already being used on the web interface

- Previously we were using the original nesting depth of the entry.
  This was aimed at providing more of the orginal context of the
  results. But currently this additional information does not provide
  as much, for the decreased legibility of the results
2022-08-01 04:01:18 +03:00
Debanjum
dbc7ca74c9
Minor cleanup of Readme. Move demo to docs/ directory
- Move Demo to docs/ directory to clean project root
- Minor updates to Readme
2022-08-01 02:46:44 +03:00
Debanjum Singh Solanky
bcaf86f25a Minor updates to Readme
- Help test workflows not triggered on PR with changes to docs
2022-08-01 02:42:48 +03:00
Debanjum Singh Solanky
f54f414893 Move demo video to docs/ directory to keep project root clean 2022-08-01 02:41:54 +03:00
Debanjum Singh Solanky
0797e9cef8 Run github workflows only when relevant paths are modified 2022-08-01 02:30:23 +03:00
Debanjum
06499da0c3
Align Config Web UI Code Layout, API with the rest of the application
- Put code to configure app via web interface under `src/interface/web` directory with the rest of the web interface code
- Make config web UI available under `/config` instead of at the generic `/ui` endpoint
- Rename `/config` API endpoint (that gets/sets config from/to yaml file) to `/config/data`
- Add Khoj App title, favicon to config web page
2022-08-01 01:53:21 +03:00
Debanjum Singh Solanky
1201bfddf3 Simplify name of config css from config-style.css to config.css 2022-08-01 01:34:00 +03:00
Debanjum Singh Solanky
075dba5d64 Use Khoj Title, Favicon in Config Page for Consistency 2022-08-01 01:27:14 +03:00
Debanjum Singh Solanky
56a4429f01 Move web interface to configure application into src/interface/web directory
- Improve code layout by ensuring all web interface specific code
  under the src/interface/web directory
- Rename config API to more specifi /config instead of /ui
- Rename config data GET, POST api to /config/data instead of /config
2022-08-01 00:53:42 +03:00
Debanjum
bb2ccec1ca
Populate type dropdown on the web interface with only enabled search types
- Previously we were statically populating types dropdown field in the web interface with all available search types
- This change populates the type dropdown field with only search types that are enabled/configured
- It queries the `/config` backend API to see which of the available search types are configured
2022-08-01 00:20:45 +03:00
Debanjum Singh Solanky
8b6058c879 Fix instantiating type field with value from URL query parameter
- Populate via `.then` after enabled search types in dropdown are
  populated
- Call to `/config` API is async and will usually complete after the value of type field is set from url
- So value of type field would earlier be overridden when search types
  dropdown is populated after the call to `/config` API completes
2022-08-01 00:04:50 +03:00
Debanjum Singh Solanky
7d7259bd92 Remove tests that validate configuring org using commandline arguments 2022-07-31 23:42:00 +03:00
Debanjum Singh Solanky
be253bab39 Populate type dropdown with only enabled search types in web interface
- Get /config API and check config for which available search types is
  populated. This gives us the list of enabled search types
- Dynamically populate search type field with enabled search types only
2022-07-31 23:42:00 +03:00
Debanjum Singh Solanky
0abd40aeb7 Only set query field when appropriate query param passed via URL
- Setting query value to default option when query param wasn't
  passed via URL was overriding placeholder text in query field

- We wanted placeholder text in field, not the query field to actually
  be populated by placeholder text

- This clears field when user starts typing query into the query field,
  instead of them having to manually delete the  default text populated
2022-07-31 22:29:23 +03:00
Debanjum Singh Solanky
17c38b526a Default config for each search types to None
- Setting up default compressed-jsonl, embeddings-file was only required
  for org search_type, while org-files and org-filter were allowed to be
  passed as command line argument
- This avoided having to set compressed-jsonl and embeddings-file via
  command line argument as well for org search type
- Now that all search types are only configurable via config file, We
  can default all search types to None. The default config for the
  rest of the search types wasn't being used anyway
2022-07-31 22:23:57 +03:00
Debanjum Singh Solanky
b83021a723 Improve code readability of merge_dicts helper method 2022-07-31 22:07:56 +03:00
Debanjum Singh Solanky
38aede68f2 Only configure org via config file for consistency across search types
- Previously org-files were configurable via cmdline args.
  Where as none of the other search types are
- This is an artifact of how the application grew
- It can be removed for better consistency and
  equal preference given all search types
2022-07-31 22:02:03 +03:00
Saba
b55159f5bd Fix URL for khoj.el quelpa setup instructions 2022-07-29 23:01:04 -04:00
Debanjum Singh Solanky
da118b3fed Simplify incremental search function used in web interface
Re-rank isn't passed to image search API in search function.
So don't need to check type in incremental_search function too
2022-07-29 23:18:01 +04:00
Debanjum Singh Solanky
3079614981 Allow set up of search form via query params in web interface
- Default search type to org, instead of images
2022-07-29 23:13:26 +04:00
Debanjum Singh Solanky
02ca2c05a1 Add Eagle Icon for Khoj to Web, Emacs Interfaces and Readme 2022-07-29 17:50:29 +04:00
Debanjum Singh Solanky
8290ef2a60 Convert Readme to Markdown from Org as better support on Github 2022-07-29 17:08:30 +04:00
Debanjum Singh Solanky
78314263a0 Add Table of Contents, Features, Performance Details to Readme 2022-07-29 17:08:17 +04:00
Debanjum
2d0d85cfda Add Incremental Search Demo to Readme 2022-07-29 06:14:24 +04:00
Debanjum Singh Solanky
ed181f47c9 Prettify rendering of org music results on Khoj web interface 2022-07-29 04:28:22 +04:00
Debanjum Singh Solanky
7e5291a38e Make org result headings at same level. Improve spacing of results
Having org-mode result headings change size based on their depth in
the source document makes is a confusing UI experience.

Improve font-size, line-spacing and margins of results to make
delineation between entries, and differntiating between entry heading
and it's body easier to visually infer.

Do not white-space: pre-line. Improves rendering of Markdown results
2022-07-29 01:55:46 +04:00
Debanjum Singh Solanky
4d5183063c Create images directory if doesn't exist, to store image search results 2022-07-28 21:30:31 +04:00
Debanjum
a29af70de5
Prettify Render of Markdown results on Web Interface
## Details
- We were previously just wrapping results from /search API into a pre formatted div field. This was not easy to read
- Use [markdown-it](https://github.com/markdown-it/markdown-it) to render markdown results from Khoj `/search` API as proper HTML

Closes #43
2022-07-28 10:13:40 -07:00
Debanjum Singh Solanky
a9bc17a6b0 Prettify Render of Markdown Results in Web Interface 2022-07-28 20:56:37 +04:00
Debanjum Singh Solanky
a6ae74f52e Move JS files like org.js into a separate assets/ directory 2022-07-28 20:46:48 +04:00
Debanjum Singh Solanky
a12eaa4ce0 Move Khoj image results into a child images/ directory 2022-07-28 20:45:12 +04:00
Debanjum
a71253e137
Support Incremental Search on Web Interface
## Support Incremental Search on Khoj Web Interface
- Use default, fast path to query /search API while user is typing
- Upgrade to cross-encoder re-ranked results once user hits enter on search box

## Improve Render of Org Results on Web Interface
- We were previously just wrapping results from /search API into a pre formatted div field. This was not easy to read
- Use [org.js](https://mooz.github.io/org-js/) to render results from Khoj `/search` API as proper HTML
- Improve org.js to render all task states, stylize task tags and make org-mode results look more like original content

Closes #42 #41
2022-07-28 09:31:57 -07:00
Debanjum
8a4ae75897
Support Incremental Search in Khoj
# Details
## Improve Search API Latency
- Improve Search API Latency by ~50-100x to <100ms
- Trade-off speed for accuracy in default, fast path of /search API by not re-ranking results using cross-encoder
- Make re-ranking of results via cross-encoder configurable via new `?&r=<false|true>` query param to /search API
- Only deep-copy entries, embeddings to apply filters if query has any filter keywords

## Support Incremental Update via Khoj Emacs Frontend
- Use default, fast path to query /search API while user is typing
- Upgrade to cross-encoder re-ranked results once user goes idle (or ends search normally)

Closes #37
2022-07-28 09:10:50 -07:00
Debanjum Singh Solanky
e8029bf415 Extract and Highlight org-mode tags in HTML render of search results 2022-07-28 19:55:15 +04:00
Debanjum Singh Solanky
c6c248df26 Improve styling of org-mode results to original alignment, line breaks 2022-07-28 19:55:15 +04:00
Debanjum Singh Solanky
9f59897eeb Highlight all org-mode task states in HTML. Not just TODO, DONE.
- Make logic to extract, mark todo state in org.js more generic
- Add default todo state styling to html
2022-07-28 19:55:15 +04:00
Debanjum Singh Solanky
f040b3f65c Stylize TODO/DONE states with CSS 2022-07-28 19:55:15 +04:00
Debanjum Singh Solanky
581b6097c7 Clean Results. Remove TOC, Heading Number and Property Drawers 2022-07-28 19:55:15 +04:00
Debanjum Singh Solanky
965a93a2f2 Add Basic HTML Rendering of Org-Mode Results 2022-07-28 19:55:15 +04:00
Debanjum Singh Solanky
1da44d4dfe Add Incremental Search to Khoj Web Interface 2022-07-28 19:55:15 +04:00
Debanjum Singh Solanky
af1dd31401 Do not pass verbose argument to image_search.query() as not supported 2022-07-28 19:52:58 +04:00
Debanjum Singh Solanky
80ac10835c Rerank results on normal minibuffer exit
In current state:
 - Rerank results:
   - If user idles while entering query OR
   - exits normally

 - Do not rerank results:
   - If user exits abnormally, e.g via C-g from query
2022-07-28 03:37:16 +04:00