- Previously we were statically populating types dropdown field in the web interface with all available search types
- This change populates the type dropdown field with only search types that are enabled/configured
- It queries the `/config` backend API to see which of the available search types are configured
- Populate via `.then` after enabled search types in dropdown are
populated
- Call to `/config` API is async and will usually complete after the value of type field is set from url
- So value of type field would earlier be overridden when search types
dropdown is populated after the call to `/config` API completes
- Get /config API and check config for which available search types is
populated. This gives us the list of enabled search types
- Dynamically populate search type field with enabled search types only
- Setting query value to default option when query param wasn't
passed via URL was overriding placeholder text in query field
- We wanted placeholder text in field, not the query field to actually
be populated by placeholder text
- This clears field when user starts typing query into the query field,
instead of them having to manually delete the default text populated
- Setting up default compressed-jsonl, embeddings-file was only required
for org search_type, while org-files and org-filter were allowed to be
passed as command line argument
- This avoided having to set compressed-jsonl and embeddings-file via
command line argument as well for org search type
- Now that all search types are only configurable via config file, We
can default all search types to None. The default config for the
rest of the search types wasn't being used anyway
- Previously org-files were configurable via cmdline args.
Where as none of the other search types are
- This is an artifact of how the application grew
- It can be removed for better consistency and
equal preference given all search types
Having org-mode result headings change size based on their depth in
the source document makes is a confusing UI experience.
Improve font-size, line-spacing and margins of results to make
delineation between entries, and differntiating between entry heading
and it's body easier to visually infer.
Do not white-space: pre-line. Improves rendering of Markdown results
## Details
- We were previously just wrapping results from /search API into a pre formatted div field. This was not easy to read
- Use [markdown-it](https://github.com/markdown-it/markdown-it) to render markdown results from Khoj `/search` API as proper HTML
Closes#43
## Support Incremental Search on Khoj Web Interface
- Use default, fast path to query /search API while user is typing
- Upgrade to cross-encoder re-ranked results once user hits enter on search box
## Improve Render of Org Results on Web Interface
- We were previously just wrapping results from /search API into a pre formatted div field. This was not easy to read
- Use [org.js](https://mooz.github.io/org-js/) to render results from Khoj `/search` API as proper HTML
- Improve org.js to render all task states, stylize task tags and make org-mode results look more like original content
Closes#42#41
# Details
## Improve Search API Latency
- Improve Search API Latency by ~50-100x to <100ms
- Trade-off speed for accuracy in default, fast path of /search API by not re-ranking results using cross-encoder
- Make re-ranking of results via cross-encoder configurable via new `?&r=<false|true>` query param to /search API
- Only deep-copy entries, embeddings to apply filters if query has any filter keywords
## Support Incremental Update via Khoj Emacs Frontend
- Use default, fast path to query /search API while user is typing
- Upgrade to cross-encoder re-ranked results once user goes idle (or ends search normally)
Closes#37
In current state:
- Rerank results:
- If user idles while entering query OR
- exits normally
- Do not rerank results:
- If user exits abnormally, e.g via C-g from query
- Rename functions to more standard, descriptive names
- Keep known, required code for incremental search
- E.g Do not set buffer local flag in hooks on minibuffer setup
- Only query when user in khoj minibuffer
- Use active-minibuffer-window and track khoj minibuffer
- (minibuffer-prompt) is not useful for our use-case here
- (For now) Run re-rank only if user idle while querying
- Do not run rerank on teardown/completion
- The reranking lag (~2s) is annoying; hit enter,
wait to see results
- Also triggered when user exits abnormally,
so C-g also results in rerank which is even more annoying
- Emacs will still hang if re-ranking gets triggered on idle but
that's better than always getting triggered. And better than not
having mechanism to get results re-ranked via cross-encoder at all
- Update khoj-simple to work cross-encoder re-ranked results like before
- Increment major version as incremental search considered a breaking
change and a major update to search capability
- Improve search speed by ~10x
Tested on corpus of 125K lines, 12.5K entries
- Allow cross-encoder to re-rank results by settings &?r=true when querying /search API
- It's an optional param that default to False
- Earlier all results were re-ranked by cross-encoder
- Making this configurable allows for much faster results, if desired
but for lower accuracy
- Formalize filters into class with can_filter() and filter() methods
- Use can_filter() method to decide whether to apply filter and
create deep copies of entries and embeddings for it
- Improve search speed for queries with no filters
as deep copying entries, embeddings takes the most time
after cross-encodes scoring when calling the /search API
Earlier we would create deep copies of entries, embeddings
even if the query did not contain any filter keywords
