1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-27 06:38:08 +00:00
Commit graph

4002 commits

Author SHA1 Message Date
Debanjum Singh Solanky
b673d26a12 Extract Entries in a standardized format across text search types
Issue:
 - Had different schema of extracted entries for symmetric_ledger vs asymmetric

 - Entry extraction for asymmetric was dirty, relying on cryptic
   indices to store raw entry vs cleaned entry meant to be passed to embeddings

 - This was pushing the load of figuring out what property to extract
   from each entry to downstream processes like the filters

 - This limited the filters to only work for asymmetric search, not for
   symmetric_ledger

- Fix
   - Use consistent format for extracted entries
     {
       'embed': entry_string_meant_to_be_passed_to_model_and_get_embeddings,
       'raw'  : raw_entry_string_meant_to_be_passed_to_use
     }

 - Result
   - Now filters can be applied across search types, and the specific
     field they should be applied on can be configured by each search
     type
2022-07-19 20:52:25 +04:00
Debanjum Singh Solanky
e66cd5bf59 Only extract transactions from Beancount
- Earlier was extracting all entries starting with dates but the other
  type of entries like account open/close, asserts etc aren't useful for
  querying
2022-07-19 19:50:58 +04:00
Debanjum Singh Solanky
732b2d287f Give the project a short, less generic name. Rename it to Khoj
- Semantic Search was just a placeholder used to test the idea out
  Didn't want to get into naming at that point of time
2022-07-19 18:26:16 +04:00
Debanjum Singh Solanky
989526ae54 Use a more accurate model for symmetric semantic search
- The all-MiniLM-L6-v2 is more accurate
  - The exact previous model isn't benchmarked but based on the
    performance of the closest model to it. Seems like the new model
    maybe similar in speed and size

- On very preliminary evaluation of the model, the new model seems
  faster, with pretty decent results
2022-07-18 20:27:26 +04:00
Debanjum Singh Solanky
4a90972e38 Use a better model for asymmetric semantic search
- The multi-qa-MiniLM-L6-cos-v1 is more extensively benchmarked[1]
- It has the right mix of model query speed, size and performance on benchmarks
- On hugging face it has way more downloads and likes than the msmarco model[2]
- On very preliminary evaluation of the model
  - It doubles the encoding speed of all entries (down from ~8min to 4mins)
  - It gave more entries that stay relevant to the query (3/5 vs 1/5 earlier)

[1]: https://www.sbert.net/docs/pretrained_models.html
[2]: https://huggingface.co/sentence-transformers
2022-07-18 20:27:26 +04:00
Debanjum Singh Solanky
5e302dbcda Fix using 1 column layout on small screens 2022-07-18 02:40:16 +04:00
Debanjum Singh Solanky
7d16b673b1 Use Single Column Layout for Small Screens on Web Interface 2022-07-18 02:08:52 +04:00
Debanjum Singh Solanky
31a221a76b Auto focus cursor on query input box to simplify, speed interactions
- Avoids having to click the query input box
- Just open page, type whatever and hit enter to do image search
  - For other search types select appropriate type from dropdown
2022-07-16 19:39:15 +04:00
Debanjum Singh Solanky
06b0c720d6 Improve Rendering of Image Search Results in Emacs
- Use shr to render image response from html in result buffer
  Earlier was using org-mode. But rendering HTML with shr seems cleaner
- Use Headings to Add highlights
- Use Random to Force fetch of Image. Similar to what was done for Web interface
- Remove trailing elisp brackets from response
- Show query match scores by image model for each image in results
2022-07-16 19:31:49 +04:00
Debanjum Singh Solanky
28ec9af589 Extract image URL location from response in elisp after API update 2022-07-16 18:43:55 +04:00
Debanjum Singh Solanky
47613cba1f Improve Landing Page Look in General and Layout for Mobile
- Ask for 6 Images to Fill Grid into 3x2 Layout
- Submit Form on Hitting Enter
2022-07-16 16:55:13 +04:00
Debanjum Singh Solanky
cf207d6ebe Add title, heading to the semantic search web interface 2022-07-16 03:44:29 +04:00
Debanjum Singh Solanky
e0d8398b27 Normalize metadata match score to work better with image match score
- Metadata match score were consistently giving higher scores by a
  factor of ~3x wrt to image match score. This was resulting in all
  results being from the metadata match with query and none from the
  image match with query.
- Scaling the metadata match scores down by scaling factor seems to
  give more consistently give a blend of results from both image and
  metadata matches
2022-07-16 03:39:33 +04:00
Debanjum Singh Solanky
a3fc82817d Log and continue on image metadata encoding error due to Tensor size mismatch 2022-07-16 03:39:19 +04:00
Debanjum Singh Solanky
f26d0ddbbd Minor fix to asymmetric search when no entries returned 2022-07-16 03:36:19 +04:00
Debanjum Singh Solanky
ca3f93e641 Add button on web interface to regenerate embeddings of specified type 2022-07-16 03:36:19 +04:00
Debanjum Singh Solanky
231cc91e14 Force reload of images every time user clicks search button
Adding a random, unused url param at the end of the img.src string
fixes the issue. As the browser thinks it's a new image and doesn't
use the image data that's already cached because of which it wasn't
even making the fetch call for the image
2022-07-16 03:36:19 +04:00
Debanjum Singh Solanky
a6aef62a99 Create Basic Landing Page to Query Semantic Search and Render Results
- Allow viewing image results returned by Semantic Search.
  Until now there wasn't any interface within the app to view image
  search results. For text results, we at least had the emacs interface

- This should help with debugging issues with image search too
  For text the Swagger interface was good enough
2022-07-16 03:36:19 +04:00
Debanjum Singh Solanky
4e27ae0577 Ease access to image result for given query by image_search
- Copy images to accessible directory
- Return URL paths to them to ease access
- This is to be used in the web interface to render image results
  directly in browser
- Return image, metadata scores for each image in response as well
  This should help get a better sense of image scores along both
  XMP metadata and whole image axis
2022-07-16 03:36:19 +04:00
Debanjum Singh Solanky
801e59a20d Allow explicit filters when querying Ledger transactions 2022-07-15 23:41:54 +04:00
Debanjum Singh Solanky
0e979587e0 Add configurable filter support to Symmetric Ledger Search 2022-07-14 23:40:41 +04:00
Debanjum Singh Solanky
50658453cd Add separate conda environment.yml for osx-arm64
Conda doesn't support using the same environment across platforms
We were able to get away with this till now because of manually
setting up the conda environment.yml
But it's more robust to just add conda environment YAML files for each
platform as necessary
2022-07-14 23:16:49 +04:00
Debanjum
152e11ac8a
Add Date Range Filter to Search
Goal
--
Allow Limiting Search to Entries in Specified Date Range

Example Queries:
---
  - _Traveled for work internationally dt>"2 years ago"_
     Finds relevant notes since start of 2020
  - _Learnt a cool new skill dt="last month"_ 
     Finds relevant notes anytime in the last month
  - _Filed my taxes dt>="Jan 1984" dt<="April 1984"_ 
    Any tax related notes between 1st Jan 1984 to 30th April of 1984

Details
--
- Parse natural language dates in query into date ranges
  - Use `dateparser` library to parse natural language dates. But tune results to return more natural date ranges
   - Example: A user asking for entries from April, requires looking for entries in the whole of April, not April 1st or April 30th
- Find all dates in entry (currently limited to YYYY-MM-DD format)
- Only perform semantic search on entries within date range specified in query by user
2022-07-14 12:15:55 -07:00
Debanjum Singh Solanky
85077bc1d1 Handle unparseable date range passed via date filter in query
- Do not reuse the same list
- Just create new list, so only parsed data is in it
2022-07-14 22:47:23 +04:00
Debanjum Singh Solanky
a60de2c02b Include date filter in asymmetic search on music as well 2022-07-14 22:37:17 +04:00
Debanjum Singh Solanky
e96253a7c1 Add dateparser library to conda environment YAML 2022-07-14 22:29:07 +04:00
Debanjum Singh Solanky
c3b3e8959d Put entry splitting regex in explicit filter into a variable for code readability 2022-07-14 22:00:10 +04:00
Debanjum Singh Solanky
3aac3c7d52 Run explicit filter on raw entry, add more terms to split entries by
- With \t Last Word in Headings was suffixed by \t and so couldn't be
filtered by
- User interacts with raw entries, so run explicit filters on raw entry
   - For semantic search using the filtered entry is cleaner, still
2022-07-14 21:54:04 +04:00
Debanjum Singh Solanky
7640e2ab0c Wrap attempt to extract dates from entry in try/catch
- Not all YYYY-MM-DD strings in entry are necessarily dates
2022-07-14 21:38:00 +04:00
Debanjum Singh Solanky
9de2097182 Fix date filter usage with multi word queries. Simplify date regex 2022-07-14 21:34:33 +04:00
Debanjum Singh Solanky
67e9366c0f Minor style fix. Use consistent/standard dates for date_filter tests 2022-07-14 20:06:39 +04:00
Debanjum Singh Solanky
dcb6fe479e Fix date_filter query, entry in query range check. Add tests for it
- Fix date_filter date_in_entry within query range check
  - Extracted_date_range is in [included_date, excluded_date) format
  - But check was checking for date_in_entry <= excluded_date
  - Fixed it to do date_in_entry < excluded_date

- Fix removal of date filter from query
- Add tests for date_filter
2022-07-14 20:01:35 +04:00
Debanjum Singh Solanky
011f81fac5 Fix date_filter to handle non overlapping date ranges 2022-07-14 18:53:38 +04:00
Debanjum Singh Solanky
70ac35b2a5 Compute Date Range to filter entries to, from Comparators, Dates in Query 2022-07-14 18:20:09 +04:00
Debanjum Singh Solanky
e6db3e3d00 Prefer Dates From Future only when specific words in date string
- Default to looking at dates from past, as most notes are from past
- Look for dates in future for cases where it's obvious query is for
  dates in the future but dateparser's parse doesn't parse it at all.
  E.g parse('5 months from now') returns nothing

- Setting PREFER_DATES_FROM_FUTURE in this case and passing just
  parse('5 months') to dateparser.parse works as expected
2022-07-14 18:13:12 +04:00
Debanjum Singh Solanky
4a201d52af Add, test date filter regex and date parsing to get natural date range 2022-07-14 16:47:32 +04:00
Debanjum Singh Solanky
b54588717f Filter for entries with dates specified by user in query
- Create Date filter
  - Users can pass dates in YYYY-MM-DD format in their query
- Use it to filter asymmetric search to user specified dates
2022-07-14 00:51:02 +04:00
Debanjum
6356feb637
Make filters applied before semantic search configurable
Reason
--
This abstraction will simplify adding other pre-search filters. E.g A date-time filter

Capabilities
--
- Multiple filters can be applied on the query, entries etc before search
- The filters to apply are configured for each type in the search controller

Details
--
- Move `explicit_filters` function into separate module under `search_filter`
- Update signature of explicit filter to take and return `query`, `entries`, `embeddings`
- Use this `explicit_filter` function from `search_filters` module in 
   `search` method in controller
- The asymmetric query method now just applies the passed filters to the
  `query`, `entries` and `embeddings` before semantic search is performed
2022-07-13 05:53:02 -07:00
Debanjum Singh Solanky
b82aef26bf Make filters to apply before semantic search configurable
Details
--
- The filters to apply are configured for each type in the search controller
- Muliple filters can be applied on the query, entries etc before search
- The asymmetric query method now just applies the passed filters to the
  query, entries and embeddings before semantic search is performed

Reason
--
This abstraction will simplify adding other pre-search filters. E.g datetime filter
2022-07-13 16:37:09 +04:00
Debanjum Singh Solanky
c92789d20a Extract explicit pre-search filter function into a separate module
Details
--
- Move explicit_filters function into separate module under search_filter
- Update signature of explicit filter to take and return query, entries, embeddings
- Use this explicit_filter func from search_filters module in query

Reason
--
Abstraction will simplify adding other pre-search filters. E.g datetime filter
2022-07-13 16:20:04 +04:00
Debanjum
589bfa9424
Run Explicit Filter on Entries, Embeddings before Semantic Search for Query
## Issue
  - Explicit filtering was being done after search by the bi-encoder
     but before re-ranking by the cross-encoder

  - This limited the quality of results being returned for queries with explicit filters. 
     The bi-encoder returned results which were going to be excluded. 
     So the burden of improving those limited results post filtering was on the
     cross-encoder, by re-ranking the remaining results to best match the query

## Fix
  - Given that the entry and its embedding are at the same index in their respective lists. 
     We know which entries map to which embedding tensors.
     So we can run the filter for blocked, required words before the bi-encoder search. 
     And limit entries, embeddings being considered for the current query

## Result
  - Semantic search by the bi-encoder returns the most relevant results 
     for the query, knowing that the results aren't going to be filtered out after. 
     So the cross-encoder shoulders less of the burden of improving the results

## Corollary
  - This pre-filtering technique allows us to apply other explicit filters
     on entries relevant for the current query, before calling search
     - E.g limit search to entries within date/time specified in query
2022-07-12 13:12:22 -07:00
Debanjum Singh Solanky
741fca0e6b Fix asymmetric search test to pass entries returned by query to collate_results 2022-07-12 18:48:49 +04:00
Debanjum Singh Solanky
6d7ab50113 Run Explicit Filter on Entries, Embeddings before Semantic Search for Query
- Issue
  - Explicit filtering was earlier being done after search by bi-encoder
    but before re-ranking by cross-encoder

  - This was limiting the quality of results being returned. As the
    bi-encoder returned results which were going to be excluded. So the
    burden of improving those limited results post filtering was on the
    cross-encoder by re-ranking the remaining results based on query

- Fix
  - Given the embeddings corresponding to an entry are at the same index
    in their respective lists. We can run the filter for blocked,
    required words before the search by the bi-encoder model. And limit
    entries, embeddings being considered for the current query

- Result
  - Semantic search by the bi-encoder gets to return most relevant
    results for the query, knowing that the results aren't going to be
    filtered out after. So the cross-encoder shoulders less of the
    burden of improving results

- Corollary
  - This pre-filtering technique allows us to apply other explicit
    filters on entries relevant for the current query
    - E.g limit search for entries within date/time specified in query
2022-07-12 18:25:42 +04:00
sabaimran
36ef37e940
Fix formatting for pytest command
Use org formatting rather than md.
2022-07-08 10:18:26 -04:00
sabaimran
d6945f4f6b
Merge pull request from debanjum/saba/fix-docker-build
Address Issues with Docker builds
2022-07-06 21:32:37 -04:00
Saba
2eb44c7a64 Correct syntax of memory limit in docker-compose.yml 2022-07-06 20:07:11 -04:00
Debanjum Singh Solanky
8bb9a49994 Cleanup Test Asymmetric Search to Fix Test
- test_regenerate_with_valid_content failed when run after test_asymmetric_search
- test_asymmetric_search did't clean the temporary update to config it had made
- This was resulting in regenerate looking for a file that didn't exist
2022-07-07 01:25:31 +04:00
Saba
7bb35ccc7e Run build on PR 2022-07-04 18:09:47 -04:00
Saba
07a56c4ab6 Add specific version for Python packages and downgrade miniconda Docker image to potentially fix build issues 2022-07-04 18:01:55 -04:00
Saba
0f88abd219 Allocate 8GB of memory to docker container. Adjust path to Dockerfile in Github action 2022-07-04 14:01:59 -04:00