### Goal
- Improve explicit filter latency to work better with incremental search
### Reasons for High Explicit Filter Latency
- Deleting entries to be excluded from existing list of entries, embeddings
- Explicit filtering on partial words during incremental search
- Creating word set for all entries on the fly during query
- Deep copying of entries, embeddings before applying filter
### Improvement Details
- **Major**
- 191a656 Use word to entry map, list comprehension to speed up explicit filter
- Use list comprehension and `torch.index_select` methods
- to speed selection of entries, embedding tensors satisfying filter
- avoid deep copy and direct manipulation of entries, embeddings
- Use word to entry map and set operations to mark entries
satisfying inclusion, exclusion filters
- c7de57b Pre-compute entry word sets to improve explicit filter query performance
- 3308e68 Cache explicitly filtered entries, embeddings by required, blocked words
- cdcee89 Wrap explicit filter words in quotes to trigger filter
- E.g `+"word_to_include"` instead of `+word_to_include`
- Signals explicit filter term completed
- Prevents latency due to incremental search with explicit filtering on partial terms
- **Minor**
- 28d3dc1 Deep copy entries, embeddings in filters. Defer till actual filtering
- 8d9f507 Load entries_by_word_set from file only once on first load of explicit filter
- 546fad5 Use regex to check for and extract include, exclude filter words from query
- b7d259b Test Explicit Include, Exclude Filters
### Results
- Improve exclude word filter latency from **20s+ to 0.02s** on 120K line notes corpus
- Code Changes
- Use list comprehension and `torch.index_select' methods
- to speed selection of entries, embedding tensors satisfying filter
- avoid deep copy of entries, embeddings
- avoid updating existing lists (of entries, embeddings)
- Use word to entry map and set operations to mark entries satisfying
inclusion, exclusion filters
- Results
- Speed up explicit filtering by two orders of magnitude
- Improve consistency of speed up across inclusion and exclusion filtering
- Only the filter knows when entries, embeddings are to be manipulated.
So move the responsibility to deep copy before manipulating entries,
embeddings to the filters
- Create deep copy in filters. Avoids creating deep copy of entries,
embeddings when filter results are being loaded from cache etc
- Do not run the more expensive explicit filter until the word to be
filtered is completed by user. This requires an end sequence marker
to identify end of explicit word filter to trigger filtering
- Space isn't a good enough delimiter as the explicit filter could be
at the end of the query in which case no space
- Stop passing verbose flag around app methods
- Minor remap of verbosity levels to match python logging framework levels
- verbose = 0 maps to logging.WARN
- verbose = 1 maps to logging.INFO
- verbose >=2 maps to logging.DEBUG
- Minor clean-up of app: unused modules, conversation file opening
- This also pushes the updated URL state to history
- Allows jumping back to the web interface after clicking on an image
and having the type set to image search
- Previously type would get reset to the default search type on
jumping back
- CLIP doesn't need full size images for generating embeddings with
decent search results. The sentence transformers docs use images
scaled to 640px width
- Benefits
- Normalize image sizes
- Increase image embeddings generation speed
- Decrease memory usage while generating embeddings from images
- 5e6625a Fix file browser to not add empty line when no file/dir selected
- 8098b8c Bring main window to Top when open from System Tray
- 1c122a8 Place window near top so buttons are not hidden by OS bottom bar
- dfe2546 Set Khoj Icon on Main Desktop Window
- 1b1f8f9 Move Splash screen text below icon. Set the text color to black
- 450f644 Fix path to remove shared libraries when packaging the Windows app
- Load Models and Embeddings onto GPU if available
- Use MPS for GPU acceleration when available
- Note: Support for [MPS](https://developer.apple.com/metal/) in Pytorch is currently in v1.13.0 nightly builds. See [Announcement](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
- Users will have to wait for PyTorch MPS support to land in stable builds
- Until then code can be tuned and tested for GPU acceleration on newer Macs
- Re-enable Tests for Image Search
- When no file selected in file browser an empty line/entry gets added
to input entries list
- Bug got introduced due to insufficient update on change to add
instead of insert
- Update is_none_or_empty helper method to also check for empty string
- It is currently on top of the splash screen icon
- Ballpark pixels to move such that text positioned below icon
- Test later to verify if text is positioned fine now
- It is a non-user configurable, app state that is set on app start
- Reduce passing unneeded arguments around. Just set device where
required by looking for ML compute device in global state
- Note: Support for MPS in Pytorch is currently in v1.13.0 nightly builds
- Users will have to wait for PyTorch MPS support to land in stable builds
- Until then the code can be tweaked and tested to make use of the GPU
acceleration on newer Macs
- Pass device to load models onto from app state.
- SentenceTransformer models accept device to load models onto during initialization
- Pass device to load corpus embeddings onto from app state
### Fix Image Search
- Do not use XMP metadata by default for image search
- It seems to be buggy currently. The returned results do not make sense with XMP metadata enabled
### Fix Image Search using Desktop App
- Fix configuring Image Search via Desktop GUI
- Set `input-directories`, instead of unused `input-files` for `content-type.image` in `khoj.yml`
- Fix running Image Search via Desktop apps.
- Previously the transformers wasn't getting packaged into the app by pyinstaller
- This is required by image search to run. So the desktop apps would fail to start when image search was enabled
- Resolves#68
- Append selected files, directories via "Add" button in Desktop GUI
- This allows selecting multiple files, directories using Desktop GUI
- Previously selecting multiple image directories had to be entered manually
### Improve Desktop App
- Show Splash Screen to Desktop on App Initialization
- The app takes a while to load during first run
- A splash screen signals that app is loading and not being unresponsive
- Note: _Pyinstaller only supports splash screens on Windows, Linux. Not on Macs._
- Add Khoj icon to the Windows, Linux app. Windows expects a `.ico` icon type
- Only exclude `libtorch_{cuda, cpu, python}` on Linux machine
- Seems those libraries are being used on Mac (and maybe Windows).
- Linux is where the app size benefits from removing these is maximum anyway
- Fix PyInstaller Warnings on App Start
- The warning show up as annoying error popups on Windows
- CLIP Image score and XMP metadata score are not combining well.
When combined they give non sensical results. Enable only once
figure how best to combine the two.
- Show scores with higher precision for image search
- Image search scores seem to be mostly be between 0.2 - 0.3 for some reason
- Higher precision scores make it easier to understand the quality
of returned results perceived by the model itself
- Allows adding multiple image directories via GUI
- Allow adding multiple files in different directories via GUI
- Previously users couldn't add multiple directories via GUI
They'd have to manually append to input field if multiple files, directories
- To clear/overwrite is much easier.
The user can just select text to delete in input area
- On Mac excluding the .dylib version of these files throws errors
Not sure why it didn't throw during testing. Maybe the libs were cached?
- Tested on Linux again. It still seems to be passing with the above
libs excluded. So going to keep those excluded for now. Unless
further testing reveals those libs are really required for app
- Issue
Fix configuring image search from Desktop GUI. It was broken before.
The Desktop GUI was updating input-files field under content-type > image.
This field is not used for image search. So image search couldn't be
configured from the Desktop GUI
- Fix
- Set input-directories when field of search type image is set from GUI
- Otherwise set input-files field in config
## Changes
- On **Debian**
- `libtorch_cuda.so` (1Gb) and `libtorch_cpu.so` (700Mb) are large shared libs
- They are available at package root and under `torch/lib` directory in the package
- We remove the unused, duplicate libraries from under `torch/lib` as only the top level libraries are used
- On **Mac**
- Remove `libtorch_{cpu,python}.dylib` under `torch/lib` directory from the Mac app.
## App Size Reduction
- **Debian amd64** app size by **42%** from **1.6Gb to 920Mb**
- **Mac arm64** app by **15%** from **190Mb to 160Mb**
- **Mac amd64** app by **33%** from **340Mb to 230Mb**
## Reference
- [Release Workflow Run](https://github.com/debanjum/khoj/actions/runs/2878104171) after changes
- [Release Workflow Run](https://github.com/debanjum/khoj/actions/runs/2869906116) before changes
