khoj/Readme.md

Khoj 🦅

A natural language search engine for your personal notes, transactions and images

Table of Contents

• Features
• Demo
  • Description
  • Analysis
• Architecture
• Setup
  • Install
  • Configure
  • Run
• Use
• Upgrade
• Troubleshoot
• Miscellaneous
• Development
  • Setup
  • Upgrade
  • Test
• Performance
  • Query Performance
  • Indexing Performance
  • Miscellaneous
• Credits

Features

• Natural: Advanced Natural language understanding using Transformer based ML Models
• Local: Your personal data stays local. All search, indexing is done on your machine*
• Incremental: Incremental search for a fast, search-as-you-type experience
• Pluggable: Modular architecture makes it easy to plug in new data sources, frontends and ML models
• Multiple Sources: Search your Org-mode and Markdown notes, Beancount transactions and Photos
• Multiple Interfaces: Search using a Web Browser, Emacs or the API

Demo

https://user-images.githubusercontent.com/6413477/181664862-31565b0a-0e64-47e1-a79a-599dfc486c74.mp4

Description

• User searches for "Setup editor"
• The demo looks for the most relevant section in this readme and the khoj.el readme
• Top result is what we are looking for, the section to Install Khoj.el on Emacs

Analysis

• The results do not have any words used in the query
  • Based on the top result it seems the re-ranking model understands that Emacs is an editor?
• The results incrementally update as the query is entered
• The results are re-ranked, for better accuracy, once user is idle

Architecture

Setup

1. Install

pip install khoj-assistant

2. Configure

• Set input-files or input-filter in each relevant content-type section of khoj_sample.yml
  • Set input-directories field in content-type.image section
• Delete content-type sections irrelevant for your use-case

3. Run

khoj config/khoj_sample.yml -vv

Loads ML model, generates embeddings and exposes API to search notes, images, transactions etc specified in config YAML

Use

• Khoj via Web
  • Open http://localhost:8000/
• Khoj via Emacs
  • Install khoj.el
  • Run M-x khoj <user-query>
• Khoj via API
  • See Khoj FastAPI Docs, Khoj FastAPI ReDocs

Upgrade

pip install --upgrade khoj-assistant

Troubleshoot

• Symptom: Errors out complaining about Tensors mismatch, null etc

  • Mitigation: Delete content-type > image section from khoj_sample.yml

• Symptom: Errors out with "Killed" in error message in Docker

  • Fix: Increase RAM available to Docker Containers in Docker Settings
  • Refer: StackOverflow Solution, Configure Resources on Docker for Mac

Miscellaneous

• The experimental chat API endpoint uses the OpenAI API
  • It is disabled by default
  • To use it add your openai-api-key to config.yml

Development

Setup

Using Docker

1. Clone

git clone https://github.com/debanjum/khoj && cd khoj

2. Configure

• Required: Update docker-compose.yml to mount your images, (org-mode or markdown) notes and beancount directories
• Optional: Edit application configuration in khoj_docker.yml

3. Run

docker-compose up -d

Note: The first run will take time. Let it run, it's mostly not hung, just generating embeddings

Using Conda

1. Install Dependencies

   • Install Conda [Required]
   • Install Exiftool [Optional]

     sudo apt -y install libimage-exiftool-perl
     

2. Install Khoj

   git clone https://github.com/debanjum/khoj && cd khoj
   conda env create -f config/environment.yml
   conda activate khoj
   

3. Configure

   • Set input-files or input-filter in each relevant content-type section of khoj_sample.yml
     • Set input-directories field in image content-type section
   • Delete content-type sections irrelevant for your use-case

4. Run Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML

   python3 -m src.main -c=config/khoj_sample.yml -vv
   

Upgrade

Using Pip

# To Upgrade To Latest Stable Release
pip install --upgrade khoj-assistant

# To Upgrade To Latest Pre-Release
pip install --upgrade --pre khoj-assistant

# To Upgrade To Specific Development Release
pip install -r testpypi khoj-assistant==0.1.5.dev491659577806

Using Docker

docker-compose build --pull

Using Conda

cd khoj
git pull origin master
conda deactivate khoj
conda env update -f config/environment.yml
conda activate khoj

Test

pytest

Performance

Query performance

• Semantic search using the bi-encoder is fairly fast at <5 ms
• Reranking using the cross-encoder is slower at <2s on 15 results. Tweak top_k to tradeoff speed for accuracy of results
• Applying explicit filters is very slow currently at ~6s. This is because the filters are rudimentary. Considerable speed-ups can be achieved using indexes etc

Indexing performance

• Indexing is more strongly impacted by the size of the source data
• Indexing 100K+ line corpus of notes takes 6 minutes
• Indexing 4000+ images takes about 15 minutes and more than 8Gb of RAM
• Once https://github.com/debanjum/khoj/issues/36 is implemented, it should only take this long on first run

Miscellaneous

• Testing done on a Mac M1 and a >100K line corpus of notes
• Search, indexing on a GPU has not been tested yet

Credits

• Multi-QA MiniLM Model, All MiniLM Model for Text Search. See SBert Documentation
• OpenAI CLIP Model for Image Search. See SBert Documentation
• Charles Cave for OrgNode Parser
• Org.js to render Org-mode results on the Web interface
• Markdown-it to render Markdown results on the Web interface
• Sven Marnach for PyExifTool