• Khoj
  • Table of Contents
  • Features
  • Demo
    • Description
    • Analysis
  • Architecture
  • Setup
    • 1. Clone
    • 2. Configure
    • 3. Run
  • Use
  • Upgrade
  • Troubleshooting
  • Miscellaneous
  • Development Setup
    • Setup on Local Machine
      • 1. Install Dependencies
      • 2. Install Khoj
      • 3. Configure
      • 4. Run
    • Upgrade On Local Machine
    • Run Unit Tests
  • Performance
    • Query performance
    • Indexing performance
    • Miscellaneous
  • Acknowledgments

Khoj

A natural language search engine for your personal notes, transactions and images

Table of Contents

• Features

• Demo

  • Description
  • Analysis
• Architecture

• Setup

  • Clone
  • Configure
  • Run
• Use
• Upgrade
• Troubleshooting
• Miscellaneous

• Development Setup

  • Setup on Local Machine
  • Upgrade on Local Machine
  • Run Unit Tests

• Performance

  • Query Performance
  • Indexing Performance
  • Miscellaneous
• Acknowledgments

Features

• Natural: Advanced Natural language understanding using Transformer based ML Models
• Local: Your personal data stays local. All search, indexing is done on your machine*
• Incremental: Incremental search for a fast, search-as-you-type experience
• Pluggable: Modular architecture makes it relatively easy to plug in new data sources, frontends and ML models
• Multiple Sources: Search your Org-mode and Markdown notes, Beancount transactions and Photos
• Multiple Interfaces: Search using a Web Browser, Emacs or the API

Demo

Description

• User searches for "Setup editor"
• The demo looks for the most relevant section in this readme and the khoj.el readme
• Top result is what we are looking for, the section to Install Khoj.el on Emacs

Analysis

• The results do not have any words used in the query

  • Based on the top result it seems the re-ranking model understands that Emacs is an editor?
• The results incrementally update as the query is entered
• The results are re-ranked, for better accuracy, once user is idle

Architecture

Setup

1. Clone

  git clone https://github.com/debanjum/khoj && cd khoj

2. Configure

• Required: Update docker-compose.yml to mount your images, (org-mode or markdown) notes and beancount directories
• Optional: Edit application configuration in sample_config.yml

3. Run

docker-compose up -d

Note: The first run will take time. Let it run, it's mostly not hung, just generating embeddings

Use

• Khoj via Web

  • Go to http://localhost:8000/ or open index.html in your browser

• Khoj via Emacs

  • Install khoj.el
  • Run M-x khoj <user-query>

• Khoj via API

  • See Khoj FastAPI Docs
  • Query
  • Regenerate Embeddings
  • Configure Application

Upgrade

  docker-compose build --pull

Troubleshooting

• Symptom: Errors out with "Killed" in error message

  • Fix: Increase RAM available to Docker Containers in Docker Settings
  • Refer: StackOverflow Solution, Configure Resources on Docker for Mac

• Symptom: Errors out complaining about Tensors mismatch, null etc

  • Mitigation: Delete content-type > image section from docker_sample_config.yml

Miscellaneous

• The experimental chat API endpoint uses the OpenAI API

  • It is disabled by default
  • To use it add your openai-api-key to config.yml

Development Setup

Setup on Local Machine

1. Install Dependencies

1. Install Python3 [Required]
2. Install Conda [Required]

3. Install Exiftool [Optional]

   sudo apt-get -y install libimage-exiftool-perl

2. Install Khoj

git clone https://github.com/debanjum/khoj && cd khoj
conda env create -f config/environment.yml
conda activate khoj

3. Configure

• Configure files/directories to search in content-type section of sample_config.yml

• To run application on test data, update file paths containing /data/ to tests/data/ in sample_config.yml

  • Example replace /data/notes/*.org with tests/data/notes/*.org

4. Run

Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML

python3 -m src.main -c=config/sample_config.yml -vv

Upgrade On Local Machine

  cd khoj
  git pull origin master
  conda deactivate khoj
  conda env update -f config/environment.yml
  conda activate khoj

Run Unit Tests

pytest

Performance

Query performance

• Semantic search using the bi-encoder is fairly fast at <5 ms
• Reranking using the cross-encoder is slower at <2s on 15 results. Tweak top_k to tradeoff speed for accuracy of results.
• Applying explicit filters is very slow currently at ~6s. This is because the filters are rudimentary. Considerable speed-ups can be achieved using indexes etc.

Indexing performance

• Indexing is more strongly impacted by the size of the source data
• Indexing 100K+ line corpus of notes takes 6 minutes
• Indexing 4000+ images takes about 15 minutes and more than 8Gb of RAM
• Once https://github.com/debanjum/khoj/issues/36 is implemented, it should only take this long on first run

Miscellaneous

• Testing done on a Mac M1 and a >100K line corpus of notes
• Search, indexing on a GPU has not been tested yet

Acknowledgments

• Multi-QA MiniLM Model, All MiniLM Model for Text Search. See SBert Documentation
• OpenAI CLIP Model for Image Search. See SBert Documentation
• Charles Cave for OrgNode Parser
• Org.js to render Org-mode results on the Web interface
• Markdown-it to render Markdown results on the Web interface
• Sven Marnach for PyExifTool