khoj/README.org

• Khoj
  • Demo
  • Setup
    • 1. Clone
    • 2. Configure
    • 3. Run
  • Use
  • Run Unit tests
  • Upgrade
  • Troubleshooting
  • Miscellaneous
  • Development Setup
    • Setup on Local Machine
      • 1. Install Dependencies
      • 2. Install Khoj
      • 3. Configure
      • 4. Run
    • Upgrade On Local Machine
  • Acknowledgments

Khoj

Allow natural language search on user content like notes, images, transactions using transformer ML models

User can interface with Khoj via the API or Emacs. All search is done locally*

Demo

Setup

1. Clone

  git clone https://github.com/debanjum/khoj && cd khoj

2. Configure

• [Required] Update docker-compose.yml to mount your images, org-mode notes and beancount directories
• [Optional] Edit application configuration in sample_config.yml

3. Run

docker-compose up -d

Note: The first run will take time. Let it run, it's mostly not hung, just generating embeddings

Use

• Khoj via API

  • See Khoj API Docs
  • Query
  • Regenerate Embeddings
  • Configure Application

• Khoj via Emacs

  • Install khoj.el
  • Run M-x khoj <user-query>

Run Unit tests

pytest

Upgrade

  docker-compose build --pull

Troubleshooting

• Symptom: Errors out with "Killed" in error message

  • Fix: Increase RAM available to Docker Containers in Docker Settings
  • Refer: StackOverflow Solution, Configure Resources on Docker for Mac

• Symptom: Errors out complaining about Tensors mismatch, null etc

  • Mitigation: Delete content-type > image section from docker_sample_config.yml

Miscellaneous

• The experimental chat API endpoint uses the OpenAI API

  • It is disabled by default
  • To use it add your openai-api-key to config.yml

Development Setup

Setup on Local Machine

1. Install Dependencies

1. Install Python3 [Required]
2. Install Conda [Required]

3. Install Exiftool [Optional]

   sudo apt-get -y install libimage-exiftool-perl

2. Install Khoj

git clone https://github.com/debanjum/khoj && cd khoj
conda env create -f config/environment.yml
conda activate khoj

3. Configure

• Configure files/directories to search in content-type section of sample_config.yml

• To run application on test data, update file paths containing /data/ to tests/data/ in sample_config.yml

  • Example replace /data/notes/*.org with tests/data/notes/*.org

4. Run

Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML

python3 -m src.main -c=config/sample_config.yml -vv

Upgrade On Local Machine

  cd khoj
  git pull origin master
  conda deactivate khoj
  conda env update -f config/environment.yml
  conda activate khoj

Acknowledgments

• Multi-QA MiniLM Model for Asymmetric Text Search. See SBert Documentation
• OpenAI CLIP Model for Image Search. See SBert Documentation
• Charles Cave for OrgNode Parser
• Sven Marnach for PyExifTool