khoj/README.org
2022-07-22 04:02:03 +04:00

4.6 KiB

https://github.com/debanjum/khoj/actions/workflows/test.yml/badge.svg https://github.com/debanjum/khoj/actions/workflows/build.yml/badge.svg

Khoj

Natural language search for your notes, transactions and images using transformer ML models

Interface with Khoj via Web, Emacs or the API. All search is done locally*

Setup

1. Clone

  git clone https://github.com/debanjum/khoj && cd khoj

2. Configure

  • [Required] Update docker-compose.yml to mount your images, (org-mode or markdown) notes and beancount directories
  • [Optional] Edit application configuration in sample_config.yml

3. Run

docker-compose up -d

Note: The first run will take time. Let it run, it's mostly not hung, just generating embeddings

Use

Run Unit tests

pytest

Upgrade

  docker-compose build --pull

Troubleshooting

  • Symptom: Errors out with "Killed" in error message

  • Symptom: Errors out complaining about Tensors mismatch, null etc

    • Mitigation: Delete content-type > image section from docker_sample_config.yml

Miscellaneous

  • The experimental chat API endpoint uses the OpenAI API

    • It is disabled by default
    • To use it add your openai-api-key to config.yml

Development Setup

Setup on Local Machine

1. Install Dependencies
  1. Install Python3 [Required]
  2. Install Conda [Required]
  3. Install Exiftool [Optional]

    sudo apt-get -y install libimage-exiftool-perl
2. Install Khoj
git clone https://github.com/debanjum/khoj && cd khoj
conda env create -f config/environment.yml
conda activate khoj
3. Configure
  • Configure files/directories to search in content-type section of sample_config.yml
  • To run application on test data, update file paths containing /data/ to tests/data/ in sample_config.yml

    • Example replace /data/notes/*.org with tests/data/notes/*.org
4. Run

Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML

python3 -m src.main -c=config/sample_config.yml -vv

Upgrade On Local Machine

  cd khoj
  git pull origin master
  conda deactivate khoj
  conda env update -f config/environment.yml
  conda activate khoj

Acknowledgments