sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-27 17:35:07 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
khoj/Readme.md
Debanjum Singh Solanky a4eb55dd00 Rename khoj config yml file to follow more specific khoj*.yml pattern 
- That is, sample_config.yml is renamed to khoj_sample.yml
- This makes the application config filename less generic,
  more easily identifiable with the application
- Update docs, app accordingly
2022-08-03 12:06:55 +03:00

8 KiB
Raw Blame History

Khoj 🦅

A natural language search engine for your personal notes, transactions and images

Table of Contents

Features

  • Natural: Advanced Natural language understanding using Transformer based ML Models
  • Local: Your personal data stays local. All search, indexing is done on your machine*
  • Incremental: Incremental search for a fast, search-as-you-type experience
  • Pluggable: Modular architecture makes it easy to plug in new data sources, frontends and ML models
  • Multiple Sources: Search your Org-mode and Markdown notes, Beancount transactions and Photos
  • Multiple Interfaces: Search using a Web Browser, Emacs or the API

Demo

https://user-images.githubusercontent.com/6413477/181664862-31565b0a-0e64-47e1-a79a-599dfc486c74.mp4

Description

Analysis

  • The results do not have any words used in the query
    • Based on the top result it seems the re-ranking model understands that Emacs is an editor?
  • The results incrementally update as the query is entered
  • The results are re-ranked, for better accuracy, once user is idle

Architecture

Setup

1. Clone

git clone https://github.com/debanjum/khoj && cd khoj

2. Configure

  • Required: Update docker-compose.yml to mount your images, (org-mode or markdown) notes and beancount directories
  • Optional: Edit application configuration in khoj_sample.yml

3. Run

docker-compose up -d

Note: The first run will take time. Let it run, it's mostly not hung, just generating embeddings

Use

Upgrade

docker-compose build --pull

Troubleshoot

  • Symptom: Errors out with "Killed" in error message
  • Symptom: Errors out complaining about Tensors mismatch, null etc
    • Mitigation: Delete content-type > image section from khoj_sample.yml

Miscellaneous

  • The experimental chat API endpoint uses the OpenAI API
    • It is disabled by default
    • To use it add your openai-api-key to config.yml

Development Setup

Setup on Local Machine

Using Pip

  1. Install Dependencies

    1. Python3, Pip [Required]
    2. Virualenv [Optional]
    3. Install Exiftool [Optional] 
      sudo apt-get -y install libimage-exiftool-perl

  2. Install Khoj

    virtualenv -m python3 .venv && source .venv/bin/activate # Optional
pip install khoj-assistant

  3. Configure

    • Configure files/directories to search in content-type section of khoj_sample.yml
    • To run application on test data, update file paths containing /data/ to tests/data/ in khoj_sample.yml
      • Example replace /data/notes/*.org with tests/data/notes/*.org

  4. Run Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML

    khoj -c=config/khoj_sample.yml -vv

Using Conda

  1. Install Dependencies

    1. Install Python3 [Required]
    2. Install Conda [Required]
    3. Install Exiftool [Optional] 
      sudo apt-get -y install libimage-exiftool-perl

  2. Install Khoj

    git clone https://github.com/debanjum/khoj && cd khoj
conda env create -f config/environment.yml
conda activate khoj

  3. Configure

    • Configure files/directories to search in content-type section of khoj_sample.yml
    • To run application on test data, update file paths containing /data/ to tests/data/ in khoj_sample.yml
      • Example replace /data/notes/*.org with tests/data/notes/*.org

  4. Run Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML

    python3 -m src.main -c=config/khoj_sample.yml -vv

Upgrade On Local Machine

Using Pip

pip install --upgrade khoj-assistant

Using Conda

cd khoj
git pull origin master
conda deactivate khoj
conda env update -f config/environment.yml
conda activate khoj

Run Unit Tests

pytest

Performance

Query performance

  • Semantic search using the bi-encoder is fairly fast at <5 ms
  • Reranking using the cross-encoder is slower at <2s on 15 results. Tweak top_k to tradeoff speed for accuracy of results
  • Applying explicit filters is very slow currently at ~6s. This is because the filters are rudimentary. Considerable speed-ups can be achieved using indexes etc

Indexing performance

  • Indexing is more strongly impacted by the size of the source data
  • Indexing 100K+ line corpus of notes takes 6 minutes
  • Indexing 4000+ images takes about 15 minutes and more than 8Gb of RAM
  • Once https://github.com/debanjum/khoj/issues/36 is implemented, it should only take this long on first run

Miscellaneous

  • Testing done on a Mac M1 and a >100K line corpus of notes
  • Search, indexing on a GPU has not been tested yet

Credits