khoj/README.md

Semantic Search

Provide natural language search on user personal content like notes, images using ML models

All data is processed locally. User can interface with semantic-search app via Emacs, API or Commandline

Dependencies

• Python3
• Miniconda

Install

git clone https://github.com/debanjum/semantic-search && cd semantic-search
conda env create -f environment.yml
conda activate semantic-search

Setup

Generate compressed JSONL from specified org-mode files

python3 processor/org-mode/org-to-jsonl.py \
--input-files ~/Notes/Schedule.org ~/Notes/Incoming.org \
--output-file .notes.jsonl.gz \
--verbose

Run

Load ML model, generate embeddings and expose API interface to run user queries on above org-mode files

python3 main.py \
--compressed-jsonl .notes.jsonl.gz \
--embeddings .notes_embeddings.pt \
--verbose

Use

• Semantic Search via Emacs

  • Install semantic-search.el
  • Run M-x semantic-search "<user-query>" or Call C-c C-s

• Call Semantic Search via API

  • GET http://localhost:8000/search?q="What is the meaning of life"

• Call Semantic Search via Python Script Directly

  python3 search_types/asymmetric.py \
  --compressed-jsonl .notes.jsonl.gz \
  --embeddings .notes_embeddings.pt \
  --results-count 5 \
  --verbose \
  --interactive
  

Acknowledgments

• MiniLM Model for Asymmetric Text Search. See SBert Documentation
• OpenAI CLIP Model for Image Search. See SBert Documentation
• Charles Cave for OrgNode Parser