khoj/README.md

Semantic Search

Provide natural language search on user personal content like notes, images using ML models

All data is processed locally. User can interface with semantic-search app via Emacs, API or Commandline

Dependencies

- Python3
- [Miniconda](https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links)

Install

```sh
git clone https://github.com/debanjum/semantic-search && cd semantic-search
conda env create -f environment.yml
conda activate semantic-search
```

Setup

Generate compressed JSONL from specified org-mode files
```sh
python3 processor/org-mode/org-to-jsonl.py \
--org-files "Schedule.org" "Incoming.org" \
--org-directory "~/Notes" \
--jsonl-file ".data/notes.jsonl" \
--compress \
--verbose
```

Run

Load ML model, generate embeddings and expose API interface to run user queries on above org-mode files sh python3 main.py -j .data/notes.jsonl.gz -e .data/notes_embeddings.pt

Use

- *Calls Semantic Search via Emacs*
  - `M-x semantic-search "<users-query"`
  - `C-c C-s`

- *Call Semantic Search via API*
  - `GET` [http://localhost:8000/search?q="What is the meaning of life"](http://localhost:8000/search?q=%22what%20is%20the%20meaning%20of%20life%22)

- *Call Semantic Search via Python Script Directly*
  ```sh
  python3 search_types/asymmetric.py \
  -j .data/notes.jsonl.gz \
  -e .data/notes_embeddings.pt \
  -n 5 \
  --verbose \
  --interactive
  ```

Acknowledgments

• Charles Cave for OrgNode Parser