mirror of
https://github.com/khoj-ai/khoj.git
synced 2024-11-27 17:35:07 +01:00
61 lines
1.7 KiB
Markdown
61 lines
1.7 KiB
Markdown
|
Semantic Search
|
||
|
|
===
|
||
|
|
> Provide natural language search on user personal content like notes, images using ML models
|
||
|
|
|
||
|
|
All data is processed locally. User can interface with semantic-search app via [Emacs](https://github.com/debanjum/emacs-semantic-search), API or Commandline
|
||
|
|
|
||
|
|
Dependencies
|
||
|
|
----
|
||
|
|
- Python3
|
||
|
|
- [Miniconda](https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links)
|
||
|
|
|
||
|
|
Install
|
||
|
|
---
|
||
|
|
```sh
|
||
|
|
git clone https://github.com/debanjum/semantic-search && cd semantic-search
|
||
|
|
conda env create -f environment.yml
|
||
|
|
conda activate semantic-search
|
||
|
|
```
|
||
|
|
|
||
|
|
Setup
|
||
|
|
---
|
||
|
|
Generate compressed JSONL from specified org-mode files
|
||
|
|
```sh
|
||
|
|
python3 processor/org-mode/org-to-jsonl.py \
|
||
|
|
--org-files "Schedule.org" "Incoming.org" \
|
||
|
|
--org-directory "~/Notes" \
|
||
|
|
--jsonl-file ".data/notes.jsonl" \
|
||
|
|
--compress \
|
||
|
|
--verbose
|
||
|
|
```
|
||
|
|
|
||
|
|
Run
|
||
|
|
---
|
||
|
|
Load ML model, generate embeddings and expose API interface to run user queries on above org-mode files
|
||
|
|
```sh
|
||
|
|
python3 main.py -j .data/notes.jsonl.gz -e .data/notes_embeddings.pt
|
||
|
|
```
|
||
|
|
|
||
|
|
Use
|
||
|
|
---
|
||
|
|
- *Calls Semantic Search via Emacs*
|
||
|
|
- `M-x semantic-search "<users-query"`
|
||
|
|
- `C-c C-s`
|
||
|
|
|
||
|
|
- *Call Semantic Search via API*
|
||
|
|
- `GET` [http://localhost:8000/search?q="What is the meaning of life"](http://localhost:8000/search?q=%22what%20is%20the%20meaning%20of%20life%22)
|
||
|
|
|
||
|
|
- *Call Semantic Search via Python Script Directly*
|
||
|
|
```sh
|
||
|
|
python3 search_types/asymmetric.py \
|
||
|
|
-j .data/notes.jsonl.gz \
|
||
|
|
-e .data/notes_embeddings.pt \
|
||
|
|
-n 5 \
|
||
|
|
--verbose \
|
||
|
|
--interactive
|
||
|
|
```
|
||
|
|
|
||
|
|
Acknowledgments
|
||
|
|
--
|
||
|
|
- Charles Cave for [OrgNode Parser](http://members.optusnet.com.au/~charles57/GTD/orgnode.html)
|