mirror of
https://github.com/khoj-ai/khoj.git
synced 2024-11-23 23:48:56 +01:00
8b29e272d3
- Remove non-standard, unnecessary argument for org-directory Pass path each file in org-files and org-files-filter argument directly - Allow shorthand -i, -o for input files, output files - Default to compress, unless user explicitly specifies not to
2.2 KiB
2.2 KiB
Semantic Search
Provide natural language search on user personal content like notes, images using ML models
All data is processed locally. User can interface with semantic-search app via Emacs, API or Commandline
Dependencies
- Python3
- Miniconda
Install
git clone https://github.com/debanjum/semantic-search && cd semantic-search
conda env create -f environment.yml
conda activate semantic-search
Setup
Generate compressed JSONL from specified org-mode files
python3 processor/org-mode/org-to-jsonl.py \
--org-files "~/Notes/Schedule.org" "~/Notes/Incoming.org" \
--jsonl-file ".notes.jsonl" \
--verbose
Run
Load ML model, generate embeddings and expose API interface to run user queries on above org-mode files
python3 main.py \
--jsonl-file .notes.jsonl.gz \
--embeddings-file .notes_embeddings.pt \
--verbose
Use
-
Semantic Search via Emacs
- Install semantic-search.el
- Run
M-x semantic-search "<user-query>"or CallC-c C-s
-
Call Semantic Search via API
-
Call Semantic Search via Python Script Directly
python3 search_types/asymmetric.py \ --jsonl-file .notes.jsonl.gz \ --embeddings-file .notes_embeddings.pt \ --results-count 5 \ --verbose \ --interactive
Acknowledgments
- MiniLM Model for Asymmetric Text Search. See SBert Documentation
- OpenAI CLIP Model for Image Search. See SBert Documentation
- Charles Cave for OrgNode Parser