mirror of
https://github.com/khoj-ai/khoj.git
synced 2024-12-03 12:23:02 +01:00
d2905c4be6
tests/ directory in project root is more standard. Just had to use absolute path for internal module imports to get it to work
47 lines
2.2 KiB
Org Mode
47 lines
2.2 KiB
Org Mode
* Semantic Search
|
|
/Allow natural language search on user content like notes, images using transformer based models/
|
|
|
|
All data is processed locally. User can interface with semantic-search app via [[./interface/emacs/semantic-search.el][Emacs]], API or Commandline
|
|
|
|
** Dependencies
|
|
- Python3
|
|
- [[https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links][Miniconda]]
|
|
|
|
** Install
|
|
#+begin_src shell
|
|
git clone https://github.com/debanjum/semantic-search && cd semantic-search
|
|
conda env create -f environment.yml
|
|
conda activate semantic-search
|
|
#+end_src
|
|
|
|
** Run
|
|
Load ML model, generate embeddings and expose API to query specified org-mode files
|
|
|
|
#+begin_src shell
|
|
python3 main.py --input-files ~/Notes/Schedule.org ~/Notes/Incoming.org --verbose
|
|
#+end_src
|
|
|
|
** Use
|
|
- *Semantic Search via Emacs*
|
|
- [[https://github.com/debanjum/semantic-search/tree/master/interface/emacs#installation][Install]] [[./interface/emacs/semantic-search.el][semantic-search.el]]
|
|
- Run ~M-x semantic-search <user-query>~ or Call ~C-c C-s~
|
|
|
|
- *Semantic Search via API*
|
|
- Query: ~GET~ [[http://localhost:8000/search?q=%22what%20is%20the%20meaning%20of%20life%22][http://localhost:8000/search?q="What is the meaning of life"]]
|
|
- Regenerate Embeddings: ~GET~ [[http://localhost:8000/regenerate][http://localhost:8000/regenerate]]
|
|
- [[http://localhost:8000/docs][Semantic Search API Docs]]
|
|
|
|
- *Call Semantic Search via Python Script Directly*
|
|
#+begin_src shell
|
|
python3 search_types/asymmetric.py \
|
|
--compressed-jsonl .notes.jsonl.gz \
|
|
--embeddings .notes_embeddings.pt \
|
|
--results-count 5 \
|
|
--verbose \
|
|
--interactive
|
|
#+end_src
|
|
|
|
** Acknowledgments
|
|
- [[https://huggingface.co/sentence-transformers/msmarco-MiniLM-L-6-v3][MiniLM Model]] for Asymmetric Text Search. See [[https://www.sbert.net/examples/applications/retrieve_rerank/README.html][SBert Documentation]]
|
|
- [[https://github.com/openai/CLIP][OpenAI CLIP Model]] for Image Search. See [[https://www.sbert.net/examples/applications/image-search/README.html][SBert Documentation]]
|
|
- Charles Cave for [[http://members.optusnet.com.au/~charles57/GTD/orgnode.html][OrgNode Parser]]
|