mirror of
https://github.com/khoj-ai/khoj.git
synced 2024-12-04 12:53:01 +01:00
78a1f4ebb4
- YAML Config
- Can specify all params[1] earlier being passed via cmd args in config YAML
- Can now also configure sentence-transformer models to use etc for search
- [1] Config params
- org files
- compressed entries file config path
- embeddings file config path
- Include sample_config.yaml
- Include sample .org file from this repos readmes
- CLI
- Configuration Priority: Config via cmd > Config via YAML > Default Config
- Test CLI, include test config.yml for the tests
- Set default type to None unless set via query param to API
Run notes search if search_enabled, also if type is None (default)
Prepares for running queries on all search types unless type
specified in API query param
- Update Readme
45 lines
2.1 KiB
Org Mode
45 lines
2.1 KiB
Org Mode
* Semantic Search
|
|
/Allow natural language search on user content like notes, images using transformer based models/
|
|
|
|
All data is processed locally. User can interface with semantic-search app via [[./src/interface/emacs/semantic-search.el][Emacs]], API or Commandline
|
|
|
|
** Dependencies
|
|
- Python3
|
|
- [[https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links][Miniconda]]
|
|
|
|
** Install
|
|
#+begin_src shell
|
|
git clone https://github.com/debanjum/semantic-search && cd semantic-search
|
|
conda env create -f environment.yml
|
|
conda activate semantic-search
|
|
#+end_src
|
|
|
|
** Run
|
|
Load ML model, generate embeddings and expose API to query specified org-mode files
|
|
|
|
#+begin_src shell
|
|
python3 src/main.py --org-files ~/Notes/Schedule.org ~/Notes/Incoming.org -c sample_config.yml --verbose
|
|
#+end_src
|
|
|
|
** Use
|
|
- *Semantic Search via Emacs*
|
|
- [[https://github.com/debanjum/semantic-search/tree/master/src/interface/emacs#installation][Install]] [[./src/interface/emacs/semantic-search.el][semantic-search.el]]
|
|
- Run ~M-x semantic-search <user-query>~ or Call ~C-c s~
|
|
|
|
- *Semantic Search via API*
|
|
- Query: ~GET~ [[http://localhost:8000/search?q=%22what%20is%20the%20meaning%20of%20life%22][http://localhost:8000/search?q="What is the meaning of life"]]
|
|
- Regenerate Embeddings: ~GET~ [[http://localhost:8000/regenerate][http://localhost:8000/regenerate]]
|
|
- [[http://localhost:8000/docs][Semantic Search API Docs]]
|
|
|
|
** Upgrade
|
|
#+begin_src shell
|
|
cd semantic-search
|
|
git pull origin master
|
|
conda env update -f environment.yml
|
|
conda activate semantic-search
|
|
#+end_src
|
|
|
|
** Acknowledgments
|
|
- [[https://huggingface.co/sentence-transformers/msmarco-MiniLM-L-6-v3][MiniLM Model]] for Asymmetric Text Search. See [[https://www.sbert.net/examples/applications/retrieve_rerank/README.html][SBert Documentation]]
|
|
- [[https://github.com/openai/CLIP][OpenAI CLIP Model]] for Image Search. See [[https://www.sbert.net/examples/applications/image-search/README.html][SBert Documentation]]
|
|
- Charles Cave for [[http://members.optusnet.com.au/~charles57/GTD/orgnode.html][OrgNode Parser]]
|