sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-30 19:03:01 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
khoj/tests/data/org/main_readme.org

48 lines
2.1 KiB
Org Mode
Raw Normal View History

Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 16:26:16 +02:00
* Khoj
Create a few basic tests
2021-08-17 12:27:16 +02:00
/Allow natural language search on user content like notes, images using transformer based models/
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 16:26:16 +02:00
All data is processed locally. User can interface with khoj app via [[./interface/emacs/khoj.el][Emacs]], API or Commandline
Create a few basic tests
2021-08-17 12:27:16 +02:00
** Dependencies
- Python3
- [[https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links][Miniconda]]
** Install
#+begin_src shell
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 16:26:16 +02:00
git clone https://github.com/debanjum/khoj && cd khoj
Create a few basic tests
2021-08-17 12:27:16 +02:00
conda env create -f environment.yml
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 16:26:16 +02:00
conda activate khoj
Create a few basic tests
2021-08-17 12:27:16 +02:00
#+end_src
** Run
Load ML model, generate embeddings and expose API to query specified org-mode files
#+begin_src shell
python3 main.py --input-files ~/Notes/Schedule.org ~/Notes/Incoming.org --verbose
#+end_src
** Use
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 16:26:16 +02:00
- *Khoj via Emacs*
- [[https://github.com/debanjum/khoj/tree/master/interface/emacs#installation][Install]] [[./interface/emacs/khoj.el][khoj.el]]
- Run ~M-x khoj <user-query>~ or Call ~C-c C-s~
Create a few basic tests
2021-08-17 12:27:16 +02:00
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 16:26:16 +02:00
- *Khoj via API*
Version Khoj API, Update frontends, tests and docs to reflect it - Split router.py into v1.0, beta and frontend (no-prefix) api modules under new router package. Version tag in main.py via prefix - Update frontends to use the versioned api endpoints - Update tests to work with versioned api endpoints - Update docs to mentioned, reference only versioned api endpoints
2022-09-14 20:22:20 +02:00
- Query: ~GET~ [[http://localhost:8000/api/v1.0/search?q=%22what%20is%20the%20meaning%20of%20life%22][http://localhost:8000/api/v1.0/search?q="What is the meaning of life"]]
- Update Index: ~GET~ [[http://localhost:8000/api/v1.0/update][http://localhost:8000/api/v1.0/update]]
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 16:26:16 +02:00
- [[http://localhost:8000/docs][Khoj API Docs]]
Create a few basic tests
2021-08-17 12:27:16 +02:00
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time
2022-07-19 16:26:16 +02:00
- *Call Khoj via Python Script Directly*
Create a few basic tests
2021-08-17 12:27:16 +02:00
#+begin_src shell
python3 search_types/asymmetric.py \
--compressed-jsonl .notes.jsonl.gz \
--embeddings .notes_embeddings.pt \
--results-count 5 \
--verbose \
--interactive
#+end_src
** Acknowledgments
Use a better model for asymmetric semantic search - The multi-qa-MiniLM-L6-cos-v1 is more extensively benchmarked[1] - It has the right mix of model query speed, size and performance on benchmarks - On hugging face it has way more downloads and likes than the msmarco model[2] - On very preliminary evaluation of the model - It doubles the encoding speed of all entries (down from ~8min to 4mins) - It gave more entries that stay relevant to the query (3/5 vs 1/5 earlier) [1]: https://www.sbert.net/docs/pretrained_models.html [2]: https://huggingface.co/sentence-transformers
2022-07-18 18:00:19 +02:00
- [[https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1][MiniLM Model]] for Asymmetric Text Search. See [[https://www.sbert.net/examples/applications/retrieve_rerank/README.html][SBert Documentation]]
Create a few basic tests
2021-08-17 12:27:16 +02:00
- [[https://github.com/openai/CLIP][OpenAI CLIP Model]] for Image Search. See [[https://www.sbert.net/examples/applications/image-search/README.html][SBert Documentation]]
- Charles Cave for [[http://members.optusnet.com.au/~charles57/GTD/orgnode.html][OrgNode Parser]]
Reference in a new issue Copy permalink