khoj/tests/data/org/main_readme.org

* Khoj
  /Allow natural language search on user content like notes, images using transformer based models/

  All data is processed locally. User can interface with khoj app via [[./interface/emacs/khoj.el][Emacs]], API or Commandline

** Dependencies
   - Python3
   - [[https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links][Miniconda]]

** Install
   #+begin_src shell
   git clone https://github.com/debanjum/khoj && cd khoj
   conda env create -f environment.yml
   conda activate khoj
   #+end_src

** Run
   Load ML model, generate embeddings and expose API to query specified org-mode files

   #+begin_src shell
   python3 main.py --input-files ~/Notes/Schedule.org ~/Notes/Incoming.org --verbose
   #+end_src

** Use
   - *Khoj via Emacs*
     - [[https://github.com/debanjum/khoj/tree/master/interface/emacs#installation][Install]] [[./interface/emacs/khoj.el][khoj.el]]
     - Run ~M-x khoj <user-query>~ or Call ~C-c C-s~

   - *Khoj via API*
     - Query: ~GET~ [[http://localhost:8000/api/search?q=%22what%20is%20the%20meaning%20of%20life%22][http://localhost:8000/api/search?q="What is the meaning of life"]]
     - Update Index: ~GET~ [[http://localhost:8000/api/update][http://localhost:8000/api/update]]
     - [[http://localhost:8000/docs][Khoj API Docs]]

   - *Call Khoj via Python Script Directly*
     #+begin_src shell
     python3 search_types/asymmetric.py \
     --compressed-jsonl .notes.jsonl.gz \
     --embeddings .notes_embeddings.pt \
     --results-count 5 \
     --verbose \
     --interactive
     #+end_src

** Acknowledgments
   - [[https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1][MiniLM Model]] for Asymmetric Text Search. See [[https://www.sbert.net/examples/applications/retrieve_rerank/README.html][SBert Documentation]]
   - [[https://github.com/openai/CLIP][OpenAI CLIP Model]] for Image Search. See [[https://www.sbert.net/examples/applications/image-search/README.html][SBert Documentation]]
   - Charles Cave for [[http://members.optusnet.com.au/~charles57/GTD/orgnode.html][OrgNode Parser]]
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time 2022-07-19 14:26:16 +00:00			`* Khoj`
Create a few basic tests 2021-08-17 10:27:16 +00:00			`/Allow natural language search on user content like notes, images using transformer based models/`

Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time 2022-07-19 14:26:16 +00:00			`All data is processed locally. User can interface with khoj app via [[./interface/emacs/khoj.el][Emacs]], API or Commandline`
Create a few basic tests 2021-08-17 10:27:16 +00:00
			`** Dependencies`
			`- Python3`
			`- [[https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links][Miniconda]]`

			`** Install`
			`#+begin_src shell`
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time 2022-07-19 14:26:16 +00:00			`git clone https://github.com/debanjum/khoj && cd khoj`
Create a few basic tests 2021-08-17 10:27:16 +00:00			`conda env create -f environment.yml`
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time 2022-07-19 14:26:16 +00:00			`conda activate khoj`
Create a few basic tests 2021-08-17 10:27:16 +00:00			`#+end_src`

			`** Run`
			`Load ML model, generate embeddings and expose API to query specified org-mode files`

			`#+begin_src shell`
			`python3 main.py --input-files ~/Notes/Schedule.org ~/Notes/Incoming.org --verbose`
			`#+end_src`

			`** Use`
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time 2022-07-19 14:26:16 +00:00			`- Khoj via Emacs`
			`- [[https://github.com/debanjum/khoj/tree/master/interface/emacs#installation][Install]] [[./interface/emacs/khoj.el][khoj.el]]`
			`- Run ~M-x khoj <user-query>~ or Call ~C-c C-s~`
Create a few basic tests 2021-08-17 10:27:16 +00:00
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time 2022-07-19 14:26:16 +00:00			`- Khoj via API`
Do not version API. Premature given current state of the codebase - Reason - All clients that currently consume the API are part of Khoj - Any breaking API changes will be fixed in clients immediately - So decoupling client from API is not required - This removes the burden of maintaining muliple versions of the API 2022-10-08 10:16:08 +00:00			`- Query: ~GET~ [[http://localhost:8000/api/search?q=%22what%20is%20the%20meaning%20of%20life%22][http://localhost:8000/api/search?q="What is the meaning of life"]]`
			`- Update Index: ~GET~ [[http://localhost:8000/api/update][http://localhost:8000/api/update]]`
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time 2022-07-19 14:26:16 +00:00			`- [[http://localhost:8000/docs][Khoj API Docs]]`
Create a few basic tests 2021-08-17 10:27:16 +00:00
Give the project a short, less generic name. Rename it to Khoj - Semantic Search was just a placeholder used to test the idea out Didn't want to get into naming at that point of time 2022-07-19 14:26:16 +00:00			`- Call Khoj via Python Script Directly`
Create a few basic tests 2021-08-17 10:27:16 +00:00			`#+begin_src shell`
			`python3 search_types/asymmetric.py \`
			`--compressed-jsonl .notes.jsonl.gz \`
			`--embeddings .notes_embeddings.pt \`
			`--results-count 5 \`
			`--verbose \`
			`--interactive`
			`#+end_src`

			`** Acknowledgments`
Use a better model for asymmetric semantic search - The multi-qa-MiniLM-L6-cos-v1 is more extensively benchmarked[1] - It has the right mix of model query speed, size and performance on benchmarks - On hugging face it has way more downloads and likes than the msmarco model[2] - On very preliminary evaluation of the model - It doubles the encoding speed of all entries (down from ~8min to 4mins) - It gave more entries that stay relevant to the query (3/5 vs 1/5 earlier) [1]: https://www.sbert.net/docs/pretrained_models.html [2]: https://huggingface.co/sentence-transformers 2022-07-18 16:00:19 +00:00			`- [[https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1][MiniLM Model]] for Asymmetric Text Search. See [[https://www.sbert.net/examples/applications/retrieve_rerank/README.html][SBert Documentation]]`
Create a few basic tests 2021-08-17 10:27:16 +00:00			`- [[https://github.com/openai/CLIP][OpenAI CLIP Model]] for Image Search. See [[https://www.sbert.net/examples/applications/image-search/README.html][SBert Documentation]]`
			`- Charles Cave for [[http://members.optusnet.com.au/~charles57/GTD/orgnode.html][OrgNode Parser]]`