From 0509854e14736e609b9bd805ffd87461ad979990 Mon Sep 17 00:00:00 2001 From: Debanjum Singh Solanky Date: Mon, 16 Aug 2021 20:00:05 -0700 Subject: [PATCH] Replace README.md with README.org. Can be used as notes for testing --- README.md | 52 ---------------------------------------------------- README.org | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 52 deletions(-) delete mode 100644 README.md create mode 100644 README.org diff --git a/README.md b/README.md deleted file mode 100644 index 815bd0e2..00000000 --- a/README.md +++ /dev/null @@ -1,52 +0,0 @@ -Semantic Search -=== -> Provide natural language search on user personal content like notes, images using ML models - -All data is processed locally. User can interface with semantic-search app via [Emacs](./interface/emacs/semantic-search.el), API or Commandline - -Dependencies ----- - - Python3 - - [Miniconda](https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links) - -Install ---- - ```sh - git clone https://github.com/debanjum/semantic-search && cd semantic-search - conda env create -f environment.yml - conda activate semantic-search - ``` - -Run ---- - Load ML model, generate embeddings and expose API to query specified org-mode files - ```sh - python3 main.py --input-files ~/Notes/Schedule.org ~/Notes/Incoming.org --verbose - ``` - -Use ---- - - *Semantic Search via Emacs* - - [Install](https://github.com/debanjum/semantic-search/tree/master/interface/emacs#installation) [semantic-search.el](./interface/emacs/semantic-search.el) - - Run `M-x semantic-search ""` or Call `C-c C-s` - - - *Semantic Search via API* - - Query: `GET` [http://localhost:8000/search?q="What is the meaning of life"](http://localhost:8000/search?q=%22what%20is%20the%20meaning%20of%20life%22) - - Regenerate Embeddings: `GET` [http://localhost:8000/regenerate](http://localhost:8000/regenerate) - - [Semantic Search API Docs](http://localhost:8000/docs) - - - *Call Semantic Search via Python Script Directly* - ```sh - python3 search_types/asymmetric.py \ - --compressed-jsonl .notes.jsonl.gz \ - --embeddings .notes_embeddings.pt \ - --results-count 5 \ - --verbose \ - --interactive - ``` - -Acknowledgments --- -- [MiniLM Model](https://huggingface.co/sentence-transformers/msmarco-MiniLM-L-6-v3) for Asymmetric Text Search. See [SBert Documentation](https://www.sbert.net/examples/applications/retrieve_rerank/README.html) -- [OpenAI CLIP Model](https://github.com/openai/CLIP) for Image Search. See [SBert Documentation](https://www.sbert.net/examples/applications/image-search/README.html) -- Charles Cave for [OrgNode Parser](http://members.optusnet.com.au/~charles57/GTD/orgnode.html) diff --git a/README.org b/README.org new file mode 100644 index 00000000..3e0b479d --- /dev/null +++ b/README.org @@ -0,0 +1,47 @@ +* Semantic Search + /Allow natural language search on user content like notes, images using transformer based models/ + + All data is processed locally. User can interface with semantic-search app via [[./interface/emacs/semantic-search.el][Emacs]], API or Commandline + +** Dependencies + - Python3 + - [[https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links][Miniconda]] + +** Install + #+begin_src shell + git clone https://github.com/debanjum/semantic-search && cd semantic-search + conda env create -f environment.yml + conda activate semantic-search + #+end_src + +** Run + Load ML model, generate embeddings and expose API to query specified org-mode files + + #+begin_src shell + python3 main.py --input-files ~/Notes/Schedule.org ~/Notes/Incoming.org --verbose + #+end_src + +** Use + - *Semantic Search via Emacs* + - [[https://github.com/debanjum/semantic-search/tree/master/interface/emacs#installation][Install]] [[./interface/emacs/semantic-search.el][semantic-search.el]] + - Run ~M-x semantic-search ~ or Call ~C-c C-s~ + + - *Semantic Search via API* + - Query: ~GET~ [[http://localhost:8000/search?q=%22what%20is%20the%20meaning%20of%20life%22][http://localhost:8000/search?q="What is the meaning of life"]] + - Regenerate Embeddings: ~GET~ [[http://localhost:8000/regenerate][http://localhost:8000/regenerate]] + - [[http://localhost:8000/docs][Semantic Search API Docs]] + + - *Call Semantic Search via Python Script Directly* + #+begin_src shell + python3 search_types/asymmetric.py \ + --compressed-jsonl .notes.jsonl.gz \ + --embeddings .notes_embeddings.pt \ + --results-count 5 \ + --verbose \ + --interactive + #+end_src + +** Acknowledgments + - [[https://huggingface.co/sentence-transformers/msmarco-MiniLM-L-6-v3][MiniLM Model]] for Asymmetric Text Search. See [[https://www.sbert.net/examples/applications/retrieve_rerank/README.html][SBert Documentation]] + - [[https://github.com/openai/CLIP][OpenAI CLIP Model]] for Image Search. See [[https://www.sbert.net/examples/applications/image-search/README.html][SBert Documentation]] + - Charles Cave for [[http://members.optusnet.com.au/~charles57/GTD/orgnode.html][OrgNode Parser]]