diff --git a/README.org b/README.org index 350e8c46..f8f4e15f 100644 --- a/README.org +++ b/README.org @@ -5,32 +5,52 @@ All data is processed locally. User can interface with semantic-search app via [[./src/interface/emacs/semantic-search.el][Emacs]], API or Commandline -** Dependencies - - Python3 - - [[https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links][Miniconda]] +** Setup -** Install - #+begin_src shell - git clone https://github.com/debanjum/semantic-search && cd semantic-search - conda env create -f environment.yml - conda activate semantic-search - #+end_src +*** Setup using Docker -*** Install Environmental Dependencies - #+begin_src shell - sudo apt-get -y install libimage-exiftool-perl - #+end_src +**** 1. Clone Repository + #+begin_src shell + git clone https://github.com/debanjum/semantic-search && cd semantic-search + #+end_src -** Configure - Configure application search types and their underlying data source/files in ~sample_config.yml~ - Use the ~sample_config.yml~ as reference +**** 2. Configure + Add Content Directories for Semantic Search to Docker-Compose + Update [[./docker-compose.yml][docker-compose.yml]] to mount your images, org-mode notes, ledger/beancount directories + If required, edit config settings in [[./docker_sample_config.yml][docker_sample_config.yml]]. -** Run - Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML +**** 3. Run + #+begin_src shell + docker-compose up -d + #+end_src - #+begin_src shell - python3 -m src.main -c=sample_config.yml -vv - #+end_src +*** Setup on Local Machine + +**** 1. Install Dependencies + 1. Install Python3 [Required[ + 2. [[https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html][Install Conda]] [Required] + 3. Install Exiftool [Optional] + #+begin_src shell + sudo apt-get -y install libimage-exiftool-perl + #+end_src + +**** 2. Install Semantic Search + #+begin_src shell + git clone https://github.com/debanjum/semantic-search && cd semantic-search + conda env create -f environment.yml + conda activate semantic-search + #+end_src + +**** 3. Configure + Configure application search types and their underlying data source/files in ~sample_config.yml~ + Use the ~sample_config.yml~ as reference + +**** 4. Run + Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML + + #+begin_src shell + python3 -m src.main -c=sample_config.yml -vv + #+end_src ** Use - *Semantic Search via Emacs* @@ -39,19 +59,29 @@ - *Semantic Search via API* - Query: ~GET~ [[http://localhost:8000/search?q=%22what%20is%20the%20meaning%20of%20life%22][http://localhost:8000/search?q="What is the meaning of life"&t=notes]] - - Regenerate Embeddings: ~GET~ [[http://localhost:8000/regenerate][http://localhost:8000/regenerate?t=image]] + - Regenerate Embeddings: ~GET~ [[http://localhost:8000/regenerate][http://localhost:8000/regenerate]] - [[http://localhost:8000/docs][Semantic Search API Docs]] + - *UI to Edit Config* + - [[https://localhost:8000/ui][Config UI]] + ** Upgrade - #+begin_src shell - cd semantic-search - git pull origin master - conda env update -f environment.yml - conda activate semantic-search - #+end_src + +*** Using Docker + #+begin_src shell + docker-compose up + #+end_src + +*** On Local Machine + #+begin_src shell + cd semantic-search + git pull origin master + conda env update -f environment.yml + conda activate semantic-search + #+end_src ** Acknowledgments - [[https://huggingface.co/sentence-transformers/msmarco-MiniLM-L-6-v3][MiniLM Model]] for Asymmetric Text Search. See [[https://www.sbert.net/examples/applications/retrieve_rerank/README.html][SBert Documentation]] - [[https://github.com/openai/CLIP][OpenAI CLIP Model]] for Image Search. See [[https://www.sbert.net/examples/applications/image-search/README.html][SBert Documentation]] - Charles Cave for [[http://members.optusnet.com.au/~charles57/GTD/orgnode.html][OrgNode Parser]] - - Sven Marnach for [[https://github.com/smarnach/pyexiftool/blob/master/exiftool.py][PyExifTool]] + - Sven Marnach for [[https://github.com/smarnach/pyexiftool/blob/master/exiftool.py][PyExifTool]] \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index 71c61286..f0cd1db8 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -20,13 +20,13 @@ services: # points to the files you want to index. # The path of the mounted directory (right hand side), # must match the path prefix in your config file. - - /path/to/notes/:/data/notes/ - - /path/to/photos/:/data/images/ - - /path/to/ledger/:/data/ledger/ - - /path/to/music/:/data/music/ + - ./tests/data/:/data/notes/ + - ./tests/data/:/data/images/ + - ./tests/data/:/data/ledger/ + - ./tests/data/:/data/music/ # It's ok if you don't have existing embeddings. - # Leave the line as is - an empty volume will be created if it doesn't exist. - - /path/to/embeddings/:/data/generated/ + # You can set this volume to point to an empty folder. + - ./tests/data/:/data/generated/ # Use 0.0.0.0 to explicitly set the host ip for the service on the container. https://pythonspeed.com/articles/docker-connection-refused/ - command: --host="0.0.0.0" --port=8000 -c=docker_sample_config.yml + command: --host="0.0.0.0" --port=8000 -c=docker_sample_config.yml -vv