sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-27 09:25:06 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 5
Mirror of khoj from Github
202 commits 12 branches 107 tags 132 MiB
Python 50.2%
TypeScript 32.9%
HTML 7.3%
Emacs Lisp 3.4%
JavaScript 3.1%
Other 3.1%
Find a file
Debanjum Singh Solanky 79c2224eaa Improve test data organization and update correspoding conftests 
- Put test data for each content type into separate directories
- Makes config.yml for docker and local host consistent
  - Prepending tests to /data in sample_config.yml makes application
    run on local host using test data
  - Allows mounting separate volume for each content type in docker-compose
- Ignore gitignore to only add tests content, not generated models or embeddings
2022-01-29 02:03:17 -05:00
.github/workflows Try cache conda build step using online docs 2021-12-15 11:48:14 +05:30
src Fix type of use_xmp_metadata to be bool, rather than str 2022-01-24 21:53:26 -05:00
tests Improve test data organization and update correspoding conftests 2022-01-29 02:03:17 -05:00
views Fix input text behavior for null/empty value fields 2021-12-04 10:45:48 -05:00
.gitignore Improve test data organization and update correspoding conftests 2022-01-29 02:03:17 -05:00
docker-compose.yml Merge sample_config, docker_sample_config yml into a single sample_config.yml 2022-01-29 01:32:12 -05:00
Dockerfile Simplify Dockerfile by removing multibuild 2022-01-24 21:54:10 -05:00
environment.yml Merge branch 'master' of github.com:debanjum/semantic-search into add-summarize-capability-to-chat-bot 2021-12-20 13:30:42 +05:30
LICENSE Add Readme, License. Update .gitignore 2021-08-15 22:52:37 -07:00
README.org Merge sample_config, docker_sample_config yml into a single sample_config.yml 2022-01-29 01:32:12 -05:00
sample_config.yml Merge sample_config, docker_sample_config yml into a single sample_config.yml 2022-01-29 01:32:12 -05:00

README.org

https://github.com/debanjum/semantic-search/actions/workflows/build.yml/badge.svg

Semantic Search

Allow natural language search on user content like notes, images, transactions using transformer based models

All search is done locally. User can interface with semantic-search app via Emacs, API or Commandline

Setup

Setup using Docker

1. Clone Repository 
  git clone https://github.com/debanjum/semantic-search && cd semantic-search
2. Configure
  • Add Content Directories for Semantic Search to Docker-Compose
  • Update docker-compose.yml to mount your images, org-mode notes, ledger/beancount directories
  • If required, edit config settings in docker_sample_config.yml.
3. Run 
docker-compose up -d
Troubleshooting
  • The first run will take time. Let it run, it's mostly not hung

  • Symptom: Errors out with "Killed" in error message

  • Symptom: Errors out complaining about Tensors mismatch, null etc

    • Mitigation: Delete content-type > image section from docker_sample_config.yml

Setup on Local Machine

1. Install Dependencies
  1. Install Python3 [Required[
  2. Install Conda [Required]

  3. Install Exiftool [Optional]

    sudo apt-get -y install libimage-exiftool-perl
2. Install Semantic Search 
git clone https://github.com/debanjum/semantic-search && cd semantic-search
conda env create -f environment.yml
conda activate semantic-search
3. Configure
  • Configure files/directories to search in content-type section of sample_config.yml

  • To run application on test data, update file paths containing /data/ to tests/data/ in sample_config.yml

    • Example replace /data/notes/*.org with tests/data/notes/*.org
4. Run

Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML

python3 -m src.main -c=sample_config.yml -vv

Use

Upgrade

On Docker 

  docker-compose build

On Local Machine 

  cd semantic-search
  git pull origin master
  conda deactivate semantic-search
  conda env update -f environment.yml
  conda activate semantic-search

Miscellaneous

  • The experimental /chat API endpoint uses the OpenAI API

    • It is disabled by default
    • To use it add your openai-api-key to config.yml

Acknowledgments