mirror of
https://github.com/khoj-ai/khoj.git
synced 2024-11-23 15:38:55 +01:00
Mirror of khoj from Github
agentaiassistantchatchatgptemacsimage-generationllama3llamacppllmobsidianobsidian-mdoffline-llmproductivityragresearchself-hostedsemantic-searchsttwhatsapp-ai
77fa8718d9
Still need quite a bit of clean-up, but this adds a working docker-compose + Dockerfile setup |
||
---|---|---|
.github/workflows | ||
src | ||
tests | ||
views | ||
.gitignore | ||
docker-compose.yml | ||
docker_sample_config.yml | ||
Dockerfile | ||
environment.yml | ||
LICENSE | ||
README.org | ||
sample_config.yml |
Semantic Search
Allow natural language search on user content like notes, images, transactions using transformer based models
All data is processed locally. User can interface with semantic-search app via Emacs, API or Commandline
Dependencies
- Python3
- Miniconda
Install
git clone https://github.com/debanjum/semantic-search && cd semantic-search
conda env create -f environment.yml
conda activate semantic-search
Install Environmental Dependencies
sudo apt-get -y install libimage-exiftool-perl
Configure
Configure application search types and their underlying data source/files in sample_config.yml
Use the sample_config.yml
as reference
Run
Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML
python3 -m src.main -c=sample_config.yml -vv
Use
-
Semantic Search via Emacs
- Install semantic-search.el
- Run
M-x semantic-search <user-query>
-
Semantic Search via API
- Query:
GET
http://localhost:8000/search?q="What is the meaning of life"&t=notes - Regenerate Embeddings:
GET
http://localhost:8000/regenerate?t=image - Semantic Search API Docs
- Query:
Upgrade
cd semantic-search
git pull origin master
conda env update -f environment.yml
conda activate semantic-search
Acknowledgments
- MiniLM Model for Asymmetric Text Search. See SBert Documentation
- OpenAI CLIP Model for Image Search. See SBert Documentation
- Charles Cave for OrgNode Parser
- Sven Marnach for PyExifTool