mirror of
https://github.com/khoj-ai/khoj.git
synced 2024-11-27 17:35:07 +01:00
79c2224eaa
- Put test data for each content type into separate directories
- Makes config.yml for docker and local host consistent
- Prepending tests to /data in sample_config.yml makes application
run on local host using test data
- Allows mounting separate volume for each content type in docker-compose
- Ignore gitignore to only add tests content, not generated models or embeddings
2.2 KiB
2.2 KiB
Semantic Search
Allow natural language search on user content like notes, images using transformer based models
All data is processed locally. User can interface with semantic-search app via Emacs, API or Commandline
Dependencies
- Python3
- Miniconda
Install
git clone https://github.com/debanjum/semantic-search && cd semantic-search
conda env create -f environment.yml
conda activate semantic-searchRun
Load ML model, generate embeddings and expose API to query specified org-mode files
python3 main.py --input-files ~/Notes/Schedule.org ~/Notes/Incoming.org --verboseUse
-
Semantic Search via Emacs
- Install semantic-search.el
- Run
M-x semantic-search <user-query>or CallC-c C-s
-
Semantic Search via API
- Query:
GEThttp://localhost:8000/search?q="What is the meaning of life" - Regenerate Embeddings:
GEThttp://localhost:8000/regenerate - Semantic Search API Docs
- Query:
-
Call Semantic Search via Python Script Directly
python3 search_types/asymmetric.py \ --compressed-jsonl .notes.jsonl.gz \ --embeddings .notes_embeddings.pt \ --results-count 5 \ --verbose \ --interactive
Acknowledgments
- MiniLM Model for Asymmetric Text Search. See SBert Documentation
- OpenAI CLIP Model for Image Search. See SBert Documentation
- Charles Cave for OrgNode Parser