sij/khoj: Mirror of khoj from Github

sij/khoj

mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-23 15:38:55 +01:00

Mirror of khoj from Github

agent ai assistant chat chatgpt emacs image-generation llama3 llamacpp llm obsidian obsidian-md offline-llm productivity rag research self-hosted semantic-search stt whatsapp-ai

Find a file

Debanjum Singh Solanky c35c6fb0b3 Reuse asymmetric.setup & input validation from asymmetric & org_to_jsonl Create asymmetric.setup method to - initialize model - generate compressed jsonl - compute embeddings put input_files, input_file_filter validation in org_to_jsonl for reuse in main.py, asymmetic.py		2021-08-17 00:45:40 -07:00
interface/emacs	Minor doc updates after merging emacs package with main repository	2021-08-16 02:02:26 -07:00
processor	Reuse asymmetric.setup & input validation from asymmetric & org_to_jsonl	2021-08-17 00:45:40 -07:00
search_type	Reuse asymmetric.setup & input validation from asymmetric & org_to_jsonl	2021-08-17 00:45:40 -07:00
utils	Allow reuse of get_absolute_path, is_none_or_empty methods	2021-08-16 16:33:43 -07:00
.gitignore	Add Readme, License. Update .gitignore	2021-08-15 22:52:37 -07:00
environment.yml	Create API interface for Semantic Search	2021-08-15 18:11:48 -07:00
LICENSE	Add Readme, License. Update .gitignore	2021-08-15 22:52:37 -07:00
main.py	Reuse asymmetric.setup & input validation from asymmetric & org_to_jsonl	2021-08-17 00:45:40 -07:00
README.org	Replace README.md with README.org. Can be used as notes for testing	2021-08-16 20:00:05 -07:00

README.org

Semantic Search
- Dependencies
- Install
- Run
- Use
- Acknowledgments

Semantic Search

Allow natural language search on user content like notes, images using transformer based models

All data is processed locally. User can interface with semantic-search app via Emacs, API or Commandline

Dependencies

Python3
Miniconda

Install

git clone https://github.com/debanjum/semantic-search && cd semantic-search
conda env create -f environment.yml
conda activate semantic-search

Run

Load ML model, generate embeddings and expose API to query specified org-mode files

python3 main.py --input-files ~/Notes/Schedule.org ~/Notes/Incoming.org --verbose

Use

Semantic Search via Emacs
- Install semantic-search.el
- Run M-x semantic-search <user-query> or Call C-c C-s
Semantic Search via API
- Query: GET http://localhost:8000/search?q="What is the meaning of life"
- Regenerate Embeddings: GET http://localhost:8000/regenerate
- Semantic Search API Docs

Call Semantic Search via Python Script Directly

python3 search_types/asymmetric.py \
--compressed-jsonl .notes.jsonl.gz \
--embeddings .notes_embeddings.pt \
--results-count 5 \
--verbose \
--interactive

Acknowledgments

MiniLM Model for Asymmetric Text Search. See SBert Documentation
OpenAI CLIP Model for Image Search. See SBert Documentation
Charles Cave for OrgNode Parser