sij/khoj

mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-23 15:38:55 +01:00

Mirror of khoj from Github

agent ai assistant chat chatgpt emacs image-generation llama3 llamacpp llm obsidian obsidian-md offline-llm productivity rag research self-hosted semantic-search stt whatsapp-ai

Find a file

Debanjum Singh Solanky 8b29e272d3 Standardize interface, better default args for org-to-json.py script - Remove non-standard, unnecessary argument for org-directory Pass path each file in org-files and org-files-filter argument directly - Allow shorthand -i, -o for input files, output files - Default to compress, unless user explicitly specifies not to		2021-08-16 11:29:08 -07:00
interface/emacs	Minor doc updates after merging emacs package with main repository	2021-08-16 02:02:26 -07:00
processor/org-mode	Standardize interface, better default args for org-to-json.py script	2021-08-16 11:29:08 -07:00
search_types	Use updated path to MiniLM bi-encoder model on hugging-face	2021-08-15 23:57:22 -07:00
utils	Change default install directory to current, fix open file code	2021-08-15 23:01:55 -07:00
.gitignore	Add Readme, License. Update .gitignore	2021-08-15 22:52:37 -07:00
environment.yml	Create API interface for Semantic Search	2021-08-15 18:11:48 -07:00
LICENSE	Add Readme, License. Update .gitignore	2021-08-15 22:52:37 -07:00
main.py	Move different search types into search_types directory	2021-08-15 19:09:50 -07:00
README.md	Standardize interface, better default args for org-to-json.py script	2021-08-16 11:29:08 -07:00

README.md

Semantic Search

Provide natural language search on user personal content like notes, images using ML models

All data is processed locally. User can interface with semantic-search app via Emacs, API or Commandline

Dependencies

Python3
Miniconda

Install

git clone https://github.com/debanjum/semantic-search && cd semantic-search
conda env create -f environment.yml
conda activate semantic-search

Setup

Generate compressed JSONL from specified org-mode files

python3 processor/org-mode/org-to-jsonl.py \
--org-files "~/Notes/Schedule.org" "~/Notes/Incoming.org" \
--jsonl-file ".notes.jsonl" \
--verbose

Run

Load ML model, generate embeddings and expose API interface to run user queries on above org-mode files

python3 main.py \
--jsonl-file .notes.jsonl.gz \
--embeddings-file .notes_embeddings.pt \
--verbose

Use

Semantic Search via Emacs
- Install semantic-search.el
- Run M-x semantic-search "<user-query>" or Call C-c C-s
Call Semantic Search via API
- GET http://localhost:8000/search?q="What is the meaning of life"

Call Semantic Search via Python Script Directly

python3 search_types/asymmetric.py \
--jsonl-file .notes.jsonl.gz \
--embeddings-file .notes_embeddings.pt \
--results-count 5 \
--verbose \
--interactive

Acknowledgments

MiniLM Model for Asymmetric Text Search. See SBert Documentation
OpenAI CLIP Model for Image Search. See SBert Documentation
Charles Cave for OrgNode Parser