khoj/README.org at d4d7dbaca60d10c9e8dcde0c290d83320f6cf7b7

sij/khoj

mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-18 02:27:10 +00:00

Debanjum Singh Solanky d4d7dbaca6 Support Natural Search on Markdown Files

- Reason:
  Allow natural search on markdown based notes, documentation,
  websites etc

- Details:
  - Create markdown processor to extract Markdown entries (identified by
    Heading) into standard jsonl format required by text_search
  - Update API, Configs to support interfacing with new markdown type
  - Update Emacs, Web clients to support interfacing with new markdown
    type via API
  - Update Readme to mentiond markdown is also supported

Closes #35

2022-07-21 22:07:05 +04:00

4.3 KiB

Raw Blame History

Khoj

Khoj

Allow natural language search on user content like notes, images, transactions using transformer ML models

User can interface with Khoj via the API or Emacs. All search is done locally*

Demo

Setup

1. Clone

  git clone https://github.com/debanjum/khoj && cd khoj

2. Configure

[Required] Update docker-compose.yml to mount your images, (org-mode or markdown) notes and beancount directories
[Optional] Edit application configuration in sample_config.yml

3. Run

docker-compose up -d

Note: The first run will take time. Let it run, it's mostly not hung, just generating embeddings

Use

Khoj via API
Khoj via Emacs
- Install khoj.el
- Run M-x khoj <user-query>

Run Unit tests

pytest

Upgrade

  docker-compose build --pull

Troubleshooting

Symptom: Errors out with "Killed" in error message
- Fix: Increase RAM available to Docker Containers in Docker Settings
- Refer: StackOverflow Solution, Configure Resources on Docker for Mac
Symptom: Errors out complaining about Tensors mismatch, null etc
- Mitigation: Delete content-type > image section from docker_sample_config.yml

Miscellaneous

The experimental chat API endpoint uses the OpenAI API
- It is disabled by default
- To use it add your openai-api-key to config.yml

Development Setup

Setup on Local Machine

1. Install Dependencies

Install Python3 [Required]
Install Conda [Required]

Install Exiftool [Optional]

sudo apt-get -y install libimage-exiftool-perl

2. Install Khoj

git clone https://github.com/debanjum/khoj && cd khoj
conda env create -f config/environment.yml
conda activate khoj

3. Configure

Configure files/directories to search in content-type section of sample_config.yml
To run application on test data, update file paths containing /data/ to tests/data/ in sample_config.yml
- Example replace /data/notes/*.org with tests/data/notes/*.org

4. Run

Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML

python3 -m src.main -c=config/sample_config.yml -vv

Upgrade On Local Machine

  cd khoj
  git pull origin master
  conda deactivate khoj
  conda env update -f config/environment.yml
  conda activate khoj

Acknowledgments

Multi-QA MiniLM Model for Asymmetric Text Search. See SBert Documentation
OpenAI CLIP Model for Image Search. See SBert Documentation
Charles Cave for OrgNode Parser
Sven Marnach for PyExifTool

4.3 KiB Raw Blame History