sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-27 17:35:07 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
Mirror of khoj from Github
352 commits 12 branches 107 tags 132 MiB
Python 50.2%
TypeScript 32.9%
HTML 7.3%
Emacs Lisp 3.4%
JavaScript 3.1%
Other 3.1%
Find a file
Debanjum a71253e137
Support Incremental Search on Web Interface 
## Support Incremental Search on Khoj Web Interface
- Use default, fast path to query /search API while user is typing
- Upgrade to cross-encoder re-ranked results once user hits enter on search box

## Improve Render of Org Results on Web Interface
- We were previously just wrapping results from /search API into a pre formatted div field. This was not easy to read
- Use [org.js](https://mooz.github.io/org-js/) to render results from Khoj `/search` API as proper HTML
- Improve org.js to render all task states, stylize task tags and make org-mode results look more like original content

Closes #42 #41
2022-07-28 09:31:57 -07:00
.github/workflows Run build on PR 2022-07-04 18:09:47 -04:00
config Create test markdown files. Use them in sample config, docker-compose 2022-07-21 22:09:44 +04:00
docs Add Khoj Architecture Diagram in Docs. Show it in the Project Readme 2022-07-26 02:09:51 +04:00
src Support Incremental Search on Web Interface 2022-07-28 09:31:57 -07:00
tests Make cross-encoder re-rank results if query param set on /search API 2022-07-26 22:56:36 +04:00
views Fix input text behavior for null/empty value fields 2021-12-04 10:45:48 -05:00
.dockerignore Make Docker ignore unnecessary files 2022-06-29 22:29:34 +04:00
.gitignore Create Basic Landing Page to Query Semantic Search and Render Results 2022-07-16 03:36:19 +04:00
demo.mp4 Add demo of semantic search to repository 2022-05-14 04:29:25 -04:00
docker-compose.yml Create test markdown files. Use them in sample config, docker-compose 2022-07-21 22:09:44 +04:00
Dockerfile Give the project a short, less generic name. Rename it to Khoj 2022-07-19 18:26:16 +04:00
LICENSE Add Readme, License. Update .gitignore 2021-08-15 22:52:37 -07:00
README.org Add Feature Section to Readme 2022-07-25 15:43:27 -07:00

README.org

https://github.com/debanjum/khoj/actions/workflows/test.yml/badge.svg https://github.com/debanjum/khoj/actions/workflows/build.yml/badge.svg

Khoj

Natural language search engine for your personal notes, transactions and images

Features

  • Advanced Natural language understanding using Transformer based ML Models
  • Your personal data stays local. All search, indexing is done on your machine*
  • Index Org-mode and Markdown notes, Beancount transactions and Photos
  • Interact with Khoj using a Web Browser, Emacs or the API.

Demo

Architecture

https://github.com/debanjum/khoj/blob/master/docs/khoj_architecture.png

Setup

1. Clone 

  git clone https://github.com/debanjum/khoj && cd khoj

2. Configure

  • Required: Update docker-compose.yml to mount your images, (org-mode or markdown) notes and beancount directories
  • Optional: Edit application configuration in sample_config.yml

3. Run 

docker-compose up -d

Note: The first run will take time. Let it run, it's mostly not hung, just generating embeddings

Use

Upgrade 

  docker-compose build --pull

Troubleshooting

  • Symptom: Errors out with "Killed" in error message

  • Symptom: Errors out complaining about Tensors mismatch, null etc

    • Mitigation: Delete content-type > image section from docker_sample_config.yml

Miscellaneous

  • The experimental chat API endpoint uses the OpenAI API

    • It is disabled by default
    • To use it add your openai-api-key to config.yml

Development Setup

Setup on Local Machine

1. Install Dependencies
  1. Install Python3 [Required]
  2. Install Conda [Required]

  3. Install Exiftool [Optional]

    sudo apt-get -y install libimage-exiftool-perl
2. Install Khoj 
git clone https://github.com/debanjum/khoj && cd khoj
conda env create -f config/environment.yml
conda activate khoj
3. Configure
  • Configure files/directories to search in content-type section of sample_config.yml

  • To run application on test data, update file paths containing /data/ to tests/data/ in sample_config.yml

    • Example replace /data/notes/*.org with tests/data/notes/*.org
4. Run

Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML

python3 -m src.main -c=config/sample_config.yml -vv

Upgrade On Local Machine 

  cd khoj
  git pull origin master
  conda deactivate khoj
  conda env update -f config/environment.yml
  conda activate khoj

Run Unit tests 

pytest

Acknowledgments