mirror of
https://github.com/khoj-ai/khoj.git
synced 2024-11-23 23:48:56 +01:00
📝 Add a documentation base for Khoj! (#333)
* Add docs for more organized, accessible information detailing Khoj setup * Delete duplicated files * Add a coverpage without enabling it. Add logo and theme * Remove obsidian README.md * Add plausible script to index.html via docsify
This commit is contained in:
sabaimran
GitHub
parent
3e59be7f1d
commit
1610d2ebd9
32 changed files with 6162 additions and 222 deletions
0
docs/.nojekyll
Normal file
0
docs/.nojekyll
Normal file
59
docs/README.md
Normal file
59
docs/README.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
<h1><img src="./assets/khoj-logo-sideways.svg" width="200" alt="Khoj Logo"></h1>
|
||||
|
||||
[](https://github.com/khoj-ai/khoj/actions/workflows/test.yml)
|
||||
[](https://github.com/khoj-ai/khoj/pkgs/container/khoj)
|
||||
[](https://pypi.org/project/khoj-assistant/)
|
||||
|
||||
# Khoj
|
||||
*An AI personal assistant for your digital brain*
|
||||
|
||||
Welcome to the Docs! This is the best place to get started with Khoj. Check out our [Github](https://github.com/khoj-ai/khoj) to dive straight into the code.
|
||||
|
||||
Khoj gives you lightning fast, offline search on your personal machine and gives you the power to talk to your notes.
|
||||
|
||||
**Supported Plugins**
|
||||
|
||||
[](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#readme)
|
||||
[](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#readme)
|
||||
|
||||
## Features
|
||||
- **Search**
|
||||
- **Local**: Your personal data stays local. All search and indexing is done on your machine. *Unlike chat which requires access to GPT.*
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
- **Chat**
|
||||
- **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers.
|
||||
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
|
||||
- **Assisted creativity**: Smoothly weave across answers retrieval and content generation
|
||||
- **General**
|
||||
- **Natural**: Advanced natural language understanding using Transformer based ML Models
|
||||
- **Pluggable**: Modular architecture makes it easy to plug in new data sources, frontends and ML models
|
||||
- **Multiple Sources**: Index your Org-mode and Markdown notes, PDF files, Github repositories, and Photos
|
||||
- **Multiple Interfaces**: Interact from your [Web Browser](./src/khoj/interface/web/index.html), [Emacs](./src/interface/emacs/khoj.el) or [Obsidian](./src/interface/obsidian/)
|
||||
|
||||
## Install
|
||||
[Click here](./setup.md) for full setup instructions.
|
||||
|
||||
## Supported data sources
|
||||
- markdown*
|
||||
- org-mode*
|
||||
- pdf*
|
||||
- images*
|
||||
- github
|
||||
- notion
|
||||
|
||||
\* These data sources are offline only.
|
||||
|
||||
If you're using Github or Notion, you can get on a waitlist for [Khoj Cloud](https://khoj.dev).
|
||||
|
||||
## Credits
|
||||
|
||||
- [Multi-QA MiniLM Model](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1), [All MiniLM Model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) for Text Search. See [SBert Documentation](https://www.sbert.net/examples/applications/retrieve_rerank/README.html)
|
||||
- [OpenAI CLIP Model](https://github.com/openai/CLIP) for Image Search. See [SBert Documentation](https://www.sbert.net/examples/applications/image-search/README.html)
|
||||
- Charles Cave for [OrgNode Parser](http://members.optusnet.com.au/~charles57/GTD/orgnode.html)
|
||||
- [Org.js](https://mooz.github.io/org-js/) to render Org-mode results on the Web interface
|
||||
- [Markdown-it](https://github.com/markdown-it/markdown-it) to render Markdown results on the Web interface
|
||||
|
||||
|
||||
[^1]: Default Khoj config file @ `~/.khoj/khoj.yml`
|
||||
|
||||
[^2]: Default Khoj url @ http://localhost:42110
|
||||
14
docs/_coverpage.md
Normal file
14
docs/_coverpage.md
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
<!-- _coverpage.md -->
|
||||
|
||||
|
||||
|
||||
> An open source, AI personal assistant for your notes
|
||||
|
||||
- Lightning fast search
|
||||
- Multi-turn chat
|
||||
- Keeps you in control of your data
|
||||
|
||||
[GitHub](https://github.com/khoj-ai/khoj)
|
||||
[Get Started](#khoj)
|
||||
|
||||
|
||||
18
docs/_sidebar.md
Normal file
18
docs/_sidebar.md
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
- Getting Started
|
||||
- [Overview](README.md)
|
||||
- [Install](setup.md)
|
||||
- [Windows Installation](windows_install.md)
|
||||
- Learn More
|
||||
- [Features](features.md)
|
||||
- [Chat](chat.md)
|
||||
- [Search](search.md)
|
||||
- [Demos](demos.md)
|
||||
- Interfaces
|
||||
- [Obsidian](obsidian.md)
|
||||
- [Emacs](emacs.md)
|
||||
- Data Sources
|
||||
- [Github](github_integration.md)
|
||||
- [Notion](notion_integration.md)
|
||||
- [Advanced](advanced.md)
|
||||
- Contributing
|
||||
- [Development](development.md)
|
||||
54
docs/advanced.md
Normal file
54
docs/advanced.md
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
|
||||
## Advanced Usage
|
||||
### Access Khoj on Mobile
|
||||
1. [Setup Khoj](#Setup) on your personal server. This can be any always-on machine, i.e an old computer, RaspberryPi(?) etc
|
||||
2. [Install](https://tailscale.com/kb/installation/) [Tailscale](tailscale.com/) on your personal server and phone
|
||||
3. Open the Khoj web interface of the server from your phone browser.<br /> It should be `http://tailscale-ip-of-server:42110` or `http://name-of-server:42110` if you've setup [MagicDNS](https://tailscale.com/kb/1081/magicdns/)
|
||||
4. Click the [Add to Homescreen](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Add_to_home_screen) button
|
||||
5. Enjoy exploring your notes, documents and images from your phone!
|
||||
|
||||
|
||||
|
||||
### Search across Different Languages
|
||||
To search for notes in multiple, different languages, you can use a [multi-lingual model](https://www.sbert.net/docs/pretrained_models.html#multi-lingual-models).<br />
|
||||
For example, the [paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) supports [50+ languages](https://www.sbert.net/docs/pretrained_models.html#:~:text=we%20used%20the%20following%2050%2B%20languages), has good search quality and speed. To use it:
|
||||
1. Manually update `search-type > asymmetric > encoder` to `paraphrase-multilingual-MiniLM-L12-v2` in your `~/.khoj/khoj.yml` file for now. See diff of `khoj.yml` below for illustration:
|
||||
```diff
|
||||
asymmetric:
|
||||
- encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1"
|
||||
+ encoder: "paraphrase-multilingual-MiniLM-L12-v2"
|
||||
cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
|
||||
model_directory: "~/.khoj/search/asymmetric/"
|
||||
```
|
||||
|
||||
2. Regenerate your content index. For example, by opening [\<khoj-url\>/api/update?t=force](http://localhost:42110/api/update?t=force)
|
||||
|
||||
### Bootstrap Khoj Search for Offline Usage later
|
||||
|
||||
You can bootstrap Khoj pre-emptively to run on machines that do not have internet access. An example use-case would be to run Khoj on an air-gapped machine.
|
||||
Note: *Only search can currently run in fully offline mode, not chat.*
|
||||
|
||||
- With Internet
|
||||
1. Manually download the [asymmetric text](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1), [symmetric text](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)and [image search](https://huggingface.co/sentence-transformers/clip-ViT-B-32) models from HuggingFace
|
||||
2. Pip install khoj (and dependencies) in an associated virtualenv. E.g `python -m venv .venv && source .venv/bin/activate && pip install khoj-assistant`
|
||||
- Without Internet
|
||||
1. Copy each of the search models into their respective folders, `asymmetric`, `symmetric` and `image` under the `~/.khoj/search/` directory on the air-gapped machine
|
||||
2. Copy the khoj virtual environment directory onto the air-gapped machine, activate the environment and start and khoj as normal. E.g `source .venv/bin/activate && khoj`
|
||||
|
||||
|
||||
## Miscellaneous
|
||||
### Set your OpenAI API key in Khoj
|
||||
If you want, Khoj can be configured to use OpenAI for search and chat.<br />
|
||||
Add your OpenAI API to Khoj by using either of the two options below:
|
||||
- Open your [Khoj settings](http://localhost:42110/config/processor/conversation), add your OpenAI API key, and click *Save*. Then go to your [Khoj settings](http://localhost:42110/config) and click `Configure`. This will refresh Khoj with your OpenAI API key.
|
||||
- Set `openai-api-key` field under `processor.conversation` section in your `khoj.yml`[^1] to your [OpenAI API key](https://beta.openai.com/account/api-keys) and restart khoj:
|
||||
```diff
|
||||
processor:
|
||||
conversation:
|
||||
- openai-api-key: # "YOUR_OPENAI_API_KEY"
|
||||
+ openai-api-key: sk-aaaaaaaaaaaaaaaaaaaaaaaahhhhhhhhhhhhhhhhhhhhhhhh
|
||||
model: "text-davinci-003"
|
||||
conversation-logfile: "~/.khoj/processor/conversation/conversation_logs.json"
|
||||
```
|
||||
|
||||
!> **Warning**: This will enable Khoj to send your query and note(s) to OpenAI for processing
|
||||
BIN
docs/assets/favicon-128x128.ico
Normal file
BIN
docs/assets/favicon-128x128.ico
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 200 KiB |
|
Before Width: | Height: | Size: 979 KiB After Width: | Height: | Size: 979 KiB |
BIN
docs/assets/khoj-logo-sideways-200.png
Normal file
BIN
docs/assets/khoj-logo-sideways-200.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
5385
docs/assets/khoj-logo-sideways.svg
Normal file
5385
docs/assets/khoj-logo-sideways.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 1.2 MiB |
|
Before Width: | Height: | Size: 350 KiB After Width: | Height: | Size: 350 KiB |
|
Before Width: | Height: | Size: 302 KiB After Width: | Height: | Size: 302 KiB |
|
Before Width: | Height: | Size: 277 KiB After Width: | Height: | Size: 277 KiB |
|
Before Width: | Height: | Size: 126 KiB After Width: | Height: | Size: 126 KiB |
|
Before Width: | Height: | Size: 544 KiB After Width: | Height: | Size: 544 KiB |
|
Before Width: | Height: | Size: 49 KiB After Width: | Height: | Size: 49 KiB |
|
Before Width: | Height: | Size: 333 KiB After Width: | Height: | Size: 333 KiB |
|
Before Width: | Height: | Size: 1.1 MiB After Width: | Height: | Size: 1.1 MiB |
|
Before Width: | Height: | Size: 378 KiB After Width: | Height: | Size: 378 KiB |
|
Before Width: | Height: | Size: 445 KiB After Width: | Height: | Size: 445 KiB |
21
docs/chat.md
Normal file
21
docs/chat.md
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
### Khoj Chat
|
||||
#### Overview
|
||||
- Creates a personal assistant for you to inquire and engage with your notes
|
||||
- Uses [ChatGPT](https://openai.com/blog/chatgpt) and [Khoj search](#khoj-search). [Offline chat](https://github.com/khoj-ai/khoj/issues/201) is coming soon.
|
||||
- Supports multi-turn conversations with the relevant notes for context
|
||||
- Shows reference notes used to generate a response
|
||||
- **Note**: *Your query and top notes from khoj search will be sent to OpenAI for processing*
|
||||
|
||||
#### Setup
|
||||
- [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
|
||||
|
||||
#### Use
|
||||
1. Open [/chat](http://localhost:42110/chat)[^2]
|
||||
2. Type your queries and see response by Khoj from your notes
|
||||
|
||||
#### Demo
|
||||
|
||||
|
||||
### Details
|
||||
1. Your query is used to retrieve the most relevant notes, if any, using Khoj search
|
||||
2. These notes, the last few messages and associated metadata is passed to ChatGPT along with your query for a response
|
||||
40
docs/demos.md
Normal file
40
docs/demos.md
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
## Demos
|
||||
|
||||
### Interfaces
|
||||
|
||||
|
||||
|
||||
### Khoj in Obsidian
|
||||
[KhojObsidian](https://github-production-user-asset-6210df.s3.amazonaws.com/6413477/240061700-3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b.mp4 ':include :type=mp4')
|
||||
|
||||
#### Installation
|
||||
|
||||
1. Install Khoj via `pip` and start Khoj backend in a terminal (Run `khoj`)
|
||||
```bash
|
||||
python -m pip install khoj-assistant
|
||||
khoj
|
||||
```
|
||||
2. Install Khoj plugin via Community Plugins settings pane on Obsidian app
|
||||
- Check the new Khoj plugin settings
|
||||
- Let Khoj backend index the markdown, pdf, Github markdown files in the current Vault
|
||||
- Open Khoj plugin on Obsidian via Search button on Left Pane
|
||||
- Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/)
|
||||
- Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
|
||||
|
||||
### Khoj in Emacs, Browser
|
||||
[KhojEmacs](https://user-images.githubusercontent.com/6413477/184735169-92c78bf1-d827-4663-9087-a1ea194b8f4b.mp4 ':include :type=mp4')
|
||||
|
||||
#### Installation
|
||||
|
||||
- Install Khoj via pip
|
||||
- Start Khoj app
|
||||
- Add this readme and [khoj.el readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs) as org-mode for Khoj to index
|
||||
- Search \"*Setup editor*\" on the Web and Emacs. Re-rank the results for better accuracy
|
||||
- Top result is what we are looking for, the [section to Install Khoj.el on Emacs](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#2-Install-Khojel)
|
||||
|
||||
#### Analysis
|
||||
|
||||
- The results do not have any words used in the query
|
||||
- *Based on the top result it seems the re-ranking model understands that Emacs is an editor?*
|
||||
- The results incrementally update as the query is entered
|
||||
- The results are re-ranked, for better accuracy, once user hits enter
|
||||
114
docs/development.md
Normal file
114
docs/development.md
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
# Development
|
||||
|
||||
## Setup
|
||||
### Using Pip
|
||||
#### 1. Install
|
||||
|
||||
```shell
|
||||
# Get Khoj Code
|
||||
git clone https://github.com/khoj-ai/khoj && cd khoj
|
||||
|
||||
# Create, Activate Virtual Environment
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
|
||||
# Install Khoj for Development
|
||||
pip install -e .[dev]
|
||||
```
|
||||
|
||||
#### 2. Run
|
||||
1. Start Khoj
|
||||
```shell
|
||||
khoj -vv
|
||||
```
|
||||
2. Configure Khoj
|
||||
- **Via the Settings UI**: Add files, directories to index the [Khoj settings](http://localhost:42110/config) UI once Khoj has started up. Once you've saved all your settings, click `Configure`.
|
||||
- **Manually**:
|
||||
- Copy the `config/khoj_sample.yml` to `~/.khoj/khoj.yml`
|
||||
- Set `input-files` or `input-filter` in each relevant `content-type` section of `~/.khoj/khoj.yml`
|
||||
- Set `input-directories` field in `image` `content-type` section
|
||||
- Delete `content-type` and `processor` sub-section(s) irrelevant for your use-case
|
||||
- Restart khoj
|
||||
|
||||
Note: Wait after configuration for khoj to Load ML model, generate embeddings and expose API to query notes, images, documents etc specified in config YAML
|
||||
|
||||
### Using Docker
|
||||
#### 1. Clone
|
||||
|
||||
```shell
|
||||
git clone https://github.com/khoj-ai/khoj && cd khoj
|
||||
```
|
||||
|
||||
#### 2. Configure
|
||||
|
||||
- **Required**: Update [docker-compose.yml](./docker-compose.yml) to mount your images, (org-mode or markdown) notes, PDFs and Github repositories
|
||||
- **Optional**: Edit application configuration in [khoj_docker.yml](./config/khoj_docker.yml)
|
||||
|
||||
#### 3. Run
|
||||
|
||||
```shell
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
*Note: The first run will take time. Let it run, it\'s mostly not hung, just generating embeddings*
|
||||
|
||||
#### 4. Upgrade
|
||||
|
||||
```shell
|
||||
docker-compose build --pull
|
||||
```
|
||||
|
||||
## Validate
|
||||
### Before Making Changes
|
||||
1. Install Git Hooks for Validation
|
||||
```shell
|
||||
pre-commit install -t pre-push -t pre-commit
|
||||
```
|
||||
- This ensures standard code formatting fixes and other checks run automatically on every commit and push
|
||||
- Note 1: If [pre-commit](https://pre-commit.com/#intro) didn't already get installed, [install it](https://pre-commit.com/#install) via `pip install pre-commit`
|
||||
- Note 2: To run the pre-commit changes manually, use `pre-commit run --hook-stage manual --all` before creating PR
|
||||
|
||||
### Before Creating PR
|
||||
|
||||
1. Run Tests. If you get an error complaining about a missing `fast_tokenizer_file`, follow the solution [in this Github issue](https://github.com/UKPLab/sentence-transformers/issues/1659).
|
||||
```shell
|
||||
pytest
|
||||
```
|
||||
|
||||
2. Run MyPy to check types
|
||||
```shell
|
||||
mypy --config-file pyproject.toml
|
||||
```
|
||||
|
||||
### After Creating PR
|
||||
- Automated [validation workflows](.github/workflows) run for every PR.
|
||||
|
||||
Ensure any issues seen by them our fixed
|
||||
|
||||
- Test the python packge created for a PR
|
||||
1. Download and extract the zipped `.whl` artifact generated from the pypi workflow run for the PR.
|
||||
2. Install (in your virtualenv) with `pip install /path/to/download*.whl>`
|
||||
3. Start and use the application to see if it works fine
|
||||
|
||||
## Create Khoj Release
|
||||
Follow the steps below to [release](https://github.com/debanjum/khoj/releases/) Khoj. This will create a stable release of Khoj on [Pypi](https://pypi.org/project/khoj-assistant/), [Melpa](https://stable.melpa.org/#%252Fkhoj) and [Obsidian](https://obsidian.md/plugins?id%253Dkhoj). It will also create desktop apps of Khoj and attach them to the latest release.
|
||||
|
||||
1. Create and tag release commit by running the bump_version script. The release commit sets version number in required metadata files.
|
||||
```shell
|
||||
./scripts/bump_version.sh -c "<release_version>"
|
||||
```
|
||||
2. Push commit and then the tag to trigger the release workflow to create Release with auto generated release notes.
|
||||
```shell
|
||||
git push origin master # push release commit to khoj repository
|
||||
git push origin <release_version> # push release tag to khoj repository
|
||||
```
|
||||
3. [Optional] Update the Release Notes to highlight new features, fixes and updates
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
|
||||
## Visualize Codebase
|
||||
|
||||
*[Interactive Visualization](https://mango-dune-07a8b7110.1.azurestaticapps.net/?repo=debanjum%2Fkhoj)*
|
||||
|
||||
|
||||
152
docs/emacs.md
Normal file
152
docs/emacs.md
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
<h1><img src="./assets/khoj-logo-sideways.svg" width="200" alt="Khoj Logo">Emacs</h1>
|
||||
|
||||
An AI personal assistance for your digital brain
|
||||
|
||||
<img src="https://stable.melpa.org/packages/khoj-badge.svg" width="150" alt="Melpa Stable Badge">
|
||||
<img src="https://melpa.org/packages/khoj-badge.svg" width="150" alt="Melpa Badge">
|
||||
|
||||
<img src="https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml/badge.svg" width="150" alt="Build Badge">
|
||||
<img src="https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml/badge.svg" width="150" alt="Test Badge">
|
||||
|
||||
|
||||
## Features
|
||||
- **Search**
|
||||
- **Natural**: Advanced natural language understanding using Transformer based ML Models
|
||||
- **Local**: Your personal data stays local. All search, indexing is done on your machine*
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
- **Chat**
|
||||
- **Faster answers**: Find answers faster than search
|
||||
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
|
||||
- **Assisted creativity**: Smoothly weave across answer retrieval and content generation
|
||||
|
||||
## Interface
|
||||
### Search UI
|
||||
|
||||
|
||||
|
||||
### Chat UI
|
||||
|
||||
|
||||
## Setup
|
||||
- /Make sure [[https://realpython.com/installing-python/][python]] and [[https://pip.pypa.io/en/stable/installation/][pip]] are installed on your machine/
|
||||
|
||||
- /khoj.el attempts to automatically install, start and configure the khoj server./
|
||||
If this fails, follow [[https://github.com/khoj-ai/khoj/tree/master/#Setup][these instructions]] to manually setup the khoj server.
|
||||
|
||||
### Direct Install
|
||||
```elisp
|
||||
M-x package-install khoj
|
||||
```
|
||||
|
||||
### Minimal Install
|
||||
Add below snippet to your Emacs config file.
|
||||
Indexes your org-agenda files, by default.
|
||||
|
||||
```elisp
|
||||
;; Install Khoj Package from MELPA Stable
|
||||
(use-package khoj
|
||||
:ensure t
|
||||
:pin melpa-stable
|
||||
:bind ("C-c s" . 'khoj)
|
||||
```
|
||||
|
||||
- Note: Install `khoj.el` from MELPA (instead of MELPA Stable) if you installed the pre-release version of khoj
|
||||
- That is, use `:pin melpa` to install khoj.el in above snippet if khoj server was installed with `--pre` flag, i.e `pip install --pre khoj-assistant`
|
||||
- Else use `:pin melpa-stable` to install khoj.el in above snippet if khoj was installed with `pip install khoj-assistant`
|
||||
- This ensures both khoj.el and khoj app are from the same version (git tagged or latest)
|
||||
|
||||
### Standard Install
|
||||
Add below snippet to your Emacs config file.
|
||||
Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat
|
||||
|
||||
```elisp
|
||||
;; Install Khoj Package from MELPA Stable
|
||||
(use-package khoj
|
||||
:ensure t
|
||||
:pin melpa-stable
|
||||
:bind ("C-c s" . 'khoj)
|
||||
:config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
|
||||
khoj-org-files '("~/docs/todo.org" "~/docs/work.org")
|
||||
khoj-openai-api-key "YOUR_OPENAI_API_KEY")) ; required to enable chat
|
||||
```
|
||||
|
||||
### With [https://github.com/raxod502/straight.el](Straight.el)
|
||||
Add below snippet to your Emacs config file.
|
||||
Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat
|
||||
|
||||
```elisp
|
||||
;; Install Khoj Package using Straight.el
|
||||
(use-package khoj
|
||||
:after org
|
||||
:straight (khoj :type git :host github :repo "khoj-ai/khoj" :files (:defaults "src/interface/emacs/khoj.el"))
|
||||
:bind ("C-c s" . 'khoj)
|
||||
:config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
|
||||
khoj-org-files '("~/docs/todo.org" "~/docs/work.org")
|
||||
khoj-openai-api-key "YOUR_OPENAI_API_KEY" ; required to enable chat)
|
||||
```
|
||||
|
||||
## Use
|
||||
### Search
|
||||
1. Hit `C-c s s` (or `M-x khoj RET s`) to open khoj search
|
||||
|
||||
2. Enter your query in natural language
|
||||
|
||||
e.g "What is the meaning of life?", "My life goals for 2023"
|
||||
|
||||
### Chat
|
||||
1. Hit `C-c s c` (or `M-x khoj RET c`) to open khoj chat
|
||||
|
||||
2. Ask questions in a natural, conversational style
|
||||
|
||||
E.g "When did I file my taxes last year?"
|
||||
|
||||
See [Khoj Chat](./README.md#khoj-chat) for more details
|
||||
|
||||
### Find Similar Entries
|
||||
This feature finds entries similar to the one you are currently on.
|
||||
1. Move cursor to the org-mode entry, markdown section or text paragraph you want to find similar entries for
|
||||
2. Hit `C-c s f` (or `M-x khoj RET f`) to find similar entries
|
||||
|
||||
### Advanced Usage
|
||||
- Add [[https://github.com/khoj-ai/khoj/#query-filters][query filters]] during search to narrow down results further
|
||||
|
||||
e.g `What is the meaning of life? -"god" +"none" dt>"last week"`
|
||||
|
||||
- Use `C-c C-o 2` to open the current result at cursor in its source org file
|
||||
- This calls `M-x org-open-at-point` on the current entry and opens the second link in the entry.
|
||||
- The second link is the entries [org-id](https://orgmode.org/manual/Handling-Links.html#FOOT28), if set, or the heading text.
|
||||
The first link is the line number of the entry in the source file. This link is less robust to file changes.
|
||||
- Note: If you have [speed keys](https://orgmode.org/manual/Speed-Keys.html) enabled, `o 2` will also work
|
||||
|
||||
### Khoj Menu
|
||||
|
||||
Hit `C-c s` (or `M-x khoj`) to open the khoj menu above. Then:
|
||||
- Hit `t` until you preferred content type is selected in the khoj menu
|
||||
|
||||
`Content Type` specifies the content to perform `Search`, `Update` or `Find Similar` actions on
|
||||
- Hit `n` twice and then enter number of results you want to see
|
||||
|
||||
`Results Count` is used by the `Search` and `Find Similar` actions
|
||||
- Hit `-f u` to `force` update the khoj content index
|
||||
|
||||
The `Force Update` switch is only used by the `Update` action
|
||||
|
||||
## Upgrade
|
||||
### Upgrade Khoj Backend
|
||||
```bash
|
||||
pip install --upgrade khoj-assistant
|
||||
```
|
||||
### Upgrade Khoj.el
|
||||
Use your Emacs package manager to upgrade `khoj.el`
|
||||
|
||||
- For `khoj.el` from MELPA
|
||||
- Method 1
|
||||
- Run `M-x package-list-packages` to list all packages
|
||||
- Press `U` on `khoj` to mark it for upgrade
|
||||
- Press `x` to execute the marked actions
|
||||
- Method 2
|
||||
- Run `M-x package-refresh-content`
|
||||
- Run `M-x package-reinstall khoj`
|
||||
|
||||
- For `khoj.el` from Straight
|
||||
- Run `M-x straight-pull-package khoj`
|
||||
13
docs/features.md
Normal file
13
docs/features.md
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
## Features
|
||||
- **Search**
|
||||
- **Local**: Your personal data stays local. All search and indexing is done on your machine. *Unlike chat which requires access to GPT.*
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
- **Chat**
|
||||
- **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers.
|
||||
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
|
||||
- **Assisted creativity**: Smoothly weave across answers retrieval and content generation
|
||||
- **General**
|
||||
- **Natural**: Advanced natural language understanding using Transformer based ML Models
|
||||
- **Pluggable**: Modular architecture makes it easy to plug in new data sources, frontends and ML models
|
||||
- **Multiple Sources**: Index your Org-mode and Markdown notes, PDF files, Github repositories, and Photos
|
||||
- **Multiple Interfaces**: Interact from your [Web Browser](https://github.com/khoj-ai/khoj/blob/master/src/khoj/interface/web/index.html), [Emacs](https://github.com/khoj-ai/khoj/blob/master//src/interface/emacs/khoj.el) or [Obsidian](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/)
|
||||
14
docs/github_integration.md
Normal file
14
docs/github_integration.md
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
# Setup the Github integration
|
||||
|
||||
The Github integration allows you to index as many repositories as you want. It's currently default configured to index Issues, Commits, and all Markdown/Org files in each repository. For large repositories, this takes a fairly long time, but it works well for smaller projects.
|
||||
|
||||
# Configure your settings
|
||||
|
||||
1. Go to [http://localhost:42110/config](http://localhost:42110/config) and enter in settings for the data sources you want to index. You'll have to specify the file paths.
|
||||
|
||||
## Use the Github plugin
|
||||
|
||||
1. Generate a [classic PAT (personal access token)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) from [Github](https://github.com/settings/tokens) with `repo` and `admin:org` scopes at least.
|
||||
2. Navigate to [http://localhost:42110/config/content_type/github](http://localhost:42110/config/content_type/github) to configure your Github settings. Enter in your PAT, along with details for each repository you want to index.
|
||||
3. Click `Save`. Go back to the settings page and click `Configure`.
|
||||
4. Go to [http://localhost:42110/](http://localhost:42110/) and start searching!
|
||||
35
docs/index.html
Normal file
35
docs/index.html
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>Document</title>
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
|
||||
<meta name="description" content="Description">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0">
|
||||
<link rel="stylesheet" href="//cdn.jsdelivr.net/npm/docsify/lib/themes/buble.css" />
|
||||
<link rel="icon" href="./assets/favicon-128x128.ico">
|
||||
</head>
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
<script>
|
||||
window.$docsify = {
|
||||
name: 'Khoj',
|
||||
repo: 'https://github.com/khoj-ai/khoj',
|
||||
loadSidebar: true,
|
||||
themeColor: '#c2a600',
|
||||
// coverpage: true,
|
||||
}
|
||||
</script>
|
||||
<!-- Docsify v4 -->
|
||||
<script src="//cdn.jsdelivr.net/npm/docsify@4"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/docsify/lib/plugins/search.min.js"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/docsify-copy-code/dist/docsify-copy-code.min.js"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/prismjs@1/components/prism-bash.min.js"></script>
|
||||
<script defer data-domain="khoj.dev" src="https://plausible.io/js/script.js"></script>
|
||||
</body>
|
||||
<style>
|
||||
video {
|
||||
max-width: 800px;
|
||||
}
|
||||
</style>
|
||||
</html>
|
||||
14
docs/notion_integration.md
Normal file
14
docs/notion_integration.md
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
## 📜 Notion Integration
|
||||
|
||||
Khoj now supports search/chat with pages in your Notion workspaces. [Notion](notion.so/) is a platform people use for taking notes, especially for collaboration.
|
||||
|
||||
We haven't setup a fancy integration with OAuth yet, so this integration still requires some effort on your end to generate an API key.
|
||||
|
||||
1. Go to https://www.notion.so/my-integrations and create a new integration called Khoj to get an API key.
|
||||
|
||||
3. Share all the workspaces that you want to integrate with the Khoj integration you just made in the previous step
|
||||
|
||||
4. In the first step, you generated an API key. Use the newly generated API Key in your Khoj settings, by default at http://localhost:8000/config/content_type/notion. Click `Save`.
|
||||
5. Click `Configure` in http://localhost:8000/config to index your Notion workspace(s).
|
||||
|
||||
That's it! You should be ready to start searching and chatting. Make sure you've configured your OpenAI API Key for chat.
|
||||
|
|
@ -1,29 +1,7 @@
|
|||
<img src="/src/khoj/interface/web/assets/icons/khoj-logo-sideways.svg" width="200" alt="Khoj Logo">Obsidian
|
||||
<h1><img src="./assets/khoj-logo-sideways.svg" width="200" alt="Khoj Logo">Obsidian</h1>
|
||||
|
||||
> An AI personal assistant for your Digital Brain in Obsidian
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Features](#Features)
|
||||
- [Demo](#Demo)
|
||||
- [Search Demo](#Search-Demo)
|
||||
- [Interfaces](#Interfaces)
|
||||
- [Search Modal](#Search-Modal)
|
||||
- [Chat Modal](#Chat-Modal)
|
||||
- [Setup](#Setup)
|
||||
- [Setup Backend](#1-Setup-Backend)
|
||||
- [Setup Plugin](#2-Setup-Plugin)
|
||||
- [Use](#Use)
|
||||
- [Search](#search)
|
||||
- [Chat](#chat)
|
||||
- [Find Similar Notes](#find-similar-notes)
|
||||
- [Upgrade](#Upgrade)
|
||||
- [Upgrade Backend](#1-Upgrade-Backend)
|
||||
- [Upgrade Plugin](#2-Upgrade-Plugin)
|
||||
- [Troubleshoot](#Troubleshoot)
|
||||
- [Visualize Codebase](#Visualize-Codebase)
|
||||
- [Implementation](#Implementation)
|
||||
|
||||
## Features
|
||||
- **Search**
|
||||
- **Natural**: Advanced natural language understanding using Transformer based ML Models
|
||||
|
|
@ -34,31 +12,6 @@
|
|||
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
|
||||
- **Assisted creativity**: Smoothly weave across answers retrieval and content generation
|
||||
|
||||
## Demo
|
||||
### Search Demo
|
||||
https://github.com/khoj-ai/khoj/assets/6413477/3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b
|
||||
|
||||
<details><summary>Description</summary>
|
||||
|
||||
1. Install Khoj via `pip` and start Khoj backend
|
||||
```shell
|
||||
python -m pip install khoj-assistant && khoj
|
||||
```
|
||||
2. Install Khoj plugin via Community Plugins settings pane on Obsidian app
|
||||
- Check the new Khoj plugin settings
|
||||
- Wait for Khoj backend to index markdown, PDF files in the current Vault
|
||||
- Open Khoj plugin on Obsidian via Search button on Left Pane
|
||||
- Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/)
|
||||
- Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
|
||||
|
||||
</details>
|
||||
|
||||
## Interfaces
|
||||
### Search Modal
|
||||
|
||||
|
||||
### Chat Modal
|
||||
|
||||
|
||||
## Setup
|
||||
- *Make sure [python](https://realpython.com/installing-python/) and [pip](https://pip.pypa.io/en/stable/installation/) are installed on your machine*
|
||||
|
|
@ -94,16 +47,14 @@ Notes:
|
|||
|
||||
See [Khoj Chat](https://github.com/khoj-ai/khoj/tree/master/#Khoj-Chat) for more details
|
||||
|
||||
|
||||
|
||||
### Search
|
||||
Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or run *Khoj: Search* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
|
||||
|
||||
*Note: Ensure the khoj server is running in the background before searching. Execute `khoj` in your terminal if it is not already running*
|
||||
|
||||
https://user-images.githubusercontent.com/6413477/218801155-cd67e8b4-a770-404a-8179-d6b61caa0f93.mp4
|
||||
[search_demo](https://user-images.githubusercontent.com/6413477/218801155-cd67e8b4-a770-404a-8179-d6b61caa0f93.mp4 ':include :type=mp4')
|
||||
|
||||
<details><summary>Query Filters</summary>
|
||||
#### Query Filters
|
||||
|
||||
Use structured query syntax to filter the natural language search results
|
||||
- **Word Filter**: Get entries that include/exclude a specified term
|
||||
|
|
@ -123,8 +74,6 @@ Use structured query syntax to filter the natural language search results
|
|||
- excluding words *"big"* and *"brother"*
|
||||
- that best match the natural language query *"what is the meaning of life?"*
|
||||
|
||||
</details>
|
||||
|
||||
### Find Similar Notes
|
||||
To see other notes similar to the current one, run *Khoj: Find Similar Notes* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
|
||||
|
||||
|
|
@ -138,6 +87,30 @@ To see other notes similar to the current one, run *Khoj: Find Similar Notes* fr
|
|||
2. Click the *Check for updates* button
|
||||
3. Click the *Update* button next to Khoj, if available
|
||||
|
||||
## Demo
|
||||
### Search Demo
|
||||
[demo](https://github-production-user-asset-6210df.s3.amazonaws.com/6413477/240061700-3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b.mp4 ':include :type=mp4')
|
||||
|
||||
#### Description
|
||||
|
||||
1. Install Khoj via `pip` and start Khoj backend
|
||||
```shell
|
||||
python -m pip install khoj-assistant && khoj
|
||||
```
|
||||
2. Install Khoj plugin via Community Plugins settings pane on Obsidian app
|
||||
- Check the new Khoj plugin settings
|
||||
- Wait for Khoj backend to index markdown, PDF files in the current Vault
|
||||
- Open Khoj plugin on Obsidian via Search button on Left Pane
|
||||
- Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/)
|
||||
- Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
|
||||
|
||||
## Interfaces
|
||||
### Search Modal
|
||||
|
||||
|
||||
### Chat Modal
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
- Open the Khoj plugin settings pane, to configure Khoj
|
||||
- Toggle Enable/Disable Khoj, if setting changes have not applied
|
||||
|
|
@ -148,7 +121,8 @@ To see other notes similar to the current one, run *Khoj: Find Similar Notes* fr
|
|||
So notes across multiple vaults **cannot** be searched at the same time
|
||||
|
||||
## Visualize Codebase
|
||||
<img src="https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_obsidian_codebase_visualization_0.2.1.png" width="700" />
|
||||
|
||||
|
||||
|
||||
## Implementation
|
||||
The plugin implements the following functionality to search your notes with Khoj:
|
||||
79
docs/search.md
Normal file
79
docs/search.md
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
## Khoj Search
|
||||
- **Khoj via Obsidian**
|
||||
- Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or Search for *Khoj: Search* in the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
|
||||
- **Khoj via Emacs**
|
||||
- Run `M-x khoj <user-query>`
|
||||
- **Khoj via Web**
|
||||
- Open <http://localhost:42110/> directly
|
||||
- **Khoj via API**
|
||||
- See the Khoj FastAPI [Swagger Docs](http://localhost:42110/docs), [ReDocs](http://localhost:42110/redocs)
|
||||
|
||||
### Query Filters
|
||||
|
||||
Use structured query syntax to filter the natural language search results
|
||||
- **Word Filter**: Get entries that include/exclude a specified term
|
||||
- Entries that contain term_to_include: `+"term_to_include"`
|
||||
- Entries that contain term_to_exclude: `-"term_to_exclude"`
|
||||
- **Date Filter**: Get entries containing dates in YYYY-MM-DD format from specified date (range)
|
||||
- Entries from April 1st 1984: `dt:"1984-04-01"`
|
||||
- Entries after March 31st 1984: `dt>="1984-04-01"`
|
||||
- Entries before April 2nd 1984 : `dt<="1984-04-01"`
|
||||
- **File Filter**: Get entries from a specified file
|
||||
- Entries from incoming.org file: `file:"incoming.org"`
|
||||
- Combined Example
|
||||
- `what is the meaning of life? file:"1984.org" dt>="1984-01-01" dt<="1985-01-01" -"big" -"brother"`
|
||||
- Adds all filters to the natural language query. It should return entries
|
||||
- from the file *1984.org*
|
||||
- containing dates from the year *1984*
|
||||
- excluding words *"big"* and *"brother"*
|
||||
- that best match the natural language query *"what is the meaning of life?"*
|
||||
|
||||
## Details
|
||||
1. Your query is used to retrieve the most relevant notes, if any, using Khoj search
|
||||
2. These notes, the last few messages and associated metadata is passed to ChatGPT along with your query for a response
|
||||
|
||||
|
||||
## Performance
|
||||
|
||||
### Query performance
|
||||
|
||||
- Semantic search using the bi-encoder is fairly fast at \<50 ms
|
||||
- Reranking using the cross-encoder is slower at \<2s on 15 results. Tweak `top_k` to tradeoff speed for accuracy of results
|
||||
- Filters in query (e.g by file, word or date) usually add \<20ms to query latency
|
||||
|
||||
### Indexing performance
|
||||
|
||||
- Indexing is more strongly impacted by the size of the source data
|
||||
- Indexing 100K+ line corpus of notes takes about 10 minutes
|
||||
- Indexing 4000+ images takes about 15 minutes and more than 8Gb of RAM
|
||||
- Note: *It should only take this long on the first run* as the index is incrementally updated
|
||||
|
||||
### Miscellaneous
|
||||
|
||||
- Testing done on a Mac M1 and a \>100K line corpus of notes
|
||||
- Search, indexing on a GPU has not been tested yet
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Use OpenAI Models for Search
|
||||
#### Setup
|
||||
1. Set `encoder-type`, `encoder` and `model-directory` under `asymmetric` and/or `symmetric` `search-type` in your `khoj.yml`[^1]:
|
||||
```diff
|
||||
asymmetric:
|
||||
- encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1"
|
||||
+ encoder: text-embedding-ada-002
|
||||
+ encoder-type: khoj.utils.models.OpenAI
|
||||
cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
|
||||
- encoder-type: sentence_transformers.SentenceTransformer
|
||||
- model_directory: "~/.khoj/search/asymmetric/"
|
||||
+ model-directory: null
|
||||
```
|
||||
2. [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
|
||||
3. Restart Khoj server to generate embeddings. It will take longer than with offline models.
|
||||
|
||||
#### Warnings
|
||||
This configuration *uses an online model*
|
||||
- It will **send all notes to OpenAI** to generate embeddings
|
||||
- **All queries will be sent to OpenAI** when you search with Khoj
|
||||
- You will be **charged by OpenAI** based on the total tokens processed
|
||||
- It *requires an active internet connection* to search and index
|
||||
98
docs/setup.md
Normal file
98
docs/setup.md
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
## Setup
|
||||
These are the general setup instructions for Khoj.
|
||||
|
||||
- Make sure [python](https://realpython.com/installing-python/) and [pip](https://pip.pypa.io/en/stable/installation/) are installed on your machine
|
||||
- Check the [Khoj.el Readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Setup) to setup Khoj with Emacs<br />
|
||||
Its simpler as it can skip the server *install*, *run* and *configure* step below.
|
||||
- Check the [Khoj Obsidian Readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#Setup) to setup Khoj with Obsidian<br />
|
||||
Its simpler as it can skip the *configure* step below.
|
||||
|
||||
### 1. Install
|
||||
Run the following command in your terminal to install the Khoj backend.
|
||||
|
||||
- On Linux/MacOS
|
||||
```shell
|
||||
python -m pip install khoj-assistant
|
||||
```
|
||||
|
||||
- On Windows
|
||||
```shell
|
||||
py -m pip install khoj-assistant
|
||||
```
|
||||
For more detailed Windows installation and troubleshooting, see [Windows Install](./windows_install.md).
|
||||
|
||||
|
||||
### 2. Run
|
||||
|
||||
Run the following commmand from your terminal to start the Khoj backend and open Khoj in your browser.
|
||||
|
||||
```shell
|
||||
khoj --gui
|
||||
```
|
||||
|
||||
Note: To start Khoj automatically in the background use [Task scheduler](https://www.windowscentral.com/how-create-automated-task-using-task-scheduler-windows-10) on Windows or [Cron](https://en.wikipedia.org/wiki/Cron) on Mac, Linux (e.g with `@reboot khoj`)
|
||||
|
||||
### 3. Configure
|
||||
1. Set `File`, `Folder` and hit `Save` in each Plugins you want to enable for Search on the Khoj config page
|
||||
2. Add your OpenAI API key to Chat Feature settings if you want to use Chat
|
||||
3. Click `Configure` and wait. The app will download ML models and index the content for search and (optionally) chat
|
||||
|
||||
### 4. Install Interface Plugins (Optional)
|
||||
Khoj exposes a web interface to search, chat and configure by default.<br />
|
||||
The optional steps below allow using Khoj from within an existing application like Obsidian or Emacs.
|
||||
|
||||
- **Khoj Obsidian**:<br />
|
||||
[Install](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#2-Setup-Plugin) the Khoj Obsidian plugin
|
||||
|
||||
- **Khoj Emacs**:<br />
|
||||
[Install](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#2-Install-Khojel) khoj.el
|
||||
|
||||
|
||||
## Upgrade
|
||||
### Upgrade Khoj Server
|
||||
```shell
|
||||
pip install --upgrade khoj-assistant
|
||||
```
|
||||
|
||||
*Note: To upgrade to the latest pre-release version of the khoj server run below command*
|
||||
```shell
|
||||
# Maps to the latest commit on the master branch
|
||||
pip install --upgrade --pre khoj-assistant
|
||||
```
|
||||
|
||||
### Upgrade Khoj on Emacs
|
||||
- Use your Emacs Package Manager to Upgrade
|
||||
- See [khoj.el readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade) for details
|
||||
|
||||
### Upgrade Khoj on Obsidian
|
||||
- Upgrade via the Community plugins tab on the settings pane in the Obsidian app
|
||||
- See the [khoj plugin readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#2-Setup-Plugin) for details
|
||||
|
||||
## Uninstall
|
||||
1. (Optional) Hit `Ctrl-C` in the terminal running the khoj server to stop it
|
||||
2. Delete the khoj directory in your home folder (i.e `~/.khoj` on Linux, Mac or `C:\Users\<your-username>\.khoj` on Windows)
|
||||
3. Uninstall the khoj server with `pip uninstall khoj-assistant`
|
||||
4. (Optional) Uninstall khoj.el or the khoj obsidian plugin in the standard way on Emacs, Obsidian
|
||||
|
||||
## Troubleshoot
|
||||
|
||||
#### Install fails while building Tokenizer dependency
|
||||
- **Details**: `pip install khoj-assistant` fails while building the `tokenizers` dependency. Complains about Rust.
|
||||
- **Fix**: Install Rust to build the tokenizers package. For example on Mac run:
|
||||
```shell
|
||||
brew install rustup
|
||||
rustup-init
|
||||
source ~/.cargo/env
|
||||
```
|
||||
- **Refer**: [Issue with Fix](https://github.com/khoj-ai/khoj/issues/82#issuecomment-1241890946) for more details
|
||||
|
||||
#### Search starts giving wonky results
|
||||
- **Fix**: Open [/api/update?force=true](http://localhost:42110/api/update?force=true)[^2] in browser to regenerate index from scratch
|
||||
- **Note**: *This is a fix for when you percieve the search results have degraded. Not if you think they've always given wonky results*
|
||||
|
||||
#### Khoj in Docker errors out with \"Killed\" in error message
|
||||
- **Fix**: Increase RAM available to Docker Containers in Docker Settings
|
||||
- **Refer**: [StackOverflow Solution](https://stackoverflow.com/a/50770267), [Configure Resources on Docker for Mac](https://docs.docker.com/desktop/mac/#resources)
|
||||
|
||||
#### Khoj errors out complaining about Tensors mismatch or null
|
||||
- **Mitigation**: Disable `image` search using the desktop GUI
|
||||
23
docs/windows_install.md
Normal file
23
docs/windows_install.md
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
# Windows Installation
|
||||
|
||||
These steps can be used to setup Khoj on a clean, new Windows 11 machine. It has been tested on a Windows VM
|
||||
|
||||
## Prerequisites
|
||||
1. Ensure you have Visual Studio C++ Build tools installed. You can download it [from Microsoft here](https://visualstudio.microsoft.com/visual-cpp-build-tools/). At the minimum, you should have the following configuration:
|
||||
<img width="1152" alt="Screenshot 2023-07-12 at 3 56 25 PM" src="https://github.com/khoj-ai/khoj/assets/65192171/b506a858-2f5e-4c85-946b-5422d83f112a">
|
||||
3. Ensure you have Python installed. You can check by running `python --version`. If you don't, install the latest version [from here](https://www.python.org/downloads/).
|
||||
- Ensure you have pip installed: `py -m ensurepip --upgrade`.
|
||||
|
||||
## Quick start
|
||||
1. Open a PowerShell terminal.
|
||||
7. Run `pip install khoj-assistant`
|
||||
8. Start Khoj with `khoj`
|
||||
|
||||
## Installation in a Virtual Environment
|
||||
Use this if you want to install with a virtual environment. This will make it much easier to manage your dependencies. You can read more about [virtual environments](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/) here.
|
||||
|
||||
1. Open a PowerShell terminal with the `Run as Administrator` privileges.
|
||||
5. Create a virtual environment: `mkdir khoj && cd khoj && py -m venv .venv`
|
||||
6. Activate the virtual environment: `.\.venv\Scripts\activate`. If you get a permissions error, then run `Set-ExecutionPolicy -ExecutionPolicy RemoteSigned`.
|
||||
7. Run `pip install khoj-assistant`
|
||||
8. Start Khoj with `khoj`
|
||||
|
|
@ -1,167 +0,0 @@
|
|||
[[https://github.com/khoj-ai/khoj/edit/master/src/interface/emacs/README.org][file:/src/khoj/interface/web/assets/icons/khoj-logo-sideways-200.png]] Emacs
|
||||
|
||||
|
||||
[[https://stable.melpa.org/#/khoj][file:https://stable.melpa.org/packages/khoj-badge.svg]] [[https://melpa.org/#/khoj][file:https://melpa.org/packages/khoj-badge.svg]] [[https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml][https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml/badge.svg?]] [[https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml][https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml/badge.svg?]]
|
||||
|
||||
/An AI personal assistant for your digital brain/
|
||||
|
||||
** Table of Contents
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#features][Features]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Interface][Interface]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Setup][Setup]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Direct-Install][Direct Install]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Minimal-Install][Minimal Install]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Standard-Install][Standard Install]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#With-Straight.el][With Straight.el]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Use][Use]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Search][Search]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Chat][Chat]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Find-similar-entries][Find Similar Entries]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Advanced-usage][Advanced Usage]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Khoj-menu][Khoj Menu]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade][Upgrade]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade-Khoj-Backend][Upgrade Backend]]
|
||||
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade-Khojel][Upgrade Khoj.el]]
|
||||
|
||||
** Features
|
||||
- *Search*
|
||||
- *Natural*: Advanced natural language understanding using Transformer based ML Models
|
||||
- *Local*: Your personal data stays local. All search, indexing is done on your machine*
|
||||
- *Incremental*: Incremental search for a fast, search-as-you-type experience
|
||||
- *Chat*
|
||||
- *Faster answers*: Find answers faster than search
|
||||
- *Iterative discovery*: Iteratively explore and (re-)discover your notes
|
||||
- *Assisted creativity*: Smoothly weave across answer retrieval and content generation
|
||||
|
||||
** Interface
|
||||
*** Search UI
|
||||
[[/docs/khoj_on_emacs.png]]
|
||||
|
||||
*** Chat UI
|
||||
[[/docs/khoj_chat_on_emacs_0.5.0.png]]
|
||||
|
||||
** Setup
|
||||
- /Make sure [[https://realpython.com/installing-python/][python]] and [[https://pip.pypa.io/en/stable/installation/][pip]] are installed on your machine/
|
||||
|
||||
- /khoj.el attempts to automatically install, start and configure the khoj server./
|
||||
If this fails, follow [[https://github.com/khoj-ai/khoj/tree/master/#Setup][these instructions]] to manually setup the khoj server.
|
||||
|
||||
*** Direct Install
|
||||
#+begin_src elisp
|
||||
M-x package-install khoj
|
||||
#+end_src
|
||||
|
||||
*** Minimal Install
|
||||
Add below snippet to your Emacs config file.
|
||||
Indexes your org-agenda files, by default.
|
||||
|
||||
#+begin_src elisp
|
||||
;; Install Khoj Package from MELPA Stable
|
||||
(use-package khoj
|
||||
:ensure t
|
||||
:pin melpa-stable
|
||||
:bind ("C-c s" . 'khoj)
|
||||
#+end_src
|
||||
|
||||
- Note: Install ~khoj.el~ from MELPA (instead of MELPA Stable) if you installed the pre-release version of khoj
|
||||
- That is, use ~:pin melpa~ to install khoj.el in above snippet if khoj server was installed with ~--pre~ flag, i.e ~pip install --pre khoj-assistant~
|
||||
- Else use ~:pin melpa-stable~ to install khoj.el in above snippet if khoj was installed with ~pip install khoj-assistant~
|
||||
- This ensures both khoj.el and khoj app are from the same version (git tagged or latest)
|
||||
|
||||
*** Standard Install
|
||||
Add below snippet to your Emacs config file.
|
||||
Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat
|
||||
|
||||
#+begin_src elisp
|
||||
;; Install Khoj Package from MELPA Stable
|
||||
(use-package khoj
|
||||
:ensure t
|
||||
:pin melpa-stable
|
||||
:bind ("C-c s" . 'khoj)
|
||||
:config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
|
||||
khoj-org-files '("~/docs/todo.org" "~/docs/work.org")
|
||||
khoj-openai-api-key "YOUR_OPENAI_API_KEY")) ; required to enable chat
|
||||
#+end_src
|
||||
|
||||
*** With [[https://github.com/raxod502/straight.el][Straight.el]]
|
||||
Add below snippet to your Emacs config file.
|
||||
Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat
|
||||
|
||||
#+begin_src elisp
|
||||
;; Install Khoj Package using Straight.el
|
||||
(use-package khoj
|
||||
:after org
|
||||
:straight (khoj :type git :host github :repo "khoj-ai/khoj" :files (:defaults "src/interface/emacs/khoj.el"))
|
||||
:bind ("C-c s" . 'khoj)
|
||||
:config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
|
||||
khoj-org-files '("~/docs/todo.org" "~/docs/work.org")
|
||||
khoj-openai-api-key "YOUR_OPENAI_API_KEY" ; required to enable chat)
|
||||
#+end_src
|
||||
|
||||
** Use
|
||||
*** Search
|
||||
1. Hit ~C-c s s~ (or ~M-x khoj RET s~) to open khoj search
|
||||
|
||||
2. Enter your query in natural language
|
||||
|
||||
e.g "What is the meaning of life?", "My life goals for 2023"
|
||||
|
||||
*** Chat
|
||||
1. Hit ~C-c s c~ (or ~M-x khoj RET c~) to open khoj chat
|
||||
|
||||
2. Ask questions in a natural, conversational style
|
||||
|
||||
E.g "When did I file my taxes last year?"
|
||||
|
||||
See [[https://github.com/khoj-ai/khoj/tree/master/#Khoj-Chat][Khoj Chat]] for more details
|
||||
|
||||
*** Find Similar Entries
|
||||
This feature finds entries similar to the one you are currently on.
|
||||
1. Move cursor to the org-mode entry, markdown section or text paragraph you want to find similar entries for
|
||||
2. Hit ~C-c s f~ (or ~M-x khoj RET f~) to find similar entries
|
||||
|
||||
*** Advanced Usage
|
||||
- Add [[https://github.com/khoj-ai/khoj/#query-filters][query filters]] during search to narrow down results further
|
||||
|
||||
e.g `What is the meaning of life? -"god" +"none" dt>"last week"`
|
||||
|
||||
- Use ~C-c C-o 2~ to open the current result at cursor in its source org file
|
||||
|
||||
- This calls ~M-x org-open-at-point~ on the current entry and opens the second link in the entry.
|
||||
- The second link is the entries [[https://orgmode.org/manual/Handling-Links.html#FOOT28][org-id]], if set, or the heading text.
|
||||
The first link is the line number of the entry in the source file. This link is less robust to file changes.
|
||||
- Note: If you have [[https://orgmode.org/manual/Speed-Keys.html][speed keys]] enabled, ~o 2~ will also work
|
||||
|
||||
*** Khoj Menu
|
||||
[[/docs/khoj_emacs_menu.png]]
|
||||
Hit ~C-c s~ (or ~M-x khoj~) to open the khoj menu above. Then:
|
||||
- Hit ~t~ until you preferred content type is selected in the khoj menu
|
||||
|
||||
~Content Type~ specifies the content to perform ~Search~, ~Update~ or ~Find Similar~ actions on
|
||||
- Hit ~n~ twice and then enter number of results you want to see
|
||||
|
||||
~Results Count~ is used by the ~Search~ and ~Find Similar~ actions
|
||||
- Hit ~-f u~ to ~force~ update the khoj content index
|
||||
|
||||
The ~Force Update~ switch is only used by the ~Update~ action
|
||||
|
||||
** Upgrade
|
||||
*** Upgrade Khoj Backend
|
||||
#+begin_src shell
|
||||
pip install --upgrade khoj-assistant
|
||||
#+end_src
|
||||
|
||||
*** Upgrade Khoj.el
|
||||
Use your Emacs package manager to upgrade ~khoj.el~
|
||||
|
||||
- For ~khoj.el~ from MELPA
|
||||
- Method 1
|
||||
- Run ~M-x package-list-packages~ to list all packages
|
||||
- Press ~U~ on ~khoj~ to mark it for upgrade
|
||||
- Press ~x~ to execute the marked actions
|
||||
- Method 2
|
||||
- Run ~M-x package-refresh-content~
|
||||
- Run ~M-x package-reinstall khoj~
|
||||
|
||||
- For ~khoj.el~ from Straight
|
||||
- Run ~M-x straight-pull-package khoj~
|
||||
Loading…
Reference in a new issue