📝 Add a documentation base for Khoj! (#333)

* Add docs for more organized, accessible information detailing Khoj setup
* Delete duplicated files
* Add a coverpage without enabling it. Add logo and theme
* Remove obsidian README.md
* Add plausible script to index.html via docsify

docs/README.md

@@ -0,0 +1,59 @@
+<h1><img src="./assets/khoj-logo-sideways.svg" width="200" alt="Khoj Logo"></h1>
+
+[![test](https://github.com/khoj-ai/khoj/actions/workflows/test.yml/badge.svg)](https://github.com/khoj-ai/khoj/actions/workflows/test.yml)
+[![dockerize](https://github.com/khoj-ai/khoj/actions/workflows/dockerize.yml/badge.svg)](https://github.com/khoj-ai/khoj/pkgs/container/khoj)
+[![pypi](https://github.com/khoj-ai/khoj/actions/workflows/pypi.yml/badge.svg)](https://pypi.org/project/khoj-assistant/)
+
+# Khoj
+*An AI personal assistant for your digital brain*
+
+Welcome to the Docs! This is the best place to get started with Khoj. Check out our [Github](https://github.com/khoj-ai/khoj) to dive straight into the code.
+
+Khoj gives you lightning fast, offline search on your personal machine and gives you the power to talk to your notes.
+
+**Supported Plugins**
+
+[![Khoj on Obsidian](https://img.shields.io/badge/Obsidian-%23483699.svg?style=for-the-badge&logo=obsidian&logoColor=white)](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#readme)
+[![Khoj on Emacs](https://img.shields.io/badge/Emacs-%237F5AB6.svg?&style=for-the-badge&logo=gnu-emacs&logoColor=white)](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#readme)
+
+## Features
+- **Search**
+  - **Local**: Your personal data stays local. All search and indexing is done on your machine. *Unlike chat which requires access to GPT.*
+  - **Incremental**: Incremental search for a fast, search-as-you-type experience
+- **Chat**
+  - **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers.
+  - **Iterative discovery**: Iteratively explore and (re-)discover your notes
+  - **Assisted creativity**: Smoothly weave across answers retrieval and content generation
+- **General**
+  - **Natural**: Advanced natural language understanding using Transformer based ML Models
+  - **Pluggable**: Modular architecture makes it easy to plug in new data sources, frontends and ML models
+  - **Multiple Sources**: Index your Org-mode and Markdown notes, PDF files, Github repositories, and Photos
+  - **Multiple Interfaces**: Interact from your [Web Browser](./src/khoj/interface/web/index.html), [Emacs](./src/interface/emacs/khoj.el) or [Obsidian](./src/interface/obsidian/)
+
+## Install
+[Click here](./setup.md) for full setup instructions.
+
+## Supported data sources
+- markdown*
+- org-mode*
+- pdf*
+- images*
+- github
+- notion
+
+\* These data sources are offline only.
+
+If you're using Github or Notion, you can get on a waitlist for [Khoj Cloud](https://khoj.dev).
+
+## Credits
+
+- [Multi-QA MiniLM Model](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1), [All MiniLM Model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) for Text Search. See [SBert Documentation](https://www.sbert.net/examples/applications/retrieve_rerank/README.html)
+- [OpenAI CLIP Model](https://github.com/openai/CLIP) for Image Search. See [SBert Documentation](https://www.sbert.net/examples/applications/image-search/README.html)
+- Charles Cave for [OrgNode Parser](http://members.optusnet.com.au/~charles57/GTD/orgnode.html)
+- [Org.js](https://mooz.github.io/org-js/) to render Org-mode results on the Web interface
+- [Markdown-it](https://github.com/markdown-it/markdown-it) to render Markdown results on the Web interface
+
+
+[^1]: Default Khoj config file @ `~/.khoj/khoj.yml`
+
+[^2]: Default Khoj url @ http://localhost:42110

docs/_coverpage.md

@@ -0,0 +1,14 @@
+<!-- _coverpage.md -->
+
+![logo](./assets/khoj-logo-sideways-200.png)
+
+> An open source, AI personal assistant for your notes
+
+- Lightning fast search
+- Multi-turn chat
+- Keeps you in control of your data
+
+[GitHub](https://github.com/khoj-ai/khoj)
+[Get Started](#khoj)
+
+![color](#f9f5de)

docs/_sidebar.md

@@ -0,0 +1,18 @@
+- Getting Started
+    - [Overview](README.md)
+    - [Install](setup.md)
+        - [Windows Installation](windows_install.md)
+- Learn More
+    - [Features](features.md)
+        - [Chat](chat.md)
+        - [Search](search.md)
+    - [Demos](demos.md)
+    - Interfaces
+        - [Obsidian](obsidian.md)
+        - [Emacs](emacs.md)
+    - Data Sources
+        - [Github](github_integration.md)
+        - [Notion](notion_integration.md)
+    - [Advanced](advanced.md)
+- Contributing
+    - [Development](development.md)

docs/advanced.md

@@ -0,0 +1,54 @@
+
+## Advanced Usage
+### Access Khoj on Mobile
+1. [Setup Khoj](#Setup) on your personal server. This can be any always-on machine, i.e an old computer, RaspberryPi(?) etc
+2. [Install](https://tailscale.com/kb/installation/) [Tailscale](tailscale.com/) on your personal server and phone
+3. Open the Khoj web interface of the server from your phone browser.<br /> It should be `http://tailscale-ip-of-server:42110` or `http://name-of-server:42110` if you've setup [MagicDNS](https://tailscale.com/kb/1081/magicdns/)
+4. Click the [Add to Homescreen](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Add_to_home_screen) button
+5. Enjoy exploring your notes, documents and images from your phone!
+
+![](https://github.com/khoj-ai/khoj/blob/master/docs/khoj_pwa_android.png?)
+
+### Search across Different Languages
+  To search for notes in multiple, different languages, you can use a [multi-lingual model](https://www.sbert.net/docs/pretrained_models.html#multi-lingual-models).<br />
+  For example, the [paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) supports [50+ languages](https://www.sbert.net/docs/pretrained_models.html#:~:text=we%20used%20the%20following%2050%2B%20languages), has good search quality and speed. To use it:
+  1. Manually update `search-type > asymmetric > encoder` to `paraphrase-multilingual-MiniLM-L12-v2` in your `~/.khoj/khoj.yml` file for now. See diff of `khoj.yml` below for illustration:
+  ```diff
+   asymmetric:
+- encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1"
++ encoder: "paraphrase-multilingual-MiniLM-L12-v2"
+     cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
+     model_directory: "~/.khoj/search/asymmetric/"
+  ```
+
+  2. Regenerate your content index. For example, by opening [\<khoj-url\>/api/update?t=force](http://localhost:42110/api/update?t=force)
+
+### Bootstrap Khoj Search for Offline Usage later
+
+  You can bootstrap Khoj pre-emptively to run on machines that do not have internet access. An example use-case would be to run Khoj on an air-gapped machine.
+  Note: *Only search can currently run in fully offline mode, not chat.*
+
+  - With Internet
+    1. Manually download the [asymmetric text](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1), [symmetric text](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)and [image search](https://huggingface.co/sentence-transformers/clip-ViT-B-32) models from HuggingFace
+    2. Pip install khoj (and dependencies) in an associated virtualenv. E.g `python -m venv .venv && source .venv/bin/activate && pip install khoj-assistant`
+  - Without Internet
+    1. Copy each of the search models into their respective folders, `asymmetric`, `symmetric` and `image` under the `~/.khoj/search/` directory on the air-gapped machine
+    2. Copy the khoj virtual environment directory onto the air-gapped machine, activate the environment and start and khoj as normal. E.g `source .venv/bin/activate && khoj`
+
+
+## Miscellaneous
+### Set your OpenAI API key in Khoj
+If you want, Khoj can be configured to use OpenAI for search and chat.<br />
+Add your OpenAI API to Khoj by using either of the two options below:
+ - Open your [Khoj settings](http://localhost:42110/config/processor/conversation), add your OpenAI API key, and click *Save*. Then go to your [Khoj settings](http://localhost:42110/config) and click `Configure`. This will refresh Khoj with your OpenAI API key.
+ - Set `openai-api-key` field under `processor.conversation` section in your `khoj.yml`[^1] to your [OpenAI API key](https://beta.openai.com/account/api-keys) and restart khoj:
+    ```diff
+    processor:
+      conversation:
+    -    openai-api-key: # "YOUR_OPENAI_API_KEY"
+    +    openai-api-key: sk-aaaaaaaaaaaaaaaaaaaaaaaahhhhhhhhhhhhhhhhhhhhhhhh
+        model: "text-davinci-003"
+        conversation-logfile: "~/.khoj/processor/conversation/conversation_logs.json"
+    ```
+
+!> **Warning**: This will enable Khoj to send your query and note(s) to OpenAI for processing

docs/chat.md

@@ -0,0 +1,21 @@
+### Khoj Chat
+#### Overview
+- Creates a personal assistant for you to inquire and engage with your notes
+- Uses [ChatGPT](https://openai.com/blog/chatgpt) and [Khoj search](#khoj-search). [Offline chat](https://github.com/khoj-ai/khoj/issues/201) is coming soon.
+- Supports multi-turn conversations with the relevant notes for context
+- Shows reference notes used to generate a response
+- **Note**: *Your query and top notes from khoj search will be sent to OpenAI for processing*
+
+#### Setup
+- [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
+
+#### Use
+1. Open [/chat](http://localhost:42110/chat)[^2]
+2. Type your queries and see response by Khoj from your notes
+
+#### Demo
+![](./assets/khoj_chat_web_interface.png)
+
+### Details
+1. Your query is used to retrieve the most relevant notes, if any, using Khoj search
+2. These notes, the last few messages and associated metadata is passed to ChatGPT along with your query for a response

docs/demos.md

@@ -0,0 +1,40 @@
+## Demos
+
+### Interfaces
+
+![](./assets/interfaces.png)
+
+### Khoj in Obsidian
+[KhojObsidian](https://github-production-user-asset-6210df.s3.amazonaws.com/6413477/240061700-3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b.mp4 ':include :type=mp4')
+
+#### Installation
+
+1. Install Khoj via `pip` and start Khoj backend in a terminal (Run `khoj`)
+    ```bash
+    python -m pip install khoj-assistant
+    khoj
+    ```
+2. Install Khoj plugin via Community Plugins settings pane on Obsidian app
+    - Check the new Khoj plugin settings
+    - Let Khoj backend index the markdown, pdf, Github markdown files in the current Vault
+    - Open Khoj plugin on Obsidian via Search button on Left Pane
+    - Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/)
+    - Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
+
+### Khoj in Emacs, Browser
+[KhojEmacs](https://user-images.githubusercontent.com/6413477/184735169-92c78bf1-d827-4663-9087-a1ea194b8f4b.mp4 ':include :type=mp4')
+
+#### Installation
+
+- Install Khoj via pip
+- Start Khoj app
+- Add this readme and [khoj.el readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs) as org-mode for Khoj to index
+- Search \"*Setup editor*\" on the Web and Emacs. Re-rank the results for better accuracy
+- Top result is what we are looking for, the [section to Install Khoj.el on Emacs](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#2-Install-Khojel)
+
+#### Analysis
+
+- The results do not have any words used in the query
+  - *Based on the top result it seems the re-ranking model understands that Emacs is an editor?*
+- The results incrementally update as the query is entered
+- The results are re-ranked, for better accuracy, once user hits enter

docs/development.md

@@ -0,0 +1,114 @@
+# Development
+
+## Setup
+### Using Pip
+#### 1. Install
+
+```shell
+# Get Khoj Code
+git clone https://github.com/khoj-ai/khoj && cd khoj
+
+# Create, Activate Virtual Environment
+python3 -m venv .venv && source .venv/bin/activate
+
+# Install Khoj for Development
+pip install -e .[dev]
+```
+
+#### 2. Run
+1. Start Khoj
+   ```shell
+   khoj -vv
+   ```
+2. Configure Khoj
+   - **Via the Settings UI**: Add files, directories to index the [Khoj settings](http://localhost:42110/config) UI once Khoj has started up. Once you've saved all your settings, click `Configure`.
+   - **Manually**:
+     - Copy the `config/khoj_sample.yml` to `~/.khoj/khoj.yml`
+     - Set `input-files` or `input-filter` in each relevant `content-type` section of `~/.khoj/khoj.yml`
+       - Set `input-directories` field in `image` `content-type` section
+     - Delete `content-type` and `processor` sub-section(s) irrelevant for your use-case
+     - Restart khoj
+
+  Note: Wait after configuration for khoj to Load ML model, generate embeddings and expose API to query notes, images, documents etc specified in config YAML
+
+### Using Docker
+#### 1. Clone
+
+```shell
+git clone https://github.com/khoj-ai/khoj && cd khoj
+```
+
+#### 2. Configure
+
+- **Required**: Update [docker-compose.yml](./docker-compose.yml) to mount your images, (org-mode or markdown) notes, PDFs and Github repositories
+- **Optional**: Edit application configuration in [khoj_docker.yml](./config/khoj_docker.yml)
+
+#### 3. Run
+
+```shell
+docker-compose up -d
+```
+
+*Note: The first run will take time. Let it run, it\'s mostly not hung, just generating embeddings*
+
+#### 4. Upgrade
+
+```shell
+docker-compose build --pull
+```
+
+## Validate
+### Before Making Changes
+1. Install Git Hooks for Validation
+   ```shell
+   pre-commit install -t pre-push -t pre-commit
+   ```
+   - This ensures standard code formatting fixes and other checks run automatically on every commit and push
+   - Note 1: If [pre-commit](https://pre-commit.com/#intro) didn't already get installed, [install it](https://pre-commit.com/#install) via `pip install pre-commit`
+   - Note 2: To run the pre-commit changes manually, use `pre-commit run --hook-stage manual --all` before creating PR
+
+### Before Creating PR
+
+1. Run Tests. If you get an error complaining about a missing `fast_tokenizer_file`, follow the solution [in this Github issue](https://github.com/UKPLab/sentence-transformers/issues/1659).
+   ```shell
+   pytest
+   ```
+
+2. Run MyPy to check types
+   ```shell
+   mypy --config-file pyproject.toml
+   ```
+
+### After Creating PR
+- Automated [validation workflows](.github/workflows) run for every PR.
+
+  Ensure any issues seen by them our fixed
+
+- Test the python packge created for a PR
+  1. Download and extract the zipped `.whl` artifact generated from the pypi workflow run for the PR.
+  2. Install (in your virtualenv) with `pip install /path/to/download*.whl>`
+  3. Start and use the application to see if it works fine
+
+## Create Khoj Release
+Follow the steps below to [release](https://github.com/debanjum/khoj/releases/) Khoj. This will create a stable release of Khoj on [Pypi](https://pypi.org/project/khoj-assistant/), [Melpa](https://stable.melpa.org/#%252Fkhoj) and [Obsidian](https://obsidian.md/plugins?id%253Dkhoj). It will also create desktop apps of Khoj and attach them to the latest release.
+
+1. Create and tag release commit by running the bump_version script. The release commit sets version number in required metadata files.
+  ```shell
+  ./scripts/bump_version.sh -c "<release_version>"
+  ```
+2. Push commit and then the tag to trigger the release workflow to create Release with auto generated release notes.
+  ```shell
+  git push origin master  # push release commit to khoj repository
+  git push origin <release_version>  # push release tag to khoj repository
+  ```
+3. [Optional] Update the Release Notes to highlight new features, fixes and updates
+
+## Architecture
+
+![](./assets/khoj_architecture.png)
+
+## Visualize Codebase
+
+*[Interactive Visualization](https://mango-dune-07a8b7110.1.azurestaticapps.net/?repo=debanjum%2Fkhoj)*
+
+![](./assets/khoj_codebase_visualization_0.2.1.png)

docs/emacs.md

@@ -0,0 +1,152 @@
+<h1><img src="./assets/khoj-logo-sideways.svg" width="200" alt="Khoj Logo">Emacs</h1>
+
+An AI personal assistance for your digital brain
+
+<img src="https://stable.melpa.org/packages/khoj-badge.svg" width="150" alt="Melpa Stable Badge">
+<img src="https://melpa.org/packages/khoj-badge.svg" width="150" alt="Melpa Badge">
+
+<img src="https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml/badge.svg" width="150" alt="Build Badge">
+<img src="https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml/badge.svg" width="150" alt="Test Badge">
+
+
+## Features
+- **Search**
+  - **Natural**: Advanced natural language understanding using Transformer based ML Models
+  - **Local**: Your personal data stays local. All search, indexing is done on your machine*
+  - **Incremental**: Incremental search for a fast, search-as-you-type experience
+- **Chat**
+  - **Faster answers**: Find answers faster than search
+  - **Iterative discovery**: Iteratively explore and (re-)discover your notes
+  - **Assisted creativity**: Smoothly weave across answer retrieval and content generation
+
+## Interface
+### Search UI
+
+![khoj_emacs](./assets/khoj_on_emacs.png)
+
+### Chat UI
+![khoj_chat_emacs](./assets/khoj_chat_on_emacs_0.5.0.png)
+
+## Setup
+- /Make sure [[https://realpython.com/installing-python/][python]] and [[https://pip.pypa.io/en/stable/installation/][pip]] are installed on your machine/
+
+- /khoj.el attempts to automatically install, start and configure the khoj server./
+  If this fails, follow [[https://github.com/khoj-ai/khoj/tree/master/#Setup][these instructions]] to manually setup the khoj server.
+
+### Direct Install
+```elisp
+M-x package-install khoj
+```
+
+### Minimal Install
+Add below snippet to your Emacs config file.
+Indexes your org-agenda files, by default.
+
+```elisp
+  ;; Install Khoj Package from MELPA Stable
+  (use-package khoj
+    :ensure t
+    :pin melpa-stable
+    :bind ("C-c s" . 'khoj)
+```
+
+- Note: Install `khoj.el` from MELPA (instead of MELPA Stable) if you installed the pre-release version of khoj
+  - That is, use `:pin melpa` to install khoj.el in above snippet if khoj server was installed with `--pre` flag, i.e `pip install --pre khoj-assistant`
+  - Else use `:pin melpa-stable` to install khoj.el in above snippet if khoj was installed with `pip install khoj-assistant`
+  - This ensures both khoj.el and khoj app are from the same version (git tagged or latest)
+
+### Standard Install
+  Add below snippet to your Emacs config file.
+  Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat
+
+```elisp
+;; Install Khoj Package from MELPA Stable
+(use-package khoj
+  :ensure t
+  :pin melpa-stable
+  :bind ("C-c s" . 'khoj)
+  :config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
+                khoj-org-files '("~/docs/todo.org" "~/docs/work.org")
+                khoj-openai-api-key "YOUR_OPENAI_API_KEY")) ; required to enable chat
+```
+
+### With [https://github.com/raxod502/straight.el](Straight.el)
+Add below snippet to your Emacs config file.
+Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat
+
+```elisp
+  ;; Install Khoj Package using Straight.el
+  (use-package khoj
+    :after org
+    :straight (khoj :type git :host github :repo "khoj-ai/khoj" :files (:defaults "src/interface/emacs/khoj.el"))
+    :bind ("C-c s" . 'khoj)
+    :config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
+                  khoj-org-files '("~/docs/todo.org" "~/docs/work.org")
+                  khoj-openai-api-key "YOUR_OPENAI_API_KEY" ; required to enable chat)
+  ```
+
+## Use
+### Search
+1. Hit  `C-c s s` (or `M-x khoj RET s`) to open khoj search
+
+2. Enter your query in natural language
+
+  e.g "What is the meaning of life?", "My life goals for 2023"
+
+### Chat
+1. Hit `C-c s c` (or `M-x khoj RET c`) to open khoj chat
+
+2. Ask questions in a natural, conversational style
+
+  E.g "When did I file my taxes last year?"
+
+  See [Khoj Chat](./README.md#khoj-chat) for more details
+
+### Find Similar Entries
+This feature finds entries similar to the one you are currently on.
+1. Move cursor to the org-mode entry, markdown section or text paragraph you want to find similar entries for
+2. Hit `C-c s f` (or `M-x khoj RET f`) to find similar entries
+
+### Advanced Usage
+- Add [[https://github.com/khoj-ai/khoj/#query-filters][query filters]] during search to narrow down results further
+
+  e.g `What is the meaning of life? -"god" +"none" dt>"last week"`
+
+- Use `C-c C-o 2` to open the current result at cursor in its source org file
+  - This calls `M-x org-open-at-point` on the current entry and opens the second link in the entry.
+  - The second link is the entries [org-id](https://orgmode.org/manual/Handling-Links.html#FOOT28), if set, or the heading text.
+    The first link is the line number of the entry in the source file. This link is less robust to file changes.
+  - Note: If you have [speed keys](https://orgmode.org/manual/Speed-Keys.html) enabled, `o 2` will also work
+
+### Khoj Menu
+![](./assets/khoj_emacs_menu.png)
+Hit `C-c s` (or `M-x khoj`) to open the khoj menu above. Then:
+- Hit `t` until you preferred content type is selected in the khoj menu
+
+  `Content Type` specifies the content to perform `Search`, `Update` or `Find Similar` actions on
+- Hit `n` twice and then enter number of results you want to see
+
+  `Results Count` is used by the `Search` and `Find Similar` actions
+- Hit `-f u` to `force` update the khoj content index
+
+  The `Force Update` switch is only used by the `Update` action
+
+## Upgrade
+### Upgrade Khoj Backend
+```bash
+pip install --upgrade khoj-assistant
+```
+### Upgrade Khoj.el
+Use your Emacs package manager to upgrade `khoj.el`
+
+- For `khoj.el` from MELPA
+  - Method 1
+    - Run `M-x package-list-packages` to list all packages
+    - Press `U` on `khoj` to mark it for upgrade
+    - Press `x` to execute the marked actions
+  - Method 2
+    - Run `M-x package-refresh-content`
+    - Run `M-x package-reinstall khoj`
+
+- For `khoj.el` from Straight
+  - Run `M-x straight-pull-package khoj`

docs/features.md

@@ -0,0 +1,13 @@
+## Features
+- **Search**
+  - **Local**: Your personal data stays local. All search and indexing is done on your machine. *Unlike chat which requires access to GPT.*
+  - **Incremental**: Incremental search for a fast, search-as-you-type experience
+- **Chat**
+  - **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers.
+  - **Iterative discovery**: Iteratively explore and (re-)discover your notes
+  - **Assisted creativity**: Smoothly weave across answers retrieval and content generation
+- **General**
+  - **Natural**: Advanced natural language understanding using Transformer based ML Models
+  - **Pluggable**: Modular architecture makes it easy to plug in new data sources, frontends and ML models
+  - **Multiple Sources**: Index your Org-mode and Markdown notes, PDF files, Github repositories, and Photos
+  - **Multiple Interfaces**: Interact from your [Web Browser](https://github.com/khoj-ai/khoj/blob/master/src/khoj/interface/web/index.html), [Emacs](https://github.com/khoj-ai/khoj/blob/master//src/interface/emacs/khoj.el) or [Obsidian](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/)

docs/github_integration.md

@@ -0,0 +1,14 @@
+# Setup the Github integration
+
+The Github integration allows you to index as many repositories as you want. It's currently default configured to index Issues, Commits, and all Markdown/Org files in each repository. For large repositories, this takes a fairly long time, but it works well for smaller projects.
+
+# Configure your settings
+
+1. Go to [http://localhost:42110/config](http://localhost:42110/config) and enter in settings for the data sources you want to index. You'll have to specify the file paths.
+
+## Use the Github plugin
+
+1. Generate a [classic PAT (personal access token)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) from [Github](https://github.com/settings/tokens) with `repo` and `admin:org` scopes at least.
+2. Navigate to [http://localhost:42110/config/content_type/github](http://localhost:42110/config/content_type/github) to configure your Github settings. Enter in your PAT, along with details for each repository you want to index.
+3. Click `Save`. Go back to the settings page and click `Configure`.
+4. Go to [http://localhost:42110/](http://localhost:42110/) and start searching!

docs/index.html

@@ -0,0 +1,35 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <title>Document</title>
+  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
+  <meta name="description" content="Description">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0">
+  <link rel="stylesheet" href="//cdn.jsdelivr.net/npm/docsify/lib/themes/buble.css" />
+  <link rel="icon" href="./assets/favicon-128x128.ico">
+</head>
+<body>
+  <div id="app"></div>
+  <script>
+    window.$docsify = {
+      name: 'Khoj',
+      repo: 'https://github.com/khoj-ai/khoj',
+      loadSidebar: true,
+      themeColor: '#c2a600',
+      // coverpage: true,
+    }
+  </script>
+  <!-- Docsify v4 -->
+  <script src="//cdn.jsdelivr.net/npm/docsify@4"></script>
+  <script src="//cdn.jsdelivr.net/npm/docsify/lib/plugins/search.min.js"></script>
+  <script src="//cdn.jsdelivr.net/npm/docsify-copy-code/dist/docsify-copy-code.min.js"></script>
+  <script src="//cdn.jsdelivr.net/npm/prismjs@1/components/prism-bash.min.js"></script>
+  <script defer data-domain="khoj.dev" src="https://plausible.io/js/script.js"></script>
+</body>
+<style>
+  video {
+    max-width: 800px;
+  }
+</style>
+</html>

docs/notion_integration.md

@@ -0,0 +1,14 @@
+## 📜 Notion Integration
+
+Khoj now supports search/chat with pages in your Notion workspaces. [Notion](notion.so/) is a platform people use for taking notes, especially for collaboration.
+
+We haven't setup a fancy integration with OAuth yet, so this integration still requires some effort on your end to generate an API key.
+
+1. Go to https://www.notion.so/my-integrations and create a new integration called Khoj to get an API key.
+![setup_new_integration](https://github.com/khoj-ai/khoj/assets/65192171/b056e057-d4dc-47dc-aad3-57b59a22c68b)
+3. Share all the workspaces that you want to integrate with the Khoj integration you just made in the previous step
+![enable_workspace](https://github.com/khoj-ai/khoj/assets/65192171/98290303-b5b8-4cb0-b32c-f68c6923a3d0)
+4. In the first step, you generated an API key. Use the newly generated API Key in your Khoj settings, by default at http://localhost:8000/config/content_type/notion. Click `Save`.
+5. Click `Configure` in http://localhost:8000/config to index your Notion workspace(s).
+
+That's it! You should be ready to start searching and chatting. Make sure you've configured your OpenAI API Key for chat.

docs/obsidian.md

@@ -1,29 +1,7 @@
-<img src="/src/khoj/interface/web/assets/icons/khoj-logo-sideways.svg" width="200" alt="Khoj Logo">Obsidian
+<h1><img src="./assets/khoj-logo-sideways.svg" width="200" alt="Khoj Logo">Obsidian</h1>
 
 > An AI personal assistant for your Digital Brain in Obsidian
 
-## Table of Contents
-
-- [Features](#Features)
-- [Demo](#Demo)
-  - [Search Demo](#Search-Demo)
-- [Interfaces](#Interfaces)
-  - [Search Modal](#Search-Modal)
-  - [Chat Modal](#Chat-Modal)
-- [Setup](#Setup)
-  - [Setup Backend](#1-Setup-Backend)
-  - [Setup Plugin](#2-Setup-Plugin)
-- [Use](#Use)
-  - [Search](#search)
-  - [Chat](#chat)
-  - [Find Similar Notes](#find-similar-notes)
-- [Upgrade](#Upgrade)
-  - [Upgrade Backend](#1-Upgrade-Backend)
-  - [Upgrade Plugin](#2-Upgrade-Plugin)
-- [Troubleshoot](#Troubleshoot)
-- [Visualize Codebase](#Visualize-Codebase)
-- [Implementation](#Implementation)
-
 ## Features
 - **Search**
   - **Natural**: Advanced natural language understanding using Transformer based ML Models
@@ -34,31 +12,6 @@
   - **Iterative discovery**: Iteratively explore and (re-)discover your notes
   - **Assisted creativity**: Smoothly weave across answers retrieval and content generation
 
-## Demo
-### Search Demo
-https://github.com/khoj-ai/khoj/assets/6413477/3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b
-
-<details><summary>Description</summary>
-
-1. Install Khoj via `pip` and start Khoj backend
-    ```shell
-    python -m pip install khoj-assistant && khoj
-    ```
-2. Install Khoj plugin via Community Plugins settings pane on Obsidian app
-    - Check the new Khoj plugin settings
-    - Wait for Khoj backend to index markdown, PDF files in the current Vault
-    - Open Khoj plugin on Obsidian via Search button on Left Pane
-    - Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/)
-    - Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
-
-</details>
-
-## Interfaces
-### Search Modal
-![](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_on_obsidian_0.2.5.png?)
-
-### Chat Modal
-![](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_chat_on_obsidian_0.6.0.png?)
 
 ## Setup
 - *Make sure [python](https://realpython.com/installing-python/) and [pip](https://pip.pypa.io/en/stable/installation/) are installed on your machine*
@@ -94,16 +47,14 @@ Notes:
 
 See [Khoj Chat](https://github.com/khoj-ai/khoj/tree/master/#Khoj-Chat) for more details
 
-![](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_chat_on_obsidian_0.6.0.png?)
-
 ### Search
 Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or run *Khoj: Search* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
 
 *Note: Ensure the khoj server is running in the background before searching. Execute `khoj` in your terminal if it is not already running*
 
-https://user-images.githubusercontent.com/6413477/218801155-cd67e8b4-a770-404a-8179-d6b61caa0f93.mp4
+[search_demo](https://user-images.githubusercontent.com/6413477/218801155-cd67e8b4-a770-404a-8179-d6b61caa0f93.mp4 ':include :type=mp4')
 
-<details><summary>Query Filters</summary>
+#### Query Filters
 
 Use structured query syntax to filter the natural language search results
 - **Word Filter**: Get entries that include/exclude a specified term
@@ -123,8 +74,6 @@ Use structured query syntax to filter the natural language search results
     - excluding words *"big"* and *"brother"*
     - that best match the natural language query *"what is the meaning of life?"*
 
-</details>
-
 ### Find Similar Notes
 To see other notes similar to the current one, run *Khoj: Find Similar Notes* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
 
@@ -138,6 +87,30 @@ To see other notes similar to the current one, run *Khoj: Find Similar Notes* fr
   2. Click the *Check for updates* button
   3. Click the *Update* button next to Khoj, if available
 
+## Demo
+### Search Demo
+[demo](https://github-production-user-asset-6210df.s3.amazonaws.com/6413477/240061700-3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b.mp4 ':include :type=mp4')
+
+#### Description
+
+1. Install Khoj via `pip` and start Khoj backend
+    ```shell
+    python -m pip install khoj-assistant && khoj
+    ```
+2. Install Khoj plugin via Community Plugins settings pane on Obsidian app
+    - Check the new Khoj plugin settings
+    - Wait for Khoj backend to index markdown, PDF files in the current Vault
+    - Open Khoj plugin on Obsidian via Search button on Left Pane
+    - Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/)
+    - Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
+
+## Interfaces
+### Search Modal
+![](./assets/khoj_on_obsidian_0.2.5.png)
+
+### Chat Modal
+![](./assets/khoj_chat_on_obsidian_0.6.0.png)
+
 ## Troubleshooting
   - Open the Khoj plugin settings pane, to configure Khoj
   - Toggle Enable/Disable Khoj, if setting changes have not applied
@@ -148,7 +121,8 @@ To see other notes similar to the current one, run *Khoj: Find Similar Notes* fr
   So notes across multiple vaults **cannot** be searched at the same time
 
 ## Visualize Codebase
-<img src="https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_obsidian_codebase_visualization_0.2.1.png" width="700" />
+
+![](./assets/khoj_obsidian_codebase_visualization_0.2.1.png)
 
 ## Implementation
 The plugin implements the following functionality to search your notes with Khoj:

docs/search.md

@@ -0,0 +1,79 @@
+## Khoj Search
+- **Khoj via Obsidian**
+  - Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or Search for *Khoj: Search* in the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
+- **Khoj via Emacs**
+  - Run `M-x khoj <user-query>`
+- **Khoj via Web**
+  - Open <http://localhost:42110/> directly
+- **Khoj via API**
+  - See the Khoj FastAPI [Swagger Docs](http://localhost:42110/docs), [ReDocs](http://localhost:42110/redocs)
+
+### Query Filters
+
+Use structured query syntax to filter the natural language search results
+- **Word Filter**: Get entries that include/exclude a specified term
+  - Entries that contain term_to_include: `+"term_to_include"`
+  - Entries that contain term_to_exclude: `-"term_to_exclude"`
+- **Date Filter**: Get entries containing dates in YYYY-MM-DD format from specified date (range)
+  - Entries from April 1st 1984: `dt:"1984-04-01"`
+  - Entries after March 31st 1984: `dt>="1984-04-01"`
+  - Entries before April 2nd 1984 : `dt<="1984-04-01"`
+- **File Filter**: Get entries from a specified file
+  - Entries from incoming.org file: `file:"incoming.org"`
+- Combined Example
+  - `what is the meaning of life? file:"1984.org" dt>="1984-01-01" dt<="1985-01-01" -"big" -"brother"`
+  - Adds all filters to the natural language query. It should return entries
+    - from the file *1984.org*
+    - containing dates from the year *1984*
+    - excluding words *"big"* and *"brother"*
+    - that best match the natural language query *"what is the meaning of life?"*
+
+## Details
+1. Your query is used to retrieve the most relevant notes, if any, using Khoj search
+2. These notes, the last few messages and associated metadata is passed to ChatGPT along with your query for a response
+
+
+## Performance
+
+### Query performance
+
+- Semantic search using the bi-encoder is fairly fast at \<50 ms
+- Reranking using the cross-encoder is slower at \<2s on 15 results. Tweak `top_k` to tradeoff speed for accuracy of results
+- Filters in query (e.g by file, word or date) usually add \<20ms to query latency
+
+### Indexing performance
+
+- Indexing is more strongly impacted by the size of the source data
+- Indexing 100K+ line corpus of notes takes about 10 minutes
+- Indexing 4000+ images takes about 15 minutes and more than 8Gb of RAM
+- Note: *It should only take this long on the first run* as the index is incrementally updated
+
+### Miscellaneous
+
+- Testing done on a Mac M1 and a \>100K line corpus of notes
+- Search, indexing on a GPU has not been tested yet
+
+## Advanced Usage
+
+### Use OpenAI Models for Search
+#### Setup
+1. Set `encoder-type`, `encoder` and `model-directory` under `asymmetric` and/or `symmetric` `search-type` in your `khoj.yml`[^1]:
+   ```diff
+      asymmetric:
+   -    encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1"
+   +    encoder: text-embedding-ada-002
+   +    encoder-type: khoj.utils.models.OpenAI
+        cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
+   -    encoder-type: sentence_transformers.SentenceTransformer
+   -    model_directory: "~/.khoj/search/asymmetric/"
+   +    model-directory: null
+   ```
+2. [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
+3. Restart Khoj server to generate embeddings. It will take longer than with offline models.
+
+#### Warnings
+  This configuration *uses an online model*
+  - It will **send all notes to OpenAI** to generate embeddings
+  - **All queries will be sent to OpenAI** when you search with Khoj
+  - You will be **charged by OpenAI** based on the total tokens processed
+  - It *requires an active internet connection* to search and index

docs/setup.md

@@ -0,0 +1,98 @@
+## Setup
+These are the general setup instructions for Khoj.
+
+- Make sure [python](https://realpython.com/installing-python/) and [pip](https://pip.pypa.io/en/stable/installation/) are installed on your machine
+- Check the [Khoj.el Readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Setup) to setup Khoj with Emacs<br />
+  Its simpler as it can skip the server *install*, *run* and *configure* step below.
+- Check the [Khoj Obsidian Readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#Setup) to setup Khoj with Obsidian<br />
+  Its simpler as it can skip the *configure* step below.
+
+### 1. Install
+Run the following command in your terminal to install the Khoj backend.
+
+- On Linux/MacOS
+  ```shell
+  python -m pip install khoj-assistant
+  ```
+
+- On Windows
+  ```shell
+  py -m pip install khoj-assistant
+  ```
+For more detailed Windows installation and troubleshooting, see [Windows Install](./windows_install.md).
+
+
+### 2. Run
+
+Run the following commmand from your terminal to start the Khoj backend and open Khoj in your browser.
+
+```shell
+khoj --gui
+```
+
+Note: To start Khoj automatically in the background use [Task scheduler](https://www.windowscentral.com/how-create-automated-task-using-task-scheduler-windows-10) on Windows or [Cron](https://en.wikipedia.org/wiki/Cron) on Mac, Linux (e.g with `@reboot khoj`)
+
+### 3. Configure
+1. Set `File`, `Folder` and hit `Save` in each Plugins you want to enable for Search on the Khoj config page
+2. Add your OpenAI API key to Chat Feature settings if you want to use Chat
+3. Click `Configure` and wait. The app will download ML models and index the content for search and (optionally) chat
+
+### 4. Install Interface Plugins (Optional)
+Khoj exposes a web interface to search, chat and configure by default.<br />
+The optional steps below allow using Khoj from within an existing application like Obsidian or Emacs.
+
+- **Khoj Obsidian**:<br />
+[Install](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#2-Setup-Plugin) the Khoj Obsidian plugin
+
+- **Khoj Emacs**:<br />
+[Install](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#2-Install-Khojel) khoj.el
+
+
+## Upgrade
+### Upgrade Khoj Server
+```shell
+pip install --upgrade khoj-assistant
+```
+
+*Note: To upgrade to the latest pre-release version of the khoj server run below command*
+```shell
+# Maps to the latest commit on the master branch
+pip install --upgrade --pre khoj-assistant
+```
+
+### Upgrade Khoj on Emacs
+- Use your Emacs Package Manager to Upgrade
+- See [khoj.el readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade) for details
+
+### Upgrade Khoj on Obsidian
+- Upgrade via the Community plugins tab on the settings pane in the Obsidian app
+- See the [khoj plugin readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#2-Setup-Plugin) for details
+
+## Uninstall
+1. (Optional) Hit `Ctrl-C` in the terminal running the khoj server to stop it
+2. Delete the khoj directory in your home folder (i.e `~/.khoj` on Linux, Mac or `C:\Users\<your-username>\.khoj` on Windows)
+3. Uninstall the khoj server with `pip uninstall khoj-assistant`
+4. (Optional) Uninstall khoj.el or the khoj obsidian plugin in the standard way on Emacs, Obsidian
+
+## Troubleshoot
+
+#### Install fails while building Tokenizer dependency
+- **Details**: `pip install khoj-assistant` fails while building the `tokenizers` dependency. Complains about Rust.
+- **Fix**: Install Rust to build the tokenizers package. For example on Mac run:
+    ```shell
+    brew install rustup
+    rustup-init
+    source ~/.cargo/env
+    ```
+- **Refer**: [Issue with Fix](https://github.com/khoj-ai/khoj/issues/82#issuecomment-1241890946) for more details
+
+#### Search starts giving wonky results
+- **Fix**: Open [/api/update?force=true](http://localhost:42110/api/update?force=true)[^2] in browser to regenerate index from scratch
+- **Note**: *This is a fix for when you percieve the search results have degraded. Not if you think they've always given wonky results*
+
+#### Khoj in Docker errors out with \"Killed\" in error message
+- **Fix**: Increase RAM available to Docker Containers in Docker Settings
+- **Refer**: [StackOverflow Solution](https://stackoverflow.com/a/50770267), [Configure Resources on Docker for Mac](https://docs.docker.com/desktop/mac/#resources)
+
+#### Khoj errors out complaining about Tensors mismatch or null
+- **Mitigation**: Disable `image` search using the desktop GUI

docs/windows_install.md

@@ -0,0 +1,23 @@
+# Windows Installation
+
+These steps can be used to setup Khoj on a clean, new Windows 11 machine. It has been tested on a Windows VM
+
+## Prerequisites
+1. Ensure you have Visual Studio C++ Build tools installed. You can download it [from Microsoft here](https://visualstudio.microsoft.com/visual-cpp-build-tools/). At the minimum, you should have the following configuration:
+    <img width="1152" alt="Screenshot 2023-07-12 at 3 56 25 PM" src="https://github.com/khoj-ai/khoj/assets/65192171/b506a858-2f5e-4c85-946b-5422d83f112a">
+3. Ensure you have Python installed. You can check by running `python --version`. If you don't, install the latest version [from here](https://www.python.org/downloads/).
+    - Ensure you have pip installed: `py -m ensurepip --upgrade`.
+
+## Quick start
+1. Open a PowerShell terminal.
+7. Run `pip install khoj-assistant`
+8. Start Khoj with `khoj`
+
+## Installation in a Virtual Environment
+Use this if you want to install with a virtual environment. This will make it much easier to manage your dependencies. You can read more about [virtual environments](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/) here.
+
+1. Open a PowerShell terminal with the `Run as Administrator` privileges.
+5. Create a virtual environment: `mkdir khoj && cd khoj && py -m venv .venv`
+6. Activate the virtual environment: `.\.venv\Scripts\activate`. If you get a permissions error, then run `Set-ExecutionPolicy -ExecutionPolicy RemoteSigned`.
+7. Run `pip install khoj-assistant`
+8. Start Khoj with `khoj`

src/interface/emacs/README.org

@@ -1,167 +0,0 @@
-[[https://github.com/khoj-ai/khoj/edit/master/src/interface/emacs/README.org][file:/src/khoj/interface/web/assets/icons/khoj-logo-sideways-200.png]] Emacs
-
-
-  [[https://stable.melpa.org/#/khoj][file:https://stable.melpa.org/packages/khoj-badge.svg]] [[https://melpa.org/#/khoj][file:https://melpa.org/packages/khoj-badge.svg]] [[https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml][https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml/badge.svg?]]  [[https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml][https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml/badge.svg?]]
-
-  /An AI personal assistant for your digital brain/
-
-** Table of Contents
-  - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#features][Features]]
-  - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Interface][Interface]]
-  - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Setup][Setup]]
-    - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Direct-Install][Direct Install]]
-    - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Minimal-Install][Minimal Install]]
-    - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Standard-Install][Standard Install]]
-    - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#With-Straight.el][With Straight.el]]
-  - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Use][Use]]
-    - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Search][Search]]
-    - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Chat][Chat]]
-    - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Find-similar-entries][Find Similar Entries]]
-    - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Advanced-usage][Advanced Usage]]
-    - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Khoj-menu][Khoj Menu]]
-  - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade][Upgrade]]
-    - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade-Khoj-Backend][Upgrade Backend]]
-    - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade-Khojel][Upgrade Khoj.el]]
-
-** Features
-  - *Search*
-    - *Natural*: Advanced natural language understanding using Transformer based ML Models
-    - *Local*: Your personal data stays local. All search, indexing is done on your machine*
-    - *Incremental*: Incremental search for a fast, search-as-you-type experience
-  - *Chat*
-    - *Faster answers*: Find answers faster than search
-    - *Iterative discovery*: Iteratively explore and (re-)discover your notes
-    - *Assisted creativity*: Smoothly weave across answer retrieval and content generation
-
-** Interface
-*** Search UI
-  [[/docs/khoj_on_emacs.png]]
-
-*** Chat UI
-  [[/docs/khoj_chat_on_emacs_0.5.0.png]]
-
-** Setup
-- /Make sure [[https://realpython.com/installing-python/][python]] and [[https://pip.pypa.io/en/stable/installation/][pip]] are installed on your machine/
-
-- /khoj.el attempts to automatically install, start and configure the khoj server./
-  If this fails, follow [[https://github.com/khoj-ai/khoj/tree/master/#Setup][these instructions]] to manually setup the khoj server.
-
-*** Direct Install
-    #+begin_src elisp
-    M-x package-install khoj
-    #+end_src
-
-*** Minimal Install
-    Add below snippet to your Emacs config file.
-    Indexes your org-agenda files, by default.
-
-    #+begin_src elisp
-      ;; Install Khoj Package from MELPA Stable
-      (use-package khoj
-        :ensure t
-        :pin melpa-stable
-        :bind ("C-c s" . 'khoj)
-    #+end_src
-
-  - Note: Install ~khoj.el~ from MELPA (instead of MELPA Stable) if you installed the pre-release version of khoj
-    - That is, use ~:pin melpa~ to install khoj.el in above snippet if khoj server was installed with ~--pre~ flag, i.e ~pip install --pre khoj-assistant~
-    - Else use ~:pin melpa-stable~ to install khoj.el in above snippet if khoj was installed with ~pip install khoj-assistant~
-    - This ensures both khoj.el and khoj app are from the same version (git tagged or latest)
-
-*** Standard Install
-    Add below snippet to your Emacs config file.
-    Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat
-
-    #+begin_src elisp
-      ;; Install Khoj Package from MELPA Stable
-      (use-package khoj
-        :ensure t
-        :pin melpa-stable
-        :bind ("C-c s" . 'khoj)
-        :config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
-                      khoj-org-files '("~/docs/todo.org" "~/docs/work.org")
-                      khoj-openai-api-key "YOUR_OPENAI_API_KEY")) ; required to enable chat
-    #+end_src
-
-*** With [[https://github.com/raxod502/straight.el][Straight.el]]
-  Add below snippet to your Emacs config file.
-  Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat
-
-  #+begin_src elisp
-    ;; Install Khoj Package using Straight.el
-    (use-package khoj
-      :after org
-      :straight (khoj :type git :host github :repo "khoj-ai/khoj" :files (:defaults "src/interface/emacs/khoj.el"))
-      :bind ("C-c s" . 'khoj)
-      :config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
-                    khoj-org-files '("~/docs/todo.org" "~/docs/work.org")
-                    khoj-openai-api-key "YOUR_OPENAI_API_KEY" ; required to enable chat)
-  #+end_src
-
-** Use
-*** Search
-  1. Hit ~C-c s s~ (or ~M-x khoj RET s~) to open khoj search
-
-  2. Enter your query in natural language
-
-    e.g "What is the meaning of life?", "My life goals for 2023"
-
-*** Chat
-  1. Hit ~C-c s c~ (or ~M-x khoj RET c~) to open khoj chat
-
-  2. Ask questions in a natural, conversational style
-
-     E.g "When did I file my taxes last year?"
-
-  See [[https://github.com/khoj-ai/khoj/tree/master/#Khoj-Chat][Khoj Chat]] for more details
-
-*** Find Similar Entries
-  This feature finds entries similar to the one you are currently on.
-  1. Move cursor to the org-mode entry, markdown section or text paragraph you want to find similar entries for
-  2. Hit ~C-c s f~ (or ~M-x khoj RET f~) to find similar entries
-
-*** Advanced Usage
-  - Add [[https://github.com/khoj-ai/khoj/#query-filters][query filters]] during search to narrow down results further
-
-    e.g `What is the meaning of life? -"god" +"none" dt>"last week"`
-
-  - Use ~C-c C-o 2~ to open the current result at cursor in its source org file
-
-    - This calls ~M-x org-open-at-point~ on the current entry and opens the second link in the entry.
-    - The second link is the entries [[https://orgmode.org/manual/Handling-Links.html#FOOT28][org-id]], if set, or the heading text.
-      The first link is the line number of the entry in the source file. This link is less robust to file changes.
-    - Note: If you have [[https://orgmode.org/manual/Speed-Keys.html][speed keys]] enabled, ~o 2~ will also work
-
-*** Khoj Menu
-  [[/docs/khoj_emacs_menu.png]]
-  Hit ~C-c s~ (or ~M-x khoj~) to open the khoj menu above. Then:
-  - Hit ~t~ until you preferred content type is selected in the khoj menu
-
-    ~Content Type~ specifies the content to perform ~Search~, ~Update~ or ~Find Similar~ actions on
-  - Hit ~n~ twice and then enter number of results you want to see
-
-    ~Results Count~ is used by the ~Search~ and ~Find Similar~ actions
-  - Hit ~-f u~ to ~force~ update the khoj content index
-
-    The ~Force Update~ switch is only used by the ~Update~ action
-
-** Upgrade
-*** Upgrade Khoj Backend
-  #+begin_src shell
-    pip install --upgrade khoj-assistant
-  #+end_src
-
-*** Upgrade Khoj.el
-  Use your Emacs package manager to upgrade ~khoj.el~
-
-  - For ~khoj.el~ from MELPA
-    - Method 1
-      - Run ~M-x package-list-packages~ to list all packages
-      - Press ~U~ on ~khoj~ to mark it for upgrade
-      - Press ~x~ to execute the marked actions
-    - Method 2
-      - Run ~M-x package-refresh-content~
-      - Run ~M-x package-reinstall khoj~
-
-  - For ~khoj.el~ from Straight
-    - Run ~M-x straight-pull-package khoj~