Merge pull request #596 from khoj-ai/chore/add-developer-documentation

Improve the developer documentation

docs/_sidebar.md

@@ -2,6 +2,8 @@
     - [Overview](README.md)
     - [Self-Host](setup.md)
     - [Demos](demos.md)
+- Contributing
+    - [Development](development.md)
 - Use
     - [Features](features.md)
         - [Chat](chat.md)
@@ -19,5 +21,3 @@
     - [Advanced](advanced.md)
     - [Performance](performance.md)
     - [Credits](credits.md)
-- Contributing
-    - [Development](development.md)

docs/development.md

@@ -1,9 +1,49 @@
 # Development
 
-## Setup
+Welcome to the development docs of Khoj! Thanks for you interesting in being a contributor ❤️. Open source contributors are a corner-store of the Khoj community. We welcome all contributions, big or small.
+
+To get started with contributing, check out the official GitHub docs on [contributing to an open-source project](https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project).
+
+Join the [Discord](https://discord.gg/WaxF3SkFPU) server and pick up click the ✅ for the question "Are you interested in becoming a contributor?" in the `#welcome-and-rules` channel. This will give you access to the `#contributors` channel where you can ask questions and get help from other contributors.
+
+If you're looking for a place to get started, check out the list of [Github Issues](https://github.com/khoj-ai/khoj/issues) with the tag `good first issue` to find issues that are good for first-time contributors.
+
+## Local Server Installation
 ### Using Pip
 #### 1. Install
 
+
+<!-- tabs:start -->
+
+#### **MacOS**
+
+```shell
+# Get Khoj Code
+git clone https://github.com/khoj-ai/khoj && cd khoj
+
+# Create, Activate Virtual Environment
+python3 -m venv .venv && source .venv/bin/activate
+
+# For MacOS or zsh users run this
+pip install -e '.[dev]'
+```
+
+#### **Windows**
+
+```shell
+# Get Khoj Code
+git clone https://github.com/khoj-ai/khoj && cd khoj
+
+# Create, Activate Virtual Environment
+python3 -m venv .venv && .venv\Scripts\activate
+
+# Install Khoj for Development
+pip install -e .[dev]
+
+```
+
+#### **Linux**
+
 ```shell
 # Get Khoj Code
 git clone https://github.com/khoj-ai/khoj && cd khoj
@@ -14,11 +54,11 @@ python3 -m venv .venv && source .venv/bin/activate
 # Install Khoj for Development
 pip install -e .[dev]
 
-# For MacOS or zsh users run this
-pip install -e .'[dev]'
-
 ```
 
+<!-- tabs:end -->
+
+
 #### 2. Run
 1. Start Khoj
    ```shell
@@ -30,6 +70,9 @@ pip install -e .'[dev]'
   Note: Wait after configuration for khoj to Load ML model, generate embeddings and expose API to query notes, images, documents etc specified in config YAML
 
 ### Using Docker
+
+Make sure you install the latest version of [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/).
+
 #### 1. Clone
 
 ```shell
@@ -38,23 +81,28 @@ git clone https://github.com/khoj-ai/khoj && cd khoj
 
 #### 2. Configure
 
-- **Required**: Update [docker-compose.yml](https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml) to mount your images, (org-mode or markdown) notes, PDFs and Github repositories
+1. Update [docker-compose.yml](https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml) to use relevant environment variables.
-- **Optional**: Edit application configuration in [khoj_docker.yml](https://github.com/khoj-ai/khoj/blob/master/config/khoj_docker.yml)
+2. Comment out the `image` line and uncomment the `build` line in the `server` service
 
 #### 3. Run
 
+This will start the Khoj server, and the database.
+
 ```shell
 docker-compose up -d
 ```
 
-*Note: The first run will take time. Let it run, it\'s mostly not hung, just generating embeddings*
-
 #### 4. Upgrade
 
+If you've made changes to the codebase, you'll need to rebuild the Docker image before running the container again.
+
 ```shell
-docker-compose build --pull
+docker-compose build --no-cache
 ```
 
+## Update clients
+In whichever clients you're using for testing, you'll need to update the server URL to point to your local server. By default, the local server URL should be `http://127.0.0.1:42110`.
+
 ## Validate
 ### Before Making Changes
 1. Install Git Hooks for Validation
@@ -67,27 +115,39 @@ docker-compose build --pull
 
 ### Before Creating PR
 
-1. Run Tests. If you get an error complaining about a missing `fast_tokenizer_file`, follow the solution [in this Github issue](https://github.com/UKPLab/sentence-transformers/issues/1659).
+!> **Note**: You should be in an active virtual environment for Khoj in order to run the unit tests and linter.
+
+1. Ensure that you have a [Github Issue](https://github.com/khoj-ai/khoj/issues) that can be linked to the PR. If not, create one. Make sure you've tagged one of the maintainers to the issue. This will ensure that the maintainers are notified of the PR and can review it. It's best discuss the code design on an existing issue or Discord thread before creating a PR. This helps get your PR merged faster.
+1. Run unit tests.
    ```shell
    pytest
    ```
-
+2. Run the linter.
-2. Run MyPy to check types
    ```shell
-   mypy --config-file pyproject.toml
+   mypy
    ```
+4. Think about how to add unit tests to verify the functionality you're adding in the PR. If you're not sure how to do this, ask for help in the Github issue or on Discord's `#contributors` channel.
 
 ### After Creating PR
-- Automated [validation workflows](.github/workflows) run for every PR.
+1. Automated [validation workflows](.github/workflows) should run for every PR. Tag one of the maintainers in the PR to trigger it.
 
-  Ensure any issues seen by them our fixed
+## Obsidian Plugin Development
+### Plugin development setup
+The core code for the Obsidian plugin is under `src/interface/obsidian`. The file `main.ts` is a good place to start.
 
-- Test the python packge created for a PR
+1. In your CLI, go to the directory `src/interface/obsidian` in the Khoj repository.
-  1. Download and extract the zipped `.whl` artifact generated from the pypi workflow run for the PR.
+2. Run `yarn install` to install the dependencies.
-  2. Install (in your virtualenv) with `pip install /path/to/download*.whl>`
+3. Run `yarn dev` to start the development server. This will continually rebuild the plugin as you make changes to the code.
-  3. Start and use the application to see if it works fine
+    - Your code changes will be outputted to a file called `main.js` in the `obsidian` directory.
 
-## Create Khoj Release
+### Loading your development plugin in Obsidian
+1. Make sure you have the Khoj plugin installed in Obsidian. [See the plugin page](https://publish.obsidian.md/hub/02+-+Community+Expansions/02.05+All+Community+Expansions/Plugins/khoj).
+1. Open Obsidian and go to your settings (gear icon in the bottom left corner)
+2. Click on 'Community Plugins' in the left panel
+3. Next to the 'Installed Plugins' heading, click on the folder icon to open the folder with the plugin's source code.
+4. Open the `khoj` folder in the file explorer that opens. You'll see a file called `main.js` in this folder. To test your changes, replace this file with the `main.js` file that was generated by the development server in the previous section.
+
+## Create Khoj Release (Only for Maintainers)
 Follow the steps below to [release](https://github.com/debanjum/khoj/releases/) Khoj. This will create a stable release of Khoj on [Pypi](https://pypi.org/project/khoj-assistant/), [Melpa](https://stable.melpa.org/#%252Fkhoj) and [Obsidian](https://obsidian.md/plugins?id%253Dkhoj). It will also create desktop apps of Khoj and attach them to the latest release.
 
 1. Create and tag release commit by running the bump_version script. The release commit sets version number in required metadata files.
@@ -114,14 +174,3 @@ Follow the steps below to [release](https://github.com/debanjum/khoj/releases/)
 ## Visualize Khoj Obsidian Plugin Codebase
 
 ![](./assets/khoj_obsidian_codebase_visualization_0.2.1.png)
-
-## Khoj Obsidian Plugin Implementation
-The plugin implements the following functionality to search your notes with Khoj:
-- [X] Open the Khoj search modal via left ribbon icon or the *Khoj: Search* command
-- [X] Render results as Markdown preview to improve readability
-- [X] Configure Khoj via the plugin setting tab on the settings page
-  - Set Obsidian Vault to Index with Khoj. Defaults to all markdown, PDF files in current Vault
-  - Set URL of Khoj backend
-  - Set Number of Search Results to show in Search Modal
-- [X] Allow reranking of result to improve search quality
-- [X] Allow Finding notes similar to current note being viewed