From 474afa5efefd1fff9f367761eb8a8671e7a790e7 Mon Sep 17 00:00:00 2001 From: Debanjum Singh Solanky Date: Fri, 2 Feb 2024 10:31:27 +0530 Subject: [PATCH] Document using OpenAI-compatible LLM API server for Khoj chat This allows using open or commerical, local or hosted LLM models that are not supported in Khoj by default. It also allows users to use other local LLM API servers that support their GPU Closes #407 --- documentation/docs/get-started/setup.mdx | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/documentation/docs/get-started/setup.mdx b/documentation/docs/get-started/setup.mdx index e1832b72..7447621c 100644 --- a/documentation/docs/get-started/setup.mdx +++ b/documentation/docs/get-started/setup.mdx @@ -264,6 +264,27 @@ You can head to http://localhost:42110 to use the web interface. You can also us ``` +## Advanced +### Use OpenAI compatible LLM API Server +Use this if you want to use non-standard, open or commercial, local or hosted LLM models for Khoj chat +1. Install an OpenAI compatible LLM API Server like [LiteLLM](https://docs.litellm.ai/docs/proxy/quick_start), [Llama-cpp-python](https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#openai-compatible-web-server) etc. +2. Set `OPENAI_API_BASE=""` environment variables before starting Khoj + +#### Sample Setup using LiteLLM and Mistral API + +```shell +# Install LiteLLM +pip install litellm[proxy] + +# Start LiteLLM and use Mistral tiny via Mistral API +export MISTRAL_API_KEY= +litellm --model mistral/mistral-tiny --drop_params + +# Set OpenAI API Base to LiteLLM server URL and start Khoj +export OPENAI_API_BASE='http://localhost:8000' +khoj --anonymous-mode +``` + ## Troubleshoot #### Install fails while building Tokenizer dependency