mirror of
https://github.com/khoj-ai/khoj.git
synced 2024-11-23 15:38:55 +01:00
Move Use OpenAI Compatible LLM Server section to existing advanced page
Add footnote on supported chat models to the self-hosting section
This commit is contained in:
Debanjum Singh Solanky
parent
523af5b3aa
commit
c40f642afa
2 changed files with 26 additions and 26 deletions
|
|
@ -175,12 +175,13 @@ To use the desktop client, you need to go to your Khoj server's settings page (h
|
|||
1. Go to http://localhost:42110/server/admin and login with your admin credentials.
|
||||
1. Go to [OpenAI settings](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/) in the server admin settings to add an OpenAI processor conversation config. This is where you set your API key. Alternatively, you can go to the [offline chat settings](http://localhost:42110/server/admin/database/offlinechatprocessorconversationconfig/) and simply create a new setting with `Enabled` set to `True`.
|
||||
2. Go to the ChatModelOptions if you want to add additional models for chat.
|
||||
- Set the `chat-model` field to a supported chat model of your choice. For example, you can specify `gpt-4` if you're using OpenAI or `mistral-7b-instruct-v0.1.Q4_0.gguf` if you're using offline chat.
|
||||
- Set the `chat-model` field to a supported chat model[^1] of your choice. For example, you can specify `gpt-4` if you're using OpenAI or `mistral-7b-instruct-v0.1.Q4_0.gguf` if you're using offline chat.
|
||||
- Make sure to set the `model-type` field to `OpenAI` or `Offline` respectively.
|
||||
- The `tokenizer` and `max-prompt-size` fields are optional. Set them only when using a non-standard model (i.e not mistral, gpt or llama2 model).
|
||||
1. Select files and folders to index [using the desktop client](/get-started/setup#2-download-the-desktop-client). When you click 'Save', the files will be sent to your server for indexing.
|
||||
- Select Notion workspaces and Github repositories to index using the web interface.
|
||||
|
||||
[^1]: Khoj, by default, can use [OpenAI GPT3.5+ chat models](https://platform.openai.com/docs/models/overview) or [GPT4All chat models that follow Llama2 Prompt Template](https://github.com/nomic-ai/gpt4all/blob/main/gpt4all-chat/metadata/models2.json). See [this section](/miscellaneous/advanced#use-openai-compatible-llm-api-server-self-hosting) to use non-standard chat models
|
||||
|
||||
:::tip[Note]
|
||||
Using Safari on Mac? You might not be able to login to the admin panel. Try using Chrome or Firefox instead.
|
||||
|
|
@ -264,29 +265,6 @@ You can head to http://localhost:42110 to use the web interface. You can also us
|
|||
</Tabs>
|
||||
```
|
||||
|
||||
## Advanced
|
||||
### Use OpenAI compatible LLM API Server
|
||||
Use this if you want to use non-standard, open or commercial, local or hosted LLM models for Khoj chat
|
||||
1. Install an OpenAI compatible LLM API Server like [LiteLLM](https://docs.litellm.ai/docs/proxy/quick_start), [Llama-cpp-python](https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#openai-compatible-web-server) etc.
|
||||
2. Set `OPENAI_API_BASE="<url-of-your-llm-server>"` environment variables before starting Khoj
|
||||
3. Add ChatModelOptions with `model-type` `OpenAI`, and `chat-model` to anything (e.g `gpt-4`) in the [Configure](#3-configure) step
|
||||
4. [Optional] Set an appropriate `tokenizer` and `max-prompt-size` relevant for the actual chat model you're using
|
||||
|
||||
#### Sample Setup using LiteLLM and Mistral API
|
||||
|
||||
```shell
|
||||
# Install LiteLLM
|
||||
pip install litellm[proxy]
|
||||
|
||||
# Start LiteLLM and use Mistral tiny via Mistral API
|
||||
export MISTRAL_API_KEY=<MISTRAL_API_KEY>
|
||||
litellm --model mistral/mistral-tiny --drop_params
|
||||
|
||||
# Set OpenAI API Base to LiteLLM server URL and start Khoj
|
||||
export OPENAI_API_BASE='http://localhost:8000'
|
||||
khoj --anonymous-mode
|
||||
```
|
||||
|
||||
## Troubleshoot
|
||||
|
||||
#### Install fails while building Tokenizer dependency
|
||||
|
|
|
|||
|
|
@ -4,13 +4,13 @@ sidebar_position: 3
|
|||
|
||||
# Advanced Usage
|
||||
|
||||
### Search across Different Languages (Self-Hosting)
|
||||
## Search across Different Languages (Self-Hosting)
|
||||
To search for notes in multiple, different languages, you can use a [multi-lingual model](https://www.sbert.net/docs/pretrained_models.html#multi-lingual-models).<br />
|
||||
For example, the [paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) supports [50+ languages](https://www.sbert.net/docs/pretrained_models.html#:~:text=we%20used%20the%20following%2050%2B%20languages), has good search quality and speed. To use it:
|
||||
1. Manually update the search config in server's admin settings page. Go to [the search config](http://localhost:42110/server/admin/database/searchmodelconfig/). Either create a new one, if none exists, or update the existing one. Set the bi_encoder to `sentence-transformers/multi-qa-MiniLM-L6-cos-v1` and the cross_encoder to `cross-encoder/ms-marco-MiniLM-L-6-v2`.
|
||||
2. Regenerate your content index from all the relevant clients. This step is very important, as you'll need to re-encode all your content with the new model.
|
||||
|
||||
### Query Filters
|
||||
## Query Filters
|
||||
|
||||
Use structured query syntax to filter entries from your knowledge based used by search results or chat responses.
|
||||
|
||||
|
|
@ -30,3 +30,25 @@ Use structured query syntax to filter entries from your knowledge based used by
|
|||
- containing dates from the year *1984*
|
||||
- excluding words *"big"* and *"brother"*
|
||||
- that best match the natural language query *"what is the meaning of life?"*
|
||||
|
||||
## Use OpenAI compatible LLM API Server (Self Hosting)
|
||||
Use this if you want to use non-standard, open or commercial, local or hosted LLM models for Khoj chat
|
||||
1. Setup your desired chat LLM by installing an OpenAI compatible LLM API Server like [LiteLLM](https://docs.litellm.ai/docs/proxy/quick_start), [llama-cpp-python](https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#openai-compatible-web-server)
|
||||
2. Set environment variable `OPENAI_API_BASE="<url-of-your-llm-server>"` before starting Khoj
|
||||
3. Add ChatModelOptions with `model-type` `OpenAI`, and `chat-model` to anything (e.g `gpt-4`) during [Config](/get-started/setup#3-configure)
|
||||
- *(Optional)* Set the `tokenizer` and `max-prompt-size` relevant to the actual chat model you're using
|
||||
|
||||
#### Sample Setup using LiteLLM and Mistral API
|
||||
|
||||
```shell
|
||||
# Install LiteLLM
|
||||
pip install litellm[proxy]
|
||||
|
||||
# Start LiteLLM and use Mistral tiny via Mistral API
|
||||
export MISTRAL_API_KEY=<MISTRAL_API_KEY>
|
||||
litellm --model mistral/mistral-tiny --drop_params
|
||||
|
||||
# Set OpenAI API Base to LiteLLM server URL and start Khoj
|
||||
export OPENAI_API_BASE='http://localhost:8000'
|
||||
khoj --anonymous-mode
|
||||
```
|
||||
|
|
|
|||
Loading…
Reference in a new issue