sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-23 23:48:56 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 16
khoj/documentation/docs/advanced/use-openai-proxy.md
Debanjum Singh Solanky 68e7c297e0 Add Advanced Self Hosting Section, Improve Self Hosting, OpenAI Proxy Docs 
- Add instructions for self-hosted users with info, warning boxes to
  avoid, fix common issues when setting up Khoj server
- Create new Advanced Self Hosting section
  - Extract Advanced Self-Hosting Sections from the Advanced Page and
    move them to separate Pages under Advanced Self Hosting section
- Improve OpenAI Proxy Docs
  - Put Ollama setup as a section under OpenAI API Proxy page instead
    of a separate page
  - Add Section to use Khoj with chat model from LM Studio
  - Update LiteLLM docs to use chat model from LM Studio
2024-06-24 16:12:20 +05:30

7.1 KiB
Raw Blame History

sidebar_position
1

Use OpenAI Proxy

:::info This is only helpful for self-hosted users. If you're using Khoj Cloud, you're limited to our first-party models. :::

:::info Khoj natively supports local LLMs available on HuggingFace in GGUF format. Using an OpenAI API proxy with Khoj maybe useful for ease of setup, trying new models or using commercial LLMs via API. :::

Khoj can use any OpenAI API compatible server including Ollama, LMStudio and LiteLLM. Configuring this allows you to use non-standard, open or commercial, local or hosted LLM models for Khoj

Combine them with Khoj can turn your favorite LLM into an AI agent. Allowing you to chat with your docs, find answers from the internet, build custom agents and run automations.

Ollama

Ollama allows you to run many popular open-source LLMs locally from your terminal. For folks comfortable with the terminal, Ollama's terminal based flows can ease setup and management of chat models.

Ollama exposes a local OpenAI API compatible server. This makes it possible to use chat models from Ollama to create your personal AI agents with Khoj.

Setup

  1. Setup Ollama: https://ollama.com/
  2. Start your preferred model with Ollama. For example, 
    ollama run llama3
  3. Create a new OpenAI Processor Conversation Config on your Khoj admin panel
    • Name: ollama
    • Api Key: any string
    • Api Base Url: http://localhost:11434/v1/ (default for Ollama)
  4. Create a new Chat Model Option on your Khoj admin panel.
    • Name: llama3 (replace with the name of your local model)
    • Model Type: Openai
    • Openai Config: <the ollama config you created in step 3>
    • Max prompt size: 1000 (replace with the max prompt size of your model)
  5. Create a new Server Chat Setting on your Khoj admin panel
    • Default model: <name of chat model option you created in step 4>
    • Summarizer model: <name of chat model option you created in step 4>
  6. Go to your config and select the model you just created in the chat model dropdown.

That's it! You should now be able to chat with your Ollama model from Khoj. If you want to add additional models running on Ollama, repeat step 6 for each model.

LM Studio

LM Studio is a desktop app to chat with open-source LLMs on your local machine. LM Studio provides a neat interface for folks comfortable with a GUI.

LM Studio can also expose an OpenAI API compatible server. This makes it possible to turn chat models from LM Studio into your personal AI agents with Khoj.

Setup

  1. Install LM Studio and download your preferred Chat Model
  2. Go to the Server Tab on LM Studio, Select your preferred Chat Model and Click the green Start Server button
  3. Create a new OpenAI Processor Conversation Config on your Khoj admin panel
    • Name: proxy-name
    • Api Key: any string
    • Api Base Url: http://localhost:1234/v1/ (default for LMStudio)
  4. Create a new Chat Model Option on your Khoj admin panel.
    • Name: llama3 (replace with the name of your local model)
    • Model Type: Openai
    • Openai Config: <the proxy config you created in step 3>
    • Max prompt size: 2000 (replace with the max prompt size of your model)
    • Tokenizer: Do not set for OpenAI, mistral, llama3 based models
  5. Create a new Server Chat Setting on your Khoj admin panel
    • Default model: <name of chat model option you created in step 4>
    • Summarizer model: <name of chat model option you created in step 4>
  6. Go to your config and select the model you just created in the chat model dropdown.

LiteLLM

LiteLLM exposes an OpenAI compatible API that proxies requests to other LLM API services. This provides a standardized API to interact with both open-source and commercial LLMs.

Using LiteLLM with Khoj makes it possible to turn any LLM behind an API into your personal AI agent.

Setup

  1. Install LiteLLM 
    pip install litellm[proxy]
  2. Start LiteLLM and use Mistral tiny via Mistral API 
    export MISTRAL_API_KEY=<MISTRAL_API_KEY>
litellm --model mistral/mistral-tiny --drop_params
  3. Create a new OpenAI Processor Conversation Config on your Khoj admin panel
    • Name: proxy-name
    • Api Key: any string
    • Api Base Url: URL of your Openai Proxy API
  4. Create a new Chat Model Option on your Khoj admin panel.
    • Name: llama3 (replace with the name of your local model)
    • Model Type: Openai
    • Openai Config: <the proxy config you created in step 3>
    • Max prompt size: 2000 (replace with the max prompt size of your model)
    • Tokenizer: Do not set for OpenAI, mistral, llama3 based models
  5. Create a new Server Chat Setting on your Khoj admin panel
    • Default model: <name of chat model option you created in step 4>
    • Summarizer model: <name of chat model option you created in step 4>
  6. Go to your config and select the model you just created in the chat model dropdown.

General

  1. Start your preferred OpenAI API compatible app
  2. Create a new OpenAI Processor Conversation Config on your Khoj admin panel
    • Name: proxy-name
    • Api Key: any string
    • Api Base Url: URL of your Openai Proxy API
  3. Create a new Chat Model Option on your Khoj admin panel.
    • Name: llama3 (replace with the name of your local model)
    • Model Type: Openai
    • Openai Config: <the proxy config you created in step 3>
    • Max prompt size: 2000 (replace with the max prompt size of your model)
    • Tokenizer: Do not set for OpenAI, mistral, llama3 based models
  4. Create a new Server Chat Setting on your Khoj admin panel
    • Default model: <name of chat model option you created in step 4>
    • Summarizer model: <name of chat model option you created in step 4>
  5. Go to your config and select the model you just created in the chat model dropdown.