sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-18 18:47:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 23
khoj/documentation/docs/advanced/use-openai-proxy.md
Debanjum 9dd3782f5c
Rename OpenAIProcessorConversationConfig DB model to more apt AiModelApi (#998) 
* Rename OpenAIProcessorConversationConfig to more apt AiModelAPI

The DB model name had drifted from what it is being used for,
a general chat api provider that supports other chat api providers like
anthropic and google chat models apart from openai based chat models.

This change renames the DB model and updates the docs to remove this
confusion.

Using Ai Model Api we catch most use-cases including chat, stt, image generation etc.
2024-12-08 18:02:29 -08:00

2.2 KiB
Raw Blame History

sidebar_position
1

Use OpenAI Proxy

:::info This is only helpful for self-hosted users. If you're using Khoj Cloud, you're limited to our first-party models. :::

:::info Khoj natively supports local LLMs available on HuggingFace in GGUF format. Using an OpenAI API proxy with Khoj maybe useful for ease of setup, trying new models or using commercial LLMs via API. :::

Khoj can use any OpenAI API compatible server including local providers like Ollama, LMStudio and LiteLLM and commercial providers like HuggingFace, OpenRouter etc. Configuring this allows you to use non-standard, open or commercial, local or hosted LLM models for Khoj

Combine them with Khoj can turn your favorite LLM into an AI agent. Allowing you to chat with your docs, find answers from the internet, build custom agents and run automations.

For specific integrations, see our Ollama, LMStudio and LiteLLM setup docs. For general instructions to setup Khoj with an OpenAI API proxy see below.

General Setup

  1. Start your preferred OpenAI API compatible app locally or get API keys from commercial AI model providers.
  2. Create a new API Model API on your Khoj admin panel
    • Name: any name
    • Api Key: any string
    • Api Base Url: URL of your Openai Proxy API
  3. Create a new Chat Model Option on your Khoj admin panel.
    • Name: llama3 (replace with the name of your local model)
    • Model Type: Openai
    • Openai Config: <the proxy config you created in step 2>
    • Max prompt size: 2000 (replace with the max prompt size of your model)
    • Tokenizer: Do not set for OpenAI, mistral, llama3 based models
  4. Go to your config and select the model you just created in the chat model dropdown.