sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-30 19:03:01 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
khoj/documentation/docs/advanced/lmstudio.md
Debanjum Singh Solanky a1c362a4f7 Expose web, desktop settings page under /settings, not /configure 
- Update references to the settings page to use new url across docs
  and code
- Rename desktop and web settings page to settigns.html instead of
  config[ure].html
2024-07-17 13:17:29 +05:30

2.1 KiB
Raw Blame History

LM Studio

:::info This is only helpful for self-hosted users. If you're using Khoj Cloud, you're limited to our first-party models. :::

:::info Khoj natively supports local LLMs available on HuggingFace in GGUF format. Using an OpenAI API proxy with Khoj maybe useful for ease of setup, trying new models or using commercial LLMs via API. :::

LM Studio is a desktop app to chat with open-source LLMs on your local machine. LM Studio provides a neat interface for folks comfortable with a GUI.

LM Studio can expose an OpenAI API compatible server. This makes it possible to turn chat models from LM Studio into your personal AI agents with Khoj.

Setup

  1. Install LM Studio and download your preferred Chat Model
  2. Go to the Server Tab on LM Studio, Select your preferred Chat Model and Click the green Start Server button
  3. Create a new OpenAI Processor Conversation Config on your Khoj admin panel
    • Name: proxy-name
    • Api Key: any string
    • Api Base Url: http://localhost:1234/v1/ (default for LMStudio)
  4. Create a new Chat Model Option on your Khoj admin panel.
    • Name: llama3 (replace with the name of your local model)
    • Model Type: Openai
    • Openai Config: <the proxy config you created in step 3>
    • Max prompt size: 2000 (replace with the max prompt size of your model)
    • Tokenizer: Do not set for OpenAI, mistral, llama3 based models
  5. Create a new Server Chat Setting on your Khoj admin panel
    • Default model: <name of chat model option you created in step 4>
    • Summarizer model: <name of chat model option you created in step 4>
  6. Go to your config and select the model you just created in the chat model dropdown.