Add and update documentation for setting up khoj with an openai proxy server or offline llm

documentation/docs/data-sources/share_your_data.md

@@ -0,0 +1,16 @@
+---
+sidebar_position: 0
+keywords: ["upload data", "upload files", "share data", "share files", "pdf ai", "ai for pdf", "ai for documents", "ai for files", "local ai pdf", "local ai documents", "local ai files"]
+
+---
+
+# Upload your data
+
+There are several ways you can get started with sharing your data with the Khoj AI.
+
+- Drag and drop your documents via [the web UI](/clients/web/#upload-documents). This is best if you have a one-off document you need to interact with.
+- Use the desktop app to [upload and sync your documents](/clients/desktop). This is best if you have a lot of documents on your computer or you need the docs to stay in sync.
+- Setup the sync options for either [Obsidian](/clients/obsidian) or [Emacs](/clients/emacs) to automatically sync your documents with Khoj. This is best if you are already using these tools and want to leverage Khoj's AI capabilities.
+- Configure your [Notion](/data-sources/notion_integration) or [Github](/data-sources/github_integration) to sync with Khoj. By providing your credentials, you can keep the data synced in the background.
+
+![demo of dragging and dropping a file](https://khoj-web-bucket.s3.amazonaws.com/drag_drop_file.gif)

documentation/docs/features/all_features.md

@@ -29,6 +29,6 @@ Khoj is available as a [Desktop app](/clients/desktop), [Emacs package](/clients
 ![](/img/khoj_clients.svg ':size=400px')
 
 ### Supported Data Sources
-Khoj can understand your org-mode, markdown, PDF, plaintext files, [Github projects](/online-data-sources/github_integration) and [Notion pages](/online-data-sources/notion_integration).
+Khoj can understand your org-mode, markdown, PDF, plaintext files, [Github projects](/data-sources/github_integration) and [Notion pages](/data-sources/notion_integration).
 
 ![](/img/khoj_datasources.svg ':size=200px')

documentation/docs/get-started/overview.md

@@ -1,6 +1,7 @@
 ---
 sidebar_position: 0
 slug: /
+keywords: ["khoj", "khoj ai", "khoj docs", "khoj documentation", "khoj features", "khoj overview", "khoj quickstart", "khoj chat", "khoj search", "khoj cloud", "khoj self-host", "khoj setup", "open source ai", "local llm", "ai copilot", "second brain ai", "ai search engine"]
 ---
 
 # Overview
@@ -28,7 +29,7 @@ Welcome to the Khoj Docs! This is the best place to get setup and explore Khoj's
 - Khoj is an open source, personal AI
 - You can [chat](/features/chat) with it about anything. It'll use files you shared with it to respond, when relevant
 - Quickly [find](/features/search) relevant notes and documents using natural language
-- It understands pdf, plaintext, markdown, org-mode files, [notion pages](/online-data-sources/notion_integration) and [github repositories](/online-data-sources/github_integration)
+- It understands pdf, plaintext, markdown, org-mode files, [notion pages](/data-sources/notion_integration) and [github repositories](/data-sources/github_integration)
 - Access it from your [Emacs](/clients/emacs), [Obsidian](/clients/obsidian), [Web browser](/clients/web) or the [Khoj Desktop app](/clients/desktop)
 - Use [cloud](https://app.khoj.dev/login) to access your Khoj anytime from anywhere, [self-host](/get-started/setup) on consumer hardware for privacy
 

documentation/docs/get-started/setup.mdx

@@ -199,15 +199,24 @@ To disable HTTPS, set the `KHOJ_NO_HTTPS` environment variable to `True`. This c
 
 ### 2. Configure
 1. Go to http://localhost:42110/server/admin and login with your admin credentials.
-    1. Go to [OpenAI settings](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/) in the server admin settings to add an OpenAI processor conversation config. This is where you set your API key. Alternatively, you can go to the [offline chat settings](http://localhost:42110/server/admin/database/offlinechatprocessorconversationconfig/) and simply create a new setting with `Enabled` set to `True`.
+#### Configure Chat Model
-    2. Go to the ChatModelOptions if you want to add additional models for chat.
+##### Configure OpenAI or a custom OpenAI-compatible proxy server
-       - Set the `chat-model` field to a supported chat model[^1] of your choice. For example, you can specify `gpt-4-turbo-preview` if you're using OpenAI or `NousResearch/Hermes-2-Pro-Mistral-7B-GGUF` if you're using offline chat.
+1. Go to the [OpenAI settings](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/) in the server admin settings to add an OpenAI processor conversation config. This is where you set your API key and server API base URL. The API base URL is optional - it's only relevant if you're using another OpenAI-compatible proxy server.
-       - Make sure to set the `model-type` field to `OpenAI` or `Offline` respectively.
+2. Go over to configure your [chat model options](http://localhost:42110/server/admin/database/chatmodeloptions/). Set the `chat-model` field to a supported chat model[^1] of your choice. For example, you can specify `gpt-4-turbo-preview` if you're using OpenAI.
-       - The `tokenizer` and `max-prompt-size` fields are optional. Set them only when using a non-standard model (i.e not mistral, gpt or llama2 model).
+    - Make sure to set the `model-type` field to `OpenAI`.
+    - The `tokenizer` and `max-prompt-size` fields are optional. Set them only if you're sure of the tokenizer or token limit for the model you're using. Contact us if you're unsure what to do here.
+
+##### Configure Offline Chat
+1. No need to setup a conversation processor config!
+2. Go over to configure your [chat model options](http://localhost:42110/server/admin/database/chatmodeloptions/). Set the `chat-model` field to a supported chat model[^1] of your choice. For example, we recommend `NousResearch/Hermes-2-Pro-Mistral-7B-GGUF`, but [any gguf model on huggingface](https://huggingface.co/models?library=gguf) should work.
+  - Make sure to set the `model-type` to `Offline`. Do not set `openai config`.
+  - The `tokenizer` and `max-prompt-size` fields are optional. Set them only when using a non-standard model (i.e not mistral, gpt or llama2 model) when you know the token limit.
+
+#### Share your data
 1. Select files and folders to index [using the desktop client](/get-started/setup#2-download-the-desktop-client). When you click 'Save', the files will be sent to your server for indexing.
     - Select Notion workspaces and Github repositories to index using the web interface.
 
-[^1]: Khoj, by default, can use [OpenAI GPT3.5+ chat models](https://platform.openai.com/docs/models/overview) or [GGUF chat models](https://huggingface.co/models?library=gguf). See [this section](/miscellaneous/advanced#use-openai-compatible-llm-api-server-self-hosting) to use non-standard chat models
+[^1]: Khoj, by default, can use [OpenAI GPT3.5+ chat models](https://platform.openai.com/docs/models/overview) or [GGUF chat models](https://huggingface.co/models?library=gguf). See [this section](/miscellaneous/advanced#use-openai-compatible-llm-api-server-self-hosting) on how to locally use OpenAI-format compatible proxy servers.
 
 :::tip[Note]
 Using Safari on Mac? You might not be able to login to the admin panel. Try using Chrome or Firefox instead.