Commit graph

46 commits

Author SHA1 Message Date
Debanjum
fe09df66b7 Make code sandbox container url accessible to Khoj container in docker compose 2024-12-11 01:14:26 -08:00
Debanjum
4bc5c1357a Upgrade server, documentation dependencies. Spell fix docker-compose.yml 2024-12-10 15:47:47 -08:00
sabaimran
3b9f4c4356 Correct negative for running prod image locally 2024-12-08 19:55:35 -08:00
sabaimran
19832a3ed0 Add note to uncomment line when using the prod image 2024-12-05 18:16:01 -08:00
Debanjum
4a7f5d1abe Set API keys in docker-compose.yml to enable web search, scrape tools 2024-11-17 15:34:36 -08:00
Debanjum
69ef6829c1 Simplify integrating Ollama, OpenAI proxies with Khoj on first run
- Integrate with Ollama or other openai compatible APIs by simply
  setting `OPENAI_API_BASE' environment variable in docker-compose etc.
- Update docs on integrating with Ollama, openai proxies on first run
- Auto populate all chat models supported by openai compatible APIs
- Auto set vision enabled for all commercial models

- Minor
  - Add huggingface cache to khoj_models volume. This is where chat
  models and (now) sentence transformer models are stored by default
  - Reduce verbosity of yarn install of web app. Otherwise hit docker
  log size limit & stops showing remaining logs after web app install
  - Suggest `ollama pull <model_name>` to start it in background
2024-11-17 02:08:20 -08:00
Debanjum
e1d0015248 Allow disabling Khoj telemetry via KHOJ_TELEMETRY_DISABLE env var 2024-11-11 19:17:39 -08:00
sabaimran
ffa7f95559 Add template for a code sandbox to the docker-compose configuration 2024-11-01 11:50:58 -07:00
Debanjum Singh Solanky
91c76d4152 Intelligently initialize a decent default set of chat model options
Given the LLM landscape is rapidly changing, providing a good default
set of options should help reduce decision fatigue to get started

Improve initialization flow during first run
- Set Google, Anthropic Chat models too
  Previously only Offline, Openai chat models could be set during init

- Add multiple chat models for each LLM provider
  Interactively set a comma separated list of models for each provider

- Auto add default chat models for each provider in non-interactive
  model if the {OPENAI,GEMINI,ANTHROPIC}_API_KEY env var is set

- Do not ask for max_tokens, tokenizer for offline models during
  initialization. Use better defaults inferred in code instead

- Explicitly set default chat model to use
  If unset, it implicitly defaults to using the first chat model.
  Make it explicit to reduce this confusion

Resolves 
2024-09-19 20:32:08 -07:00
Debanjum Singh Solanky
f177723711 Add default server configuration on first run in non-interactive mode
This should configure Khoj with decent default configurations via
Docker and avoid needing to configure Khoj via admin page to start
using dockerized Khoj

Update default max prompt size set during khoj initialization
as online chat model are cheaper and offline chat models have larger
context now
2024-09-19 15:12:55 -07:00
LV Nilesh
77cc1cd42f
Update docker-compose.yml
Handles deprecation of version reference
2024-08-25 17:05:47 -07:00
Debanjum Singh Solanky
732332a3c5 Spell fix s/e.g/e.g./ across code, tests and docs 2024-06-24 15:24:45 +05:30
sabaimran
6d10f98498 Add additional lines for KHOJ_NO_HTTPS and KHOJ_DOMAIN in the docker-compose 2024-06-01 21:48:43 +05:30
sabaimran
880368635e Set default value of KHOJ_DEBUG to False in the docker-compose file 2024-03-01 21:51:13 +05:30
Debanjum Singh Solanky
795ec9eb55 Add KHOJ_prefix to server admin credentials environment variables 2023-11-14 16:13:13 -08:00
sabaimran
75e5a6b6de Remove all the example mounted volumes as they're no longer required in the new architecture 2023-11-14 12:31:24 -08:00
sabaimran
20ce3d0c78 Update default docker compose configuration with Khoj local mode 2023-11-14 12:21:26 -08:00
sabaimran
5f3f6b7c61
[Multi-User Part 5]: Add a production Docker file and use a gunicorn configuration with it ()
- Add a productionized setup for the Khoj server using `gunicorn` with multiple workers for handling requests
- Add a new Dockerfile meant for production config at `ghcr.io/khoj-ai/khoj:prod`; the existing Docker config should remain the same
2023-10-26 13:15:31 -07:00
sabaimran
216acf545f
[Multi-User Part 1]: Enable storage of settings for plaintext files based on user account ()
- Partition configuration for indexing local data based on user accounts
- Store indexed data in an underlying postgres db using the `pgvector` extension
- Add migrations for all relevant user data and embeddings generation. Very little performance optimization has been done for the lookup time
- Apply filters using SQL queries
- Start removing many server-level configuration settings
- Configure GitHub test actions to run during any PR. Update the test action to run in a containerized environment with a DB.
- Update the Docker image and docker-compose.yml to work with the new application design
2023-10-26 09:42:29 -07:00
sabaimran
343854752c
Improve docker builds for local hosting ()
* Remove GPT4All dependency in pyproject.toml and use multiplatform builds in the dockerization setup in GH actions
* Move configure_search method into indexer
* Add conditional installation for gpt4all
* Add hint to go to localhost:42110 in the docs. Addresses 
2023-09-08 17:07:26 -07:00
Debanjum Singh Solanky
28df08b907 Fix configure openai processor for khoj docker
Store khoj search models and embeddings in default location in docker
container under /root/.khoj
2023-07-30 02:07:33 -07:00
Debanjum Singh Solanky
f664a74e77 Update Khoj server to run on non standard port, 42110 instead of 8000
Resolves 
2023-07-10 21:27:58 -07:00
Debanjum Singh Solanky
0f993b332e Drop support for Ledger as a separate content type
Khoj will soon get a generic text indexing content type. This along
with a file filter should suffice for searching through Ledger
transactions, if required.

Having a specific content type for niche use-case like ledger isn't
useful. Removing unused content types will reduce khoj code to manage.
2023-07-02 16:57:49 -07:00
Debanjum Singh Solanky
c9db5321e7 Remove unused org-music as an indexable content type from Khoj
Org-music was just a custom content type that worked with org-music.
It was mostly only useful for me.

Cleaning up that code will reduce number of content types for khoj to
manage.
2023-07-02 16:21:21 -07:00
sabaimran
f0f6390366 Make --no-gui the default behavior of Khoj and update corresponding documentation 2023-07-01 19:07:59 -07:00
Debanjum Singh Solanky
69d4fa6525 Rename project links across repo from debanjum/khoj to khoj-ai/khoj 2023-06-21 00:13:21 -07:00
Debanjum Singh Solanky
acd14a5e41 Wire up PDF to jsonl processor to Khoj server layer (API, config)
- Specify PDF content to index via khoj.yml
- Index PDF content on app start, reconfigure
- Expose PDF as a search type via API
2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky
051f0e3fb5 Add, configure and run pre-commit locally and in test workflow 2023-02-17 13:31:36 -06:00
Debanjum Singh Solanky
5d3aeba22f Use --no-gui flag on starting Khoj from docker-compose
As the GUI wouldn't work when run from a docker container
2022-09-08 10:37:39 +03:00
Debanjum Singh Solanky
a876b652d8 Rename khoj_sample.yml to more specific khoj_docker.yml
- Update docker-compose.yml to start container using khoj_docker.yml
- Use /data/org in input-filter for content-type > org
2022-08-04 22:42:05 +03:00
Debanjum Singh Solanky
f50f343f73 Rename org-mode test data directory to more specific org/ from notes/ 2022-08-04 22:29:57 +03:00
Debanjum Singh Solanky
84adf0c568 Use published docker image to run khoj service using docker-compose 2022-08-04 02:17:01 +03:00
Debanjum Singh Solanky
a4eb55dd00 Rename khoj config yml file to follow more specific khoj*.yml pattern
- That is, sample_config.yml is renamed to khoj_sample.yml
- This makes the application config filename less generic,
  more easily identifiable with the application
- Update docs, app accordingly
2022-08-03 12:06:55 +03:00
Debanjum Singh Solanky
1f4b5ac112 Create test markdown files. Use them in sample config, docker-compose 2022-07-21 22:09:44 +04:00
Saba
2eb44c7a64 Correct syntax of memory limit in docker-compose.yml 2022-07-06 20:07:11 -04:00
Saba
0f88abd219 Allocate 8GB of memory to docker container. Adjust path to Dockerfile in Github action 2022-07-04 14:01:59 -04:00
Saba
092d0f2f21 Move Dockerfile to project root to avoid permissions issues. Allocate more memory to docker-compose to avoid OOM 2022-07-04 12:33:55 -04:00
Debanjum Singh Solanky
c31abad0a6 Mount embeddings to /data/embeddings for directory naming consistency
- Keeps directory paths consistent between host and container volumes
- Consistency simplifies documentation and updates required to setup
  sample_config.yml for local installation
2022-01-29 03:24:02 -05:00
Debanjum Singh Solanky
b0067fc32e Store docker, conda, semantic-search configuration in a config directory
- Improves organization of config files required for application
- Declutters the application root directory from configs
2022-01-29 02:41:11 -05:00
Debanjum Singh Solanky
3e889760c7 Merge sample_config, docker_sample_config yml into a single sample_config.yml
- Update readme to indicate how to update the new sample_config to run on test data
2022-01-29 01:32:12 -05:00
Debanjum Singh Solanky
6ed667aed0 Add Troubleshooting Section, Minor Fixes to Readme 2022-01-29 00:11:40 -05:00
Saba
1ba7fa66e5 Update README and default folders in docker_sample_config.yml
- Add instruction to using Docker with README
- Use the ./tests/data folder in docker_sample_conifg.yml so it can work right away for users
2022-01-28 23:20:50 -05:00
Saba
9802023c79 Clean up docker-compose
- Mount the local directory to /app
- Reformat the file paths to generically indicate what their purpose is
- Add comments to assist users who wasnt to modify properties themselves
2022-01-24 14:10:18 -05:00
Saba
66d08ab5df Rename web to server in docker-compose.yml 2022-01-24 00:14:01 -05:00
Saba
77fa8718d9 Working example with docker-compose
Still need quite a bit of clean-up, but this adds a working docker-compose + Dockerfile setup
2022-01-23 23:44:38 -05:00
Saba
875188dc6f Initialize working on to add Docker support
- Add a Dockerfile which uses an Ubuntu image to install relevant dependencies (exif) and uses a Miniconda image for setting up/reusing the conda environment
- Add a dummy docker-compose file
2022-01-23 14:57:28 -05:00