Commit graph

43 commits

Author SHA1 Message Date
Debanjum
4a5646c8da Cache docker layers, nextjs builds in dockerize github workflow 2024-11-24 21:06:22 -08:00
Debanjum
f51e0f7859 Make Next.js create production builds of web app for Docker images 2024-11-24 13:59:40 -08:00
Debanjum
78d8ca49ec Skip Nvidia GPU python packages during Server install in Dockerfiles 2024-11-24 13:59:39 -08:00
Debanjum
37887a175a Speed up Docker image builds using multi-stage parallel pipelines
Decouple web app, server builds in parallel to speed up Docker
builds
2024-11-24 12:48:30 -08:00
Debanjum
04bb3d6f15 Fix libmusl error using pre-built llama-cpp-python wheel via Docker
Seems like llama-cpp-python pre-built wheels need libmusl. Otherwise
you run into runtime errors on Khoj startup via Docker.
2024-11-23 18:46:44 -08:00
Debanjum
15ae22bdcf Use pre-built llama-cpp-python wheel in Khoj docker images
Reduces build time and resolves FileNotFoundError 'ninja' during
llama-cpp-python local build.
2024-11-23 14:38:07 -08:00
Debanjum
69ef6829c1 Simplify integrating Ollama, OpenAI proxies with Khoj on first run
- Integrate with Ollama or other openai compatible APIs by simply
  setting `OPENAI_API_BASE' environment variable in docker-compose etc.
- Update docs on integrating with Ollama, openai proxies on first run
- Auto populate all chat models supported by openai compatible APIs
- Auto set vision enabled for all commercial models

- Minor
  - Add huggingface cache to khoj_models volume. This is where chat
  models and (now) sentence transformer models are stored by default
  - Reduce verbosity of yarn install of web app. Otherwise hit docker
  log size limit & stops showing remaining logs after web app install
  - Suggest `ollama pull <model_name>` to start it in background
2024-11-17 02:08:20 -08:00
Debanjum Singh Solanky
1e8ce52d98 Reduce size of Khoj Docker images by removing layers and caches
- Align Dockerfile and prod.Dockerfile code
- Reduce Docker image size by 25% by reducing Docker layers and
  removing package caches
2024-09-29 04:06:35 -07:00
sabaimran
e694c82343
Fix Docker build issues with yarn / next /node ()
* Rollback node version being installed from nodesource to node 20
2024-07-19 19:11:29 +05:30
sabaimran
1af9dbb083 Switch node/yarn install steps to use more native installation patterns 2024-07-19 17:10:08 +05:30
sabaimran
6d5ca5a3e1 yarn clean cache before build 2024-07-19 16:06:38 +05:30
sabaimran
7f0d1bd414 Add verbose logs when outputing yarn install steps 2024-07-19 15:48:43 +05:30
Debanjum Singh Solanky
30d60aaae9 Add, fix Khoj Docker container labels 2024-07-17 10:41:17 +05:30
kxnarak
43413cd21f add dependencies required by the RapidOCR python package 2024-07-08 18:26:19 +05:30
sabaimran
a53178cab9
Add developer support for using next.js to serve generated static files ()
To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road.

Dev instructions for using the next.js setup are in the added README.

This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up.

This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
2024-06-22 20:12:41 +05:30
Debanjum Singh Solanky
4007c871ae Remove unused git dependency from Docker images 2024-02-16 17:41:09 +05:30
Debanjum Singh Solanky
08012c71b1 Update Dockerfile with swig system package required by PyMuPDF 2024-01-17 19:24:27 +05:30
Debanjum Singh Solanky
d3d47dce0b Allow setting Khoj app version during docker build via build-args
This will allow troubleshooting by getting the actual khoj version
being used. Previously it was always set to a static 0.0.0 version

Command to build Khoj docker image with dynamically set current app version:
`docker-compose build server --build-arg VERSION=$(pipx run hatch version)'
2023-12-22 16:47:13 +05:30
sabaimran
5fac39afed Fix PYTHONPATH reference in order to maintain appropriate package imports 2023-11-22 20:35:11 -08:00
sabaimran
a474c31e02 Move the django app into the src/khoj folder for better organization and functionality
- Our pypi package currently does not work because the django app and associated database is not included. To remedy this issue, move the app into the src/khoj folder. This has the added benefit of improved organization of the codebase, as all server related code is now in a single folder
- Update associated file paths and system references
2023-11-21 10:56:04 -08:00
sabaimran
216acf545f
[Multi-User Part 1]: Enable storage of settings for plaintext files based on user account ()
- Partition configuration for indexing local data based on user accounts
- Store indexed data in an underlying postgres db using the `pgvector` extension
- Add migrations for all relevant user data and embeddings generation. Very little performance optimization has been done for the lookup time
- Apply filters using SQL queries
- Start removing many server-level configuration settings
- Configure GitHub test actions to run during any PR. Update the test action to run in a containerized environment with a DB.
- Update the Docker image and docker-compose.yml to work with the new application design
2023-10-26 09:42:29 -07:00
sabaimran
c125995d94
[Multi-User]: Part 0 - Add support for logging in with Google ()
* Add concept of user authentication to the request session via GoogleUser
2023-10-14 19:39:13 -07:00
sabaimran
efe5e09c3a Use jammy for docker base image due to dependency issue with arm64 image 2023-09-18 15:38:18 -07:00
sabaimran
6df728c445 Move bash command in Dockerfile into single line 2023-09-18 15:13:11 -07:00
Debanjum Singh Solanky
9c76150895 Migrate from PyQT6 to PySide6 2023-07-11 18:43:44 -07:00
Debanjum Singh Solanky
6308388dfc Install Khoj on Docker from local app instead of pulling from github
Just use a random static version for Khoj on the Docker as otherwise
the hatch vcs dynamic versioning requires the .git directory in the
docker image too
2023-07-11 00:41:05 -07:00
Debanjum Singh Solanky
802472cd99 Reduce Khoj Docker image size by 2Gb by not caching pip packages
Resolve 
2023-07-10 23:27:02 -07:00
Debanjum Singh Solanky
5da6a5e669 Build docker image using latest khoj from git master
- Previous state
  Ideally docker image should use latest app code available locally.
  But this is better than the previous state where the latest Docker
  image was being built using older khoj package published to pypi

  This would happen because the workflow to publish the khoj-assistant
  pypi package runs in parallel to the dockerize workflow so the latest
  khoj pypi package isn't published before the latest docker image is
  built on master

- Updated state
  Now at least the docker image published via the dockerize github
  workflow will be built using the latest khoj code on github
2023-06-26 20:16:07 -07:00
Debanjum Singh Solanky
69d4fa6525 Rename project links across repo from debanjum/khoj to khoj-ai/khoj 2023-06-21 00:13:21 -07:00
Debanjum Singh Solanky
f57d7bf5ad Use pypi khoj to fix docker builds and dockerize github workflow
- Instead of building the package locally like before
  The issue started since moving to dynamic git based versioning with hatch-vcs
  This should reduce image size of docker builds too

- Also move to ubuntu image since pyqt6 builds available on it, so do
  not need to build it locally for image

- This s
2023-02-19 01:57:01 -06:00
Debanjum Singh Solanky
47c2cc63e1 Automate uploading Obsidian artifacts to new releases 2023-02-17 19:57:44 -06:00
Debanjum Singh Solanky
bf1ae038cb Get XMP metadata from image using Pillow. Remove ExifTool dependency
- Pillow already supports reading XMP metadata from Images
- Removes need to maintain my fork of unmaintained PyExiftool
  - This also removes dependency on system Exiftool package for
    XMP metadata extraction
- Add test to verify XMP metadata extracted from test images
- Remove references to Exiftool from Documentation
2022-09-16 00:48:45 +03:00
Debanjum Singh Solanky
17354aaffd Install pyqt system package in Docker image to get qt dependencies
Otherwise app start fails with pyqt package import related errors.
See  for bug
2022-09-08 10:39:11 +03:00
Debanjum Singh Solanky
84adf0c568 Use published docker image to run khoj service using docker-compose 2022-08-04 02:17:01 +03:00
Debanjum Singh Solanky
ee65809dc6 Update Dockerfile to use Pip instead of Conda to install application
- Reduce size of app copied to container by adding unneeded
  directories to .dockerignore. E.g Ignore docs, tests, pip build, dist

- Update MANIFEST.ini to include web, emacs interface directories
  Web directory is used by the web interface which will be exposed by
  the docker container
2022-08-04 00:14:25 +03:00
Debanjum Singh Solanky
732b2d287f Give the project a short, less generic name. Rename it to Khoj
- Semantic Search was just a placeholder used to test the idea out
  Didn't want to get into naming at that point of time
2022-07-19 18:26:16 +04:00
Saba
07a56c4ab6 Add specific version for Python packages and downgrade miniconda Docker image to potentially fix build issues 2022-07-04 18:01:55 -04:00
Saba
092d0f2f21 Move Dockerfile to project root to avoid permissions issues. Allocate more memory to docker-compose to avoid OOM 2022-07-04 12:33:55 -04:00
Debanjum Singh Solanky
b0067fc32e Store docker, conda, semantic-search configuration in a config directory
- Improves organization of config files required for application
- Declutters the application root directory from configs
2022-01-29 02:41:11 -05:00
Saba
52e701b3c2 Simplify Dockerfile by removing multibuild
- Install exiftool dependency directly in the miniconda image
2022-01-24 21:54:10 -05:00
Saba
4ae8c15170 Clean the Dockerfile
- Use /app as the working directory
- Clarify comment to explain why the ENTRYPOINT is constructed as it is
- Move explanations for the argument to docker-compose, where it's set
- Copy required artifacts from the first build image into the subsequent one (exiftool)
2022-01-24 14:08:55 -05:00
Saba
77fa8718d9 Working example with docker-compose
Still need quite a bit of clean-up, but this adds a working docker-compose + Dockerfile setup
2022-01-23 23:44:38 -05:00
Saba
875188dc6f Initialize working on to add Docker support
- Add a Dockerfile which uses an Ubuntu image to install relevant dependencies (exif) and uses a Miniconda image for setting up/reusing the conda environment
- Add a dummy docker-compose file
2022-01-23 14:57:28 -05:00