sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-18 18:47:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 23
khoj/prod.Dockerfile

64 lines
2.1 KiB
Text
Raw Normal View History

Reduce size of Khoj Docker images by removing layers and caches - Align Dockerfile and prod.Dockerfile code - Reduce Docker image size by 25% by reducing Docker layers and removing package caches
2024-09-29 10:52:11 +00:00
# syntax=docker/dockerfile:1
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
FROM ubuntu:jammy AS base
Reduce size of Khoj Docker images by removing layers and caches - Align Dockerfile and prod.Dockerfile code - Reduce Docker image size by 25% by reducing Docker layers and removing package caches
2024-09-29 10:52:11 +00:00
LABEL homepage="https://khoj.dev"
LABEL repository="https://github.com/khoj-ai/khoj"
Switch node/yarn install steps to use more native installation patterns
2024-07-19 11:40:08 +00:00
LABEL org.opencontainers.image.source="https://github.com/khoj-ai/khoj"
Reduce size of Khoj Docker images by removing layers and caches - Align Dockerfile and prod.Dockerfile code - Reduce Docker image size by 25% by reducing Docker layers and removing package caches
2024-09-29 10:52:11 +00:00
LABEL org.opencontainers.image.description="Your second brain, containerized for multi-user, cloud deployment"
[Multi-User Part 5]: Add a production Docker file and use a gunicorn configuration with it (#514) - Add a productionized setup for the Khoj server using `gunicorn` with multiple workers for handling requests - Add a new Dockerfile meant for production config at `ghcr.io/khoj-ai/khoj:prod`; the existing Docker config should remain the same
2023-10-26 20:15:31 +00:00
# Install System Dependencies
Reduce size of Khoj Docker images by removing layers and caches - Align Dockerfile and prod.Dockerfile code - Reduce Docker image size by 25% by reducing Docker layers and removing package caches
2024-09-29 10:52:11 +00:00
RUN apt update -y && apt -y install \
python3-pip \
libsqlite3-0 \
ffmpeg \
libsm6 \
libxext6 \
swig \
Fix apt install for musl-dev in prod.Dockerfile
2024-11-24 05:05:30 +00:00
curl \
Fix libmusl error using pre-built llama-cpp-python wheel in prod Docker
2024-11-24 04:47:41 +00:00
# Required by llama-cpp-python pre-built wheels. See #1628
Fix apt install for musl-dev in prod.Dockerfile
2024-11-24 05:05:30 +00:00
musl-dev && \
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1 && \
# Clean up
apt clean && rm -rf /var/lib/apt/lists/*
[Multi-User Part 5]: Add a production Docker file and use a gunicorn configuration with it (#514) - Add a productionized setup for the Khoj server using `gunicorn` with multiple workers for handling requests - Add a new Dockerfile meant for production config at `ghcr.io/khoj-ai/khoj:prod`; the existing Docker config should remain the same
2023-10-26 20:15:31 +00:00
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
# Build Server
FROM base AS server-deps
Reduce size of Khoj Docker images by removing layers and caches - Align Dockerfile and prod.Dockerfile code - Reduce Docker image size by 25% by reducing Docker layers and removing package caches
2024-09-29 10:52:11 +00:00
WORKDIR /app
[Multi-User Part 5]: Add a production Docker file and use a gunicorn configuration with it (#514) - Add a productionized setup for the Khoj server using `gunicorn` with multiple workers for handling requests - Add a new Dockerfile meant for production config at `ghcr.io/khoj-ai/khoj:prod`; the existing Docker config should remain the same
2023-10-26 20:15:31 +00:00
COPY pyproject.toml .
COPY README.md .
Allow setting Khoj app version during docker build via build-args This will allow troubleshooting by getting the actual khoj version being used. Previously it was always set to a static 0.0.0 version Command to build Khoj docker image with dynamically set current app version: `docker-compose build server --build-arg VERSION=$(pipx run hatch version)'
2023-12-21 11:37:08 +00:00
ARG VERSION=0.0.0
Skip Nvidia GPU python packages during Server install in Dockerfiles
2024-11-24 06:24:45 +00:00
# use the pre-built llama-cpp-python, torch cpu wheel
ENV PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu https://abetlen.github.io/llama-cpp-python/whl/cpu"
# avoid downloading unused cuda specific python packages
ENV CUDA_VISIBLE_DEVICES=""
Allow setting Khoj app version during docker build via build-args This will allow troubleshooting by getting the actual khoj version being used. Previously it was always set to a static 0.0.0 version Command to build Khoj docker image with dynamically set current app version: `docker-compose build server --build-arg VERSION=$(pipx run hatch version)'
2023-12-21 11:37:08 +00:00
RUN sed -i "s/dynamic = \\[\"version\"\\]/version = \"$VERSION\"/" pyproject.toml && \
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
pip install --no-cache-dir .[prod]
[Multi-User Part 5]: Add a production Docker file and use a gunicorn configuration with it (#514) - Add a productionized setup for the Khoj server using `gunicorn` with multiple workers for handling requests - Add a new Dockerfile meant for production config at `ghcr.io/khoj-ai/khoj:prod`; the existing Docker config should remain the same
2023-10-26 20:15:31 +00:00
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
# Build Web App
FROM node:20-alpine AS web-app
Make Next.js create production builds of web app for Docker images
2024-11-24 20:59:17 +00:00
# Set build optimization env vars
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
Add developer support for using next.js to serve generated static files (#814) To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road. Dev instructions for using the next.js setup are in the added README. This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up. This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
2024-06-22 14:42:41 +00:00
WORKDIR /app/src/interface/web
Cache docker layers, nextjs builds in dockerize github workflow
2024-11-24 21:10:17 +00:00
# Install dependencies first (cache layer)
COPY src/interface/web/package.json src/interface/web/yarn.lock ./
RUN yarn install --frozen-lockfile
# Copy source and build
COPY src/interface/web/. ./
RUN yarn build
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
# Merge the Server and Web App into a Single Image
FROM base
ENV PYTHONPATH=/app/src
Add developer support for using next.js to serve generated static files (#814) To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road. Dev instructions for using the next.js setup are in the added README. This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up. This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
2024-06-22 14:42:41 +00:00
WORKDIR /app
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
COPY --from=server-deps /usr/local/lib/python3.10/dist-packages /usr/local/lib/python3.10/dist-packages
COPY --from=web-app /app/src/interface/web/out ./src/khoj/interface/built
COPY . .
RUN cd src && python3 khoj/manage.py collectstatic --noinput
Add developer support for using next.js to serve generated static files (#814) To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road. Dev instructions for using the next.js setup are in the added README. This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up. This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
2024-06-22 14:42:41 +00:00
[Multi-User Part 5]: Add a production Docker file and use a gunicorn configuration with it (#514) - Add a productionized setup for the Khoj server using `gunicorn` with multiple workers for handling requests - Add a new Dockerfile meant for production config at `ghcr.io/khoj-ai/khoj:prod`; the existing Docker config should remain the same
2023-10-26 20:15:31 +00:00
# Run the Application
# There are more arguments required for the application to run,
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
# but those should be passed in through the docker-compose.yml file.
[Multi-User Part 5]: Add a production Docker file and use a gunicorn configuration with it (#514) - Add a productionized setup for the Khoj server using `gunicorn` with multiple workers for handling requests - Add a new Dockerfile meant for production config at `ghcr.io/khoj-ai/khoj:prod`; the existing Docker config should remain the same
2023-10-26 20:15:31 +00:00
ARG PORT
EXPOSE ${PORT}
Reduce size of Khoj Docker images by removing layers and caches - Align Dockerfile and prod.Dockerfile code - Reduce Docker image size by 25% by reducing Docker layers and removing package caches
2024-09-29 10:52:11 +00:00
ENTRYPOINT ["gunicorn", "-c", "gunicorn-config.py", "src.khoj.main:app"]
Reference in a new issue Copy permalink