sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-18 18:47:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 23
khoj/Dockerfile

61 lines
2 KiB
Text
Raw Normal View History

Initialize working on #20 to add Docker support - Add a Dockerfile which uses an Ubuntu image to install relevant dependencies (exif) and uses a Miniconda image for setting up/reusing the conda environment - Add a dummy docker-compose file
2022-01-23 19:57:28 +00:00
# syntax=docker/dockerfile:1
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
FROM ubuntu:jammy AS base
Switch node/yarn install steps to use more native installation patterns
2024-07-19 11:40:08 +00:00
LABEL homepage="https://khoj.dev"
LABEL repository="https://github.com/khoj-ai/khoj"
LABEL org.opencontainers.image.source="https://github.com/khoj-ai/khoj"
Reduce size of Khoj Docker images by removing layers and caches - Align Dockerfile and prod.Dockerfile code - Reduce Docker image size by 25% by reducing Docker layers and removing package caches
2024-09-29 10:52:11 +00:00
LABEL org.opencontainers.image.description="Your second brain, containerized for personal, local deployment."
Initialize working on #20 to add Docker support - Add a Dockerfile which uses an Ubuntu image to install relevant dependencies (exif) and uses a Miniconda image for setting up/reusing the conda environment - Add a dummy docker-compose file
2022-01-23 19:57:28 +00:00
Update Dockerfile to use Pip instead of Conda to install application - Reduce size of app copied to container by adding unneeded directories to .dockerignore. E.g Ignore docs, tests, pip build, dist - Update MANIFEST.ini to include web, emacs interface directories Web directory is used by the web interface which will be exposed by the docker container
2022-08-03 20:00:20 +00:00
# Install System Dependencies
Reduce size of Khoj Docker images by removing layers and caches - Align Dockerfile and prod.Dockerfile code - Reduce Docker image size by 25% by reducing Docker layers and removing package caches
2024-09-29 10:52:11 +00:00
RUN apt update -y && apt -y install \
python3-pip \
swig \
curl \
# Required by RapidOCR
libgl1 \
libglx-mesa0 \
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
libglib2.0-0 \
Fix libmusl error using pre-built llama-cpp-python wheel via Docker Seems like llama-cpp-python pre-built wheels need libmusl. Otherwise you run into runtime errors on Khoj startup via Docker.
2024-11-24 02:46:44 +00:00
# Required by llama-cpp-python pre-built wheels. See #1628
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
musl-dev && \
ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1 && \
# Clean up
apt clean && rm -rf /var/lib/apt/lists/*
add dependencies required by the RapidOCR python package
2024-07-08 12:50:46 +00:00
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
# Build Server
FROM base AS server-deps
Add developer support for using next.js to serve generated static files (#814) To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road. Dev instructions for using the next.js setup are in the added README. This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up. This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
2024-06-22 14:42:41 +00:00
WORKDIR /app
[Multi-User Part 1]: Enable storage of settings for plaintext files based on user account (#498) - Partition configuration for indexing local data based on user accounts - Store indexed data in an underlying postgres db using the `pgvector` extension - Add migrations for all relevant user data and embeddings generation. Very little performance optimization has been done for the lookup time - Apply filters using SQL queries - Start removing many server-level configuration settings - Configure GitHub test actions to run during any PR. Update the test action to run in a containerized environment with a DB. - Update the Docker image and docker-compose.yml to work with the new application design
2023-10-26 16:42:29 +00:00
COPY pyproject.toml .
COPY README.md .
Allow setting Khoj app version during docker build via build-args This will allow troubleshooting by getting the actual khoj version being used. Previously it was always set to a static 0.0.0 version Command to build Khoj docker image with dynamically set current app version: `docker-compose build server --build-arg VERSION=$(pipx run hatch version)'
2023-12-21 11:37:08 +00:00
ARG VERSION=0.0.0
Skip Nvidia GPU python packages during Server install in Dockerfiles
2024-11-24 06:24:45 +00:00
# use the pre-built llama-cpp-python, torch cpu wheel
ENV PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu https://abetlen.github.io/llama-cpp-python/whl/cpu"
# avoid downloading unused cuda specific python packages
ENV CUDA_VISIBLE_DEVICES=""
Allow setting Khoj app version during docker build via build-args This will allow troubleshooting by getting the actual khoj version being used. Previously it was always set to a static 0.0.0 version Command to build Khoj docker image with dynamically set current app version: `docker-compose build server --build-arg VERSION=$(pipx run hatch version)'
2023-12-21 11:37:08 +00:00
RUN sed -i "s/dynamic = \\[\"version\"\\]/version = \"$VERSION\"/" pyproject.toml && \
Install Khoj on Docker from local app instead of pulling from github Just use a random static version for Khoj on the Docker as otherwise the hatch vcs dynamic versioning requires the .git directory in the docker image too
2023-07-11 07:41:05 +00:00
pip install --no-cache-dir .
Initialize working on #20 to add Docker support - Add a Dockerfile which uses an Ubuntu image to install relevant dependencies (exif) and uses a Miniconda image for setting up/reusing the conda environment - Add a dummy docker-compose file
2022-01-23 19:57:28 +00:00
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
# Build Web App
FROM node:20-alpine AS web-app
Make Next.js create production builds of web app for Docker images
2024-11-24 20:59:17 +00:00
# Set build optimization env vars
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
COPY src/interface/web /app/src/interface/web
Add developer support for using next.js to serve generated static files (#814) To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road. Dev instructions for using the next.js setup are in the added README. This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up. This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
2024-06-22 14:42:41 +00:00
WORKDIR /app/src/interface/web
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
RUN yarn install --frozen-lockfile && \
yarn build
# Merge the Server and Web App into a Single Image
FROM base
ENV PYTHONPATH=/app/src
Add developer support for using next.js to serve generated static files (#814) To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road. Dev instructions for using the next.js setup are in the added README. This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up. This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
2024-06-22 14:42:41 +00:00
WORKDIR /app
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
COPY --from=server-deps /usr/local/lib/python3.10/dist-packages /usr/local/lib/python3.10/dist-packages
COPY --from=web-app /app/src/interface/web/out ./src/khoj/interface/built
COPY . .
RUN cd src && python3 khoj/manage.py collectstatic --noinput
Add developer support for using next.js to serve generated static files (#814) To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road. Dev instructions for using the next.js setup are in the added README. This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up. This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
2024-06-22 14:42:41 +00:00
Update Dockerfile to use Pip instead of Conda to install application - Reduce size of app copied to container by adding unneeded directories to .dockerignore. E.g Ignore docs, tests, pip build, dist - Update MANIFEST.ini to include web, emacs interface directories Web directory is used by the web interface which will be exposed by the docker container
2022-08-03 20:00:20 +00:00
# Run the Application
# There are more arguments required for the application to run,
Speed up Docker image builds using multi-stage parallel pipelines Decouple web app, server builds in parallel to speed up Docker builds
2024-11-24 05:13:51 +00:00
# but those should be passed in through the docker-compose.yml file.
Update Dockerfile to use Pip instead of Conda to install application - Reduce size of app copied to container by adding unneeded directories to .dockerignore. E.g Ignore docs, tests, pip build, dist - Update MANIFEST.ini to include web, emacs interface directories Web directory is used by the web interface which will be exposed by the docker container
2022-08-03 20:00:20 +00:00
ARG PORT
EXPOSE ${PORT}
[Multi-User Part 1]: Enable storage of settings for plaintext files based on user account (#498) - Partition configuration for indexing local data based on user accounts - Store indexed data in an underlying postgres db using the `pgvector` extension - Add migrations for all relevant user data and embeddings generation. Very little performance optimization has been done for the lookup time - Apply filters using SQL queries - Start removing many server-level configuration settings - Configure GitHub test actions to run during any PR. Update the test action to run in a containerized environment with a DB. - Update the Docker image and docker-compose.yml to work with the new application design
2023-10-26 16:42:29 +00:00
ENTRYPOINT ["python3", "src/khoj/main.py"]
Reference in a new issue Copy permalink