To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road.
Dev instructions for using the next.js setup are in the added README.
This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up.
This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
This will allow troubleshooting by getting the actual khoj version
being used. Previously it was always set to a static 0.0.0 version
Command to build Khoj docker image with dynamically set current app version:
`docker-compose build server --build-arg VERSION=$(pipx run hatch version)'
- Our pypi package currently does not work because the django app and associated database is not included. To remedy this issue, move the app into the src/khoj folder. This has the added benefit of improved organization of the codebase, as all server related code is now in a single folder
- Update associated file paths and system references
- Partition configuration for indexing local data based on user accounts
- Store indexed data in an underlying postgres db using the `pgvector` extension
- Add migrations for all relevant user data and embeddings generation. Very little performance optimization has been done for the lookup time
- Apply filters using SQL queries
- Start removing many server-level configuration settings
- Configure GitHub test actions to run during any PR. Update the test action to run in a containerized environment with a DB.
- Update the Docker image and docker-compose.yml to work with the new application design
Just use a random static version for Khoj on the Docker as otherwise
the hatch vcs dynamic versioning requires the .git directory in the
docker image too
- Previous state
Ideally docker image should use latest app code available locally.
But this is better than the previous state where the latest Docker
image was being built using older khoj package published to pypi
This would happen because the workflow to publish the khoj-assistant
pypi package runs in parallel to the dockerize workflow so the latest
khoj pypi package isn't published before the latest docker image is
built on master
- Updated state
Now at least the docker image published via the dockerize github
workflow will be built using the latest khoj code on github
- Instead of building the package locally like before
The issue started since moving to dynamic git based versioning with hatch-vcs
This should reduce image size of docker builds too
- Also move to ubuntu image since pyqt6 builds available on it, so do
not need to build it locally for image
- This s
- Pillow already supports reading XMP metadata from Images
- Removes need to maintain my fork of unmaintained PyExiftool
- This also removes dependency on system Exiftool package for
XMP metadata extraction
- Add test to verify XMP metadata extracted from test images
- Remove references to Exiftool from Documentation
- Reduce size of app copied to container by adding unneeded
directories to .dockerignore. E.g Ignore docs, tests, pip build, dist
- Update MANIFEST.ini to include web, emacs interface directories
Web directory is used by the web interface which will be exposed by
the docker container
- Use /app as the working directory
- Clarify comment to explain why the ENTRYPOINT is constructed as it is
- Move explanations for the argument to docker-compose, where it's set
- Copy required artifacts from the first build image into the subsequent one (exiftool)
- Add a Dockerfile which uses an Ubuntu image to install relevant dependencies (exif) and uses a Miniconda image for setting up/reusing the conda environment
- Add a dummy docker-compose file
