khoj/.github/workflows/pypi.yml

name: pypi

on:
  push:
    tags:
      - "*"
    branches:
      - 'master'
    paths:
      - src/khoj/**
      - src/interface/web/**
      - pyproject.toml
      - .github/workflows/pypi.yml
  pull_request:
    branches:
      - 'master'
    paths:
      - src/khoj/**
      - src/interface/web/**
      - pyproject.toml
      - .github/workflows/pypi.yml

jobs:
  publish:
    name: Publish Python Package to PyPI
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Set up Python 3.11
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: ⬇️ Install Application
        run: python -m pip install --upgrade pip && pip install --upgrade .

      - name: Install the Next.js application
        run: |
          yarn install
        working-directory: src/interface/web

      - name: Build & export static Next.js app to Django static assets
        run: |
          yarn ciexport
        working-directory: src/interface/web

      - name: ⚙️ Build Python Package
        run: |
          # Setup Environment for Reproducible Builds
          export PYTHONHASHSEED=42
          export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
          rm -rf dist

          # Build PyPi Package
          pipx run build

      - name: 🌡️ Validate Python Package
        run: |
          # Validate PyPi Package
          pipx run check-wheel-contents dist/*.whl --ignore W004
          pipx run twine check dist/*

      - name: ⏫ Upload Python Package Artifacts
        uses: actions/upload-artifact@v3
        with:
          name: khoj-assistant
          path: dist/*.whl

      - name: 📦 Publish Python Package to PyPI
        if: startsWith(github.ref, 'refs/tags') || github.ref == 'refs/heads/master'
        uses: pypa/gh-action-pypi-publish@v1.8.14
Improve workflow to publish khoj to pypi - Use emoji's to improve visual indicator of action step - Rename to pypi instead of the more ambiguous publish name Publish could mean publish docker image, publish to pypi, MELPA or Obsidian plugin - Update workflow badge, link pypi badge to khoj pypi package page - Use pypa official github action to upload package to (test) pypi instead of doing it manually using twine - Upload python package artifact for easier access for testing. As uploading to testpypi doesn't work for PRs by others from forked repos 2023-02-15 02:47:42 +01:00			`name: pypi`
Create Workflow to Publish Application to (test) PyPI 2022-08-03 20:57:27 +02:00
			`on:`
Fix publish of pip package to PyPi on git tag 2022-08-15 05:47:36 +02:00			`push:`
			`tags:`
Update the publish workflow to also run on any tag push 2023-01-05 00:46:48 +01:00			`- "*"`
Create Workflow to Publish Application to (test) PyPI 2022-08-03 20:57:27 +02:00			`branches:`
			`- 'master'`
			`paths:`
Use the src/ layout to fix packaging Khoj for PyPi - Why The khoj pypi packages should be installed in `khoj' directory. Previously it was being installed into `src' directory, which is a generic top level directory name that is discouraged from being used - Changes - move src/* to src/khoj/* - update `setup.py' to `find_packages' in `src' instead of project root - rename imports to form `from khoj.*' in complete project - update `constants.web_directory' path to use `khoj' directory - rename root logger to `khoj' in `main.py' - fix image_search tests to use the newly rename `khoj' logger - update config, docs, workflows to reference new path `src/khoj' 2023-02-14 21:50:51 +01:00			`- src/khoj/**`
Add developer support for using next.js to serve generated static files (#814) To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road. Dev instructions for using the next.js setup are in the added README. This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up. This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json. 2024-06-22 16:42:41 +02:00			`- src/interface/web/**`
Fix authors, homepage URL in pyproject.toml and workflow triggers 2023-02-16 10:12:04 +01:00			`- pyproject.toml`
Improve workflow to publish khoj to pypi - Use emoji's to improve visual indicator of action step - Rename to pypi instead of the more ambiguous publish name Publish could mean publish docker image, publish to pypi, MELPA or Obsidian plugin - Update workflow badge, link pypi badge to khoj pypi package page - Use pypa official github action to upload package to (test) pypi instead of doing it manually using twine - Upload python package artifact for easier access for testing. As uploading to testpypi doesn't work for PRs by others from forked repos 2023-02-15 02:47:42 +01:00			`- .github/workflows/pypi.yml`
Enable the publish workflow for PRs created in the main repo The publish workflow was previously disabled for PRs in commit d1945c5ba8537811e49f1fd5c5b5793e83f0d04d 2023-02-14 20:47:25 +01:00			`pull_request:`
			`branches:`
			`- 'master'`
			`paths:`
Use the src/ layout to fix packaging Khoj for PyPi - Why The khoj pypi packages should be installed in `khoj' directory. Previously it was being installed into `src' directory, which is a generic top level directory name that is discouraged from being used - Changes - move src/* to src/khoj/* - update `setup.py' to `find_packages' in `src' instead of project root - rename imports to form `from khoj.*' in complete project - update `constants.web_directory' path to use `khoj' directory - rename root logger to `khoj' in `main.py' - fix image_search tests to use the newly rename `khoj' logger - update config, docs, workflows to reference new path `src/khoj' 2023-02-14 21:50:51 +01:00			`- src/khoj/**`
Add developer support for using next.js to serve generated static files (#814) To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road. Dev instructions for using the next.js setup are in the added README. This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up. This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json. 2024-06-22 16:42:41 +02:00			`- src/interface/web/**`
Fix authors, homepage URL in pyproject.toml and workflow triggers 2023-02-16 10:12:04 +01:00			`- pyproject.toml`
Improve workflow to publish khoj to pypi - Use emoji's to improve visual indicator of action step - Rename to pypi instead of the more ambiguous publish name Publish could mean publish docker image, publish to pypi, MELPA or Obsidian plugin - Update workflow badge, link pypi badge to khoj pypi package page - Use pypa official github action to upload package to (test) pypi instead of doing it manually using twine - Upload python package artifact for easier access for testing. As uploading to testpypi doesn't work for PRs by others from forked repos 2023-02-15 02:47:42 +01:00			`- .github/workflows/pypi.yml`
Create Workflow to Publish Application to (test) PyPI 2022-08-03 20:57:27 +02:00
			`jobs:`
Publish to Test PyPI on Push to PR or Master Branch 2022-08-04 02:49:32 +02:00			`publish:`
Improve workflow to publish khoj to pypi - Use emoji's to improve visual indicator of action step - Rename to pypi instead of the more ambiguous publish name Publish could mean publish docker image, publish to pypi, MELPA or Obsidian plugin - Update workflow badge, link pypi badge to khoj pypi package page - Use pypa official github action to upload package to (test) pypi instead of doing it manually using twine - Upload python package artifact for easier access for testing. As uploading to testpypi doesn't work for PRs by others from forked repos 2023-02-15 02:47:42 +01:00			`name: Publish Python Package to PyPI`
Update stale Khoj pypi package metadata Use latest License, Intended Audience and Dev Status 2024-03-28 19:28:15 +01:00			`runs-on: ubuntu-latest`
Use OIDC TrustedPublisher to publish khoj python package to PyPi 2024-03-28 18:28:36 +01:00			`permissions:`
			`id-token: write`
Create Workflow to Publish Application to (test) PyPI 2022-08-03 20:57:27 +02:00			`steps:`
			`- uses: actions/checkout@v3`
Improve workflow to publish khoj to pypi - Use emoji's to improve visual indicator of action step - Rename to pypi instead of the more ambiguous publish name Publish could mean publish docker image, publish to pypi, MELPA or Obsidian plugin - Update workflow badge, link pypi badge to khoj pypi package page - Use pypa official github action to upload package to (test) pypi instead of doing it manually using twine - Upload python package artifact for easier access for testing. As uploading to testpypi doesn't work for PRs by others from forked repos 2023-02-15 02:47:42 +01:00			`with:`
			`fetch-depth: 0`
Create Workflow to Publish Application to (test) PyPI 2022-08-03 20:57:27 +02:00
Increase minimum python required in the pyproject, use python 3.11 for building the wheel in the workflow 2024-03-28 07:49:07 +01:00			`- name: Set up Python 3.11`
Create Workflow to Publish Application to (test) PyPI 2022-08-03 20:57:27 +02:00			`uses: actions/setup-python@v4`
			`with:`
Increase minimum python required in the pyproject, use python 3.11 for building the wheel in the workflow 2024-03-28 07:49:07 +01:00			`python-version: '3.11'`
Create Workflow to Publish Application to (test) PyPI 2022-08-03 20:57:27 +02:00
Emojify step names in workflows. Stop publishing to TestPyPi from PR 2023-03-01 17:56:39 +01:00			`- name: ⬇️ Install Application`
Fix workflow to publish Khoj python package to PyPi 2023-02-15 04:44:00 +01:00			`run: python -m pip install --upgrade pip && pip install --upgrade .`
Create Workflow to Publish Application to (test) PyPI 2022-08-03 20:57:27 +02:00
Add developer support for using next.js to serve generated static files (#814) To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road. Dev instructions for using the next.js setup are in the added README. This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up. This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json. 2024-06-22 16:42:41 +02:00			`- name: Install the Next.js application`
			`run: \|`
			`yarn install`
			`working-directory: src/interface/web`

			`- name: Build & export static Next.js app to Django static assets`
			`run: \|`
			`yarn ciexport`
			`working-directory: src/interface/web`

Improve workflow to publish khoj to pypi - Use emoji's to improve visual indicator of action step - Rename to pypi instead of the more ambiguous publish name Publish could mean publish docker image, publish to pypi, MELPA or Obsidian plugin - Update workflow badge, link pypi badge to khoj pypi package page - Use pypa official github action to upload package to (test) pypi instead of doing it manually using twine - Upload python package artifact for easier access for testing. As uploading to testpypi doesn't work for PRs by others from forked repos 2023-02-15 02:47:42 +01:00			`- name: ⚙️ Build Python Package`
			`run: \|`
Fix setup of Reproducible Build environment in publish workflow - Note: Reproducible builds have not been validated. This is just preliminary work to get there. Further testing and fixes maybe required 2022-08-19 20:00:12 +02:00			`# Setup Environment for Reproducible Builds`
			`export PYTHONHASHSEED=42`
			`export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)`
Fix workflow to publish Khoj python package to PyPi 2023-02-15 04:44:00 +01:00			`rm -rf dist`
Fix setup of Reproducible Build environment in publish workflow - Note: Reproducible builds have not been validated. This is just preliminary work to get there. Further testing and fixes maybe required 2022-08-19 20:00:12 +02:00
Check wheel contents in workflow before publishing it to PyPI 2023-02-14 20:41:33 +01:00			`# Build PyPi Package`
Fix workflow to publish Khoj python package to PyPi 2023-02-15 04:44:00 +01:00			`pipx run build`
Check wheel contents in workflow before publishing it to PyPI 2023-02-14 20:41:33 +01:00
Emojify step names in workflows. Stop publishing to TestPyPi from PR 2023-03-01 17:56:39 +01:00			`- name: 🌡️ Validate Python Package`
Improve workflow to publish khoj to pypi - Use emoji's to improve visual indicator of action step - Rename to pypi instead of the more ambiguous publish name Publish could mean publish docker image, publish to pypi, MELPA or Obsidian plugin - Update workflow badge, link pypi badge to khoj pypi package page - Use pypa official github action to upload package to (test) pypi instead of doing it manually using twine - Upload python package artifact for easier access for testing. As uploading to testpypi doesn't work for PRs by others from forked repos 2023-02-15 02:47:42 +01:00			`run: \|`
Check wheel contents in workflow before publishing it to PyPI 2023-02-14 20:41:33 +01:00			`# Validate PyPi Package`
Ignore-check step failed due to unrecognized code. Try using capital letters for indicator 2023-11-21 20:33:43 +01:00			`pipx run check-wheel-contents dist/*.whl --ignore W004`
Fix workflow to publish Khoj python package to PyPi 2023-02-15 04:44:00 +01:00			`pipx run twine check dist/*`
Check wheel contents in workflow before publishing it to PyPI 2023-02-14 20:41:33 +01:00
Improve workflow to publish khoj to pypi - Use emoji's to improve visual indicator of action step - Rename to pypi instead of the more ambiguous publish name Publish could mean publish docker image, publish to pypi, MELPA or Obsidian plugin - Update workflow badge, link pypi badge to khoj pypi package page - Use pypa official github action to upload package to (test) pypi instead of doing it manually using twine - Upload python package artifact for easier access for testing. As uploading to testpypi doesn't work for PRs by others from forked repos 2023-02-15 02:47:42 +01:00			`- name: ⏫ Upload Python Package Artifacts`
Fix workflow to publish Khoj python package to PyPi 2023-02-15 04:44:00 +01:00			`uses: actions/upload-artifact@v3`
Improve workflow to publish khoj to pypi - Use emoji's to improve visual indicator of action step - Rename to pypi instead of the more ambiguous publish name Publish could mean publish docker image, publish to pypi, MELPA or Obsidian plugin - Update workflow badge, link pypi badge to khoj pypi package page - Use pypa official github action to upload package to (test) pypi instead of doing it manually using twine - Upload python package artifact for easier access for testing. As uploading to testpypi doesn't work for PRs by others from forked repos 2023-02-15 02:47:42 +01:00			`with:`
Fix workflow to publish Khoj python package to PyPi 2023-02-15 04:44:00 +01:00			`name: khoj-assistant`
			`path: dist/*.whl`
Improve workflow to publish khoj to pypi - Use emoji's to improve visual indicator of action step - Rename to pypi instead of the more ambiguous publish name Publish could mean publish docker image, publish to pypi, MELPA or Obsidian plugin - Update workflow badge, link pypi badge to khoj pypi package page - Use pypa official github action to upload package to (test) pypi instead of doing it manually using twine - Upload python package artifact for easier access for testing. As uploading to testpypi doesn't work for PRs by others from forked repos 2023-02-15 02:47:42 +01:00
Fix workflow to publish Khoj python package to PyPi 2023-02-15 04:44:00 +01:00			`- name: 📦 Publish Python Package to PyPI`
Improve workflow to publish khoj to pypi - Use emoji's to improve visual indicator of action step - Rename to pypi instead of the more ambiguous publish name Publish could mean publish docker image, publish to pypi, MELPA or Obsidian plugin - Update workflow badge, link pypi badge to khoj pypi package page - Use pypa official github action to upload package to (test) pypi instead of doing it manually using twine - Upload python package artifact for easier access for testing. As uploading to testpypi doesn't work for PRs by others from forked repos 2023-02-15 02:47:42 +01:00			`if: startsWith(github.ref, 'refs/tags') \|\| github.ref == 'refs/heads/master'`
Specify version of pypi gh publish action 2024-03-28 08:17:31 +01:00			`uses: pypa/gh-action-pypi-publish@v1.8.14`