Commit graph

163 commits

Author SHA1 Message Date
Debanjum Singh Solanky
7614718204 Build Khoj cloud docker image for arm64 architecture too 2024-09-10 13:57:09 -07:00
Debanjum Singh Solanky
5d59acd1f4 Stop pushing deprecated khoj-assistant package to pypi
- Also skip uploading package version to it already exists on pypi
  This happens when a release is new khoj tagged release is created
2024-08-20 11:43:02 -07:00
sabaimran
d4a8ff0683 Support workflow dispatch events for running the pypi.yml job 2024-08-06 15:55:39 +05:30
sabaimran
24d0fdb262 Fix directory referenceds in pypi.yml configuration for compiled folder 2024-08-06 13:38:34 +05:30
sabaimran
23b1b36f8c Fix directory referenceds in pypi.yml configuration for compiled folder 2024-08-06 13:31:42 +05:30
sabaimran
694f551625 Fix mkdir step when copying generated files 2024-08-06 10:17:56 +05:30
sabaimran
e9f9d92989 Try to manually copy the built files into where the src directory should be for the pypi build 2024-08-06 10:05:06 +05:30
sabaimran
e9d6899fc2 Change the way the export is created for the pypi package in order to transfer static files out of the tmp shell 2024-08-05 22:46:54 +05:30
Debanjum Singh Solanky
583fa3c188 Migrate the pypi package to khoj project name. Update references
- Deprecate khoj-assistant pypi package. Use more accurate and
  succinct pypi project name, khoj
- Update references to sye khoj pypi package in docs and code instead
  of the legacy khoj-assistant pypi package
- Update pypi workflow to publish to both khoj, khoj-assistant for now
- Update stale python 3.9 support mentioned in our pyproject. Can't
  support python 3.9 as depend on latest django which support >=3.10
2024-07-17 10:41:16 +05:30
sabaimran
260aa61818 Remove tests for python3.9 2024-07-09 12:28:11 +05:30
sabaimran
38090b2553 In dockerize.yml file, revert the added configuration 2024-06-22 21:11:25 +05:30
sabaimran
a53178cab9
Add developer support for using next.js to serve generated static files ()
To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road.

Dev instructions for using the next.js setup are in the added README.

This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up.

This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.
2024-06-22 20:12:41 +05:30
sabaimran
7100614de5
Add support for rendering math equations in the web view ()
- Add parsing logic for LaTeX-format math equations in the web chat
- Add placeholder delimiters when converting the markdown to HTML in order to avoid removing the escaped characters
- Add the `<!DOCTYPE html>` specification to the page
2024-05-04 15:59:17 +05:30
Debanjum Singh Solanky
e3deb29f8e Upgrade khoj.el workflow to use Python 3.11 2024-04-07 11:24:07 +05:30
Debanjum Singh Solanky
14fbf594b2 Support using Python 3.12 with Khoj
- RapidOCR for indexing image PDFs doesn't currently support python 3.12.
  It's an optional dependency anyway, so only install it if python < 3.12
- Run unit tests with python version 3.12 as well

Resolves 
2024-04-07 11:23:44 +05:30
Debanjum Singh Solanky
90c5b3c410 Update stale Khoj pypi package metadata
Use latest License, Intended Audience and Dev Status
2024-03-29 00:06:55 +05:30
Debanjum Singh Solanky
a374288cea Use OIDC TrustedPublisher to publish khoj python package to PyPi 2024-03-28 22:58:36 +05:30
sabaimran
6cb38d92c0 Specify version of pypi gh publish action 2024-03-28 12:47:31 +05:30
sabaimran
56da96b2e9 Increase minimum python required in the pyproject, use python 3.11 for building the wheel in the workflow 2024-03-28 12:19:07 +05:30
Debanjum Singh Solanky
667b975400 Free space on Github workflow VM to build Khoj docker images 2024-02-06 23:37:51 +05:30
Debanjum Singh Solanky
2f1bb5c2c8 Upload Desktop App Artifacts to Github Release 2024-01-18 11:40:04 +05:30
sabaimran
e71ebb8068 Standardize issue templates and make them easier to use 2024-01-18 10:54:05 +05:30
sabaimran
efb4bd6780 Add a template for feature requests 2024-01-18 10:38:53 +05:30
sabaimran
6165ae56c2
Update bug report issue template
- collect info about OS, device, server, client, and prompt to include any relevant data
2024-01-18 10:35:02 +05:30
sabaimran
ce53bc52c5 Modify permissions of the GITHUB_TOKEN for publishing to gh-pages 2024-01-07 20:53:57 +05:30
sabaimran
740453fa18 Use documentation folder for building project and uploading data 2024-01-07 20:50:15 +05:30
sabaimran
2be7c84203 Enter documentation repository before running yarn build 2024-01-07 20:46:21 +05:30
sabaimran
ad95e88838 Update node version in github action 2024-01-07 20:41:24 +05:30
sabaimran
bd9aa578f4 Add a yarn.lock file and use for node.js setup 2024-01-07 20:36:02 +05:30
sabaimran
9b991eb4fe
Migrate to using docusaurus, rather than docsify for documentation ()
* Add docusaurus documentation (to replace the docsify setup
* Remove older docs
* Specify documentation as the gh pages build action working directory
2024-01-07 20:28:15 +05:30
Debanjum Singh Solanky
074123b9b9 Merge cloud, local dockerize workflows
- Delete unused config directory
2023-12-22 17:11:52 +05:30
Debanjum Singh Solanky
8d1e988059 Update tagging of the docker image on release, push to master & PR
- Tag docker image with `tag_name' on release (i.e tag push)
- Else tag with 'pre' on push to master
- Else tag with 'dev' on push to PR branch

- Only tag the latest release with release tag
  Previously the latest commit on master was being tagged with the
  latest tag. This doesn't sync with the release cadence of the rest
  of Khoj
2023-12-22 17:01:31 +05:30
Debanjum Singh Solanky
b5ae64cb3c Dynamically set Khoj app version in the Dockerization Github workflows 2023-12-22 17:01:31 +05:30
Debanjum Singh Solanky
9d4bfdf47c Merge Github workflows to dockerize for production 2023-11-30 17:18:13 -08:00
sabaimran
61f6b8c0d4 Ignore-check step failed due to unrecognized code. Try using capital letters for indicator 2023-11-21 11:33:43 -08:00
sabaimran
38144a7a69 pull_request path should be src/khoj rather than src/ 2023-11-21 11:33:07 -08:00
sabaimran
8932fc0c36 Ignore w004 check to bypass pypi warnings for check-wheel-contents
- PyPi doesn't like to have files that start with numbers, however all of the generated django migration files start with numbers. To accommodate, skip this check.
- Refer to https://pypi.org/project/check-wheel-contents/ for documentation and recommendation
2023-11-21 11:12:50 -08:00
Debanjum Singh Solanky
68ac1e0193 Automate Desktop app builds on new release or push to master branch 2023-11-16 16:09:03 -08:00
sabaimran
36d200580b Use a different name for the production-config containers 2023-11-16 10:28:28 -08:00
sabaimran
ba633c4015 Only build the production docker image when pushing to master 2023-11-16 09:24:57 -08:00
sabaimran
cf74fa4a70 Allow dockerize step to run for prod from PR temporarily 2023-11-15 17:04:48 -08:00
sabaimran
992e54c218 Rename dockerize dev action to b emore accurate 2023-11-15 14:09:28 -08:00
sabaimran
99f5a6082e Add a dev workflow for GitHub actions, change the production workflow to only kick off when pushed to master 2023-11-15 14:07:25 -08:00
sabaimran
3d6e8d53fe Try adding dependencies for libgl in order to run OCR in github action unit tests 2023-11-05 15:09:40 -08:00
sabaimran
5f3f6b7c61
[Multi-User Part 5]: Add a production Docker file and use a gunicorn configuration with it ()
- Add a productionized setup for the Khoj server using `gunicorn` with multiple workers for handling requests
- Add a new Dockerfile meant for production config at `ghcr.io/khoj-ai/khoj:prod`; the existing Docker config should remain the same
2023-10-26 13:15:31 -07:00
sabaimran
216acf545f
[Multi-User Part 1]: Enable storage of settings for plaintext files based on user account ()
- Partition configuration for indexing local data based on user accounts
- Store indexed data in an underlying postgres db using the `pgvector` extension
- Add migrations for all relevant user data and embeddings generation. Very little performance optimization has been done for the lookup time
- Apply filters using SQL queries
- Start removing many server-level configuration settings
- Configure GitHub test actions to run during any PR. Update the test action to run in a containerized environment with a DB.
- Update the Docker image and docker-compose.yml to work with the new application design
2023-10-26 09:42:29 -07:00
sabaimran
343854752c
Improve docker builds for local hosting ()
* Remove GPT4All dependency in pyproject.toml and use multiplatform builds in the dockerization setup in GH actions
* Move configure_search method into indexer
* Add conditional installation for gpt4all
* Add hint to go to localhost:42110 in the docs. Addresses 
2023-09-08 17:07:26 -07:00
sabaimran
dccfae3853
Remove PySide dependency and deprecate desktop builds ()
* Remove PySide, gui option from code
* Remove pyside 6 dependency from code
* Remove workflows which build desktop applications
* Update unit tests and update line in documentation
* Remove additional references to pyinstaller, gui
* Add uninstall steps to normal uninstall instructions
2023-09-07 11:36:27 -07:00
Debanjum Singh Solanky
34d5cd2bd8 Increase pytests workflow timeout duration to reduce intermittent failures
The test workflow fails regularly with an OperationCancelled error.
This is an intermittent failure that gets resolved on running the
failed workflows a few times.
2023-08-16 20:00:36 -07:00
Debanjum Singh Solanky
84d774ea34 Retain desktop builds for 3 days to allow user tests
Upgrade minimum tiktoken version to work for encoding gpt4
2023-08-08 23:02:13 -07:00