203 commits

Author	SHA1	Message	Date
sabaimran	fd90842d38	Bump postgresql server dev version to 16 for latest ubuntu	2025-01-22 19:07:54 -08:00
sabaimran	8fe08eecce	add --break-system-packages to bypass venv requirement	2025-01-20 00:21:27 -08:00
sabaimran	bf58d9430b	downgrade postgres server pkg to 16	2025-01-20 00:15:56 -08:00
sabaimran	95ad1f936e	upgrade postgres server to 17	2025-01-20 00:10:20 -08:00
sabaimran	a214bd4100	upgrade pg server dev version to 15	2025-01-20 00:05:35 -08:00
sabaimran	82ff74cfa9	Run on container with ubuntu latest for pytest gh action workflow	2025-01-19 23:57:57 -08:00
sabaimran	af9e906cb5	Use python3 instead of python when running pip install commands in gh actions	2025-01-17 17:48:42 -08:00
Debanjum	6bd9f6bb61	Give a shorter, simpler name to github workflow to deploy docs	2025-01-12 10:54:56 +07:00
sabaimran	bac90ad69d	Upgrade deploy-pages action to vv4	2025-01-09 19:04:31 -08:00
Debanjum	2069f571c8	Upgrade upload-artifact gh action to v4 as <=v3 deprecated ... This started failing github workflow jobs	2025-01-10 00:41:24 +07:00
sabaimran	92144c8102	Remove release step in todesktop flow, since we need to run releases manually now ... - Leaving it commented out for the time being so we can revisit automating this later	2024-12-17 16:02:45 -08:00
Debanjum	10bd56d2b9	Attest Khoj pypi package by upgrading pypi publish gh action ... - Print hash in CI to ease verifying ci built python package matches khoj package published on pypi - Newer pypi publish github action should speed up workflow by ~30s	2024-12-17 13:40:39 -08:00
Debanjum	df15f00243	Tag docker images with latest tag in dockerize workflow on release	2024-12-17 13:18:51 -08:00
sabaimran	f6abfcfa6b	Use latest release version for pypi gh action to publish	2024-12-17 12:19:42 -08:00
sabaimran	e74e922cea	Update file path of python installation	2024-12-12 16:50:32 -08:00
Debanjum	2db7a1ca6b	Restart code sandbox on crash in eval github workflow (#1007) ... See e3fed3750b for corresponding change to use pm2 to auto-restart code sandbox	2024-12-12 14:32:03 -08:00
Debanjum	9eb863e964	Restart code sandbox on crash in eval github workflow	2024-12-12 11:28:54 -08:00
Debanjum	59008ae90e	Use buildx to create multi platform docker image	2024-12-11 00:21:29 -08:00
Debanjum	ec797bc6b8	Build docker imgs on native arch runners to avoid manifest list error ... This also avoids the need to use --amend and annotate steps when creating the multi-arch docker images	2024-12-10 23:16:36 -08:00
Debanjum	5f7b13df2d	Fix new docker tags in workflow to not include forward slashes	2024-12-10 22:55:33 -08:00
Debanjum	ba6237b5c0	Fix to create multi-arch builds. Stop docker image overwrites in workflow	2024-12-10 21:08:17 -08:00
sabaimran	44ede26e67	Temporarily disable cloud arm builds while we disambiguate the build issues	2024-12-10 20:00:59 -08:00
sabaimran	9c403d24e1	Fix reference to directory in the eval workflow for starting terrarium	2024-12-08 13:03:05 -08:00
sabaimran	6940c6379b	Add sudo when running installations in order to install relevant packages ... add --legacy-peer-deps temporarily to see if it helps mitigate the issue	2024-12-08 11:11:13 -08:00
sabaimran	4c4b7120c6	Use Khoj terrarium fork instead of building from official Cohere repo	2024-12-08 11:06:33 -08:00
sabaimran	2dfd163430	Add more explicity run strategies in the runner matrix	2024-11-28 19:31:34 -08:00
sabaimran	80cd902c86	Since linux/amd64 images aren't being created, try setting a custom description on the image ... Refer to this GH documentation on working with multi arch images in the container registry: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#adding-a-description-to-multi-arch-images	2024-11-28 19:14:06 -08:00
Debanjum	29e801c381	Add MATH500 dataset to eval ... Evaluate simpler MATH500 responses with gemini 1.5 flash This improves both the speed and cost of running this eval	2024-11-28 12:48:25 -08:00
Debanjum	22aef9bf53	Add GPQA (diamond) dataset to eval	2024-11-28 12:48:25 -08:00
Debanjum	8cb0db0051	Fix llama-cpp-python install by pytest github workflow ... - Use pre-built wheels for torch and llama-cpp-python - Install and link musl as it's used by llama-cpp-python pre-built wheel instead of glibc - Join Install git and Install Dependencies steps in pytest workflow To remove unnecessary steps	2024-11-26 02:04:36 -08:00
Debanjum	e088fcbc7b	Build for arm64 on arm64 runner. Parallelize arm64, x64 docker builds ... - Building arm64 image on an ubuntu arm64 runner reduces `yarn build' step time by 75% from 12mins to 3mins. - This is because no QEMU emulation for arm64 on x86 is required now - Parallelizing x64 and arm64 platform builds halves build time on top - Revert to use standard ubuntu-latest runner as large x64 runner doesn't give much more speed improvements This results an effective additional 50%-66% reduction in build time on top of #987. So a full dockerize workflow run now takes *10 mins* vs previous 35+mins. This is a total of *72% improvement* in max dockerize run time. Get additional speed improvements when docker layer cache hit.	2024-11-24 23:18:55 -08:00
Debanjum	4a5646c8da	Cache docker layers, nextjs builds in dockerize github workflow	2024-11-24 21:06:22 -08:00
Debanjum	9848d89d03	Try build docker images with github high cpu, ram runner	2024-11-23 19:09:36 -08:00
Debanjum	8dd2122817	Set sample size to 200 for automated eval runs as well	2024-11-23 14:48:38 -08:00
Debanjum	50d8405981	Enable khoj to use terrarium code sandbox as tool in eval workflow	2024-11-20 14:19:27 -08:00
Debanjum	ffbd0ae3a5	Fix eval github workflow to run on releases, i.e on tags push	2024-11-20 12:57:42 -08:00
Debanjum	a2ccf6f59f	Fix github workflow to start Khoj, connect to PG and upload results ... - Do not trigger tests to run in ci on update to evals	2024-11-18 04:25:15 -08:00
Debanjum	7c0fd71bfd	Add GitHub workflow to quiz Khoj across modes and specified evals (#982) ... - Evaluate khoj on random 200 questions from each of google frames and openai simpleqa benchmarks across *general*, *default* and *research* modes - Run eval with Gemini 1.5 Flash as test giver and Gemini 1.5 Pro as test evaluator models - Trigger eval workflow on release or manually - Make dataset, khoj mode and sample size configurable when triggered via manual workflow - Enable Web search, webpage read tools during evaluation	2024-11-18 02:19:30 -08:00
sabaimran	63a2b5b3c4	Remove tools cache in dockerize.yml workflow	2024-09-29 00:27:37 -07:00
sabaimran	e306e6ca94	Fix file paths used for pypi wheel building	2024-09-22 12:42:08 -07:00
Debanjum Singh Solanky	7614718204	Build Khoj cloud docker image for arm64 architecture too	2024-09-10 13:57:09 -07:00
Debanjum Singh Solanky	5d59acd1f4	Stop pushing deprecated khoj-assistant package to pypi ... - Also skip uploading package version to it already exists on pypi This happens when a release is new khoj tagged release is created	2024-08-20 11:43:02 -07:00
sabaimran	d4a8ff0683	Support workflow dispatch events for running the pypi.yml job	2024-08-06 15:55:39 +05:30
sabaimran	24d0fdb262	Fix directory referenceds in pypi.yml configuration for compiled folder	2024-08-06 13:38:34 +05:30
sabaimran	23b1b36f8c	Fix directory referenceds in pypi.yml configuration for compiled folder	2024-08-06 13:31:42 +05:30
sabaimran	694f551625	Fix mkdir step when copying generated files	2024-08-06 10:17:56 +05:30
sabaimran	e9f9d92989	Try to manually copy the built files into where the src directory should be for the pypi build	2024-08-06 10:05:06 +05:30
sabaimran	e9d6899fc2	Change the way the export is created for the pypi package in order to transfer static files out of the tmp shell	2024-08-05 22:46:54 +05:30
Debanjum Singh Solanky	583fa3c188	Migrate the pypi package to khoj project name. Update references ... - Deprecate khoj-assistant pypi package. Use more accurate and succinct pypi project name, khoj - Update references to sye khoj pypi package in docs and code instead of the legacy khoj-assistant pypi package - Update pypi workflow to publish to both khoj, khoj-assistant for now - Update stale python 3.9 support mentioned in our pyproject. Can't support python 3.9 as depend on latest django which support >=3.10	2024-07-17 10:41:16 +05:30
sabaimran	260aa61818	Remove tests for python3.9	2024-07-09 12:28:11 +05:30