sabaimran
2dfd163430
Add more explicity run strategies in the runner matrix
2024-11-28 19:31:34 -08:00
sabaimran
80cd902c86
Since linux/amd64 images aren't being created, try setting a custom description on the image
...
Refer to this GH documentation on working with multi arch images in the container registry:
https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#adding-a-description-to-multi-arch-images
2024-11-28 19:14:06 -08:00
sabaimran
40d8a7a581
Release Khoj version 1.30.9
2024-11-28 18:45:50 -08:00
sabaimran
87aa653c7f
Add additional steps in prod.Dockerfile to ensure dependencies are copied over
2024-11-28 18:37:08 -08:00
sabaimran
d91935c880
Initial commit of a functional but not yet elegant prototype for this concept
2024-11-28 17:28:23 -08:00
Debanjum
a552543f4f
Use json5 to parse llm generated questions to query docs and web
...
build khoj.el / build (push) Waiting to run
desktop / 🖥️ Build, Release Desktop App (push) Waiting to run
dockerize / Publish Khoj Docker Images (push) Waiting to run
build and deploy github pages for documentation / deploy (push) Waiting to run
pypi / Publish Python Package to PyPI (push) Waiting to run
test khoj.el / test (27.1) (push) Waiting to run
test khoj.el / test (27.2) (push) Waiting to run
test khoj.el / test (28.1) (push) Waiting to run
test khoj.el / test (28.2) (push) Waiting to run
test khoj.el / test (snapshot) (push) Waiting to run
test / Run Tests (push) Has been cancelled
pre-commit / Setup Application and Lint (push) Has been cancelled
json5 is more forgiving, handles double quotes, newlines in raw json
string
2024-11-28 14:35:34 -08:00
Debanjum
0a69af4f61
Update to latest ToDesktop runtime
2024-11-28 13:56:14 -08:00
Debanjum
1d0fe141dc
Release Khoj version 1.30.8
2024-11-28 13:37:30 -08:00
Debanjum
29e801c381
Add MATH500 dataset to eval
...
Evaluate simpler MATH500 responses with gemini 1.5 flash
This improves both the speed and cost of running this eval
2024-11-28 12:48:25 -08:00
Debanjum
22aef9bf53
Add GPQA (diamond) dataset to eval
2024-11-28 12:48:25 -08:00
Debanjum
f1190ccf32
Improve parsing complex json strings returned by LLM ( #989 )
...
- Improve escaping to load complex json objects
- Fallback to a more forgiving [json5](https://json5.org/ ) loader if `json.loads` cannot parse complex json str
This should reduce failures to pick research tool and run code by agent
2024-11-28 11:01:39 -08:00
Debanjum
8c120a5139
Fallback to json5 loader if json.loads cannot parse complex json str
...
JSON5 spec is more flexible, try to load using a fast json5 parser if
the stricter json.loads from the standard library can't load the
raw complex json string into a python dictionary/list
2024-11-26 21:17:00 -08:00
Debanjum
70b7e7c73a
Improve load of complex json objects. Use it to pick tool, run code
...
Gemini doesn't work well when trying to output json objects. Using it
to output raw json strings with complex, multi-line structures
requires more intense clean-up of raw json string for parsing
2024-11-26 17:37:57 -08:00
Debanjum
8cb0db0051
Fix llama-cpp-python install by pytest github workflow
...
dockerize / Publish Khoj Docker Images (push) Has been cancelled
build and deploy github pages for documentation / deploy (push) Has been cancelled
pre-commit / Setup Application and Lint (push) Has been cancelled
pypi / Publish Python Package to PyPI (push) Has been cancelled
test / Run Tests (push) Has been cancelled
- Use pre-built wheels for torch and llama-cpp-python
- Install and link musl as it's used by llama-cpp-python pre-built
wheel instead of glibc
- Join Install git and Install Dependencies steps in pytest workflow
To remove unnecessary steps
2024-11-26 02:04:36 -08:00
Debanjum
29315f44e7
Add assetlinks.json to link android app to app.khoj.dev domain
...
Add sha cert of android upload, signing keys to open debug, prod apps
as TWA in fullscreen on android phones
2024-11-26 01:57:54 -08:00
Debanjum
a97a45bf20
Align agent personality with recently updated khoj personality
...
See update to Khoj personality in commit
6eb59464da
2024-11-26 00:06:16 -08:00
Debanjum
e088fcbc7b
Build for arm64 on arm64 runner. Parallelize arm64, x64 docker builds
...
dockerize / Publish Khoj Docker Images (push) Has been cancelled
build and deploy github pages for documentation / deploy (push) Has been cancelled
pypi / Publish Python Package to PyPI (push) Has been cancelled
- Building arm64 image on an ubuntu arm64 runner reduces `yarn build'
step time by 75% from 12mins to 3mins.
- This is because no QEMU emulation for arm64 on x86 is required now
- Parallelizing x64 and arm64 platform builds halves build time on top
- Revert to use standard ubuntu-latest runner as large x64 runner
doesn't give much more speed improvements
This results an effective additional 50%-66% reduction in build time
on top of #987 .
So a full dockerize workflow run now takes *10 mins* vs previous 35+mins.
This is a total of *72% improvement* in max dockerize run time.
Get additional speed improvements when docker layer cache hit.
2024-11-24 23:18:55 -08:00
Debanjum
5723a3778e
Speed up Docker image builds using multi-stage parallel pipelines ( #987 )
...
## Objective
Improve build speed and size of khoj docker images
## Changes
### Improve docker image build speeds
- Decouple web app and server build steps
- Build the web app and server in parallel
- Cache docker layers for reuse across dockerize github workflow runs
- Split Docker build layers for improved cacheability (e.g separate `yarn install` and `yarn build` steps)
### Reduce size of khoj docker images
- Use an up-to-date `.dockerignore` to exclude unnecessary directories
- Do not installing cuda python packages for cpu builds
### Improve web app builds
- Use consistent mechanism to get fonts for web app
- Make tailwind extensions production instead of dev dependencies
- Make next.js create production builds for the web app (via `NODE_ENV=production` env var)
2024-11-24 21:49:46 -08:00
Debanjum
4a5646c8da
Cache docker layers, nextjs builds in dockerize github workflow
2024-11-24 21:06:22 -08:00
Debanjum
6a39651ad3
Standardize loading fonts locally across pages on web app
2024-11-24 20:41:15 -08:00
sabaimran
9368699b2c
Migrate the pre-commit config
dockerize / Publish Khoj Docker Images (push) Waiting to run
build and deploy github pages for documentation / deploy (push) Waiting to run
pypi / Publish Python Package to PyPI (push) Waiting to run
test / Run Tests (push) Has been cancelled
pre-commit / Setup Application and Lint (push) Has been cancelled
2024-11-24 14:54:26 -08:00
sabaimran
6eb59464da
Add additional reinforcement to coax gemini into giving a minimum helpful response
2024-11-24 14:53:53 -08:00
sabaimran
15f062b34a
Remove print statement for agent style map
2024-11-24 14:53:53 -08:00
sabaimran
d7e68a2d1b
Wait for iplcodata to load before first message
...
- Fix the console khoj ai ascii art
- Remove some not so good suggested prompt
2024-11-24 14:53:53 -08:00
Debanjum
f51e0f7859
Make Next.js create production builds of web app for Docker images
2024-11-24 13:59:40 -08:00
Debanjum
710e00ad9e
Make tailwind extensions prod, instead of dev, deps of web app
2024-11-24 13:59:40 -08:00
Debanjum
4b486ea5f6
Exclude unnecessary directories from final docker builds
2024-11-24 13:59:40 -08:00
Debanjum
78d8ca49ec
Skip Nvidia GPU python packages during Server install in Dockerfiles
2024-11-24 13:59:39 -08:00
Debanjum
37887a175a
Speed up Docker image builds using multi-stage parallel pipelines
...
Decouple web app, server builds in parallel to speed up Docker
builds
2024-11-24 12:48:30 -08:00
Debanjum
7c77d65d35
Improve logic to disable telemetry via KHOJ_TELEMETRY_DISABLE env var
...
dockerize / Publish Khoj Docker Images (push) Waiting to run
build and deploy github pages for documentation / deploy (push) Waiting to run
pre-commit / Setup Application and Lint (push) Waiting to run
pypi / Publish Python Package to PyPI (push) Waiting to run
test / Run Tests (push) Waiting to run
build khoj.el / build (push) Has been cancelled
desktop / 🖥️ Build, Release Desktop App (push) Has been cancelled
test khoj.el / test (27.1) (push) Has been cancelled
test khoj.el / test (27.2) (push) Has been cancelled
test khoj.el / test (28.1) (push) Has been cancelled
test khoj.el / test (28.2) (push) Has been cancelled
test khoj.el / test (snapshot) (push) Has been cancelled
The newly added KHOJ_TELEMETRY_DISABLE env var knob to disable
telemetry should override old config mechanism when set
2024-11-24 00:54:16 -08:00
sabaimran
2d683898c2
Release Khoj version 1.30.7
2024-11-23 22:51:10 -08:00
sabaimran
914ff994f7
Fix cost addition to chat_metadata
build khoj.el / build (push) Waiting to run
desktop / 🖥️ Build, Release Desktop App (push) Waiting to run
dockerize / Publish Khoj Docker Images (push) Waiting to run
build and deploy github pages for documentation / deploy (push) Waiting to run
pre-commit / Setup Application and Lint (push) Waiting to run
pypi / Publish Python Package to PyPI (push) Waiting to run
test / Run Tests (push) Waiting to run
test khoj.el / test (27.1) (push) Waiting to run
test khoj.el / test (27.2) (push) Waiting to run
test khoj.el / test (28.1) (push) Waiting to run
test khoj.el / test (28.2) (push) Waiting to run
test khoj.el / test (snapshot) (push) Waiting to run
2024-11-23 22:50:45 -08:00
Debanjum
caaa127dcf
Release Khoj version 1.30.6
2024-11-23 21:07:00 -08:00
Debanjum
57b8273002
Fix apt install for musl-dev in prod.Dockerfile
2024-11-23 21:06:09 -08:00
Debanjum
8f966b11ec
Release Khoj version 1.30.5
2024-11-23 20:49:05 -08:00
Debanjum
498895a47d
Fix libmusl error using pre-built llama-cpp-python wheel in prod Docker
2024-11-23 20:47:41 -08:00
Debanjum
e5b211a743
Release Khoj version 1.30.4
2024-11-23 19:48:21 -08:00
Debanjum
9848d89d03
Try build docker images with github high cpu, ram runner
2024-11-23 19:09:36 -08:00
Debanjum
04bb3d6f15
Fix libmusl error using pre-built llama-cpp-python wheel via Docker
...
Seems like llama-cpp-python pre-built wheels need libmusl. Otherwise
you run into runtime errors on Khoj startup via Docker.
2024-11-23 18:46:44 -08:00
Debanjum
8dd2122817
Set sample size to 200 for automated eval runs as well
2024-11-23 14:48:38 -08:00
Debanjum
c4ef31d86f
Release Khoj version 1.30.3
build khoj.el / build (push) Waiting to run
desktop / 🖥️ Build, Release Desktop App (push) Waiting to run
dockerize / Publish Khoj Docker Images (push) Waiting to run
build and deploy github pages for documentation / deploy (push) Waiting to run
pypi / Publish Python Package to PyPI (push) Waiting to run
test khoj.el / test (27.1) (push) Waiting to run
test khoj.el / test (27.2) (push) Waiting to run
test khoj.el / test (28.1) (push) Waiting to run
test khoj.el / test (28.2) (push) Waiting to run
test khoj.el / test (snapshot) (push) Waiting to run
2024-11-23 14:40:06 -08:00
Debanjum
15ae22bdcf
Use pre-built llama-cpp-python wheel in Khoj docker images
...
Reduces build time and resolves FileNotFoundError 'ninja' during
llama-cpp-python local build.
2024-11-23 14:38:07 -08:00
sabaimran
4ac49ca90f
Release Khoj version 1.30.2
2024-11-23 12:00:28 -08:00
sabaimran
eb1b21baaa
Add a new sign in modal that is triggered from the login prompt screen, rather than redirecting user to another screen to sign in
2024-11-23 11:55:34 -08:00
Debanjum
5aa5cb1941
Add "New" section with latest updates to Readme
build and deploy github pages for documentation / deploy (push) Waiting to run
2024-11-23 01:36:50 -08:00
sabaimran
7f5bf35806
Disambiguate renewal_date type. Previously, being used as None, False, and Datetime in different places.
dockerize / Publish Khoj Docker Images (push) Waiting to run
build and deploy github pages for documentation / deploy (push) Waiting to run
pypi / Publish Python Package to PyPI (push) Waiting to run
pre-commit / Setup Application and Lint (push) Has been cancelled
test / Run Tests (push) Has been cancelled
2024-11-22 12:06:20 -08:00
sabaimran
5e8c824ecc
Improve the experience for finding past conversation
...
- add a conversation title search filter, and an agents filter, for finding conversations
- in the chat session api, return relevant agent style data
2024-11-22 12:03:01 -08:00
sabaimran
a761865724
Fix handling of customer.subscription.updated event to process new renewal end date
2024-11-22 12:03:01 -08:00
sabaimran
6a054d884b
Add quicker/easier filtering on auth
2024-11-22 12:03:01 -08:00
Debanjum
b9a889ab69
Fix Khoj responses when code generated charts in response context
...
The current fix should improve Khoj responses when charts in response
context. It truncates code context before sharing with response chat actors.
Previously Khoj would respond with it not being able to create chart
but than have a generated chart in it's response in default mode.
The truncate code context was added to research chat actor for
decision making but it wasn't added to conversation response
generation chat actors.
When khoj generated charts with code for its response, the images in
the context would exceed context window limits.
So the truncation logic to drop all past context, including chat
history, context gathered for current response.
This would result in chat response generator 'forgetting' all for the
current response when code generated images, charts in response context.
2024-11-21 14:43:52 -08:00
