Commit graph

4084 commits

Author SHA1 Message Date
Debanjum
354dc12b3b
Style the Admin Panel with a modern theme and Khoj branding (#999)
Some checks failed
dockerize / Publish Khoj Docker Images (push) Waiting to run
build and deploy github pages for documentation / deploy (push) Waiting to run
pypi / Publish Python Package to PyPI (push) Waiting to run
pre-commit / Setup Application and Lint (push) Has been cancelled
test / Run Tests (push) Has been cancelled
Overview
- The default django admin panel UI looks pretty dated and didn't
  have any Khoj specific branding
- Used the Unfold Django admin panel theme for a modern look
- Used the Khoj logo and name in Admin panel title, headings, favicons

Details:
All models shown on Admin panel need to inherit from unfold's
ModelAdmin to get styling applied. So

- Make all models on Admin panel inherit from unfold's ModelAdmin
- Subclassed UserAdmin to inherit from unfold's ModelAdmin
- Deregistered the unused Auth Group model from the Admin panel
  We can add it back when its actually used. Avoid confusion for now
- Explicitly register DjangoJobExecution on admin panel and again make
  it inherit from the unfold.admin.ModelAdmin
2024-12-04 23:53:43 -08:00
Matias Forbord
9cc79c0fb7
Fix broken doc links to query filters from emacs docs page (#1000)
* docs: repair query filters link

* change docs link to be relative
2024-12-04 23:52:27 -08:00
sabaimran
8953ac03ec Rename additional context for llm response to program execution context 2024-12-04 18:43:41 -08:00
sabaimran
886fe4a0c9 Merge branch 'master' of github.com:khoj-ai/khoj into features/allow-multi-outputs-in-chat 2024-12-03 21:37:00 -08:00
sabaimran
df5e34615a Fix processing of images field when construct chat messages 2024-12-03 21:26:55 -08:00
sabaimran
3552032827 Rename additional context to additional_context_for_llm_response 2024-12-03 21:23:15 -08:00
sabaimran
d507894546 Simplify the desktop app
- Make the desktop app mainly a file-syncing client for users who have lots of documents that they need to share with Khoj. This is because the web app provides a fairly robust chat client which can be used by anyone on their computer.
- The chat client in the desktop app had significantly drifted from our current brand / them, and didn't provide enough value add to update. Later, we will make it easier to install the existing web app as a desktop PWA.
2024-12-02 15:54:05 -08:00
Debanjum
9f7cb335c5
Create Android app for Khoj (#991)
Some checks failed
pypi / Publish Python Package to PyPI (push) Has been cancelled
dockerize / Publish Khoj Docker Images (push) Has been cancelled
build and deploy github pages for documentation / deploy (push) Has been cancelled
Use bubblewrap to publish the Khoj Progressive Web App (PWA) as a Trusted Web Activity (TWA) to the Android Play Store
2024-12-02 15:45:04 -08:00
Debanjum
ee28d7f125 Fix Android Studio build warnings by using newer gradle, mavenCentral 2024-12-02 12:48:33 -08:00
Debanjum
5d6fb07066 Fix app icons, orientation. Improve name, id, description in webmanifest
- Use bubblewrap generated splash screen, notification icons from
  1200x1200 high res khoj icon in assets.khoj.dev.
- Discard bubblewrap generated launcher icons.
- Fix orientation to respect phone orientation settings. "any" was not.
2024-12-02 12:48:25 -08:00
Debanjum
147c8e9115 Release v3 with high-res splash screen. More details in web, app manifest
- Add 512, 192 Khoj maskable icons to web app manifest for android rendering
- Add id, categories etc suggested by pwabuilder
- Use higher quality icon images for splash screen than what
  bubblewrap creates by default
2024-12-02 11:28:35 -08:00
Debanjum
d333e10e64 Encode request params as utf-8 to fix multibyte char error in khoj.el
Some checks failed
dockerize / Publish Khoj Docker Images (push) Waiting to run
build and deploy github pages for documentation / deploy (push) Waiting to run
pypi / Publish Python Package to PyPI (push) Waiting to run
test / Run Tests (push) Has been cancelled
test khoj.el / test (27.1) (push) Has been cancelled
build khoj.el / build (push) Has been cancelled
pre-commit / Setup Application and Lint (push) Has been cancelled
test khoj.el / test (27.2) (push) Has been cancelled
test khoj.el / test (28.1) (push) Has been cancelled
test khoj.el / test (28.2) (push) Has been cancelled
test khoj.el / test (snapshot) (push) Has been cancelled
Encode api key in header, POST request body and GET query param for
search as utf-8 to avoid the multibyte char in request issue when
making API calls from khoj.el to khoj server.

Resolves #935
2024-12-02 02:00:14 -08:00
Debanjum
db29894038
Do not wrap filepath in Path to fix indexing markdown files on Windows (#993)
### Issue
- Path with / are converted to \\ on Windows using the `Path' operator.
- The `markdown_to_entries' module was trying to normalize file paths with`Path'  for some reason.
  This would store the file paths in DB Entry differently than the file to entries map if Khoj ran on Windows.
  That'd result in a KeyError when trying to look up the entry file path from `file_to_text_map' in the `text_to_entries:update_embeddings()' function.

### Fix
- Removing the unnecessary OS dependent Path normalization in `markdown_to_entries' should keep the file path storage consistent across `file_to_text_map' var, `FileObjectAdaptor', `Entry' DB tables on Windows for Markdown files as well.

This issue will affect users hosting Khoj server on Windows and attempting to index markdown files.

Resolves #984
2024-12-02 01:02:58 -08:00
Debanjum
47c926b0ff Add more typing to org|md_to_entries. Remove redundant f-string wraps
- Add type hints to improve maintainability of stabilzed indexing code
- It shouldn't be necessary to wrap string variables in an f-string

This change aims to improve code quality. It should not affect
functionality.
2024-12-01 23:02:52 -08:00
Debanjum
dffdd81345 Do not wrap filepath in Path to fix indexing markdown files on Windows
Issue
- Path with / are converted to \\ on Windows using the Path operator.
- The markdown to entries method for some reason was doing this.
  This would store the file paths in DB entry differently than the file
  to entries map. Resulting in a KeyError when trying to look up the
  entry file path from file_to_text_map in the
  text_to_entries:update_embeddings() function.

Fix
- Removing the unnecessary OS dependendent Path normalization in
  markdown_to_entries should keep the file path storage consistent
  across file_to_text_map var, FileObjectAdaptor, Entry DB tables on
  Windows for Markdown files as well

This issue would only affect users hosting Khoj server on Windows and
attempting to index markdown files.

Resolves #984
2024-12-01 23:00:31 -08:00
sabaimran
c87fce5930 Add a migration to use the new image storage format for past conversations
- Added it to the Django migrations so that it auto-triggers when someone updates their server and starts it up again for the first time. This will require that they update their clients as well in order to view/consume image content.
- Remove server-side references in the code that allow to parse the text-to-image intent as it will no longer be necessary, given the chat logs will be migrated
2024-12-01 18:35:31 -08:00
Debanjum
9e0a2c7a98 Restrict generated chat title to 200 chars limit allowed for chat slug
Some checks are pending
dockerize / Publish Khoj Docker Images (push) Waiting to run
build and deploy github pages for documentation / deploy (push) Waiting to run
pre-commit / Setup Application and Lint (push) Waiting to run
pypi / Publish Python Package to PyPI (push) Waiting to run
test / Run Tests (push) Waiting to run
2024-11-30 19:12:03 -08:00
Debanjum
8b8e2be82d Only create subscription object when it does not exist for user
This avoid unnecessarily throwing an internal server error when the
user tries to sign-up using multiple mechanisms (e.g first by email, then
by google oauth)
2024-11-30 19:08:34 -08:00
Debanjum
fc6be543bd Improve GPQA eval prompt to imrpove parsing answer from Khoj response
Some checks are pending
build and deploy github pages for documentation / deploy (push) Waiting to run
pre-commit / Setup Application and Lint (push) Waiting to run
2024-11-30 17:21:09 -08:00
sabaimran
00f48dc1e8 If in the new images format, show the response text in obsidian instead of the inferred query 2024-11-30 14:39:51 -08:00
sabaimran
224abd14e0 Only add the image_url to the constructed chat message if it is a url 2024-11-30 14:39:27 -08:00
sabaimran
991577aa17 Allow a None turnId to accommodate historic chat messages 2024-11-30 14:39:08 -08:00
sabaimran
a539761c49 Fix processing of excalidrawdiagram in json response chunking 2024-11-30 12:35:13 -08:00
sabaimran
dc4a9ee3e1 Ensure that the generated assets are maintained in the chat window after streaming is completed. 2024-11-30 12:31:20 -08:00
sabaimran
e3aee50cf3 Fix parsing of generated_asset response 2024-11-29 18:41:53 -08:00
sabaimran
2b32f0e80d Remove commented out code blocks 2024-11-29 18:11:50 -08:00
sabaimran
df855adc98 Update response handling in Obsidian to work with new format 2024-11-29 18:10:47 -08:00
sabaimran
512cf535e0 Collapse train of thought when completed during live stream 2024-11-29 18:10:35 -08:00
sabaimran
a0b00ce4a1 Don't include null attributes when filling in stored conversation metadata
- Prompt adjustments to indicate to LLM what context it has
2024-11-29 18:10:14 -08:00
sabaimran
c5329d76ba Merge branch 'master' of github.com:khoj-ai/khoj into features/allow-multi-outputs-in-chat 2024-11-29 14:12:03 -08:00
sabaimran
46f647d91d Improve image rendering for khoj generated images. FIx typing of stored excalidraw image. 2024-11-29 14:11:48 -08:00
Debanjum
fdf69b7049 Publish second version with new upload key 2024-11-28 22:04:10 -08:00
Debanjum
faf15072b6 Create first version of Khoj Android app from PWA using Bubblewrap 2024-11-28 22:04:10 -08:00
sabaimran
4f6d1211ba Fix additional context type in anthropic chat 2024-11-28 20:16:36 -08:00
sabaimran
6f408948d3 Fix typing of generated_fiels parameters 2024-11-28 20:15:10 -08:00
sabaimran
439b18c21f Release Khoj version 1.30.10
Some checks failed
build khoj.el / build (push) Has been cancelled
desktop / 🖥️ Build, Release Desktop App (push) Has been cancelled
dockerize / Publish Khoj Docker Images (push) Has been cancelled
build and deploy github pages for documentation / deploy (push) Has been cancelled
pypi / Publish Python Package to PyPI (push) Has been cancelled
test khoj.el / test (27.1) (push) Has been cancelled
test khoj.el / test (27.2) (push) Has been cancelled
test khoj.el / test (28.1) (push) Has been cancelled
test khoj.el / test (28.2) (push) Has been cancelled
test khoj.el / test (snapshot) (push) Has been cancelled
2024-11-28 19:43:06 -08:00
sabaimran
2dfd163430 Add more explicity run strategies in the runner matrix 2024-11-28 19:31:34 -08:00
sabaimran
80cd902c86 Since linux/amd64 images aren't being created, try setting a custom description on the image
Refer to this GH documentation on working with multi arch images in the container registry:
https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#adding-a-description-to-multi-arch-images
2024-11-28 19:14:06 -08:00
sabaimran
40d8a7a581 Release Khoj version 1.30.9 2024-11-28 18:45:50 -08:00
sabaimran
87aa653c7f Add additional steps in prod.Dockerfile to ensure dependencies are copied over 2024-11-28 18:37:08 -08:00
sabaimran
d91935c880 Initial commit of a functional but not yet elegant prototype for this concept 2024-11-28 17:28:23 -08:00
Debanjum
a552543f4f Use json5 to parse llm generated questions to query docs and web
Some checks failed
build khoj.el / build (push) Waiting to run
desktop / 🖥️ Build, Release Desktop App (push) Waiting to run
dockerize / Publish Khoj Docker Images (push) Waiting to run
build and deploy github pages for documentation / deploy (push) Waiting to run
pypi / Publish Python Package to PyPI (push) Waiting to run
test khoj.el / test (27.1) (push) Waiting to run
test khoj.el / test (27.2) (push) Waiting to run
test khoj.el / test (28.1) (push) Waiting to run
test khoj.el / test (28.2) (push) Waiting to run
test khoj.el / test (snapshot) (push) Waiting to run
test / Run Tests (push) Has been cancelled
pre-commit / Setup Application and Lint (push) Has been cancelled
json5 is more forgiving, handles double quotes, newlines in raw json
string
2024-11-28 14:35:34 -08:00
Debanjum
0a69af4f61 Update to latest ToDesktop runtime 2024-11-28 13:56:14 -08:00
Debanjum
1d0fe141dc Release Khoj version 1.30.8 2024-11-28 13:37:30 -08:00
Debanjum
29e801c381 Add MATH500 dataset to eval
Evaluate simpler MATH500 responses with gemini 1.5 flash

This improves both the speed and cost of running this eval
2024-11-28 12:48:25 -08:00
Debanjum
22aef9bf53 Add GPQA (diamond) dataset to eval 2024-11-28 12:48:25 -08:00
Debanjum
f1190ccf32
Improve parsing complex json strings returned by LLM (#989)
- Improve escaping to load complex json objects
- Fallback to a more forgiving [json5](https://json5.org/) loader if `json.loads` cannot parse complex json str

This should reduce failures to pick research tool and run code by agent
2024-11-28 11:01:39 -08:00
Debanjum
8c120a5139 Fallback to json5 loader if json.loads cannot parse complex json str
JSON5 spec is more flexible, try to load using a fast json5 parser if
the stricter json.loads from the standard library can't load the
raw complex json string into a python dictionary/list
2024-11-26 21:17:00 -08:00
Debanjum
70b7e7c73a Improve load of complex json objects. Use it to pick tool, run code
Gemini doesn't work well when trying to output json objects. Using it
to output raw json strings with complex, multi-line structures
requires more intense clean-up of raw json string for parsing
2024-11-26 17:37:57 -08:00
Debanjum
8cb0db0051 Fix llama-cpp-python install by pytest github workflow
Some checks failed
dockerize / Publish Khoj Docker Images (push) Has been cancelled
build and deploy github pages for documentation / deploy (push) Has been cancelled
pre-commit / Setup Application and Lint (push) Has been cancelled
pypi / Publish Python Package to PyPI (push) Has been cancelled
test / Run Tests (push) Has been cancelled
- Use pre-built wheels for torch and llama-cpp-python
- Install and link musl as it's used by llama-cpp-python pre-built
  wheel instead of glibc
- Join Install git and Install Dependencies steps in pytest workflow
  To remove unnecessary steps
2024-11-26 02:04:36 -08:00