sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-27 17:35:07 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

Add multi-user support to Khoj and use Postgres for backend storage (#549)

Browse source 
- Adds support for multiple users to be connected to the same Khoj instance using their Google login credentials
- Moves storage solution from in-memory json data to a Postgres db. This stores all relevant information, including accounts, embeddings, chat history, server side chat configuration
- Adds the concept of a Khoj server admin for configuring instance-wide settings regarding search model, and chat configuration
- Miscellaneous updates and fixes to the UX, including chat references, colors, and an updated config page
- Adds billing to allow users to subscribe to the cloud service easily
- Adds a separate GitHub action for building the dockerized production (tag `prod`) and dev (tag `dev`) images, separate from the image used for local building. The production image uses `gunicorn` with multiple workers to run the server.
- Updates all clients (Obsidian, Emacs, Desktop) to follow the client/server architecture. The server no longer reads from the file system at all; it only accepts data via the indexer API. In line with that, removes the functionality to configure org, markdown, plaintext, or other file-specific settings in the server. Only leaves GitHub and Notion for server-side configuration.
- Changes license to GNU AGPLv3

Resolves #467 
Resolves #488 
Resolves #303 
Resolves #345 
Resolves #195 
Resolves #280 
Resolves #461 
Closes #259 
Resolves #351
Resolves #301
Resolves #296
This commit is contained in:
sabaimran 2023-11-16 11:48:01 -08:00 committed by GitHub
commit e8a13f0813
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
147 changed files with 8704 additions and 4450 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

43
.github/workflows/dockerize_dev.yml vendored Normal file
View file

@ -0,0 +1,43 @@
name: dockerize-dev
on:
pull_request:
paths:
- src/khoj/**
- config/**
- pyproject.toml
- prod.Dockerfile
- .github/workflows/dockerize_dev.yml
workflow_dispatch:
env:
DOCKER_IMAGE_TAG: 'dev'
jobs:
build:
name: Build Production Docker Image, Push to Container Registry
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.PAT }}
- name: 📦 Build and Push Docker Image
uses: docker/build-push-action@v2
with:
context: .
file: prod.Dockerfile
platforms: linux/amd64
push: true
tags: ghcr.io/${{ github.repository }}-cloud:${{ env.DOCKER_IMAGE_TAG }}
build-args: |
PORT=42110

47
.github/workflows/dockerize_production.yml vendored Normal file
View file

@ -0,0 +1,47 @@
name: dockerize-prod
on:
push:
tags:
- "*"
branches:
- master
paths:
- src/khoj/**
- config/**
- pyproject.toml
- prod.Dockerfile
- .github/workflows/dockerize_production.yml
workflow_dispatch:
env:
DOCKER_IMAGE_TAG: ${{ github.ref == 'refs/heads/master' && 'latest' || github.ref_name }}
jobs:
build:
name: Build Production Docker Image, Push to Container Registry
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.PAT }}
- name: 📦 Build and Push Docker Image
uses: docker/build-push-action@v2
with:
context: .
file: prod.Dockerfile
platforms: linux/amd64
push: true
tags: ghcr.io/${{ github.repository }}-cloud:${{ env.DOCKER_IMAGE_TAG }}
build-args: |
PORT=42110

48
.github/workflows/pre-commit.yml vendored Normal file
View file

@ -0,0 +1,48 @@
name: pre-commit
on:
pull_request:
paths:
- src/**
- tests/**
- config/**
- pyproject.toml
- .pre-commit-config.yml
- .github/workflows/test.yml
push:
branches:
- master
paths:
- src/khoj/**
- tests/**
- config/**
- pyproject.toml
- .pre-commit-config.yml
- .github/workflows/test.yml
jobs:
test:
name: Run Tests
runs-on: ubuntu-latest
strategy:
fail-fast: false
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.11
- name: ⏬️ Install Dependencies
run: |
sudo apt update && sudo apt install -y libegl1
python -m pip install --upgrade pip
- name: ⬇️ Install Application
run: pip install --upgrade .[dev]
- name: 🌡️ Validate Application
run: pre-commit run --hook-stage manual --all

50
.github/workflows/test.yml vendored
View file

@ -2,10 +2,8 @@ name: test
on:
pull_request:
branches:
- 'master'
paths:
- src/khoj/**
- src/**
- tests/**
- config/**
- pyproject.toml
@ -13,7 +11,7 @@ on:
- .github/workflows/test.yml
push:
branches:
- 'master'
- master
paths:
- src/khoj/**
- tests/**
@ -26,6 +24,7 @@ jobs:
test:
name: Run Tests
runs-on: ubuntu-latest
container: ubuntu:jammy
strategy:
fail-fast: false
matrix:
@ -33,6 +32,17 @@ jobs:
- '3.9'
- '3.10'
- '3.11'
services:
postgres:
image: ankane/pgvector
env:
POSTGRES_PASSWORD: postgres
POSTGRES_USER: postgres
ports:
- 5432:5432
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
steps:
- uses: actions/checkout@v3
with:
@ -43,17 +53,37 @@ jobs:
with:
python-version: ${{ matrix.python_version }}
- name: ⏬️ Install Dependencies
- name: Install Git
run: |
sudo apt update && sudo apt install -y libegl1
apt update && apt install -y git
- name: ⏬️ Install Dependencies
env:
DEBIAN_FRONTEND: noninteractive
run: |
apt update && apt install -y libegl1 sqlite3 libsqlite3-dev libsqlite3-0 ffmpeg libsm6 libxext6
- name: ⬇️ Install Postgres
env:
DEBIAN_FRONTEND: noninteractive
run : |
apt install -y postgresql postgresql-client && apt install -y postgresql-server-dev-14
- name: ⬇️ Install pip
run: |
apt install -y python3-pip
python -m ensurepip --upgrade
python -m pip install --upgrade pip
- name: ⬇️ Install Application
run: pip install --upgrade .[dev]
- name: 🌡️ Validate Application
run: pre-commit run --hook-stage manual --all
run: sed -i 's/dynamic = \["version"\]/version = "0.0.0"/' pyproject.toml && pip install --upgrade .[dev]
- name: 🧪 Test Application
env:
POSTGRES_HOST: postgres
POSTGRES_PORT: 5432
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
run: pytest
timeout-minutes: 10

3
.gitignore vendored
View file

@ -21,7 +21,8 @@ todesktop.json
khoj_assistant.egg-info
/config/khoj*.yml
.pytest_cache
khoj.log
*.log
static
# Obsidian plugin artifacts
# ---

13
Dockerfile
View file

@ -5,14 +5,23 @@ LABEL org.opencontainers.image.source https://github.com/khoj-ai/khoj
# Install System Dependencies
RUN apt update -y && apt -y install python3-pip git
WORKDIR /app
# Install Application
COPY . .
COPY pyproject.toml .
COPY README.md .
RUN sed -i 's/dynamic = \["version"\]/version = "0.0.0"/' pyproject.toml && \
pip install --no-cache-dir .
# Copy Source Code
COPY . .
# Set the PYTHONPATH environment variable in order for it to find the Django app.
ENV PYTHONPATH=/app/src:$PYTHONPATH
# Run the Application
# There are more arguments required for the application to run,
# but these should be passed in through the docker-compose.yml file.
ARG PORT
EXPOSE ${PORT}
ENTRYPOINT ["khoj"]
ENTRYPOINT ["python3", "src/khoj/main.py"]

152
LICENSE
View file

@ -1,23 +1,21 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
@ -26,44 +24,34 @@ them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
@ -72,7 +60,7 @@ modification follow.
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
@ -549,35 +537,45 @@ to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
@ -619,3 +617,45 @@ Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.

53
docker-compose.yml
View file

@ -1,7 +1,29 @@
version: "3.9"
services:
database:
image: ankane/pgvector
ports:
- "5432:5432"
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
volumes:
- khoj_db:/var/lib/postgresql/data/
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 30s
timeout: 10s
retries: 5
server:
depends_on:
database:
condition: service_healthy
# Use the following line to use the latest version of khoj. Otherwise, it will build from source.
image: ghcr.io/khoj-ai/khoj:latest
# Uncomment the following line to build from source. This will take a few minutes. Comment the next two lines out if you want to use the offiicial image.
# build:
# context: .
ports:
# If changing the local port (left hand side), no other changes required.
# If changing the remote port (right hand side),
@ -10,26 +32,23 @@ services:
- "42110:42110"
working_dir: /app
volumes:
- .:/app
# These mounted volumes hold the raw data that should be indexed for search.
# The path in your local directory (left hand side)
# points to the files you want to index.
# The path of the mounted directory (right hand side),
# must match the path prefix in your config file.
- ./tests/data/org/:/data/org/
- ./tests/data/images/:/data/images/
- ./tests/data/markdown/:/data/markdown/
- ./tests/data/pdf/:/data/pdf/
# Embeddings and models are populated after the first run
# You can set these volumes to point to empty directories on host
- ./tests/data/embeddings/:/root/.khoj/content/
- ./tests/data/models/:/root/.khoj/search/
- khoj_config:/root/.khoj/
- sentence_tranformer_models:/root/.cache/torch/sentence_transformers
- khoj_models:/root/.cache/torch/sentence_transformers
# Use 0.0.0.0 to explicitly set the host ip for the service on the container. https://pythonspeed.com/articles/docker-connection-refused/
command: --host="0.0.0.0" --port=42110 -vv
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_HOST=database
- POSTGRES_PORT=5432
- KHOJ_DJANGO_SECRET_KEY=secret
- KHOJ_DEBUG=True
- KHOJ_ADMIN_EMAIL=username@example.com
- KHOJ_ADMIN_PASSWORD=password
command: --host="0.0.0.0" --port=42110 -vv --anonymous-mode
volumes:
khoj_config:
sentence_tranformer_models:
khoj_db:
khoj_models:

2
docs/github_integration.md
View file

@ -9,6 +9,6 @@ The Github integration allows you to index as many repositories as you want. It'
## Use the Github plugin
1. Generate a [classic PAT (personal access token)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) from [Github](https://github.com/settings/tokens) with `repo` and `admin:org` scopes at least.
2. Navigate to [http://localhost:42110/config/content_type/github](http://localhost:42110/config/content_type/github) to configure your Github settings. Enter in your PAT, along with details for each repository you want to index.
2. Navigate to [http://localhost:42110/config/content-source/github](http://localhost:42110/config/content-source/github) to configure your Github settings. Enter in your PAT, along with details for each repository you want to index.
3. Click `Save`. Go back to the settings page and click `Configure`.
4. Go to [http://localhost:42110/](http://localhost:42110/) and start searching!

2
docs/notion_integration.md
View file

@ -8,7 +8,7 @@ We haven't setup a fancy integration with OAuth yet, so this integration still r
![setup_new_integration](https://github.com/khoj-ai/khoj/assets/65192171/b056e057-d4dc-47dc-aad3-57b59a22c68b)
3. Share all the workspaces that you want to integrate with the Khoj integration you just made in the previous step
![enable_workspace](https://github.com/khoj-ai/khoj/assets/65192171/98290303-b5b8-4cb0-b32c-f68c6923a3d0)
4. In the first step, you generated an API key. Use the newly generated API Key in your Khoj settings, by default at http://localhost:42110/config/content_type/notion. Click `Save`.
4. In the first step, you generated an API key. Use the newly generated API Key in your Khoj settings, by default at http://localhost:42110/config/content-source/notion. Click `Save`.
5. Click `Configure` in http://localhost:42110/config to index your Notion workspace(s).
That's it! You should be ready to start searching and chatting. Make sure you've configured your OpenAI API Key for chat.

10
gunicorn-config.py Normal file
View file

@ -0,0 +1,10 @@
import multiprocessing
bind = "0.0.0.0:42110"
workers = 4
worker_class = "uvicorn.workers.UvicornWorker"
timeout = 120
keep_alive = 60
accesslog = "access.log"
errorlog = "error.log"
loglevel = "debug"

30
prod.Dockerfile Normal file
View file

@ -0,0 +1,30 @@
# Use Nvidia's latest Ubuntu 22.04 image as the base image
FROM nvidia/cuda:12.2.0-devel-ubuntu22.04
LABEL org.opencontainers.image.source https://github.com/khoj-ai/khoj
# Install System Dependencies
RUN apt update -y && apt -y install python3-pip git libsqlite3-0 ffmpeg libsm6 libxext6
WORKDIR /app
# Install Application
COPY pyproject.toml .
COPY README.md .
RUN sed -i 's/dynamic = \["version"\]/version = "0.0.0"/' pyproject.toml && \
TMPDIR=/home/cache/ pip install --cache-dir=/home/cache/ -e .
# Copy Source Code
COPY . .
RUN apt install vim -y
# Set the PYTHONPATH environment variable in order for it to find the Django app.
ENV PYTHONPATH=/app/src:$PYTHONPATH
# Run the Application
# There are more arguments required for the application to run,
# but these should be passed in through the docker-compose.yml file.
ARG PORT
EXPOSE ${PORT}
ENTRYPOINT [ "gunicorn", "-c", "gunicorn-config.py", "src.khoj.main:app" ]

22
pyproject.toml
View file

@ -54,14 +54,27 @@ dependencies = [
"transformers >= 4.28.0",
"torch == 2.0.1",
"uvicorn == 0.17.6",
"aiohttp == 3.8.5",
"langchain >= 0.0.187",
"aiohttp == 3.8.6",
"langchain >= 0.0.331",
"requests >= 2.26.0",
"bs4 >= 0.0.1",
"anyio == 3.7.1",
"pymupdf >= 1.23.3",
"pymupdf >= 1.23.5",
"django == 4.2.5",
"authlib == 1.2.1",
"gpt4all >= 2.0.0; platform_system == 'Linux' and platform_machine == 'x86_64'",
"gpt4all >= 2.0.0; platform_system == 'Windows' or platform_system == 'Darwin'",
"itsdangerous == 2.1.2",
"httpx == 0.25.0",
"pgvector == 0.2.3",
"psycopg2-binary == 2.9.9",
"google-auth == 2.23.3",
"python-multipart == 0.0.6",
"gunicorn == 21.2.0",
"lxml == 4.9.3",
"tzdata == 2023.3",
"rapidocr-onnxruntime == 1.3.8",
"stripe == 7.3.0",
]
dynamic = ["version"]
@ -81,12 +94,15 @@ test = [
"factory-boy >= 3.2.1",
"trio >= 0.22.0",
"pytest-xdist",
"psutil >= 5.8.0",
]
dev = [
"khoj-assistant[test]",
"mypy >= 1.0.1",
"black >= 23.1.0",
"pre-commit >= 3.0.4",
"pytest-django == 4.5.2",
"pytest-asyncio == 0.21.1",
]
[tool.hatch.version]

6
pytest.ini Normal file
View file

@ -0,0 +1,6 @@
[pytest]
DJANGO_SETTINGS_MODULE = app.settings
pythonpath = . src
testpaths = tests
markers =
chatquality: marks tests as chatquality (deselect with '-m "not chatquality"')

94
src/app/README.md Normal file
View file

@ -0,0 +1,94 @@
# Django App
Khoj uses Django as the backend framework primarily for its powerful ORM and the admin interface. The Django app is located in the `src/app` directory. We have one installed app, under the `/database/` directory. This app is responsible for all the database related operations and holds all of our models. You can find the extensive Django documentation [here](https://docs.djangoproject.com/en/4.2/) 🌈.
## Setup (Docker)
### Prerequisites
1. Ensure you have [Docker](https://docs.docker.com/get-docker/) installed.
2. Ensure you have [Docker Compose](https://docs.docker.com/compose/install/) installed.
### Run
Using the `docker-compose.yml` file in the root directory, you can run the Khoj app using the following command:
```bash
docker-compose up
```
## Setup (Local)
### Install Postgres (with PgVector)
#### MacOS
- Install the [Postgres.app](https://postgresapp.com/).
#### Debian, Ubuntu
From [official instructions](https://wiki.postgresql.org/wiki/Apt)
```bash
sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
sudo apt install postgres-16 postgresql-16-pgvector
```
#### Windows
- Use the [recommended installer](https://www.postgresql.org/download/windows/)
#### From Source
1. Follow instructions to [Install Postgres](https://www.postgresql.org/download/)
2. Follow instructions to [Install PgVector](https://github.com/pgvector/pgvector#installation) in case you need to manually install it. Reproduced instructions below for convenience.
```bash
cd /tmp
git clone --branch v0.5.1 https://github.com/pgvector/pgvector.git
cd pgvector
make
make install # may need sudo
```
### Create the Khoj database
#### MacOS
```bash
createdb khoj -U postgres
```
#### Debian, Ubuntu
```bash
sudo -u postgres createdb khoj
```
- [Optional] To set default postgres user's password
- Execute `ALTER USER postgres PASSWORD 'my_secure_password';` using `psql`
- Run `export $POSTGRES_PASSWORD=my_secure_password` in your terminal for Khoj to use it later
### Install Khoj
```bash
pip install -e '.[dev]'
```
### Make Khoj DB migrations
This command will create the migrations for the database app. This command should be run whenever a new db model is added to the database app or an existing db model is modified (updated or deleted).
```bash
python3 src/manage.py makemigrations
```
### Run Khoj DB migrations
This command will run any pending migrations in your application.
```bash
python3 src/manage.py migrate
```
### Start Khoj Server
While we're using Django for the ORM, we're still using the FastAPI server for the API. This command automatically scaffolds the Django application in the backend.
*Note: Anonymous mode bypasses authentication for local, single-user usage.*
```bash
python3 src/khoj/main.py --anonymous-mode
```

0
src/khoj/processor/jsonl/__init__.py → src/app/__init__.py
View file

152
src/app/settings.py Normal file
View file

@ -0,0 +1,152 @@
"""
Django settings for app project.
Generated by 'django-admin startproject' using Django 4.2.5.
For more information on this file, see
https://docs.djangoproject.com/en/4.2/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/4.2/ref/settings/
"""
from pathlib import Path
import os
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent.parent
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/4.2/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = os.getenv("KHOJ_DJANGO_SECRET_KEY")
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = os.getenv("KHOJ_DEBUG", "False") == "True"
ALLOWED_HOSTS = [".khoj.dev", "localhost", "127.0.0.1", "[::1]", "beta.khoj.dev"]
CSRF_TRUSTED_ORIGINS = [
"https://app.khoj.dev",
"https://beta.khoj.dev",
"https://khoj.dev",
"https://*.khoj.dev",
]
COOKIE_SAMESITE = "None"
if DEBUG:
SESSION_COOKIE_DOMAIN = "localhost"
CSRF_COOKIE_DOMAIN = "localhost"
else:
SESSION_COOKIE_DOMAIN = "khoj.dev"
CSRF_COOKIE_DOMAIN = "khoj.dev"
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
COOKIE_SAMESITE = "None"
SESSION_COOKIE_SAMESITE = "None"
# Application definition
INSTALLED_APPS = [
"django.contrib.auth",
"django.contrib.contenttypes",
"database.apps.DatabaseConfig",
"django.contrib.admin",
"django.contrib.sessions",
"django.contrib.messages",
"django.contrib.staticfiles",
]
MIDDLEWARE = [
"django.middleware.security.SecurityMiddleware",
"django.contrib.sessions.middleware.SessionMiddleware",
"django.middleware.common.CommonMiddleware",
"django.middleware.csrf.CsrfViewMiddleware",
"django.contrib.auth.middleware.AuthenticationMiddleware",
"django.contrib.messages.middleware.MessageMiddleware",
"django.middleware.clickjacking.XFrameOptionsMiddleware",
]
ROOT_URLCONF = "app.urls"
TEMPLATES = [
{
"BACKEND": "django.template.backends.django.DjangoTemplates",
"APP_DIRS": True,
"DIRS": [os.path.join(BASE_DIR, "templates"), os.path.join(BASE_DIR, "templates", "account")],
"OPTIONS": {
"context_processors": [
"django.template.context_processors.debug",
"django.template.context_processors.request",
"django.contrib.auth.context_processors.auth",
"django.contrib.messages.context_processors.messages",
],
},
},
]
WSGI_APPLICATION = "app.wsgi.application"
# Database
# https://docs.djangoproject.com/en/4.2/ref/settings/#databases
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"HOST": os.getenv("POSTGRES_HOST", "localhost"),
"PORT": os.getenv("POSTGRES_PORT", "5432"),
"USER": os.getenv("POSTGRES_USER", "postgres"),
"NAME": os.getenv("POSTGRES_DB", "khoj"),
"PASSWORD": os.getenv("POSTGRES_PASSWORD", "postgres"),
}
}
# User Settings
AUTH_USER_MODEL = "database.KhojUser"
# Password validation
# https://docs.djangoproject.com/en/4.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
"NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator",
},
{
"NAME": "django.contrib.auth.password_validation.MinimumLengthValidator",
},
{
"NAME": "django.contrib.auth.password_validation.CommonPasswordValidator",
},
{
"NAME": "django.contrib.auth.password_validation.NumericPasswordValidator",
},
]
# Internationalization
# https://docs.djangoproject.com/en/4.2/topics/i18n/
LANGUAGE_CODE = "en-us"
TIME_ZONE = "UTC"
USE_I18N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.2/howto/static-files/
STATIC_ROOT = BASE_DIR / "static"
STATICFILES_DIRS = [BASE_DIR / "src/khoj/interface/web"]
STATIC_URL = "/static/"
# Default primary key field type
# https://docs.djangoproject.com/en/4.2/ref/settings/#default-auto-field
DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"

25
src/app/urls.py Normal file
View file

@ -0,0 +1,25 @@
"""
URL configuration for app project.
The `urlpatterns` list routes URLs to views. For more information please see:
https://docs.djangoproject.com/en/4.2/topics/http/urls/
Examples:
Function views
1. Add an import: from my_app import views
2. Add a URL to urlpatterns: path('', views.home, name='home')
Class-based views
1. Add an import: from other_app.views import Home
2. Add a URL to urlpatterns: path('', Home.as_view(), name='home')
Including another URLconf
1. Import the include() function: from django.urls import include, path
2. Add a URL to urlpatterns: path('blog/', include('blog.urls'))
"""
from django.contrib import admin
from django.urls import path
from django.contrib.staticfiles.urls import staticfiles_urlpatterns
urlpatterns = [
path("admin/", admin.site.urls),
]
urlpatterns += staticfiles_urlpatterns()

16
src/app/wsgi.py Normal file
View file

@ -0,0 +1,16 @@
"""
WSGI config for app project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/4.2/howto/deployment/wsgi/
"""
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.settings")
application = get_wsgi_application()

0
src/database/__init__.py Normal file
View file

462
src/database/adapters/__init__.py Normal file
View file

@ -0,0 +1,462 @@
import math
from typing import Optional, Type, List
from datetime import date, datetime
import secrets
from typing import Type, List
from datetime import date, timezone
from django.db import models
from django.contrib.sessions.backends.db import SessionStore
from pgvector.django import CosineDistance
from django.db.models.manager import BaseManager
from django.db.models import Q
from torch import Tensor
# Import sync_to_async from Django Channels
from asgiref.sync import sync_to_async
from fastapi import HTTPException
from database.models import (
KhojUser,
GoogleUser,
KhojApiUser,
NotionConfig,
GithubConfig,
Entry,
GithubRepoConfig,
Conversation,
ChatModelOptions,
SearchModelConfig,
Subscription,
UserConversationConfig,
OpenAIProcessorConversationConfig,
OfflineChatProcessorConversationConfig,
)
from khoj.utils.helpers import generate_random_name
from khoj.search_filter.word_filter import WordFilter
from khoj.search_filter.file_filter import FileFilter
from khoj.search_filter.date_filter import DateFilter
async def set_notion_config(token: str, user: KhojUser):
notion_config = await NotionConfig.objects.filter(user=user).afirst()
if not notion_config:
notion_config = await NotionConfig.objects.acreate(token=token, user=user)
else:
notion_config.token = token
await notion_config.asave()
return notion_config
async def create_khoj_token(user: KhojUser, name=None):
"Create Khoj API key for user"
token = f"kk-{secrets.token_urlsafe(32)}"
name = name or f"{generate_random_name().title()}"
return await KhojApiUser.objects.acreate(token=token, user=user, name=name)
def get_khoj_tokens(user: KhojUser):
"Get all Khoj API keys for user"
return list(KhojApiUser.objects.filter(user=user))
async def delete_khoj_token(user: KhojUser, token: str):
"Delete Khoj API Key for user"
await KhojApiUser.objects.filter(token=token, user=user).adelete()
async def get_or_create_user(token: dict) -> KhojUser:
user = await get_user_by_token(token)
if not user:
user = await create_user_by_google_token(token)
return user
async def create_user_by_google_token(token: dict) -> KhojUser:
user, _ = await KhojUser.objects.filter(email=token.get("email")).aupdate_or_create(
defaults={"username": token.get("email"), "email": token.get("email")}
)
await user.asave()
await GoogleUser.objects.acreate(
sub=token.get("sub"),
azp=token.get("azp"),
email=token.get("email"),
name=token.get("name"),
given_name=token.get("given_name"),
family_name=token.get("family_name"),
picture=token.get("picture"),
locale=token.get("locale"),
user=user,
)
await Subscription.objects.acreate(user=user, type="trial")
return user
def get_user_subscription(email: str) -> Optional[Subscription]:
return Subscription.objects.filter(user__email=email).first()
async def set_user_subscription(
email: str, is_recurring=None, renewal_date=None, type="standard"
) -> Optional[Subscription]:
user_subscription = await Subscription.objects.filter(user__email=email).afirst()
if not user_subscription:
user = await get_user_by_email(email)
if not user:
return None
user_subscription = await Subscription.objects.acreate(
user=user, type=type, is_recurring=is_recurring, renewal_date=renewal_date
)
return user_subscription
elif user_subscription:
user_subscription.type = type
if is_recurring is not None:
user_subscription.is_recurring = is_recurring
if renewal_date is False:
user_subscription.renewal_date = None
elif renewal_date is not None:
user_subscription.renewal_date = renewal_date
await user_subscription.asave()
return user_subscription
else:
return None
def get_user_subscription_state(email: str) -> str:
"""Get subscription state of user
Valid state transitions: trial -> subscribed <-> unsubscribed OR expired
"""
user_subscription = Subscription.objects.filter(user__email=email).first()
if not user_subscription:
return "trial"
elif user_subscription.type == Subscription.Type.TRIAL:
return "trial"
elif user_subscription.is_recurring and user_subscription.renewal_date >= datetime.now(tz=timezone.utc):
return "subscribed"
elif not user_subscription.is_recurring and user_subscription.renewal_date >= datetime.now(tz=timezone.utc):
return "unsubscribed"
elif not user_subscription.is_recurring and user_subscription.renewal_date < datetime.now(tz=timezone.utc):
return "expired"
return "invalid"
async def get_user_by_email(email: str) -> KhojUser:
return await KhojUser.objects.filter(email=email).afirst()
async def get_user_by_token(token: dict) -> KhojUser:
google_user = await GoogleUser.objects.filter(sub=token.get("sub")).select_related("user").afirst()
if not google_user:
return None
return google_user.user
async def retrieve_user(session_id: str) -> KhojUser:
session = SessionStore(session_key=session_id)
if not await sync_to_async(session.exists)(session_key=session_id):
raise HTTPException(status_code=401, detail="Invalid session")
session_data = await sync_to_async(session.load)()
user = await KhojUser.objects.filter(id=session_data.get("_auth_user_id")).afirst()
if not user:
raise HTTPException(status_code=401, detail="Invalid user")
return user
def get_all_users() -> BaseManager[KhojUser]:
return KhojUser.objects.all()
def get_user_github_config(user: KhojUser):
config = GithubConfig.objects.filter(user=user).prefetch_related("githubrepoconfig").first()
return config
def get_user_notion_config(user: KhojUser):
config = NotionConfig.objects.filter(user=user).first()
return config
async def set_text_content_config(user: KhojUser, object: Type[models.Model], updated_config):
deduped_files = list(set(updated_config.input_files)) if updated_config.input_files else None
deduped_filters = list(set(updated_config.input_filter)) if updated_config.input_filter else None
await object.objects.filter(user=user).adelete()
await object.objects.acreate(
input_files=deduped_files,
input_filter=deduped_filters,
index_heading_entries=updated_config.index_heading_entries,
user=user,
)
async def set_user_github_config(user: KhojUser, pat_token: str, repos: list):
config = await GithubConfig.objects.filter(user=user).afirst()
if not config:
config = await GithubConfig.objects.acreate(pat_token=pat_token, user=user)
else:
config.pat_token = pat_token
await config.asave()
await config.githubrepoconfig.all().adelete()
for repo in repos:
await GithubRepoConfig.objects.acreate(
name=repo["name"], owner=repo["owner"], branch=repo["branch"], github_config=config
)
return config
def get_or_create_search_model():
search_model = SearchModelConfig.objects.filter().first()
if not search_model:
search_model = SearchModelConfig.objects.create()
return search_model
class ConversationAdapters:
@staticmethod
def get_conversation_by_user(user: KhojUser):
conversation = Conversation.objects.filter(user=user)
if conversation.exists():
return conversation.first()
return Conversation.objects.create(user=user)
@staticmethod
async def aget_conversation_by_user(user: KhojUser):
conversation = Conversation.objects.filter(user=user)
if await conversation.aexists():
return await conversation.afirst()
return await Conversation.objects.acreate(user=user)
@staticmethod
def has_any_conversation_config(user: KhojUser):
return ChatModelOptions.objects.filter(user=user).exists()
@staticmethod
def get_openai_conversation_config():
return OpenAIProcessorConversationConfig.objects.filter().first()
@staticmethod
def get_offline_chat_conversation_config():
return OfflineChatProcessorConversationConfig.objects.filter().first()
@staticmethod
def has_valid_offline_conversation_config():
return OfflineChatProcessorConversationConfig.objects.filter(enabled=True).exists()
@staticmethod
def has_valid_openai_conversation_config():
return OpenAIProcessorConversationConfig.objects.filter().exists()
@staticmethod
async def aset_user_conversation_processor(user: KhojUser, conversation_processor_config_id: int):
config = await ChatModelOptions.objects.filter(id=conversation_processor_config_id).afirst()
if not config:
return None
new_config = await UserConversationConfig.objects.aupdate_or_create(user=user, defaults={"setting": config})
return new_config
@staticmethod
def get_conversation_config(user: KhojUser):
config = UserConversationConfig.objects.filter(user=user).first()
if not config:
return None
return config.setting
@staticmethod
def get_default_conversation_config():
return ChatModelOptions.objects.filter().first()
@staticmethod
def save_conversation(user: KhojUser, conversation_log: dict):
conversation = Conversation.objects.filter(user=user)
if conversation.exists():
conversation.update(conversation_log=conversation_log)
else:
Conversation.objects.create(user=user, conversation_log=conversation_log)
@staticmethod
def get_conversation_processor_options():
return ChatModelOptions.objects.all()
@staticmethod
def set_conversation_processor_config(user: KhojUser, new_config: ChatModelOptions):
user_conversation_config, _ = UserConversationConfig.objects.get_or_create(user=user)
user_conversation_config.setting = new_config
user_conversation_config.save()
@staticmethod
def has_offline_chat():
return OfflineChatProcessorConversationConfig.objects.filter(enabled=True).exists()
@staticmethod
async def ahas_offline_chat():
return await OfflineChatProcessorConversationConfig.objects.filter(enabled=True).aexists()
@staticmethod
async def get_offline_chat():
return await ChatModelOptions.objects.filter(model_type="offline").afirst()
@staticmethod
async def aget_user_conversation_config(user: KhojUser):
config = await UserConversationConfig.objects.filter(user=user).prefetch_related("setting").afirst()
if not config:
return None
return config.setting
@staticmethod
async def has_openai_chat():
return await OpenAIProcessorConversationConfig.objects.filter().aexists()
@staticmethod
async def get_openai_chat():
return await ChatModelOptions.objects.filter(model_type="openai").afirst()
@staticmethod
async def get_openai_chat_config():
return await OpenAIProcessorConversationConfig.objects.filter().afirst()
@staticmethod
async def aget_default_conversation_config():
return await ChatModelOptions.objects.filter().afirst()
class EntryAdapters:
word_filer = WordFilter()
file_filter = FileFilter()
date_filter = DateFilter()
@staticmethod
def does_entry_exist(user: KhojUser, hashed_value: str) -> bool:
return Entry.objects.filter(user=user, hashed_value=hashed_value).exists()
@staticmethod
def delete_entry_by_file(user: KhojUser, file_path: str):
deleted_count, _ = Entry.objects.filter(user=user, file_path=file_path).delete()
return deleted_count
@staticmethod
def delete_all_entries_by_type(user: KhojUser, file_type: str = None):
if file_type is None:
deleted_count, _ = Entry.objects.filter(user=user).delete()
else:
deleted_count, _ = Entry.objects.filter(user=user, file_type=file_type).delete()
return deleted_count
@staticmethod
def delete_all_entries(user: KhojUser, file_source: str = None):
if file_source is None:
deleted_count, _ = Entry.objects.filter(user=user).delete()
else:
deleted_count, _ = Entry.objects.filter(user=user, file_source=file_source).delete()
return deleted_count
@staticmethod
def get_existing_entry_hashes_by_file(user: KhojUser, file_path: str):
return Entry.objects.filter(user=user, file_path=file_path).values_list("hashed_value", flat=True)
@staticmethod
def delete_entry_by_hash(user: KhojUser, hashed_values: List[str]):
Entry.objects.filter(user=user, hashed_value__in=hashed_values).delete()
@staticmethod
def get_entries_by_date_filter(entry: BaseManager[Entry], start_date: date, end_date: date):
return entry.filter(
entrydates__date__gte=start_date,
entrydates__date__lte=end_date,
)
@staticmethod
def user_has_entries(user: KhojUser):
return Entry.objects.filter(user=user).exists()
@staticmethod
async def adelete_entry_by_file(user: KhojUser, file_path: str):
return await Entry.objects.filter(user=user, file_path=file_path).adelete()
@staticmethod
def aget_all_filenames_by_source(user: KhojUser, file_source: str):
return (
Entry.objects.filter(user=user, file_source=file_source)
.distinct("file_path")
.values_list("file_path", flat=True)
)
@staticmethod
async def adelete_all_entries(user: KhojUser):
return await Entry.objects.filter(user=user).adelete()
@staticmethod
def apply_filters(user: KhojUser, query: str, file_type_filter: str = None):
q_filter_terms = Q()
explicit_word_terms = EntryAdapters.word_filer.get_filter_terms(query)
file_filters = EntryAdapters.file_filter.get_filter_terms(query)
date_filters = EntryAdapters.date_filter.get_query_date_range(query)
if len(explicit_word_terms) == 0 and len(file_filters) == 0 and len(date_filters) == 0:
return Entry.objects.filter(user=user)
for term in explicit_word_terms:
if term.startswith("+"):
q_filter_terms &= Q(raw__icontains=term[1:])
elif term.startswith("-"):
q_filter_terms &= ~Q(raw__icontains=term[1:])
q_file_filter_terms = Q()
if len(file_filters) > 0:
for term in file_filters:
q_file_filter_terms |= Q(file_path__regex=term)
q_filter_terms &= q_file_filter_terms
if len(date_filters) > 0:
min_date, max_date = date_filters
if min_date is not None:
# Convert the min_date timestamp to yyyy-mm-dd format
formatted_min_date = date.fromtimestamp(min_date).strftime("%Y-%m-%d")
q_filter_terms &= Q(embeddings_dates__date__gte=formatted_min_date)
if max_date is not None:
# Convert the max_date timestamp to yyyy-mm-dd format
formatted_max_date = date.fromtimestamp(max_date).strftime("%Y-%m-%d")
q_filter_terms &= Q(embeddings_dates__date__lte=formatted_max_date)
relevant_entries = Entry.objects.filter(user=user).filter(
q_filter_terms,
)
if file_type_filter:
relevant_entries = relevant_entries.filter(file_type=file_type_filter)
return relevant_entries
@staticmethod
def search_with_embeddings(
user: KhojUser,
embeddings: Tensor,
max_results: int = 10,
file_type_filter: str = None,
raw_query: str = None,
max_distance: float = math.inf,
):
relevant_entries = EntryAdapters.apply_filters(user, raw_query, file_type_filter)
relevant_entries = relevant_entries.filter(user=user).annotate(
distance=CosineDistance("embeddings", embeddings)
)
relevant_entries = relevant_entries.filter(distance__lte=max_distance)
if file_type_filter:
relevant_entries = relevant_entries.filter(file_type=file_type_filter)
relevant_entries = relevant_entries.order_by("distance")
return relevant_entries[:max_results]
@staticmethod
def get_unique_file_types(user: KhojUser):
return Entry.objects.filter(user=user).values_list("file_type", flat=True).distinct()
@staticmethod
def get_unique_file_sources(user: KhojUser):
return Entry.objects.filter(user=user).values_list("file_source", flat=True).distinct().all()

21
src/database/admin.py Normal file
View file

@ -0,0 +1,21 @@
from django.contrib import admin
from django.contrib.auth.admin import UserAdmin
# Register your models here.
from database.models import (
KhojUser,
ChatModelOptions,
OpenAIProcessorConversationConfig,
OfflineChatProcessorConversationConfig,
SearchModelConfig,
Subscription,
)
admin.site.register(KhojUser, UserAdmin)
admin.site.register(ChatModelOptions)
admin.site.register(OpenAIProcessorConversationConfig)
admin.site.register(OfflineChatProcessorConversationConfig)
admin.site.register(SearchModelConfig)
admin.site.register(Subscription)

6
src/database/apps.py Normal file
View file

@ -0,0 +1,6 @@
from django.apps import AppConfig
class DatabaseConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "database"

98
src/database/migrations/0001_khojuser.py Normal file
View file

@ -0,0 +1,98 @@
# Generated by Django 4.2.5 on 2023-09-14 19:00
import django.contrib.auth.models
import django.contrib.auth.validators
from django.db import migrations, models
import django.utils.timezone
class Migration(migrations.Migration):
initial = True
dependencies = [
("auth", "0012_alter_user_first_name_max_length"),
]
run_before = [
("admin", "0001_initial"),
]
operations = [
migrations.CreateModel(
name="KhojUser",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("password", models.CharField(max_length=128, verbose_name="password")),
("last_login", models.DateTimeField(blank=True, null=True, verbose_name="last login")),
(
"is_superuser",
models.BooleanField(
default=False,
help_text="Designates that this user has all permissions without explicitly assigning them.",
verbose_name="superuser status",
),
),
(
"username",
models.CharField(
error_messages={"unique": "A user with that username already exists."},
help_text="Required. 150 characters or fewer. Letters, digits and @/./+/-/_ only.",
max_length=150,
unique=True,
validators=[django.contrib.auth.validators.UnicodeUsernameValidator()],
verbose_name="username",
),
),
("first_name", models.CharField(blank=True, max_length=150, verbose_name="first name")),
("last_name", models.CharField(blank=True, max_length=150, verbose_name="last name")),
("email", models.EmailField(blank=True, max_length=254, verbose_name="email address")),
(
"is_staff",
models.BooleanField(
default=False,
help_text="Designates whether the user can log into this admin site.",
verbose_name="staff status",
),
),
(
"is_active",
models.BooleanField(
default=True,
help_text="Designates whether this user should be treated as active. Unselect this instead of deleting accounts.",
verbose_name="active",
),
),
("date_joined", models.DateTimeField(default=django.utils.timezone.now, verbose_name="date joined")),
(
"groups",
models.ManyToManyField(
blank=True,
help_text="The groups this user belongs to. A user will get all permissions granted to each of their groups.",
related_name="user_set",
related_query_name="user",
to="auth.group",
verbose_name="groups",
),
),
(
"user_permissions",
models.ManyToManyField(
blank=True,
help_text="Specific permissions for this user.",
related_name="user_set",
related_query_name="user",
to="auth.permission",
verbose_name="user permissions",
),
),
],
options={
"verbose_name": "user",
"verbose_name_plural": "users",
"abstract": False,
},
managers=[
("objects", django.contrib.auth.models.UserManager()),
],
),
]

32
src/database/migrations/0002_googleuser.py Normal file
View file

@ -0,0 +1,32 @@
# Generated by Django 4.2.4 on 2023-09-18 23:24
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("database", "0001_khojuser"),
]
operations = [
migrations.CreateModel(
name="GoogleUser",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("sub", models.CharField(max_length=200)),
("azp", models.CharField(max_length=200)),
("email", models.CharField(max_length=200)),
("name", models.CharField(max_length=200)),
("given_name", models.CharField(max_length=200)),
("family_name", models.CharField(max_length=200)),
("picture", models.CharField(max_length=200)),
("locale", models.CharField(max_length=200)),
(
"user",
models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL),
),
],
),
]

10
src/database/migrations/0003_vector_extension.py Normal file
View file

@ -0,0 +1,10 @@
from django.db import migrations
from pgvector.django import VectorExtension
class Migration(migrations.Migration):
dependencies = [
("database", "0002_googleuser"),
]
operations = [VectorExtension()]

180
src/database/migrations/0004_content_types_and_more.py Normal file
View file

@ -0,0 +1,180 @@
# Generated by Django 4.2.5 on 2023-10-11 22:24
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
import pgvector.django
import uuid
class Migration(migrations.Migration):
dependencies = [
("database", "0003_vector_extension"),
]
operations = [
migrations.CreateModel(
name="GithubConfig",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("pat_token", models.CharField(max_length=200)),
],
options={
"abstract": False,
},
),
migrations.AddField(
model_name="khojuser",
name="uuid",
field=models.UUIDField(default=1234, verbose_name=models.UUIDField(default=uuid.uuid4, editable=False)),
preserve_default=False,
),
migrations.CreateModel(
name="NotionConfig",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("token", models.CharField(max_length=200)),
("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
],
options={
"abstract": False,
},
),
migrations.CreateModel(
name="LocalPlaintextConfig",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("input_files", models.JSONField(default=list, null=True)),
("input_filter", models.JSONField(default=list, null=True)),
("index_heading_entries", models.BooleanField(default=False)),
("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
],
options={
"abstract": False,
},
),
migrations.CreateModel(
name="LocalPdfConfig",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("input_files", models.JSONField(default=list, null=True)),
("input_filter", models.JSONField(default=list, null=True)),
("index_heading_entries", models.BooleanField(default=False)),
("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
],
options={
"abstract": False,
},
),
migrations.CreateModel(
name="LocalOrgConfig",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("input_files", models.JSONField(default=list, null=True)),
("input_filter", models.JSONField(default=list, null=True)),
("index_heading_entries", models.BooleanField(default=False)),
("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
],
options={
"abstract": False,
},
),
migrations.CreateModel(
name="LocalMarkdownConfig",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("input_files", models.JSONField(default=list, null=True)),
("input_filter", models.JSONField(default=list, null=True)),
("index_heading_entries", models.BooleanField(default=False)),
("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
],
options={
"abstract": False,
},
),
migrations.CreateModel(
name="GithubRepoConfig",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("name", models.CharField(max_length=200)),
("owner", models.CharField(max_length=200)),
("branch", models.CharField(max_length=200)),
(
"github_config",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="githubrepoconfig",
to="database.githubconfig",
),
),
],
options={
"abstract": False,
},
),
migrations.AddField(
model_name="githubconfig",
name="user",
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL),
),
migrations.CreateModel(
name="Embeddings",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("embeddings", pgvector.django.VectorField(dimensions=384)),
("raw", models.TextField()),
("compiled", models.TextField()),
("heading", models.CharField(blank=True, default=None, max_length=1000, null=True)),
(
"file_type",
models.CharField(
choices=[
("image", "Image"),
("pdf", "Pdf"),
("plaintext", "Plaintext"),
("markdown", "Markdown"),
("org", "Org"),
("notion", "Notion"),
("github", "Github"),
("conversation", "Conversation"),
],
default="plaintext",
max_length=30,
),
),
("file_path", models.CharField(blank=True, default=None, max_length=400, null=True)),
("file_name", models.CharField(blank=True, default=None, max_length=400, null=True)),
("url", models.URLField(blank=True, default=None, max_length=400, null=True)),
("hashed_value", models.CharField(max_length=100)),
(
"user",
models.ForeignKey(
blank=True,
default=None,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to=settings.AUTH_USER_MODEL,
),
),
],
options={
"abstract": False,
},
),
]

18
src/database/migrations/0005_embeddings_corpus_id.py Normal file
View file

@ -0,0 +1,18 @@
# Generated by Django 4.2.5 on 2023-10-13 02:39
from django.db import migrations, models
import uuid
class Migration(migrations.Migration):
dependencies = [
("database", "0004_content_types_and_more"),
]
operations = [
migrations.AddField(
model_name="embeddings",
name="corpus_id",
field=models.UUIDField(default=uuid.uuid4, editable=False),
),
]

33
src/database/migrations/0006_embeddingsdates.py Normal file
View file

@ -0,0 +1,33 @@
# Generated by Django 4.2.5 on 2023-10-13 19:28
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("database", "0005_embeddings_corpus_id"),
]
operations = [
migrations.CreateModel(
name="EmbeddingsDates",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("date", models.DateField()),
(
"embeddings",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="embeddings_dates",
to="database.embeddings",
),
),
],
options={
"indexes": [models.Index(fields=["date"], name="database_em_date_a1ba47_idx")],
},
),
]

27
src/database/migrations/0007_add_conversation.py Normal file
View file

@ -0,0 +1,27 @@
# Generated by Django 4.2.5 on 2023-10-18 05:31
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("database", "0006_embeddingsdates"),
]
operations = [
migrations.CreateModel(
name="Conversation",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("conversation_log", models.JSONField()),
("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
],
options={
"abstract": False,
},
),
]

17
src/database/migrations/0008_alter_conversation_conversation_log.py Normal file
View file

@ -0,0 +1,17 @@
# Generated by Django 4.2.5 on 2023-10-18 16:46
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("database", "0007_add_conversation"),
]
operations = [
migrations.AlterField(
model_name="conversation",
name="conversation_log",
field=models.JSONField(default=dict),
),
]

24
src/database/migrations/0009_khojapiuser.py Normal file
View file

@ -0,0 +1,24 @@
# Generated by Django 4.2.5 on 2023-10-26 17:02
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("database", "0008_alter_conversation_conversation_log"),
]
operations = [
migrations.CreateModel(
name="KhojApiUser",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("token", models.CharField(max_length=50, unique=True)),
("name", models.CharField(max_length=50)),
("accessed_at", models.DateTimeField(default=None, null=True)),
("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
],
),
]

83
src/database/migrations/0010_chatmodeloptions_and_more.py Normal file
View file

@ -0,0 +1,83 @@
# Generated by Django 4.2.4 on 2023-11-01 17:41
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("database", "0009_khojapiuser"),
]
operations = [
migrations.CreateModel(
name="ChatModelOptions",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("max_prompt_size", models.IntegerField(blank=True, default=None, null=True)),
("tokenizer", models.CharField(blank=True, default=None, max_length=200, null=True)),
("chat_model", models.CharField(blank=True, default=None, max_length=200, null=True)),
(
"model_type",
models.CharField(
choices=[("openai", "Openai"), ("offline", "Offline")], default="openai", max_length=200
),
),
],
options={
"abstract": False,
},
),
migrations.CreateModel(
name="OfflineChatProcessorConversationConfig",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("enabled", models.BooleanField(default=False)),
],
options={
"abstract": False,
},
),
migrations.CreateModel(
name="OpenAIProcessorConversationConfig",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("api_key", models.CharField(max_length=200)),
],
options={
"abstract": False,
},
),
migrations.CreateModel(
name="UserConversationConfig",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
(
"setting",
models.ForeignKey(
blank=True,
default=None,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="database.chatmodeloptions",
),
),
(
"user",
models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL),
),
],
options={
"abstract": False,
},
),
]

30
src/database/migrations/0010_rename_embeddings_entry_and_more.py Normal file
View file

@ -0,0 +1,30 @@
# Generated by Django 4.2.5 on 2023-10-26 23:52
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("database", "0009_khojapiuser"),
]
operations = [
migrations.RenameModel(
old_name="Embeddings",
new_name="Entry",
),
migrations.RenameModel(
old_name="EmbeddingsDates",
new_name="EntryDates",
),
migrations.RenameField(
model_name="entrydates",
old_name="embeddings",
new_name="entry",
),
migrations.RenameIndex(
model_name="entrydates",
new_name="database_en_date_8d823c_idx",
old_name="database_em_date_a1ba47_idx",
),
]

12
src/database/migrations/0011_merge_20231102_0138.py Normal file
View file

@ -0,0 +1,12 @@
# Generated by Django 4.2.5 on 2023-11-02 01:38
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("database", "0010_chatmodeloptions_and_more"),
("database", "0010_rename_embeddings_entry_and_more"),
]
operations = []

21
src/database/migrations/0012_entry_file_source.py Normal file
View file

@ -0,0 +1,21 @@
# Generated by Django 4.2.5 on 2023-11-07 07:24
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("database", "0011_merge_20231102_0138"),
]
operations = [
migrations.AddField(
model_name="entry",
name="file_source",
field=models.CharField(
choices=[("computer", "Computer"), ("notion", "Notion"), ("github", "Github")],
default="computer",
max_length=30,
),
),
]

37
src/database/migrations/0013_subscription.py Normal file
View file

@ -0,0 +1,37 @@
# Generated by Django 4.2.5 on 2023-11-09 01:27
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("database", "0012_entry_file_source"),
]
operations = [
migrations.CreateModel(
name="Subscription",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
(
"type",
models.CharField(
choices=[("trial", "Trial"), ("standard", "Standard")], default="trial", max_length=20
),
),
("is_recurring", models.BooleanField(default=False)),
("renewal_date", models.DateTimeField(default=None, null=True)),
(
"user",
models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL),
),
],
options={
"abstract": False,
},
),
]

17
src/database/migrations/0014_alter_googleuser_picture.py Normal file
View file

@ -0,0 +1,17 @@
# Generated by Django 4.2.5 on 2023-11-09 08:11
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("database", "0013_subscription"),
]
operations = [
migrations.AlterField(
model_name="googleuser",
name="picture",
field=models.CharField(default=None, max_length=200, null=True),
),
]

21
src/database/migrations/0015_alter_subscription_user.py Normal file
View file

@ -0,0 +1,21 @@
# Generated by Django 4.2.5 on 2023-11-11 05:39
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("database", "0014_alter_googleuser_picture"),
]
operations = [
migrations.AlterField(
model_name="subscription",
name="user",
field=models.OneToOneField(
on_delete=django.db.models.deletion.CASCADE, related_name="subscription", to=settings.AUTH_USER_MODEL
),
),
]

17
src/database/migrations/0016_alter_subscription_renewal_date.py Normal file
View file

@ -0,0 +1,17 @@
# Generated by Django 4.2.5 on 2023-11-11 06:15
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("database", "0015_alter_subscription_user"),
]
operations = [
migrations.AlterField(
model_name="subscription",
name="renewal_date",
field=models.DateTimeField(blank=True, default=None, null=True),
),
]

32
src/database/migrations/0017_searchmodel.py Normal file
View file

@ -0,0 +1,32 @@
# Generated by Django 4.2.5 on 2023-11-14 23:25
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("database", "0016_alter_subscription_renewal_date"),
]
operations = [
migrations.CreateModel(
name="SearchModel",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("name", models.CharField(default="default", max_length=200)),
("model_type", models.CharField(choices=[("text", "Text")], default="text", max_length=200)),
("bi_encoder", models.CharField(default="thenlper/gte-small", max_length=200)),
(
"cross_encoder",
models.CharField(
blank=True, default="cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=200, null=True
),
),
],
options={
"abstract": False,
},
),
]

30
src/database/migrations/0018_searchmodelconfig_delete_searchmodel.py Normal file
View file

@ -0,0 +1,30 @@
# Generated by Django 4.2.5 on 2023-11-16 01:13
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("database", "0017_searchmodel"),
]
operations = [
migrations.CreateModel(
name="SearchModelConfig",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("name", models.CharField(default="default", max_length=200)),
("model_type", models.CharField(choices=[("text", "Text")], default="text", max_length=200)),
("bi_encoder", models.CharField(default="thenlper/gte-small", max_length=200)),
("cross_encoder", models.CharField(default="cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=200)),
],
options={
"abstract": False,
},
),
migrations.DeleteModel(
name="SearchModel",
),
]

0
src/database/migrations/__init__.py Normal file
View file

181
src/database/models/__init__.py Normal file
View file

@ -0,0 +1,181 @@
import uuid
from django.db import models
from django.contrib.auth.models import AbstractUser
from pgvector.django import VectorField
class BaseModel(models.Model):
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
class Meta:
abstract = True
class KhojUser(AbstractUser):
uuid = models.UUIDField(models.UUIDField(default=uuid.uuid4, editable=False))
def save(self, *args, **kwargs):
if not self.uuid:
self.uuid = uuid.uuid4()
super().save(*args, **kwargs)
class GoogleUser(models.Model):
user = models.OneToOneField(KhojUser, on_delete=models.CASCADE)
sub = models.CharField(max_length=200)
azp = models.CharField(max_length=200)
email = models.CharField(max_length=200)
name = models.CharField(max_length=200)
given_name = models.CharField(max_length=200)
family_name = models.CharField(max_length=200)
picture = models.CharField(max_length=200, null=True, default=None)
locale = models.CharField(max_length=200)
def __str__(self):
return self.name
class KhojApiUser(models.Model):
"""User issued API tokens to authenticate Khoj clients"""
user = models.ForeignKey(KhojUser, on_delete=models.CASCADE)
token = models.CharField(max_length=50, unique=True)
name = models.CharField(max_length=50)
accessed_at = models.DateTimeField(null=True, default=None)
class Subscription(BaseModel):
class Type(models.TextChoices):
TRIAL = "trial"
STANDARD = "standard"
user = models.OneToOneField(KhojUser, on_delete=models.CASCADE, related_name="subscription")
type = models.CharField(max_length=20, choices=Type.choices, default=Type.TRIAL)
is_recurring = models.BooleanField(default=False)
renewal_date = models.DateTimeField(null=True, default=None, blank=True)
class NotionConfig(BaseModel):
token = models.CharField(max_length=200)
user = models.ForeignKey(KhojUser, on_delete=models.CASCADE)
class GithubConfig(BaseModel):
pat_token = models.CharField(max_length=200)
user = models.ForeignKey(KhojUser, on_delete=models.CASCADE)
class GithubRepoConfig(BaseModel):
name = models.CharField(max_length=200)
owner = models.CharField(max_length=200)
branch = models.CharField(max_length=200)
github_config = models.ForeignKey(GithubConfig, on_delete=models.CASCADE, related_name="githubrepoconfig")
class LocalOrgConfig(BaseModel):
input_files = models.JSONField(default=list, null=True)
input_filter = models.JSONField(default=list, null=True)
index_heading_entries = models.BooleanField(default=False)
user = models.ForeignKey(KhojUser, on_delete=models.CASCADE)
class LocalMarkdownConfig(BaseModel):
input_files = models.JSONField(default=list, null=True)
input_filter = models.JSONField(default=list, null=True)
index_heading_entries = models.BooleanField(default=False)
user = models.ForeignKey(KhojUser, on_delete=models.CASCADE)
class LocalPdfConfig(BaseModel):
input_files = models.JSONField(default=list, null=True)
input_filter = models.JSONField(default=list, null=True)
index_heading_entries = models.BooleanField(default=False)
user = models.ForeignKey(KhojUser, on_delete=models.CASCADE)
class LocalPlaintextConfig(BaseModel):
input_files = models.JSONField(default=list, null=True)
input_filter = models.JSONField(default=list, null=True)
index_heading_entries = models.BooleanField(default=False)
user = models.ForeignKey(KhojUser, on_delete=models.CASCADE)
class SearchModelConfig(BaseModel):
class ModelType(models.TextChoices):
TEXT = "text"
name = models.CharField(max_length=200, default="default")
model_type = models.CharField(max_length=200, choices=ModelType.choices, default=ModelType.TEXT)
bi_encoder = models.CharField(max_length=200, default="thenlper/gte-small")
cross_encoder = models.CharField(max_length=200, default="cross-encoder/ms-marco-MiniLM-L-6-v2")
class OpenAIProcessorConversationConfig(BaseModel):
api_key = models.CharField(max_length=200)
class OfflineChatProcessorConversationConfig(BaseModel):
enabled = models.BooleanField(default=False)
class ChatModelOptions(BaseModel):
class ModelType(models.TextChoices):
OPENAI = "openai"
OFFLINE = "offline"
max_prompt_size = models.IntegerField(default=None, null=True, blank=True)
tokenizer = models.CharField(max_length=200, default=None, null=True, blank=True)
chat_model = models.CharField(max_length=200, default=None, null=True, blank=True)
model_type = models.CharField(max_length=200, choices=ModelType.choices, default=ModelType.OPENAI)
class UserConversationConfig(BaseModel):
user = models.OneToOneField(KhojUser, on_delete=models.CASCADE)
setting = models.ForeignKey(ChatModelOptions, on_delete=models.CASCADE, default=None, null=True, blank=True)
class Conversation(BaseModel):
user = models.ForeignKey(KhojUser, on_delete=models.CASCADE)
conversation_log = models.JSONField(default=dict)
class Entry(BaseModel):
class EntryType(models.TextChoices):
IMAGE = "image"
PDF = "pdf"
PLAINTEXT = "plaintext"
MARKDOWN = "markdown"
ORG = "org"
NOTION = "notion"
GITHUB = "github"
CONVERSATION = "conversation"
class EntrySource(models.TextChoices):
COMPUTER = "computer"
NOTION = "notion"
GITHUB = "github"
user = models.ForeignKey(KhojUser, on_delete=models.CASCADE, default=None, null=True, blank=True)
embeddings = VectorField(dimensions=384)
raw = models.TextField()
compiled = models.TextField()
heading = models.CharField(max_length=1000, default=None, null=True, blank=True)
file_source = models.CharField(max_length=30, choices=EntrySource.choices, default=EntrySource.COMPUTER)
file_type = models.CharField(max_length=30, choices=EntryType.choices, default=EntryType.PLAINTEXT)
file_path = models.CharField(max_length=400, default=None, null=True, blank=True)
file_name = models.CharField(max_length=400, default=None, null=True, blank=True)
url = models.URLField(max_length=400, default=None, null=True, blank=True)
hashed_value = models.CharField(max_length=100)
corpus_id = models.UUIDField(default=uuid.uuid4, editable=False)
class EntryDates(BaseModel):
date = models.DateField()
entry = models.ForeignKey(Entry, on_delete=models.CASCADE, related_name="embeddings_dates")
class Meta:
indexes = [
models.Index(fields=["date"]),
]

3
src/database/tests.py Normal file
View file

@ -0,0 +1,3 @@
from django.test import TestCase
# Create your tests here.

88
src/interface/desktop/about.html Normal file
View file

@ -0,0 +1,88 @@
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0 maximum-scale=1.0">
<title>Khoj - About</title>
<link rel="icon" type="image/png" sizes="128x128" href="./assets/icons/favicon-128x128.png">
<link rel="manifest" href="/static/khoj_chat.webmanifest">
<link rel="stylesheet" href="./assets/khoj.css">
</head>
<script type="text/javascript" src="./utils.js"></script>
<style>
html, body {
height: 100%;
width: 100%;
padding: 0px;
margin: 0px;
}
body {
display: grid;
grid-template-rows: auto;
background: var(--background-color);
color: var(--main-text-color);
text-align: center;
font-family: roboto, karma, segoe ui, sans-serif;
font-size: small;
font-weight: 300;
line-height: 1.5em;
}
header > *,
body > * {
padding: 0px;
margin: 0px;
}
header > * {
margin-top: 20px;
}
img {
width: 100px;
height: 100px;
margin-top: 32px;
}
p {
font-size: 14px;
}
#about-page-version {
margin: 0;
}
.button {
display: block;
width: 60%;
padding: 10px 16px;
margin: 10px auto;
background-color: var(--primary);
border: none;
border-radius: 8px;
cursor: pointer;
transition: background-color 0.3s;
}
.button:hover {
background-color: var(--primary-hover);
}
footer {
font-size: 10px;
color: slategray;
margin-top: 10px;
}
</style>
<body>
<header>
<img id="logo" src="./assets/icons/favicon-128x128.png" alt="Khoj Logo">
<p id="about-page-title"><b>Khoj for Desktop</b>
<p id="about-page-version"></p>
</header>
<div class="action">
<button class="button" onclick="window.open('https://khoj.dev/terms-of-service', '_blank')">Terms of Service</button>
<button class="button" onclick="window.open('https://khoj.dev/privacy-policy', '_blank')">Privacy Policy</button>
</div>
<footer>
© 2023 Khoj Inc. All rights reserved.
</footer>
</body>
</html>

BIN
src/interface/desktop/assets/icons/favicon-20x20.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 1.6 KiB

4
src/interface/desktop/assets/icons/key.svg Normal file
View file

@ -0,0 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="800px" height="800px" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fill-rule="evenodd" clip-rule="evenodd" d="M22 8.29344C22 11.7692 19.1708 14.5869 15.6807 14.5869C15.0439 14.5869 13.5939 14.4405 12.8885 13.8551L12.0067 14.7333C11.4883 15.2496 11.6283 15.4016 11.8589 15.652C11.9551 15.7565 12.0672 15.8781 12.1537 16.0505C12.1537 16.0505 12.8885 17.075 12.1537 18.0995C11.7128 18.6849 10.4783 19.5045 9.06754 18.0995L8.77362 18.3922C8.77362 18.3922 9.65538 19.4167 8.92058 20.4412C8.4797 21.0267 7.30403 21.6121 6.27531 20.5876L5.2466 21.6121C4.54119 22.3146 3.67905 21.9048 3.33616 21.6121L2.45441 20.7339C1.63143 19.9143 2.1115 19.0264 2.45441 18.6849L10.0963 11.0743C10.0963 11.0743 9.3615 9.90338 9.3615 8.29344C9.3615 4.81767 12.1907 2 15.6807 2C19.1708 2 22 4.81767 22 8.29344ZM15.681 10.4889C16.8984 10.4889 17.8853 9.50601 17.8853 8.29353C17.8853 7.08105 16.8984 6.09814 15.681 6.09814C14.4635 6.09814 13.4766 7.08105 13.4766 8.29353C13.4766 9.50601 14.4635 10.4889 15.681 10.4889Z" fill="#1C274C"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 1.1 KiB

BIN
src/interface/desktop/assets/icons/khoj-logo-sideways-500.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 36 KiB

After

Width:  |  Height:  |  Size: 29 KiB

5
src/interface/desktop/assets/icons/link.svg
View file

@ -1,5 +1,4 @@
<?xml version="1.0" encoding="utf-8"?><!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
<?xml version="1.0" encoding="utf-8"?>
<svg width="800px" height="800px" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M15.197 3.35462C16.8703 1.67483 19.4476 1.53865 20.9536 3.05046C22.4596 4.56228 22.3239 7.14956 20.6506 8.82935L18.2268 11.2626M10.0464 14C8.54044 12.4882 8.67609 9.90087 10.3494 8.22108L12.5 6.06212" stroke="#1C274C" stroke-width="1.5" stroke-linecap="round"/>
<path d="M13.9536 10C15.4596 11.5118 15.3239 14.0991 13.6506 15.7789L11.2268 18.2121L8.80299 20.6454C7.12969 22.3252 4.55237 22.4613 3.0464 20.9495C1.54043 19.4377 1.67609 16.8504 3.34939 15.1706L5.77323 12.7373" stroke="#1C274C" stroke-width="1.5" stroke-linecap="round"/>
<path d="M9.16488 17.6505C8.92513 17.8743 8.73958 18.0241 8.54996 18.1336C7.62175 18.6695 6.47816 18.6695 5.54996 18.1336C5.20791 17.9361 4.87912 17.6073 4.22153 16.9498C3.56394 16.2922 3.23514 15.9634 3.03767 15.6213C2.50177 14.6931 2.50177 13.5495 3.03767 12.6213C3.23514 12.2793 3.56394 11.9505 4.22153 11.2929L7.04996 8.46448C7.70755 7.80689 8.03634 7.47809 8.37838 7.28062C9.30659 6.74472 10.4502 6.74472 11.3784 7.28061C11.7204 7.47809 12.0492 7.80689 12.7068 8.46448C13.3644 9.12207 13.6932 9.45086 13.8907 9.7929C14.4266 10.7211 14.4266 11.8647 13.8907 12.7929C13.7812 12.9825 13.6314 13.1681 13.4075 13.4078M10.5919 10.5922C10.368 10.8319 10.2182 11.0175 10.1087 11.2071C9.57284 12.1353 9.57284 13.2789 10.1087 14.2071C10.3062 14.5492 10.635 14.878 11.2926 15.5355C11.9502 16.1931 12.279 16.5219 12.621 16.7194C13.5492 17.2553 14.6928 17.2553 15.621 16.7194C15.9631 16.5219 16.2919 16.1931 16.9495 15.5355L19.7779 12.7071C20.4355 12.0495 20.7643 11.7207 20.9617 11.3787C21.4976 10.4505 21.4976 9.30689 20.9617 8.37869C20.7643 8.03665 20.4355 7.70785 19.7779 7.05026C19.1203 6.39267 18.7915 6.06388 18.4495 5.8664C17.5212 5.3305 16.3777 5.3305 15.4495 5.8664C15.2598 5.97588 15.0743 6.12571 14.8345 6.34955" stroke="#000000" stroke-width="2" stroke-linecap="round"/>
</svg>

Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 777 B

After

Width:  |  Height:  |  Size: 1.4 KiB

51
src/interface/desktop/assets/khoj.css
View file

@ -2,29 +2,44 @@
/* Can be forced with data-theme="light" */
[data-theme="light"],
:root:not([data-theme="dark"]) {
--primary: #ffb300;
--primary-hover: #ffa000;
--primary: #fee285;
--primary-hover: #fcc50b;
--primary-focus: rgba(255, 179, 0, 0.125);
--primary-inverse: rgba(0, 0, 0, 0.75);
--background-color: #f5f4f3;
--main-text-color: #475569;
--water: #44b9da;
--leaf: #7b990a;
--flower: #d1684e;
}
/* Amber Dark scheme (Auto) */
/* Automatically enabled if user has Dark mode enabled */
@media only screen and (prefers-color-scheme: dark) {
:root:not([data-theme]) {
--primary: #ffb300;
--primary-hover: #ffc107;
--primary: #fee285;
--primary-hover: #fcc50b;
--primary-focus: rgba(255, 179, 0, 0.25);
--primary-inverse: rgba(0, 0, 0, 0.75);
--background-color: #f5f4f3;
--main-text-color: #475569;
--water: #44b9da;
--leaf: #7b990a;
--flower: #d1684e;
}
}
/* Amber Dark scheme (Forced) */
/* Enabled if forced with data-theme="dark" */
[data-theme="dark"] {
--primary: #ffb300;
--primary-hover: #ffc107;
--primary: #fee285;
--primary-hover: #fcc50b;
--primary-focus: rgba(255, 179, 0, 0.25);
--primary-inverse: rgba(0, 0, 0, 0.75);
--background-color: #f5f4f3;
--main-text-color: #475569;
--water: #44b9da;
--leaf: #7b990a;
--flower: #d1684e;
}
/* Amber (Common styles) */
:root {
@ -37,8 +52,10 @@
.khoj-configure {
display: grid;
grid-template-columns: 1fr;
padding: 0 24px;
font-family: roboto, karma, segoe ui, sans-serif;
font-weight: 300;
}
.khoj-header {
display: grid;
grid-auto-flow: column;
@ -64,7 +81,7 @@ a.khoj-logo {
}
.khoj-nav a {
color: #333;
color: var(--main-text-color);
text-decoration: none;
font-size: small;
font-weight: normal;
@ -75,8 +92,9 @@ a.khoj-logo {
}
.khoj-nav a:hover {
background-color: var(--primary-hover);
color: var(--main-text-color);
}
.khoj-nav-selected {
a.khoj-nav-selected {
background-color: var(--primary);
}
img.khoj-logo {
@ -85,21 +103,6 @@ img.khoj-logo {
justify-self: center;
}
a.khoj-banner {
color: black;
text-decoration: none;
}
p.khoj-banner {
font-size: small;
margin: 0;
padding: 10px;
}
p#khoj-banner {
display: inline;
}
@media only screen and (max-width: 600px) {
div.khoj-header {
display: grid;

991
src/interface/desktop/assets/three.min.js vendored Normal file
View file

File diff suppressed because one or more lines are too long

290
src/interface/desktop/chat.html
View file

@ -8,6 +8,8 @@
<link rel="manifest" href="/static/khoj_chat.webmanifest">
<link rel="stylesheet" href="./assets/khoj.css">
</head>
<script src="./utils.js"></script>
<script>
let chatOptions = [];
function copyProgrammaticOutput(event) {
@ -32,32 +34,101 @@
let escaped_ref = reference.replaceAll('"', '&quot;');
// Generate HTML for Chat Reference
return `<sup><abbr title="${escaped_ref}" tabindex="0">${index}</abbr></sup>`;
let short_ref = escaped_ref.slice(0, 100);
short_ref = short_ref.length < escaped_ref.length ? short_ref + "..." : short_ref;
let referenceButton = document.createElement('button');
referenceButton.innerHTML = short_ref;
referenceButton.id = `ref-${index}`;
referenceButton.classList.add("reference-button");
referenceButton.classList.add("collapsed");
referenceButton.tabIndex = 0;
// Add event listener to toggle full reference on click
referenceButton.addEventListener('click', function() {
console.log(`Toggling ref-${index}`)
if (this.classList.contains("collapsed")) {
this.classList.remove("collapsed");
this.classList.add("expanded");
this.innerHTML = escaped_ref;
} else {
this.classList.add("collapsed");
this.classList.remove("expanded");
this.innerHTML = short_ref;
}
});
return referenceButton;
}
function renderMessage(message, by, dt=null) {
function renderMessage(message, by, dt=null, annotations=null) {
let message_time = formatDate(dt ?? new Date());
let by_name = by == "khoj" ? "🏮 Khoj" : "🤔 You";
let formattedMessage = formatHTMLMessage(message);
// Generate HTML for Chat Message and Append to Chat Body
document.getElementById("chat-body").innerHTML += `
<div data-meta="${by_name} at ${message_time}" class="chat-message ${by}">
<div class="chat-message-text ${by}">${formattedMessage}</div>
</div>
`;
let chatBody = document.getElementById("chat-body");
// Create a new div for the chat message
let chatMessage = document.createElement('div');
chatMessage.className = `chat-message ${by}`;
chatMessage.dataset.meta = `${by_name} at ${message_time}`;
// Create a new div for the chat message text and append it to the chat message
let chatMessageText = document.createElement('div');
chatMessageText.className = `chat-message-text ${by}`;
chatMessageText.innerHTML = formattedMessage;
chatMessage.appendChild(chatMessageText);
// Append annotations div to the chat message
if (annotations) {
chatMessageText.appendChild(annotations);
}
// Append chat message div to chat body
chatBody.appendChild(chatMessage);
// Scroll to bottom of chat-body element
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
chatBody.scrollTop = chatBody.scrollHeight;
}
function renderMessageWithReference(message, by, context=null, dt=null) {
let references = '';
if (context) {
references = context
.map((reference, index) => generateReference(reference, index))
.join("<sup>,</sup>");
if (context == null || context.length == 0) {
renderMessage(message, by, dt);
return;
}
renderMessage(message+references, by, dt);
let references = document.createElement('div');
let referenceExpandButton = document.createElement('button');
referenceExpandButton.classList.add("reference-expand-button");
let expandButtonText = context.length == 1 ? "1 reference" : `${context.length} references`;
referenceExpandButton.innerHTML = expandButtonText;
references.appendChild(referenceExpandButton);
let referenceSection = document.createElement('div');
referenceSection.classList.add("reference-section");
referenceSection.classList.add("collapsed");
referenceExpandButton.addEventListener('click', function() {
if (referenceSection.classList.contains("collapsed")) {
referenceSection.classList.remove("collapsed");
referenceSection.classList.add("expanded");
} else {
referenceSection.classList.add("collapsed");
referenceSection.classList.remove("expanded");
}
});
references.classList.add("references");
if (context) {
for (let index in context) {
let reference = context[index];
let polishedReference = generateReference(reference, index);
referenceSection.appendChild(polishedReference);
}
}
references.appendChild(referenceSection);
renderMessage(message, by, dt, references);
}
function formatHTMLMessage(htmlMessage) {
@ -66,6 +137,8 @@
// Replace any ** with <b> and __ with <u>
newHTML = newHTML.replace(/\*\*([\s\S]*?)\*\*/g, '<b>$1</b>');
newHTML = newHTML.replace(/__([\s\S]*?)__/g, '<u>$1</u>');
// Remove any text between <s>[INST] and </s> tags. These are spurious instructions for the AI chat model.
newHTML = newHTML.replace(/<s>\[INST\].+(<\/s>)?/g, '');
return newHTML;
}
@ -89,6 +162,8 @@
// Generate backend API URL to execute query
let url = `${hostURL}/api/chat?q=${encodeURIComponent(query)}&n=${resultsCount}&client=web&stream=true`;
const khojToken = await window.tokenAPI.getToken();
const headers = { 'Authorization': `Bearer ${khojToken}` };
let chat_body = document.getElementById("chat-body");
let new_response = document.createElement("div");
@ -113,10 +188,11 @@
chatInput.classList.remove("option-enabled");
// Call specified Khoj API which returns a streamed response of type text/plain
fetch(url)
.then(response => {
fetch(url, { headers })
.then(response => {
const reader = response.body.getReader();
const decoder = new TextDecoder();
let references = null;
function readStream() {
reader.read().then(({ done, value }) => {
@ -124,7 +200,8 @@
// Evaluate the contents of new_response_text.innerHTML after all the data has been streamed
const currentHTML = newResponseText.innerHTML;
newResponseText.innerHTML = formatHTMLMessage(currentHTML);
newResponseText.appendChild(references);
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
return;
}
@ -137,11 +214,36 @@
const rawReference = chunk.split("### compiled references:")[1];
const rawReferenceAsJson = JSON.parse(rawReference);
let polishedReference = rawReferenceAsJson.map((reference, index) => generateReference(reference, index))
.join("<sup>,</sup>");
references = document.createElement('div');
references.classList.add("references");
newResponseText.innerHTML += polishedReference;
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
let referenceExpandButton = document.createElement('button');
referenceExpandButton.classList.add("reference-expand-button");
let expandButtonText = rawReferenceAsJson.length == 1 ? "1 reference" : `${rawReferenceAsJson.length} references`;
referenceExpandButton.innerHTML = expandButtonText;
references.appendChild(referenceExpandButton);
let referenceSection = document.createElement('div');
referenceSection.classList.add("reference-section");
referenceSection.classList.add("collapsed");
referenceExpandButton.addEventListener('click', function() {
if (referenceSection.classList.contains("collapsed")) {
referenceSection.classList.remove("collapsed");
referenceSection.classList.add("expanded");
} else {
referenceSection.classList.add("collapsed");
referenceSection.classList.remove("expanded");
}
});
rawReferenceAsJson.forEach((reference, index) => {
let polishedReference = generateReference(reference, index);
referenceSection.appendChild(polishedReference);
});
references.appendChild(referenceSection);
readStream();
} else {
// Display response from Khoj
@ -164,6 +266,7 @@
function incrementalChat(event) {
if (!event.shiftKey && event.key === 'Enter') {
event.preventDefault();
chat();
}
}
@ -217,12 +320,23 @@
async function loadChat() {
const hostURL = await window.hostURLAPI.getURL();
fetch(`${hostURL}/api/chat/history?client=web`)
const khojToken = await window.tokenAPI.getToken();
const headers = { 'Authorization': `Bearer ${khojToken}` };
fetch(`${hostURL}/api/chat/history?client=web`, { headers })
.then(response => response.json())
.then(data => {
if (data.detail) {
// If the server returns a 500 error with detail, render a setup hint.
renderMessage("Hi 👋🏾, to get started you have two options:<ol><li><b>Use OpenAI</b>: <ol><li>Get your <a class='inline-chat-link' href='https://platform.openai.com/account/api-keys'>OpenAI API key</a></li><li>Save it in the Khoj <a class='inline-chat-link' href='/config/processor/conversation/openai'>chat settings</a></li><li>Click Configure on the Khoj <a class='inline-chat-link' href='/config'>settings page</a></li></ol></li><li><b>Enable offline chat</b>: <ol><li>Go to the Khoj <a class='inline-chat-link' href='/config'>settings page</a> and enable offline chat</li></ol></li></ol>", "khoj");
first_run_message = `Hi 👋🏾, to get started:
<ol>
<li>Generate an API token in the <a class='inline-chat-link' href="#" onclick="window.navigateAPI.navigateToWebSettings()">Khoj Web settings</a></li>
<li>Paste it into the API Key field in the <a class='inline-chat-link' href="#" onclick="window.navigateAPI.navigateToSettings()">Khoj Desktop settings</a></li>
</ol>`
.trim()
.replace(/(\r\n|\n|\r)/gm, "");
renderMessage(first_run_message, "khoj");
// Disable chat input field and update placeholder text
document.getElementById("chat-input").setAttribute("disabled", "disabled");
@ -243,7 +357,7 @@
return;
});
fetch(`${hostURL}/api/chat/options`)
fetch(`${hostURL}/api/chat/options`, { headers })
.then(response => response.json())
.then(data => {
// Render chat options, if any
@ -264,17 +378,18 @@
}
</script>
<body>
<div id="khoj-banner-container" class="khoj-banner-container">
<div id="khoj-empty-container" class="khoj-empty-container">
</div>
<!--Add Header Logo and Nav Pane-->
<div class="khoj-header">
<a class="khoj-logo" href="/">
<img class="khoj-logo" src="./assets/icons/khoj-logo-sideways-500.png" alt="Khoj"></img>
</a>
<nav class="khoj-nav">
<a class="khoj-nav khoj-nav-selected" href="./chat.html">Chat</a>
<a class="khoj-nav" href="./index.html">Search</a>
<a class="khoj-nav" href="./config.html">⚙️</a>
<a class="khoj-nav khoj-nav-selected" href="./chat.html">💬 Chat</a>
<a class="khoj-nav" href="./search.html">🔎 Search</a>
<a class="khoj-nav" href="./config.html">⚙️ Settings</a>
</nav>
</div>
@ -284,8 +399,7 @@
<!-- Chat Footer -->
<div id="chat-footer">
<div id="chat-tooltip" style="display: none;"></div>
<textarea id="chat-input" class="option" oninput="onChatInput()" onkeyup=incrementalChat(event) autofocus="autofocus" placeholder="Type / to see a list of commands, or just type your questions and hit enter.">
</textarea>
<textarea id="chat-input" class="option" oninput="onChatInput()" onkeydown=incrementalChat(event) autofocus="autofocus" placeholder="Type / to see a list of commands, or just type your questions and hit enter."></textarea>
</div>
</body>
@ -298,8 +412,8 @@
}
body {
display: grid;
background: #fff;
color: #475569;
background: var(--background-color);
color: var(--main-text-color);
text-align: center;
font-family: roboto, karma, segoe ui, sans-serif;
font-size: small;
@ -433,6 +547,83 @@
box-shadow: 0 0 12px rgb(119, 156, 46);
}
div.collapsed {
display: none;
}
div.expanded {
display: block;
}
div.reference {
display: grid;
grid-template-rows: auto;
grid-auto-flow: row;
grid-column-gap: 10px;
grid-row-gap: 10px;
margin: 10px;
}
div.expanded.reference-section {
display: grid;
grid-template-rows: auto;
grid-auto-flow: row;
grid-column-gap: 10px;
grid-row-gap: 10px;
margin: 10px;
}
button.reference-button {
background: var(--background-color);
color: var(--main-text-color);
border: 1px solid var(--main-text-color);
border-radius: 5px;
padding: 5px;
font-size: 14px;
font-weight: 300;
line-height: 1.5em;
cursor: pointer;
transition: background 0.2s ease-in-out;
text-align: left;
max-height: 75px;
transition: max-height 0.3s ease-in-out;
overflow: hidden;
}
button.reference-button.expanded {
max-height: none;
}
button.reference-button::before {
content: "▶";
margin-right: 5px;
display: inline-block;
transition: transform 0.3s ease-in-out;
}
button.reference-button:active:before,
button.reference-button[aria-expanded="true"]::before {
transform: rotate(90deg);
}
button.reference-expand-button {
background: var(--background-color);
color: var(--main-text-color);
border: 1px dotted var(--main-text-color);
border-radius: 5px;
padding: 5px;
font-size: 14px;
font-weight: 300;
line-height: 1.5em;
cursor: pointer;
transition: background 0.4s ease-in-out;
text-align: left;
}
button.reference-expand-button:hover {
background: var(--primary-hover);
}
.option-enabled:focus {
outline: none !important;
border:1px solid #475569;
@ -445,6 +636,11 @@
border-bottom: 1px dotted #475569;
}
div.khoj-empty-container {
padding: 0;
margin: 0;
}
@media (pointer: coarse), (hover: none) {
abbr[title] {
position: relative;
@ -481,12 +677,6 @@
margin: 4px;
grid-template-columns: auto;
}
a.khoj-banner {
display: block;
}
p.khoj-banner {
padding: 0;
}
}
@media only screen and (min-width: 600px) {
body {
@ -498,11 +688,6 @@
}
}
div.khoj-banner-container {
padding: 0px;
margin: 0px;
}
div#chat-tooltip {
text-align: left;
font-size: medium;
@ -524,23 +709,6 @@
text-align: center;
}
button#khoj-banner-submit,
input#khoj-banner-email {
padding: 10px;
border-radius: 5px;
border: 1px solid #475569;
background: #f9fafc;
}
button#khoj-banner-submit:hover,
input#khoj-banner-email:hover {
box-shadow: 0 0 11px #aaa;
}
div.khoj-banner-container-hidden {
margin: 0px;
padding: 0px;
}
div.programmatic-output {
background-color: #f5f5f5;
border: 1px solid #ddd;

250
src/interface/desktop/config.html
View file

@ -2,89 +2,105 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0 maximum-scale=1.0">
<title>Khoj - Search</title>
<title>Khoj - Settings</title>
<link rel="icon" type="image/png" sizes="128x128" href="./assets/icons/favicon-128x128.png">
<link rel="manifest" href="./khoj.webmanifest">
<link rel="stylesheet" href="./assets/khoj.css">
</head>
<script type="text/javascript" src="./assets/org.min.js"></script>
<script type="text/javascript" src="./assets/markdown-it.min.js"></script>
<script src="./utils.js"></script>
<body>
<div class="page">
<!--Add Header Logo and Nav Pane-->
<div class="khoj-header">
<a class="khoj-logo" href="./index.html">
<img class="khoj-logo" src="./assets/icons/khoj-logo-sideways-500.png" alt="Khoj"></img>
</a>
<nav class="khoj-nav">
<a class="khoj-nav" href="./chat.html">Chat</a>
<a class="khoj-nav" href="./index.html">Search</a>
<a class="khoj-nav khoj-nav-selected" href="./config.html">⚙️</a>
</nav>
</div>
<!--Add Header Logo and Nav Pane-->
<div class="khoj-header">
<a class="khoj-logo" href="./chat.html">
<img class="khoj-logo" src="./assets/icons/khoj-logo-sideways-500.png" alt="Khoj"></img>
</a>
<nav class="khoj-nav">
<a class="khoj-nav" href="./chat.html">💬 Chat</a>
<a class="khoj-nav" href="./search.html">🔎 Search</a>
<a class="khoj-nav khoj-nav-selected" href="./config.html">⚙️ Settings</a>
</nav>
</div>
<div class="section-cards">
<div class="card configuration">
<div class="card-title-row">
<img class="card-icon" src="./assets/icons/link.svg" alt="File">
<h3 class="card-title">
Host
</h3>
<div class="card-description-row">
<div class="card configuration">
<div class="card-title-row">
<img class="card-icon" src="./assets/icons/link.svg" alt="Khoj Server URL">
<h3 class="card-title">
Server URL
</h3>
</div>
<div class="card-description-row">
<input id="khoj-host-url" class="card-input" type="text">
</div>
<div class="card-title-row">
<img class="card-icon" src="./assets/icons/key.svg" alt="Khoj Access Key">
<h3 class="card-title">
API Key
</h3>
</div>
<div class="card-description-row">
<input id="khoj-access-key" class="card-input" type="text" placeholder="Enter API key to access your Khoj">
</div>
</div>
<div class="card-description-row">
<input id="khoj-host-url" class="card-input" type="text">
</div>
<div class="card-title-row">
<img class="card-icon" src="./assets/icons/plaintext.svg" alt="File">
<h3 class="card-title">
Files
<button id="toggle-files" class="card-button">
<svg id="toggle-files-svg" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 5v14M5 12l7 7 7-7"></path></svg>
</div>
<div class="card-description-row">
<div class="card configuration">
<div class="card-title-row">
<img class="card-icon" src="./assets/icons/plaintext.svg" alt="File">
<h3 class="card-title">
Files
<button id="toggle-files" class="card-button">
<svg id="toggle-files-svg" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 5v14M5 12l7 7 7-7"></path></svg>
</button>
</h3>
</div>
<div class="card-description-row">
<div id="current-files"></div>
</div>
<div class="card-action-row">
<button id="update-file" class="card-button">
Add
<img class="add-files-icon" src="./assets/icons/circular-add.svg" alt="Add">
</button>
</h3>
</div>
</div>
<div class="card-description-row">
<div id="current-files"></div>
</div>
<div class="card-action-row">
<button id="update-file" class="card-button">
Add
<img class="add-files-icon" src="./assets/icons/circular-add.svg" alt="Add">
</button>
</div>
<div class="card-title-row">
<img class="card-icon" src="./assets/icons/folder.svg" alt="Folder">
<h3 class="card-title">
Folders
<button id="toggle-folders" class="card-button">
<svg id="toggle-folders-svg" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 5v14M5 12l7 7 7-7"></path></svg>
</div>
<div class="card-description-row">
<div class="card configuration">
<div class="card-title-row">
<img class="card-icon" src="./assets/icons/folder.svg" alt="Folder">
<h3 class="card-title">
Folders
<button id="toggle-folders" class="card-button">
<svg id="toggle-folders-svg" xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 5v14M5 12l7 7 7-7"></path></svg>
</button>
</h3>
</div>
<div class="card-description-row">
<div id="current-folders"></div>
</div>
<div class="card-action-row">
<button id="update-folder" class="card-button">
Add
<img class="add-files-icon" src="./assets/icons/circular-add.svg" alt="Add">
</button>
</h3>
</div>
</div>
</div>
<div class="section-action-row">
<div class="card-description-row">
<button id="sync-force" class="sync-data">💾 Save</button>
</div>
<div class="card-description-row">
<div id="current-folders"></div>
</div>
<div class="card-action-row">
<button id="update-folder" class="card-button">
Add
<img class="add-files-icon" src="./assets/icons/circular-add.svg" alt="Add">
</button>
</div>
<div class="card-description-row">
<button id="sync-data">Sync</button>
</div>
<div class="card-description-row sync-force-toggle">
<input id="sync-force" type="checkbox" name="sync-force" value="force">
<label for="sync-force">Force Sync</label>
</div>
<div id="loading-bar" style="display: none;">
</div>
<div class="card-description-row">
<div id="sync-status"></div>
<button id="delete-all" class="sync-data">🗑️ Delete All</button>
</div>
</div>
<div id="loading-bar" style="display: none;"></div>
<div class="card-description-row">
<div id="sync-status"></div>
</div>
</div>
</body>
@ -93,7 +109,7 @@
body {
display: grid;
grid-template-columns: 1fr;
grid-template-rows: 1fr auto auto auto minmax(80px, 100%);
grid-template-rows: 1fr auto;
font-size: small!important;
}
body > * {
@ -104,8 +120,7 @@
body {
display: grid;
grid-template-columns: 1fr min(70vw, 100%) 1fr;
grid-template-rows: 1fr auto auto auto minmax(80px, 100%);
padding-top: 60vw;
grid-template-rows: 80px auto;
}
body > * {
grid-column: 2;
@ -114,7 +129,7 @@
body, input {
padding: 0px;
margin: 0px;
background: #fff;
background: var(--background-color);
color: #475569;
font-family: roboto, karma, segoe ui, sans-serif;
font-size: small;
@ -126,11 +141,6 @@
margin: 10px;
}
div.page {
padding: 0px;
margin: 0px;
}
svg {
transition: transform 0.3s ease-in-out;
}
@ -167,19 +177,19 @@
}
}
#khoj-host-url {
.card-input {
padding: 4px;
box-shadow: 0 0 2px 1px rgba(0, 0, 0, 0.2);
box-shadow: 0 0 2px 1px rgba(0, 0, 0, 0.3);
border: none;
width: 450px;
}
.card {
display: grid;
/* grid-template-rows: repeat(3, 1fr); */
gap: 8px;
padding: 24px 16px;
width: 100%;
background: white;
width: 450px;
background: var(--background-color);
border: 1px solid rgb(229, 229, 229);
border-radius: 4px;
box-shadow: 0px 1px 3px 0px rgba(0,0,0,0.1),0px 1px 2px -1px rgba(0,0,0,0.1);
@ -188,15 +198,15 @@
.section-cards {
display: grid;
grid-template-columns: repeat(1, 1fr);
gap: 16px;
justify-items: start;
justify-items: center;
margin: 0;
width: auto;
}
div.configuration {
width: auto;
.section-action-row {
display: grid;
grid-auto-flow: column;
gap: 16px;
height: fit-content;
}
.card-title-row {
@ -247,7 +257,7 @@
}
.primary-button {
border: none;
color: white;
color: var(--background-color);
padding: 15px 32px;
text-align: center;
text-decoration: none;
@ -256,7 +266,7 @@
}
button.card-button.disabled {
color: rgb(255, 136, 136);
color: var(--flower);
background: transparent;
font-size: small;
cursor: pointer;
@ -268,11 +278,7 @@
}
button.card-button.happy {
color: rgb(0, 146, 0);
}
button.card-button.happy {
color: rgb(0, 146, 0);
color: var(--leaf);
}
img.configured-icon {
@ -296,13 +302,14 @@
div.folder-element {
display: grid;
grid-template-columns: auto 1fr;
box-shadow: 0 0 2px 1px rgba(0, 0, 0, 0.2);
border: 1px solid rgb(229, 229, 229);
border-radius: 4px;
box-shadow: 0px 1px 3px 0px rgba(0,0,0,0.1),0px 1px 2px -1px rgba(0,0,0,0.8);
padding: 4px;
margin-bottom: 8px;
}
div.content-name {
width: 500px;
overflow-wrap: break-word;
}
@ -315,7 +322,7 @@
background-color: rgb(253 214 214);
border-radius: 3px;
border: none;
color: rgb(207, 67, 59);
color: var(--flower);
padding: 4px;
}
@ -324,14 +331,14 @@
background-color: rgb(255 235 235);
border-radius: 3px;
border: none;
color: rgb(207, 67, 59);
color: var(--flower);
padding: 4px;
cursor: pointer;
}
#sync-data {
background-color: #ffb300;
button.sync-data {
background-color: var(--primary);
border: none;
color: white;
color: var(--main-text-color);
padding: 12px;
text-align: center;
text-decoration: none;
@ -340,43 +347,20 @@
border-radius: 4px;
cursor: pointer;
transition: background-color 0.3s ease;
box-shadow: 0px 5px 0px #f9f5de;
box-shadow: 0px 5px 0px var(--background-color);
}
#sync-data:hover {
background-color: #ffcc00;
box-shadow: 0px 3px 0px #f9f5de;
button.sync-data:hover {
background-color: var(--primary-hover);
box-shadow: 0px 3px 0px var(--background-color);
}
.sync-force-toggle {
align-content: center;
display: grid;
grid-auto-flow: column;
gap: 4px;
}
</style>
<script>
var khojBannerSubmit = document.getElementById("khoj-banner-submit");
khojBannerSubmit?.addEventListener("click", function(event) {
event.preventDefault();
var email = document.getElementById("khoj-banner-email").value;
fetch("https://app.khoj.dev/beta/users/", {
method: "POST",
body: JSON.stringify({
email: email
}),
headers: {
"Content-Type": "application/json"
}
}).then(function(response) {
return response.json();
}).then(function(data) {
console.log(data);
if (data.user != null) {
document.getElementById("khoj-banner").innerHTML = "Thanks for signing up. We'll be in touch soon! 🚀";
document.getElementById("khoj-banner-submit").remove();
} else {
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
}
}).catch(function(error) {
console.log(error);
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
});
});
</script>
<script src="./renderer.js"></script>
</html>

129
src/interface/desktop/loading-animation.js Normal file
View file

@ -0,0 +1,129 @@
var $wrap = document.getElementById('loading-animation'),
canvassize = 380,
length = 40,
radius = 6.8,
rotatevalue = 0.02,
acceleration = 0,
animatestep = 0,
toend = false,
pi2 = Math.PI*2,
group = new THREE.Group(),
mesh, ringcover, ring,
camera, scene, renderer;
camera = new THREE.PerspectiveCamera(65, 1, 1, 10000);
camera.position.z = 120;
scene = new THREE.Scene();
// scene.add(new THREE.AxisHelper(30));
scene.add(group);
mesh = new THREE.Mesh(
new THREE.TubeGeometry(new (THREE.Curve.create(function() {},
function(percent) {
var x = length*Math.sin(pi2*percent),
y = radius*Math.cos(pi2*3*percent),
z, t;
t = percent%0.25/0.25;
t = percent%0.25-(2*(1-t)*t* -0.0185 +t*t*0.25);
if (Math.floor(percent/0.25) == 0 || Math.floor(percent/0.25) == 2) {
t *= -1;
}
z = radius*Math.sin(pi2*2* (percent-t));
return new THREE.Vector3(x, y, z);
}
))(), 200, 1.1, 2, true),
new THREE.MeshBasicMaterial({
color: 0xfcc50b
// , wireframe: true
})
);
group.add(mesh);
ringcover = new THREE.Mesh(new THREE.PlaneGeometry(50, 15, 1), new THREE.MeshBasicMaterial({color: 0xd1684e, opacity: 0, transparent: true}));
ringcover.position.x = length+1;
ringcover.rotation.y = Math.PI/2;
group.add(ringcover);
ring = new THREE.Mesh(new THREE.RingGeometry(4.3, 5.55, 32), new THREE.MeshBasicMaterial({color: 0xfcc50b, opacity: 0, transparent: true}));
ring.position.x = length+1.1;
ring.rotation.y = Math.PI/2;
group.add(ring);
// fake shadow
(function() {
var plain, i;
for (i = 0; i < 10; i++) {
plain = new THREE.Mesh(new THREE.PlaneGeometry(length*2+1, radius*3, 1), new THREE.MeshBasicMaterial({color: 0xd1684e, transparent: true, opacity: 0.15}));
plain.position.z = -2.5+i*0.5;
group.add(plain);
}
})();
renderer = new THREE.WebGLRenderer({
antialias: true
});
renderer.setPixelRatio(window.devicePixelRatio);
renderer.setSize(canvassize, canvassize);
renderer.setClearColor('#d1684e');
$wrap.appendChild(renderer.domElement);
function start() {
toend = true;
}
function back() {
toend = false;
}
function tilt(percent) {
group.rotation.y = percent*0.5;
}
function render() {
var progress;
animatestep = Math.max(0, Math.min(240, toend ? animatestep+1 : animatestep-4));
acceleration = easing(animatestep, 0, 1, 240);
if (acceleration > 0.35) {
progress = (acceleration-0.35)/0.65;
group.rotation.y = -Math.PI/2 *progress;
group.position.z = 20*progress;
progress = Math.max(0, (acceleration-0.99)/0.01);
mesh.material.opacity = 1-progress;
ringcover.material.opacity = ring.material.opacity = progress;
ring.scale.x = ring.scale.y = 0.9 + 0.1*progress;
}
renderer.render(scene, camera);
}
function animate() {
mesh.rotation.x += rotatevalue + acceleration*Math.sin(Math.PI*acceleration);
render();
requestAnimationFrame(animate);
}
function easing(t, b, c, d) {
if ((t /= d/2) < 1)
return c/2*t*t+b;
return c/2*((t-=2)*t*t+2)+b;
}
animate();
setTimeout(start, 30);

170
src/interface/desktop/main.js
View file

@ -1,5 +1,6 @@
const { app, BrowserWindow, ipcMain } = require('electron');
const { app, BrowserWindow, ipcMain, Tray, Menu, nativeImage, shell } = require('electron');
const todesktop = require("@todesktop/runtime");
const khojPackage = require('./package.json');
todesktop.init();
@ -9,7 +10,7 @@ const {dialog} = require('electron');
const cron = require('cron').CronJob;
const axios = require('axios');
const KHOJ_URL = 'http://127.0.0.1:42110'
const KHOJ_URL = 'https://app.khoj.dev';
const Store = require('electron-store');
@ -42,6 +43,10 @@ const schema = {
},
default: []
},
khojToken: {
type: 'string',
default: ''
},
hostURL: {
type: 'string',
default: KHOJ_URL
@ -62,8 +67,8 @@ const schema = {
}
};
let syncing = false;
var state = {}
const store = new Store({ schema });
console.log(store);
@ -106,6 +111,15 @@ function filenameToMimeType (filename) {
}
function pushDataToKhoj (regenerate = false) {
// Don't sync if token or hostURL is not set or if already syncing
if (store.get('khojToken') === '' || store.get('hostURL') === '' || syncing === true) {
const win = BrowserWindow.getAllWindows()[0];
if (win) win.webContents.send('update-state', state);
return;
} else {
syncing = true;
}
let filesToPush = [];
const files = store.get('files') || [];
const folders = store.get('folders') || [];
@ -168,7 +182,7 @@ function pushDataToKhoj (regenerate = false) {
if (!!formData?.entries()?.next().value) {
const hostURL = store.get('hostURL') || KHOJ_URL;
const headers = {
'x-api-key': 'secret'
'Authorization': `Bearer ${store.get("khojToken")}`
};
axios.post(`${hostURL}/api/v1/index/update?force=${regenerate}&client=desktop`, formData, { headers })
.then(response => {
@ -188,11 +202,13 @@ function pushDataToKhoj (regenerate = false) {
})
.finally(() => {
// Syncing complete
syncing = false;
const win = BrowserWindow.getAllWindows()[0];
if (win) win.webContents.send('update-state', state);
});
} else {
// Syncing complete
syncing = false;
const win = BrowserWindow.getAllWindows()[0];
if (win) win.webContents.send('update-state', state);
}
@ -246,6 +262,15 @@ async function handleFileOpen (type) {
}
}
async function getToken () {
return store.get('khojToken');
}
async function setToken (event, token) {
store.set('khojToken', token);
return store.get('khojToken');
}
async function getFiles () {
return store.get('files');
}
@ -255,6 +280,12 @@ async function getFolders () {
}
async function setURL (event, url) {
// Sanitize the URL. Remove trailing slash if present. Add http:// if not present.
url = url.replace(/\/$/, "");
if (!url.match(/^[a-zA-Z]+:\/\//)) {
url = `http://${url}`;
}
store.set('hostURL', url);
return store.get('hostURL');
}
@ -287,10 +318,26 @@ async function syncData (regenerate = false) {
}
}
const createWindow = () => {
const win = new BrowserWindow({
async function deleteAllFiles () {
try {
store.set('files', []);
store.set('folders', []);
pushDataToKhoj(true);
const date = new Date();
console.log('Pushing data to Khoj at: ', date);
} catch (err) {
console.error(err);
}
}
let firstRun = true;
let win = null;
const createWindow = (tab = 'chat.html') => {
win = new BrowserWindow({
width: 800,
height: 800,
show: false,
// titleBarStyle: 'hidden',
webPreferences: {
preload: path.join(__dirname, 'preload.js'),
@ -311,12 +358,30 @@ const createWindow = () => {
win.setResizable(true);
win.setOpacity(0.95);
win.setBackgroundColor('#FFFFFF');
win.setBackgroundColor('#f5f4f3');
win.setHasShadow(true);
job.start();
win.loadFile('index.html')
win.loadFile(tab)
if (firstRun === true) {
firstRun = false;
// Create splash screen
var splash = new BrowserWindow({width: 400, height: 400, transparent: true, frame: false, alwaysOnTop: true});
splash.setOpacity(1.0);
splash.setBackgroundColor('#d16b4e');
splash.loadFile('splash.html');
// Show splash screen on app load
win.once('ready-to-show', () => {
setTimeout(function(){ splash.close(); win.show(); }, 4500);
});
} else {
// Show main window directly if not first run
win.once('ready-to-show', () => { win.show(); });
}
}
app.whenReady().then(() => {
@ -331,6 +396,14 @@ app.whenReady().then(() => {
event.reply('update-state', arg);
});
ipcMain.on('navigate', (event, page) => {
win.loadFile(page);
});
ipcMain.on('navigateToWebApp', (event, page) => {
shell.openExternal(`${store.get('hostURL')}/${page}`);
});
ipcMain.handle('getFiles', getFiles);
ipcMain.handle('getFolders', getFolders);
@ -340,19 +413,24 @@ app.whenReady().then(() => {
ipcMain.handle('setURL', setURL);
ipcMain.handle('getURL', getURL);
ipcMain.handle('setToken', setToken);
ipcMain.handle('getToken', getToken);
ipcMain.handle('syncData', (event, regenerate) => {
syncData(regenerate);
});
ipcMain.handle('deleteAllFiles', deleteAllFiles);
createWindow()
app.setAboutPanelOptions({
applicationName: "Khoj",
applicationVersion: "0.0.1",
version: "0.0.1",
authors: "Khoj Team",
applicationVersion: khojPackage.version,
version: khojPackage.version,
authors: "Saba Imran, Debanjum Singh Solanky and contributors",
website: "https://khoj.dev",
iconPath: path.join(__dirname, 'assets', 'khoj.png')
copyright: "GPL v3",
iconPath: path.join(__dirname, 'assets', 'icons', 'favicon-128x128.png')
});
app.on('ready', async() => {
@ -375,3 +453,71 @@ app.whenReady().then(() => {
app.on('window-all-closed', () => {
if (process.platform !== 'darwin') app.quit()
})
/*
** About Page
*/
let aboutWindow;
function openAboutWindow() {
if (aboutWindow) { aboutWindow.focus(); return; }
aboutWindow = new BrowserWindow({
width: 400,
height: 400,
titleBarStyle: 'hidden',
show: false,
webPreferences: {
preload: path.join(__dirname, 'preload.js'),
nodeIntegration: true,
},
});
aboutWindow.loadFile('about.html');
// Pass OS, Khoj version to About page
aboutWindow.webContents.on('did-finish-load', () => {
aboutWindow.webContents.send('appInfo', { version: khojPackage.version, platform: process.platform });
});
// Open links in external browser
aboutWindow.webContents.setWindowOpenHandler(({ url }) => {
shell.openExternal(url);
return { action: 'deny' };
});
aboutWindow.once('ready-to-show', () => { aboutWindow.show(); });
aboutWindow.on('closed', () => { aboutWindow = null; });
}
/*
** System Tray Icon
*/
let tray
openWindow = (page) => {
if (BrowserWindow.getAllWindows().length === 0) {
createWindow(page);
} else {
win.loadFile(page); win.show();
}
}
app.whenReady().then(() => {
const icon = nativeImage.createFromPath('assets/icons/favicon-20x20.png')
tray = new Tray(icon)
const contextMenu = Menu.buildFromTemplate([
{ label: 'Chat', type: 'normal', click: () => { openWindow('chat.html'); }},
{ label: 'Search', type: 'normal', click: () => { openWindow('search.html') }},
{ label: 'Configure', type: 'normal', click: () => { openWindow('config.html') }},
{ type: 'separator' },
{ label: 'About Khoj', type: 'normal', click: () => { openAboutWindow(); } },
{ label: 'Quit', type: 'normal', click: () => { app.quit() } }
])
tray.setToolTip('Khoj')
tray.setContextMenu(contextMenu)
})

4
src/interface/desktop/package.json
View file

@ -10,14 +10,14 @@
"main": "main.js",
"private": false,
"devDependencies": {
"electron": "25.8.1"
"electron": "25.8.4"
},
"scripts": {
"start": "yarn electron ."
},
"dependencies": {
"@todesktop/runtime": "^1.3.0",
"axios": "^1.5.0",
"axios": "^1.6.0",
"cron": "^2.4.3",
"electron-store": "^8.1.0",
"fs": "^0.0.1-security"

17
src/interface/desktop/preload.js
View file

@ -45,5 +45,20 @@ contextBridge.exposeInMainWorld('hostURLAPI', {
})
contextBridge.exposeInMainWorld('syncDataAPI', {
syncData: (regenerate) => ipcRenderer.invoke('syncData', regenerate)
syncData: (regenerate) => ipcRenderer.invoke('syncData', regenerate),
deleteAllFiles: () => ipcRenderer.invoke('deleteAllFiles')
})
contextBridge.exposeInMainWorld('tokenAPI', {
setToken: (token) => ipcRenderer.invoke('setToken', token),
getToken: () => ipcRenderer.invoke('getToken')
})
contextBridge.exposeInMainWorld('appInfoAPI', {
getInfo: (callback) => ipcRenderer.on('appInfo', callback)
})
contextBridge.exposeInMainWorld('navigateAPI', {
navigateToSettings: () => ipcRenderer.send('navigate', 'config.html'),
navigateToWebSettings: () => ipcRenderer.send('navigateToWebApp', 'config'),
})

35
src/interface/desktop/renderer.js
View file

@ -61,6 +61,7 @@ toggleFoldersButton.addEventListener('click', () => {
function makeFileElement(file) {
let fileElement = document.createElement("div");
fileElement.classList.add("file-element");
let fileNameElement = document.createElement("div");
fileNameElement.classList.add("content-name");
fileNameElement.innerHTML = file.path;
@ -82,6 +83,7 @@ function makeFileElement(file) {
function makeFolderElement(folder) {
let folderElement = document.createElement("div");
folderElement.classList.add("folder-element");
let folderNameElement = document.createElement("div");
folderNameElement.classList.add("content-name");
folderNameElement.innerHTML = folder.path;
@ -153,11 +155,14 @@ window.updateStateAPI.onUpdateState((event, state) => {
loadingBar.style.display = 'none';
let syncStatusElement = document.getElementById("sync-status");
const currentTime = new Date();
nextSyncTime = new Date();
nextSyncTime.setMinutes(Math.ceil((nextSyncTime.getMinutes() + 1) / 10) * 10);
if (state.completed == false) {
syncStatusElement.innerHTML = `Sync was unsuccessful at ${currentTime.toLocaleTimeString()}. Contact team@khoj.dev to report this issue.`;
return;
}
syncStatusElement.innerHTML = `Last synced at ${currentTime.toLocaleTimeString()}`;
const options = { hour: '2-digit', minute: '2-digit' };
syncStatusElement.innerHTML = `⏱️ Synced at ${currentTime.toLocaleTimeString(undefined, options)}. Next sync at ${nextSyncTime.toLocaleTimeString(undefined, options)}.`;
});
const urlInput = document.getElementById('khoj-host-url');
@ -174,6 +179,7 @@ urlInput.addEventListener('blur', async () => {
new URL(urlInputValue);
} catch (e) {
console.log(e);
alert('Please enter a valid URL');
return;
}
@ -181,10 +187,25 @@ urlInput.addEventListener('blur', async () => {
urlInput.value = url;
});
const syncButton = document.getElementById('sync-data');
const syncForceToggle = document.getElementById('sync-force');
syncButton.addEventListener('click', async () => {
loadingBar.style.display = 'block';
const regenerate = syncForceToggle.checked;
await window.syncDataAPI.syncData(regenerate);
const khojKeyInput = document.getElementById('khoj-access-key');
(async function() {
const token = await window.tokenAPI.getToken();
khojKeyInput.value = token;
})();
khojKeyInput.addEventListener('blur', async () => {
const token = await window.tokenAPI.setToken(khojKeyInput.value.trim());
khojKeyInput.value = token;
});
const syncForceButton = document.getElementById('sync-force');
syncForceButton.addEventListener('click', async () => {
loadingBar.style.display = 'block';
await window.syncDataAPI.syncData(true);
});
const deleteAllButton = document.getElementById('delete-all');
deleteAllButton.addEventListener('click', async () => {
loadingBar.style.display = 'block';
await window.syncDataAPI.deleteAllFiles();
});

92
src/interface/desktop/index.html → src/interface/desktop/search.html
View file

@ -10,6 +10,7 @@
</head>
<script type="text/javascript" src="./assets/org.min.js"></script>
<script type="text/javascript" src="./assets/markdown-it.min.js"></script>
<script src="./utils.js"></script>
<script>
function render_image(item) {
@ -94,6 +95,15 @@
}).join("\n");
}
function render_xml(query, data) {
return data.map(function (item) {
return `<div class="results-xml">` +
`<b><a href="${item.additional.file}">${item.additional.heading}</a></b>` +
`<xml>${item.entry}</xml>` +
`</div>`
}).join("\n");
}
function render_multiple(query, data, type) {
let html = "";
data.forEach(item => {
@ -113,6 +123,8 @@
html += `<div class="results-notion">` + `<b><a href="${item.additional.file}">${item.additional.heading}</a></b>` + `<p>${item.entry}</p>` + `</div>`;
} else if (item.additional.file.endsWith(".html")) {
html += render_html(query, [item]);
} else if (item.additional.file.endsWith(".xml")) {
html += render_xml(query, [item])
} else {
html += `<div class="results-plugin">` + `<b><a href="${item.additional.file}">${item.additional.heading}</a></b>` + `<p>${item.entry}</p>` + `</div>`;
}
@ -170,10 +182,12 @@
// Execute Search and Render Results
url = await createRequestUrl(query, type, results_count || 5, rerank);
fetch(url)
const khojToken = await window.tokenAPI.getToken();
const headers = { 'Authorization': `Bearer ${khojToken}` };
fetch(url, { headers })
.then(response => response.json())
.then(data => {
console.log(data);
document.getElementById("results").innerHTML = render_results(data, query, type);
});
}
@ -192,9 +206,11 @@
async function populate_type_dropdown() {
const hostURL = await window.hostURLAPI.getURL();
const khojToken = await window.tokenAPI.getToken();
const headers = { 'Authorization': `Bearer ${khojToken}` };
// Populate type dropdown field with enabled content types only
fetch(`${hostURL}/api/config/types`)
fetch(`${hostURL}/api/config/types`, { headers })
.then(response => response.json())
.then(enabled_types => {
// Show warning if no content types are enabled
@ -247,9 +263,9 @@
<img class="khoj-logo" src="./assets/icons/khoj-logo-sideways-500.png" alt="Khoj"></img>
</a>
<nav class="khoj-nav">
<a class="khoj-nav" href="./chat.html">Chat</a>
<a class="khoj-nav khoj-nav-selected" href="./index.html">Search</a>
<a class="khoj-nav" href="./config.html">⚙️</a>
<a class="khoj-nav" href="./chat.html">💬 Chat</a>
<a class="khoj-nav khoj-nav-selected" href="./search.html">🔎 Search</a>
<a class="khoj-nav" href="./config.html">⚙️ Settings</a>
</nav>
</div>
@ -286,8 +302,8 @@
body {
padding: 0px;
margin: 0px;
background: #fff;
color: #475569;
background: var(--background-color);
color: var(--main-text-color);
font-family: roboto, karma, segoe ui, sans-serif;
font-size: small;
font-weight: 300;
@ -419,14 +435,6 @@
max-width: 90%;
}
div.khoj-banner-container {
background: linear-gradient(-45deg, #FFC107, #FF9800, #FF5722, #FF9800, #FFC107);
background-size: 400% 400%;
animation: gradient 15s ease infinite;
text-align: center;
padding: 10px;
}
@keyframes gradient {
0% {
background-position: 0% 50%;
@ -443,57 +451,5 @@
text-align: center;
}
button#khoj-banner-submit,
input#khoj-banner-email {
padding: 10px;
border-radius: 5px;
border: 1px solid #475569;
background: #f9fafc;
}
button#khoj-banner-submit:hover,
input#khoj-banner-email:hover {
box-shadow: 0 0 11px #aaa;
}
@media only screen and (max-width: 600px) {
a.khoj-banner {
display: block;
}
p.khoj-banner {
padding: 0;
}
}
</style>
<script>
var khojBannerSubmit = document.getElementById("khoj-banner-submit");
khojBannerSubmit?.addEventListener("click", function(event) {
event.preventDefault();
var email = document.getElementById("khoj-banner-email").value;
fetch("https://app.khoj.dev/beta/users/", {
method: "POST",
body: JSON.stringify({
email: email
}),
headers: {
"Content-Type": "application/json"
}
}).then(function(response) {
return response.json();
}).then(function(data) {
console.log(data);
if (data.user != null) {
document.getElementById("khoj-banner").innerHTML = "Thanks for signing up. We'll be in touch soon! 🚀";
document.getElementById("khoj-banner-submit").remove();
} else {
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
}
}).catch(function(error) {
console.log(error);
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
});
});
</script>
</html>

15
src/interface/desktop/splash.html Normal file
View file

@ -0,0 +1,15 @@
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0 maximum-scale=1.0">
<title>Khoj</title>
<link rel="icon" type="image/png" sizes="128x128" href="./assets/icons/favicon-128x128.png">
<link rel="manifest" href="./khoj.webmanifest">
</head>
<script type="text/javascript" src="./assets/three.min.js"></script>
<body>
<div id="loading-animation"></div>
</body>
<script src="./loading-animation.js"></script>
</html>

26
src/interface/desktop/utils.js Normal file
View file

@ -0,0 +1,26 @@
console.log(`%c %s`, "font-family:monospace", `
__ __ __ __ ______ __ _____ __
/\\ \\/ / /\\ \\_\\ \\ /\\ __ \\ /\\ \\ /\\ __ \\ /\\ \\
\\ \\ _"-. \\ \\ __ \\ \\ \\ \\/\\ \\ _\\_\\ \\ \\ \\ __ \\ \\ \\ \\
\\ \\_\\ \\_\\ \\ \\_\\ \\_\\ \\ \\_____\\ /\\_____\\ \\ \\_\\ \\_\\ \\ \\_\\
\\/_/\\/_/ \\/_/\\/_/ \\/_____/ \\/_____/ \\/_/\\/_/ \\/_/
Greetings traveller,
I am ✨Khoj✨, your open-source, personal AI copilot.
See my source code at https://github.com/khoj-ai/khoj
Read my operating manual at https://docs.khoj.dev
`);
window.appInfoAPI.getInfo((_, info) => {
let khojVersionElement = document.getElementById("about-page-version");
if (khojVersionElement) {
khojVersionElement.innerHTML = `<code>${info.version}</code>`;
}
let khojTitleElement = document.getElementById("about-page-title");
if (khojTitleElement) {
khojTitleElement.innerHTML = '<b>Khoj for ' + (info.platform === 'win32' ? 'Windows' : info.platform === 'darwin' ? 'macOS' : 'Linux') + '</b>';
}
});

16
src/interface/desktop/yarn.lock
View file

@ -163,10 +163,10 @@ atomically@^1.7.0:
resolved "https://registry.yarnpkg.com/atomically/-/atomically-1.7.0.tgz#c07a0458432ea6dbc9a3506fffa424b48bccaafe"
integrity sha512-Xcz9l0z7y9yQ9rdDaxlmaI4uJHf/T8g9hOEzJcsEqX2SjCj4J20uK7+ldkDHMbpJDK76wF7xEIgxc/vSlsfw5w==
axios@^1.5.0:
version "1.5.0"
resolved "https://registry.yarnpkg.com/axios/-/axios-1.5.0.tgz#f02e4af823e2e46a9768cfc74691fdd0517ea267"
integrity sha512-D4DdjDo5CY50Qms0qGQTTw6Q44jl7zRwY7bthds06pUGfChBCTcQs+N743eFWGEd6pRTMd6A+I87aWyFV5wiZQ==
axios@^1.6.0:
version "1.6.2"
resolved "https://registry.yarnpkg.com/axios/-/axios-1.6.2.tgz#de67d42c755b571d3e698df1b6504cde9b0ee9f2"
integrity sha512-7i24Ri4pmDRfJTR7LDBhsOTtcm+9kjX5WiY1X3wIisx6G9So3pfMkEiU7emUBe46oceVImccTEM3k6C5dbVW8A==
dependencies:
follow-redirects "^1.15.0"
form-data "^4.0.0"
@ -379,10 +379,10 @@ electron-updater@^4.6.1:
lodash.isequal "^4.5.0"
semver "^7.3.5"
electron@25.8.1:
version "25.8.1"
resolved "https://registry.yarnpkg.com/electron/-/electron-25.8.1.tgz#092fab5a833db4d9240d4d6f36218cf7ca954f86"
integrity sha512-GtcP1nMrROZfFg0+mhyj1hamrHvukfF6of2B/pcWxmWkd5FVY1NJib0tlhiorFZRzQN5Z+APLPr7aMolt7i2AQ==
electron@25.8.4:
version "25.8.4"
resolved "https://registry.yarnpkg.com/electron/-/electron-25.8.4.tgz#b50877aac7d96323920437baf309ad86382cb455"
integrity sha512-hUYS3RGdaa6E1UWnzeGnsdsBYOggwMMg4WGxNGvAoWtmRrr6J1BsjFW/yRq4WsJHJce2HdzQXtz4OGXV6yUCLg==
dependencies:
"@electron/get" "^2.0.0"
"@types/node" "^18.11.18"

145
src/interface/emacs/khoj.el
View file

@ -93,7 +93,7 @@
:group 'khoj
:type 'number)
(defcustom khoj-server-api-key "secret"
(defcustom khoj-api-key nil
"API Key to Khoj server."
:group 'khoj
:type 'string)
@ -246,26 +246,6 @@ for example), set this to the full interpreter path."
:type '(repeat string)
:group 'khoj)
(defcustom khoj-chat-model "gpt-3.5-turbo"
"Specify chat model to use for chat with khoj."
:type 'string
:group 'khoj)
(defcustom khoj-openai-api-key nil
"OpenAI API key used to configure chat on khoj server."
:type 'string
:group 'khoj)
(defcustom khoj-chat-offline nil
"Use offline model to chat with khoj."
:type 'boolean
:group 'khoj)
(defcustom khoj-offline-chat-model nil
"Specify chat model to use for offline chat with khoj."
:type 'string
:group 'khoj)
(defcustom khoj-auto-setup t
"Automate install, configure and start of khoj server.
Auto invokes setup steps on calling main entrypoint."
@ -319,8 +299,7 @@ Auto invokes setup steps on calling main entrypoint."
:filter (lambda (process msg)
(cond ((string-match (format "Uvicorn running on %s" khoj-server-url) msg)
(progn
(setq khoj--server-ready? t)
(khoj--server-configure)))
(setq khoj--server-ready? t)))
((string-match "Batches: " msg)
(when (string-match "\\([0-9]+\\.[0-9]+\\|\\([0-9]+\\)\\)%?" msg)
(message "khoj.el: %s updating index %s"
@ -383,106 +362,13 @@ Auto invokes setup steps on calling main entrypoint."
(when (not (khoj--server-started?))
(khoj--server-start)))
(defun khoj--get-directory-from-config (config keys &optional level)
"Extract directory under specified KEYS in CONFIG and trim it to LEVEL.
CONFIG is json obtained from Khoj config API."
(let ((item config))
(dolist (key keys)
(setq item (cdr (assoc key item))))
(-> item
(split-string "/")
(butlast (or level nil))
(string-join "/"))))
(defun khoj--server-configure ()
"Configure the Khoj server for search and chat."
(interactive)
(let* ((url-request-method "GET")
(current-config
(with-temp-buffer
(url-insert-file-contents (format "%s/api/config/data" khoj-server-url))
(ignore-error json-end-of-file (json-parse-buffer :object-type 'alist :array-type 'list :null-object json-null :false-object json-false))))
(default-config
(with-temp-buffer
(url-insert-file-contents (format "%s/api/config/data/default" khoj-server-url))
(ignore-error json-end-of-file (json-parse-buffer :object-type 'alist :array-type 'list :null-object json-null :false-object json-false))))
(default-chat-dir (khoj--get-directory-from-config default-config '(processor conversation conversation-logfile)))
(chat-model (or khoj-chat-model (alist-get 'chat-model (alist-get 'openai (alist-get 'conversation (alist-get 'processor default-config))))))
(enable-offline-chat (or khoj-chat-offline (alist-get 'enable-offline-chat (alist-get 'offline-chat (alist-get 'conversation (alist-get 'processor default-config))))))
(offline-chat-model (or khoj-offline-chat-model (alist-get 'chat-model (alist-get 'offline-chat (alist-get 'conversation (alist-get 'processor default-config))))))
(config (or current-config default-config)))
;; Configure processors
(cond
((not khoj-openai-api-key)
(let* ((processor (assoc 'processor config))
(conversation (assoc 'conversation processor))
(openai (assoc 'openai conversation)))
(when openai
;; Unset the `openai' field in the khoj conversation processor config
(message "khoj.el: Disable Chat using OpenAI as your OpenAI API key got removed from config")
(setcdr conversation (delq openai (cdr conversation)))
(push conversation (cdr processor))
(push processor config))))
;; If khoj backend isn't configured yet
((not current-config)
(message "khoj.el: Khoj not configured yet.")
(setq config (delq (assoc 'processor config) config))
(cl-pushnew `(processor . ((conversation . ((conversation-logfile . ,(format "%s/conversation.json" default-chat-dir))
(offline-chat . ((enable-offline-chat . ,enable-offline-chat)
(chat-model . ,offline-chat-model)))
(openai . ((chat-model . ,chat-model)
(api-key . ,khoj-openai-api-key)))))))
config))
;; Else if chat isn't configured in khoj backend
((not (alist-get 'conversation (alist-get 'processor config)))
(message "khoj.el: Chat not configured yet.")
(let ((new-processor-type (alist-get 'processor config)))
(setq new-processor-type (delq (assoc 'conversation new-processor-type) new-processor-type))
(cl-pushnew `(conversation . ((conversation-logfile . ,(format "%s/conversation.json" default-chat-dir))
(offline-chat . ((enable-offline-chat . ,enable-offline-chat)
(chat-model . ,offline-chat-model)))
(openai . ((chat-model . ,chat-model)
(api-key . ,khoj-openai-api-key)))))
new-processor-type)
(setq config (delq (assoc 'processor config) config))
(cl-pushnew `(processor . ,new-processor-type) config)))
;; Else if chat configuration in khoj backend has gone stale
((not (and (equal (alist-get 'api-key (alist-get 'openai (alist-get 'conversation (alist-get 'processor config)))) khoj-openai-api-key)
(equal (alist-get 'chat-model (alist-get 'openai (alist-get 'conversation (alist-get 'processor config)))) khoj-chat-model)
(equal (alist-get 'enable-offline-chat (alist-get 'offline-chat (alist-get 'conversation (alist-get 'processor config)))) enable-offline-chat)
(equal (alist-get 'chat-model (alist-get 'offline-chat (alist-get 'conversation (alist-get 'processor config)))) offline-chat-model)))
(message "khoj.el: Chat configuration has gone stale.")
(let* ((chat-directory (khoj--get-directory-from-config config '(processor conversation conversation-logfile)))
(new-processor-type (alist-get 'processor config)))
(setq new-processor-type (delq (assoc 'conversation new-processor-type) new-processor-type))
(cl-pushnew `(conversation . ((conversation-logfile . ,(format "%s/conversation.json" chat-directory))
(offline-chat . ((enable-offline-chat . ,enable-offline-chat)
(chat-model . ,offline-chat-model)))
(openai . ((chat-model . ,khoj-chat-model)
(api-key . ,khoj-openai-api-key)))))
new-processor-type)
(setq config (delq (assoc 'processor config) config))
(cl-pushnew `(processor . ,new-processor-type) config))))
;; Update server with latest configuration, if required
(cond ((not current-config)
(khoj--post-new-config config)
(message "khoj.el: ⚙️ Generated new khoj server configuration."))
((not (equal config current-config))
(khoj--post-new-config config)
(message "khoj.el: ⚙️ Updated khoj server configuration.")))))
(defun khoj-setup (&optional interact)
"Install, start and configure Khoj server. Get permission if INTERACT is non-nil."
"Install and start Khoj server. Get permission if INTERACT is non-nil."
(interactive "p")
;; Setup khoj server if not running
(let* ((not-started (not (khoj--server-started?)))
(permitted (if (and not-started interact)
(y-or-n-p "Could not connect to Khoj server. Should I install, start and configure it for you?")
(y-or-n-p "Could not connect to Khoj server. Should I install, start it for you?")
t)))
;; If user permits setup of khoj server from khoj.el
(when permitted
@ -491,12 +377,9 @@ CONFIG is json obtained from Khoj config API."
(khoj--server-setup))
;; Wait until server is ready
;; As server can be started but not ready to use/configure
;; As server can be started but not ready to use
(while (not khoj--server-ready?)
(sit-for 0.5))
;; Configure server once it's ready
(khoj--server-configure))))
(sit-for 0.5)))))
;; -------------------
@ -516,7 +399,7 @@ CONFIG is json obtained from Khoj config API."
(let ((url-request-method "POST")
(url-request-data (khoj--render-files-as-request-body files-to-index khoj--indexed-files boundary))
(url-request-extra-headers `(("content-type" . ,(format "multipart/form-data; boundary=%s" boundary))
("x-api-key" . ,khoj-server-api-key))))
("Authorization" . ,(format "Bearer %s" khoj-api-key)))))
(with-current-buffer
(url-retrieve (format "%s/api/v1/index/update?%s&force=%s&client=emacs" khoj-server-url type-query (or force "false"))
;; render response from indexing API endpoint on server
@ -690,19 +573,22 @@ Use `BOUNDARY' to separate files. This is sent to Khoj server as a POST request.
"Configure khoj server with provided CONFIG."
;; POST provided config to khoj server
(let ((url-request-method "POST")
(url-request-extra-headers '(("Content-Type" . "application/json")))
(url-request-extra-headers `(("Content-Type" . "application/json")
("Authorization" . ,(format "Bearer %s" khoj-api-key))))
(url-request-data (encode-coding-string (json-encode-alist config) 'utf-8))
(config-url (format "%s/api/config/data" khoj-server-url)))
(with-current-buffer (url-retrieve-synchronously config-url)
(buffer-string)))
;; Update index on khoj server after configuration update
(let ((khoj--server-ready? nil))
(let ((khoj--server-ready? nil)
(url-request-extra-headers `(("Authorization" . ,(format "\"Bearer %s\"" khoj-api-key)))))
(url-retrieve (format "%s/api/update?client=emacs" khoj-server-url) #'identity)))
(defun khoj--get-enabled-content-types ()
"Get content types enabled for search from API."
(let ((config-url (format "%s/api/config/types" khoj-server-url))
(url-request-method "GET"))
(url-request-method "GET")
(url-request-extra-headers `(("Authorization" . ,(format "Bearer %s" khoj-api-key)))))
(with-temp-buffer
(url-insert-file-contents config-url)
(thread-last
@ -722,7 +608,8 @@ Render results in BUFFER-NAME using QUERY, CONTENT-TYPE."
;; get json response from api
(with-current-buffer buffer-name
(let ((inhibit-read-only t)
(url-request-method "GET"))
(url-request-method "GET")
(url-request-extra-headers `(("Authorization" . ,(format "Bearer %s" khoj-api-key)))))
(erase-buffer)
(url-insert-file-contents query-url)))
;; render json response into formatted entries
@ -848,6 +735,7 @@ Render results in BUFFER-NAME using QUERY, CONTENT-TYPE."
"Send QUERY to Khoj Chat API."
(let* ((url-request-method "GET")
(encoded-query (url-hexify-string query))
(url-request-extra-headers `(("Authorization" . ,(format "Bearer %s" khoj-api-key))))
(query-url (format "%s/api/chat?q=%s&n=%s&client=emacs" khoj-server-url encoded-query khoj-results-count)))
(with-temp-buffer
(condition-case ex
@ -862,6 +750,7 @@ Render results in BUFFER-NAME using QUERY, CONTENT-TYPE."
(defun khoj--get-chat-history-api ()
"Send QUERY to Khoj Chat History API."
(let* ((url-request-method "GET")
(url-request-extra-headers `(("Authorization" . ,(format "Bearer %s" khoj-api-key))))
(query-url (format "%s/api/chat/history?client=emacs" khoj-server-url)))
(with-temp-buffer
(condition-case ex

6
src/interface/obsidian/src/chat_modal.ts
View file

@ -142,7 +142,8 @@ export class KhojChatModal extends Modal {
async getChatHistory(): Promise<void> {
// Get chat history from Khoj backend
let chatUrl = `${this.setting.khojUrl}/api/chat/history?client=obsidian`;
let response = await request(chatUrl);
let headers = { "Authorization": `Bearer ${this.setting.khojApiKey}` };
let response = await request({ url: chatUrl, headers: headers });
let chatLogs = JSON.parse(response).response;
chatLogs.forEach((chatLog: any) => {
this.renderMessageWithReferences(chatLog.message, chatLog.by, chatLog.context, new Date(chatLog.created));
@ -168,7 +169,8 @@ export class KhojChatModal extends Modal {
method: "GET",
headers: {
"Access-Control-Allow-Origin": "*",
"Content-Type": "text/event-stream"
"Content-Type": "text/event-stream",
"Authorization": `Bearer ${this.setting.khojApiKey}`,
},
})

31
src/interface/obsidian/src/main.ts
View file

@ -1,8 +1,8 @@
import { Notice, Plugin, TFile } from 'obsidian';
import { Notice, Plugin, request } from 'obsidian';
import { KhojSetting, KhojSettingTab, DEFAULT_SETTINGS } from 'src/settings'
import { KhojSearchModal } from 'src/search_modal'
import { KhojChatModal } from 'src/chat_modal'
import { configureKhojBackend, updateContentIndex } from './utils';
import { updateContentIndex } from './utils';
export default class Khoj extends Plugin {
@ -39,9 +39,9 @@ export default class Khoj extends Plugin {
id: 'chat',
name: 'Chat',
checkCallback: (checking) => {
if (!checking && this.settings.connectedToBackend && (!!this.settings.openaiApiKey || this.settings.enableOfflineChat))
if (!checking && this.settings.connectedToBackend)
new KhojChatModal(this.app, this.settings).open();
return !!this.settings.openaiApiKey || this.settings.enableOfflineChat;
return this.settings.connectedToBackend;
}
});
@ -70,16 +70,27 @@ export default class Khoj extends Plugin {
// Load khoj obsidian plugin settings
this.settings = Object.assign({}, DEFAULT_SETTINGS, await this.loadData());
if (this.settings.autoConfigure) {
// Load, configure khoj server settings
await configureKhojBackend(this.app.vault, this.settings);
// Check if khoj backend is configured, note if cannot connect to backend
let headers = { "Authorization": `Bearer ${this.settings.khojApiKey}` };
if (this.settings.khojUrl === "https://app.khoj.dev") {
if (this.settings.khojApiKey === "") {
new Notice(`❗️Khoj API key is not configured. Please visit https://app.khoj.dev to get an API key.`);
return;
}
await request({ url: this.settings.khojUrl ,method: "GET", headers: headers })
.then(response => {
this.settings.connectedToBackend = true;
})
.catch(error => {
this.settings.connectedToBackend = false;
new Notice(`❗️Ensure Khoj backend is running and Khoj URL is pointing to it in the plugin settings.\n\n${error}`);
});
}
}
async saveSettings() {
if (this.settings.autoConfigure) {
await configureKhojBackend(this.app.vault, this.settings, false);
}
this.saveData(this.settings);
}

5
src/interface/obsidian/src/search_modal.ts
View file

@ -90,10 +90,11 @@ export class KhojSearchModal extends SuggestModal<SearchResult> {
// Query Khoj backend for search results
let encodedQuery = encodeURIComponent(query);
let searchUrl = `${this.setting.khojUrl}/api/search?q=${encodedQuery}&n=${this.setting.resultsCount}&r=${this.rerank}&client=obsidian`;
let headers = { 'Authorization': `Bearer ${this.setting.khojApiKey}` }
// Get search results for markdown and pdf files
let mdResponse = await request(`${searchUrl}&t=markdown`);
let pdfResponse = await request(`${searchUrl}&t=pdf`);
let mdResponse = await request({ url: `${searchUrl}&t=markdown`, headers: headers });
let pdfResponse = await request({ url: `${searchUrl}&t=pdf`, headers: headers });
// Parse search results
let mdData = JSON.parse(mdResponse)

29
src/interface/obsidian/src/settings.ts
View file

@ -3,22 +3,20 @@ import Khoj from 'src/main';
import { updateContentIndex } from './utils';
export interface KhojSetting {
enableOfflineChat: boolean;
openaiApiKey: string;
resultsCount: number;
khojUrl: string;
khojApiKey: string;
connectedToBackend: boolean;
autoConfigure: boolean;
lastSyncedFiles: TFile[];
}
export const DEFAULT_SETTINGS: KhojSetting = {
enableOfflineChat: false,
resultsCount: 6,
khojUrl: 'http://127.0.0.1:42110',
khojApiKey: '',
connectedToBackend: false,
autoConfigure: true,
openaiApiKey: '',
lastSyncedFiles: []
}
@ -49,21 +47,12 @@ export class KhojSettingTab extends PluginSettingTab {
containerEl.firstElementChild?.setText(this.getBackendStatusMessage());
}));
new Setting(containerEl)
.setName('OpenAI API Key')
.setDesc('Use OpenAI for Khoj Chat with your API key.')
.setName('Khoj API Key')
.setDesc('Use Khoj Cloud with your Khoj API Key')
.addText(text => text
.setValue(`${this.plugin.settings.openaiApiKey}`)
.setValue(`${this.plugin.settings.khojApiKey}`)
.onChange(async (value) => {
this.plugin.settings.openaiApiKey = value.trim();
await this.plugin.saveSettings();
}));
new Setting(containerEl)
.setName('Enable Offline Chat')
.setDesc('Chat privately without an internet connection. Enabling this will use offline chat even if OpenAI is configured.')
.addToggle(toggle => toggle
.setValue(this.plugin.settings.enableOfflineChat)
.onChange(async (value) => {
this.plugin.settings.enableOfflineChat = value;
this.plugin.settings.khojApiKey = value.trim();
await this.plugin.saveSettings();
}));
new Setting(containerEl)
@ -78,8 +67,8 @@ export class KhojSettingTab extends PluginSettingTab {
await this.plugin.saveSettings();
}));
new Setting(containerEl)
.setName('Auto Configure')
.setDesc('Automatically configure the Khoj backend.')
.setName('Auto Sync')
.setDesc('Automatically index your vault with Khoj.')
.addToggle(toggle => toggle
.setValue(this.plugin.settings.autoConfigure)
.onChange(async (value) => {
@ -88,7 +77,7 @@ export class KhojSettingTab extends PluginSettingTab {
}));
let indexVaultSetting = new Setting(containerEl);
indexVaultSetting
.setName('Index Vault')
.setName('Force Sync')
.setDesc('Manually force Khoj to re-index your Obsidian Vault.')
.addButton(button => button
.setButtonText('Update')

118
src/interface/obsidian/src/utils.ts
View file

@ -1,4 +1,4 @@
import { FileSystemAdapter, Notice, RequestUrlParam, request, Vault, Modal, TFile } from 'obsidian';
import { FileSystemAdapter, Notice, Vault, Modal, TFile } from 'obsidian';
import { KhojSetting } from 'src/settings'
export function getVaultAbsolutePath(vault: Vault): string {
@ -9,26 +9,6 @@ export function getVaultAbsolutePath(vault: Vault): string {
return '';
}
type OpenAIType = null | {
"chat-model": string;
"api-key": string;
};
type OfflineChatType = null | {
"chat-model": string;
"enable-offline-chat": boolean;
};
interface ProcessorData {
conversation: {
"conversation-logfile": string;
openai: OpenAIType;
"offline-chat": OfflineChatType;
"tokenizer": null | string;
"max-prompt-size": null | number;
};
}
function fileExtensionToMimeType (extension: string): string {
switch (extension) {
case 'pdf':
@ -78,7 +58,7 @@ export async function updateContentIndex(vault: Vault, setting: KhojSetting, las
const response = await fetch(`${setting.khojUrl}/api/v1/index/update?force=${regenerate}&client=obsidian`, {
method: 'POST',
headers: {
'x-api-key': 'secret',
'Authorization': `Bearer ${setting.khojApiKey}`,
},
body: formData,
});
@ -92,100 +72,6 @@ export async function updateContentIndex(vault: Vault, setting: KhojSetting, las
return files;
}
export async function configureKhojBackend(vault: Vault, setting: KhojSetting, notify: boolean = true) {
let khojConfigUrl = `${setting.khojUrl}/api/config/data`;
// Check if khoj backend is configured, note if cannot connect to backend
let khoj_already_configured = await request(khojConfigUrl)
.then(response => {
setting.connectedToBackend = true;
return response !== "null"
})
.catch(error => {
setting.connectedToBackend = false;
if (notify)
new Notice(`❗️Ensure Khoj backend is running and Khoj URL is pointing to it in the plugin settings.\n\n${error}`);
})
// Short-circuit configuring khoj if unable to connect to khoj backend
if (!setting.connectedToBackend) return;
// Set index name from the path of the current vault
// Get default config fields from khoj backend
let defaultConfig = await request(`${khojConfigUrl}/default`).then(response => JSON.parse(response));
let khojDefaultChatDirectory = getIndexDirectoryFromBackendConfig(defaultConfig["processor"]["conversation"]["conversation-logfile"]);
let khojDefaultOpenAIChatModelName = defaultConfig["processor"]["conversation"]["openai"]["chat-model"];
let khojDefaultOfflineChatModelName = defaultConfig["processor"]["conversation"]["offline-chat"]["chat-model"];
// Get current config if khoj backend configured, else get default config from khoj backend
await request(khoj_already_configured ? khojConfigUrl : `${khojConfigUrl}/default`)
.then(response => JSON.parse(response))
.then(data => {
let conversationLogFile = data?.["processor"]?.["conversation"]?.["conversation-logfile"] ?? `${khojDefaultChatDirectory}/conversation.json`;
let processorData: ProcessorData = {
"conversation": {
"conversation-logfile": conversationLogFile,
"openai": null,
"offline-chat": {
"chat-model": khojDefaultOfflineChatModelName,
"enable-offline-chat": setting.enableOfflineChat,
},
"tokenizer": null,
"max-prompt-size": null,
}
}
// If the Open AI API Key was configured in the plugin settings
if (!!setting.openaiApiKey) {
let openAIChatModel = data?.["processor"]?.["conversation"]?.["openai"]?.["chat-model"] ?? khojDefaultOpenAIChatModelName;
processorData = {
"conversation": {
"conversation-logfile": conversationLogFile,
"openai": {
"chat-model": openAIChatModel,
"api-key": setting.openaiApiKey,
},
"offline-chat": {
"chat-model": khojDefaultOfflineChatModelName,
"enable-offline-chat": setting.enableOfflineChat,
},
"tokenizer": null,
"max-prompt-size": null,
},
}
}
// Set khoj processor config to conversation processor config
data["processor"] = processorData;
// Save updated config and refresh index on khoj backend
updateKhojBackend(setting.khojUrl, data);
if (!khoj_already_configured)
console.log(`Khoj: Created khoj backend config:\n${JSON.stringify(data)}`)
else
console.log(`Khoj: Updated khoj backend config:\n${JSON.stringify(data)}`)
})
.catch(error => {
if (notify)
new Notice(`❗️Failed to configure Khoj backend. Contact developer on Github.\n\nError: ${error}`);
})
}
export async function updateKhojBackend(khojUrl: string, khojConfig: Object) {
// POST khojConfig to khojConfigUrl
let requestContent: RequestUrlParam = {
url: `${khojUrl}/api/config/data`,
body: JSON.stringify(khojConfig),
method: 'POST',
contentType: 'application/json',
};
// Save khojConfig on khoj backend at khojConfigUrl
request(requestContent);
}
function getIndexDirectoryFromBackendConfig(filepath: string) {
return filepath.split("/").slice(0, -1).join("/");
}
export async function createNote(name: string, newLeaf = false): Promise<void> {
try {
let pathPrefix: string

2
src/interface/obsidian/styles.css
View file

@ -8,7 +8,7 @@ If your plugin does not need CSS, delete this file.
*/
:root {
--khoj-chat-primary: #ffb300;
--khoj-chat-primary: #fee285;
--khoj-chat-dark-grey: #475569;
}

268
src/khoj/configure.py
View file

@ -1,43 +1,97 @@
# Standard Packages
import sys
import logging
import json
from enum import Enum
from typing import Optional
import requests
import os
# External Packages
import schedule
from fastapi.staticfiles import StaticFiles
from starlette.middleware.sessions import SessionMiddleware
from starlette.middleware.authentication import AuthenticationMiddleware
from starlette.requests import HTTPConnection
from starlette.authentication import (
AuthCredentials,
AuthenticationBackend,
SimpleUser,
UnauthenticatedUser,
)
# Internal Packages
from database.models import KhojUser, Subscription
from database.adapters import get_all_users, get_or_create_search_model
from khoj.processor.embeddings import CrossEncoderModel, EmbeddingsModel
from khoj.routers.indexer import configure_content, load_content, configure_search
from khoj.utils import constants, state
from khoj.utils.config import (
SearchType,
ProcessorConfigModel,
ConversationProcessorConfigModel,
)
from khoj.utils.helpers import resolve_absolute_path, merge_dicts
from khoj.utils.fs_syncer import collect_files
from khoj.utils.rawconfig import FullConfig, OfflineChatProcessorConfig, ProcessorConfig, ConversationProcessorConfig
from khoj.routers.indexer import configure_content, load_content, configure_search
from khoj.utils.rawconfig import FullConfig
logger = logging.getLogger(__name__)
def initialize_server(config: Optional[FullConfig], required=False):
if config is None and required:
logger.error(
f"🚨 Exiting as Khoj is not configured.\nConfigure it via http://{state.host}:{state.port}/config or by editing {state.config_file}."
)
sys.exit(1)
elif config is None:
logger.warning(
f"🚨 Khoj is not configured.\nConfigure it via http://{state.host}:{state.port}/config, plugins or by editing {state.config_file}."
)
return None
class AuthenticatedKhojUser(SimpleUser):
def __init__(self, user):
self.object = user
super().__init__(user.email)
class UserAuthenticationBackend(AuthenticationBackend):
def __init__(
self,
):
from database.models import KhojUser, KhojApiUser
self.khojuser_manager = KhojUser.objects
self.khojapiuser_manager = KhojApiUser.objects
self._initialize_default_user()
super().__init__()
def _initialize_default_user(self):
if not self.khojuser_manager.filter(username="default").exists():
default_user = self.khojuser_manager.create_user(
username="default",
email="default@example.com",
password="default",
)
Subscription.objects.create(user=default_user, type="standard", renewal_date="2100-04-01")
async def authenticate(self, request: HTTPConnection):
current_user = request.session.get("user")
if current_user and current_user.get("email"):
user = (
await self.khojuser_manager.filter(email=current_user.get("email"))
.prefetch_related("subscription")
.afirst()
)
if user:
return AuthCredentials(["authenticated"]), AuthenticatedKhojUser(user)
if len(request.headers.get("Authorization", "").split("Bearer ")) == 2:
# Get bearer token from header
bearer_token = request.headers["Authorization"].split("Bearer ")[1]
# Get user owning token
user_with_token = (
await self.khojapiuser_manager.filter(token=bearer_token)
.select_related("user")
.prefetch_related("user__subscription")
.afirst()
)
if user_with_token:
return AuthCredentials(["authenticated"]), AuthenticatedKhojUser(user_with_token.user)
if state.anonymous_mode:
user = await self.khojuser_manager.filter(username="default").prefetch_related("subscription").afirst()
if user:
return AuthCredentials(["authenticated"]), AuthenticatedKhojUser(user)
return AuthCredentials(), UnauthenticatedUser()
def initialize_server(config: Optional[FullConfig]):
try:
configure_server(config, init=True)
except Exception as e:
@ -45,32 +99,30 @@ def initialize_server(config: Optional[FullConfig], required=False):
def configure_server(
config: FullConfig, regenerate: bool = False, search_type: Optional[SearchType] = None, init=False
config: FullConfig,
regenerate: bool = False,
search_type: Optional[SearchType] = None,
init=False,
user: KhojUser = None,
):
# Update Config
if config == None:
logger.info(f"🚨 Khoj is not configured.\nInitializing it with a default config.")
config = FullConfig()
state.config = config
# Initialize Processor from Config
try:
state.processor_config = configure_processor(state.config.processor)
except Exception as e:
logger.error(f"🚨 Failed to configure processor", exc_info=True)
raise e
# Initialize Search Models from Config and initialize content
try:
state.config_lock.acquire()
state.SearchType = configure_search_types(state.config)
state.embeddings_model = EmbeddingsModel(get_or_create_search_model().bi_encoder)
state.cross_encoder_model = CrossEncoderModel(get_or_create_search_model().cross_encoder)
state.SearchType = configure_search_types()
state.search_models = configure_search(state.search_models, state.config.search_type)
initialize_content(regenerate, search_type, init)
initialize_content(regenerate, search_type, init, user)
except Exception as e:
logger.error(f"🚨 Failed to configure search models", exc_info=True)
raise e
finally:
state.config_lock.release()
def initialize_content(regenerate: bool, search_type: Optional[SearchType] = None, init=False):
def initialize_content(regenerate: bool, search_type: Optional[SearchType] = None, init=False, user: KhojUser = None):
# Initialize Content from Config
if state.search_models:
try:
@ -79,17 +131,19 @@ def initialize_content(regenerate: bool, search_type: Optional[SearchType] = Non
state.content_index = load_content(state.config.content_type, state.content_index, state.search_models)
else:
logger.info("📬 Updating content index...")
all_files = collect_files(state.config.content_type)
state.content_index = configure_content(
all_files = collect_files(user=user)
state.content_index, status = configure_content(
state.content_index,
state.config.content_type,
all_files,
state.search_models,
regenerate,
search_type,
user=user,
)
if not status:
raise RuntimeError("Failed to update content index")
except Exception as e:
logger.error(f"🚨 Failed to index content", exc_info=True)
raise e
@ -99,134 +153,50 @@ def configure_routes(app):
from khoj.routers.api_beta import api_beta
from khoj.routers.web_client import web_client
from khoj.routers.indexer import indexer
from khoj.routers.auth import auth_router
from khoj.routers.subscription import subscription_router
app.mount("/static", StaticFiles(directory=constants.web_directory), name="static")
app.include_router(api, prefix="/api")
app.include_router(api_beta, prefix="/api/beta")
app.include_router(indexer, prefix="/api/v1/index")
if state.billing_enabled:
logger.info("💳 Enabled Billing")
app.include_router(subscription_router, prefix="/api/subscription")
app.include_router(web_client)
app.include_router(auth_router, prefix="/auth")
if not state.demo:
def configure_middleware(app):
app.add_middleware(AuthenticationMiddleware, backend=UserAuthenticationBackend())
app.add_middleware(SessionMiddleware, secret_key=os.environ.get("KHOJ_DJANGO_SECRET_KEY", "!secret"))
@schedule.repeat(schedule.every(61).minutes)
def update_search_index():
try:
logger.info("📬 Updating content index via Scheduler")
all_files = collect_files(state.config.content_type)
state.content_index = configure_content(
state.content_index, state.config.content_type, all_files, state.search_models
@schedule.repeat(schedule.every(61).minutes)
def update_search_index():
try:
logger.info("📬 Updating content index via Scheduler")
for user in get_all_users():
all_files = collect_files(user=user)
state.content_index, success = configure_content(
state.content_index, state.config.content_type, all_files, state.search_models, user=user
)
logger.info("📪 Content index updated via Scheduler")
except Exception as e:
logger.error(f"🚨 Error updating content index via Scheduler: {e}", exc_info=True)
all_files = collect_files(user=None)
state.content_index, success = configure_content(
state.content_index, state.config.content_type, all_files, state.search_models, user=None
)
if not success:
raise RuntimeError("Failed to update content index")
logger.info("📪 Content index updated via Scheduler")
except Exception as e:
logger.error(f"🚨 Error updating content index via Scheduler: {e}", exc_info=True)
def configure_search_types(config: FullConfig):
def configure_search_types():
# Extract core search types
core_search_types = {e.name: e.value for e in SearchType}
# Extract configured plugin search types
plugin_search_types = {}
if config.content_type and config.content_type.plugins:
plugin_search_types = {plugin_type: plugin_type for plugin_type in config.content_type.plugins.keys()}
# Dynamically generate search type enum by merging core search types with configured plugin search types
return Enum("SearchType", merge_dicts(core_search_types, plugin_search_types))
def configure_processor(
processor_config: Optional[ProcessorConfig], state_processor_config: Optional[ProcessorConfigModel] = None
):
if not processor_config:
logger.warning("🚨 No Processor configuration available.")
return None
processor = ProcessorConfigModel()
# Initialize Conversation Processor
logger.info("💬 Setting up conversation processor")
processor.conversation = configure_conversation_processor(processor_config, state_processor_config)
return processor
def configure_conversation_processor(
processor_config: Optional[ProcessorConfig], state_processor_config: Optional[ProcessorConfigModel] = None
):
if (
not processor_config
or not processor_config.conversation
or not processor_config.conversation.conversation_logfile
):
default_config = constants.default_config
default_conversation_logfile = resolve_absolute_path(
default_config["processor"]["conversation"]["conversation-logfile"] # type: ignore
)
conversation_logfile = resolve_absolute_path(default_conversation_logfile)
conversation_config = processor_config.conversation if processor_config else None
conversation_processor = ConversationProcessorConfigModel(
conversation_config=ConversationProcessorConfig(
conversation_logfile=conversation_logfile,
openai=(conversation_config.openai if (conversation_config is not None) else None),
offline_chat=conversation_config.offline_chat if conversation_config else OfflineChatProcessorConfig(),
)
)
else:
conversation_processor = ConversationProcessorConfigModel(
conversation_config=processor_config.conversation,
)
conversation_logfile = resolve_absolute_path(conversation_processor.conversation_logfile)
# Load Conversation Logs from Disk
if state_processor_config and state_processor_config.conversation and state_processor_config.conversation.meta_log:
conversation_processor.meta_log = state_processor_config.conversation.meta_log
conversation_processor.chat_session = state_processor_config.conversation.chat_session
logger.debug(f"Loaded conversation logs from state")
return conversation_processor
if conversation_logfile.is_file():
# Load Metadata Logs from Conversation Logfile
with conversation_logfile.open("r") as f:
conversation_processor.meta_log = json.load(f)
logger.debug(f"Loaded conversation logs from {conversation_logfile}")
else:
# Initialize Conversation Logs
conversation_processor.meta_log = {}
conversation_processor.chat_session = []
return conversation_processor
@schedule.repeat(schedule.every(17).minutes)
def save_chat_session():
# No need to create empty log file
if not (
state.processor_config
and state.processor_config.conversation
and state.processor_config.conversation.meta_log
and state.processor_config.conversation.chat_session
):
return
# Summarize Conversation Logs for this Session
conversation_log = state.processor_config.conversation.meta_log
session = {
"session-start": conversation_log.get("session", [{"session-end": 0}])[-1]["session-end"],
"session-end": len(conversation_log["chat"]),
}
if "session" in conversation_log:
conversation_log["session"].append(session)
else:
conversation_log["session"] = [session]
# Save Conversation Metadata Logs to Disk
conversation_logfile = resolve_absolute_path(state.processor_config.conversation.conversation_logfile)
conversation_logfile.parent.mkdir(parents=True, exist_ok=True) # create conversation directory if doesn't exist
with open(conversation_logfile, "w+", encoding="utf-8") as logfile:
json.dump(conversation_log, logfile, indent=2)
state.processor_config.conversation.chat_session = []
logger.info("📩 Saved current chat session to conversation logs")
return Enum("SearchType", core_search_types)
@schedule.repeat(schedule.every(59).minutes)

BIN
src/khoj/interface/web/assets/icons/computer.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 10 KiB

1
src/khoj/interface/web/assets/icons/copy-solid.svg Normal file
View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Pro 6.4.2 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license (Commercial License) Copyright 2023 Fonticons, Inc. --><path d="M208 0H332.1c12.7 0 24.9 5.1 33.9 14.1l67.9 67.9c9 9 14.1 21.2 14.1 33.9V336c0 26.5-21.5 48-48 48H208c-26.5 0-48-21.5-48-48V48c0-26.5 21.5-48 48-48zM48 128h80v64H64V448H256V416h64v48c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V176c0-26.5 21.5-48 48-48z"/></svg>
Side by side

After

Width:  |  Height:  |  Size: 503 B

BIN
src/khoj/interface/web/assets/icons/credit-card.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 19 KiB

4
src/khoj/interface/web/assets/icons/key.svg Normal file
View file

@ -0,0 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="800px" height="800px" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fill-rule="evenodd" clip-rule="evenodd" d="M22 8.29344C22 11.7692 19.1708 14.5869 15.6807 14.5869C15.0439 14.5869 13.5939 14.4405 12.8885 13.8551L12.0067 14.7333C11.4883 15.2496 11.6283 15.4016 11.8589 15.652C11.9551 15.7565 12.0672 15.8781 12.1537 16.0505C12.1537 16.0505 12.8885 17.075 12.1537 18.0995C11.7128 18.6849 10.4783 19.5045 9.06754 18.0995L8.77362 18.3922C8.77362 18.3922 9.65538 19.4167 8.92058 20.4412C8.4797 21.0267 7.30403 21.6121 6.27531 20.5876L5.2466 21.6121C4.54119 22.3146 3.67905 21.9048 3.33616 21.6121L2.45441 20.7339C1.63143 19.9143 2.1115 19.0264 2.45441 18.6849L10.0963 11.0743C10.0963 11.0743 9.3615 9.90338 9.3615 8.29344C9.3615 4.81767 12.1907 2 15.6807 2C19.1708 2 22 4.81767 22 8.29344ZM15.681 10.4889C16.8984 10.4889 17.8853 9.50601 17.8853 8.29353C17.8853 7.08105 16.8984 6.09814 15.681 6.09814C14.4635 6.09814 13.4766 7.08105 13.4766 8.29353C13.4766 9.50601 14.4635 10.4889 15.681 10.4889Z" fill="#1C274C"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 1.1 KiB

BIN
src/khoj/interface/web/assets/icons/khoj-logo-sideways-500.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 36 KiB

After

Width:  |  Height:  |  Size: 29 KiB

1
src/khoj/interface/web/assets/icons/trash-solid.svg Normal file
View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Pro 6.4.2 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license (Commercial License) Copyright 2023 Fonticons, Inc. --><path d="M135.2 17.7L128 32H32C14.3 32 0 46.3 0 64S14.3 96 32 96H416c17.7 0 32-14.3 32-32s-14.3-32-32-32H320l-7.2-14.3C307.4 6.8 296.3 0 284.2 0H163.8c-12.1 0-23.2 6.8-28.6 17.7zM416 128H32L53.2 467c1.6 25.3 22.6 45 47.9 45H346.9c25.3 0 46.3-19.7 47.9-45L416 128z"/></svg>
Side by side

After

Width:  |  Height:  |  Size: 503 B

123
src/khoj/interface/web/assets/khoj.css
View file

@ -2,29 +2,44 @@
/* Can be forced with data-theme="light" */
[data-theme="light"],
:root:not([data-theme="dark"]) {
--primary: #ffb300;
--primary-hover: #ffa000;
--primary: #fee285;
--primary-hover: #fcc50b;
--primary-focus: rgba(255, 179, 0, 0.125);
--primary-inverse: rgba(0, 0, 0, 0.75);
--background-color: #f5f4f3;
--main-text-color: #475569;
--water: #44b9da;
--leaf: #7b990a;
--flower: #ffaeae;
}
/* Amber Dark scheme (Auto) */
/* Automatically enabled if user has Dark mode enabled */
@media only screen and (prefers-color-scheme: dark) {
:root:not([data-theme]) {
--primary: #ffb300;
--primary-hover: #ffc107;
--primary: #fee285;
--primary-hover: #fcc50b;
--primary-focus: rgba(255, 179, 0, 0.25);
--primary-inverse: rgba(0, 0, 0, 0.75);
--background-color: #f5f4f3;
--main-text-color: #475569;
--water: #44b9da;
--leaf: #7b990a;
--flower: #ffaeae;
}
}
/* Amber Dark scheme (Forced) */
/* Enabled if forced with data-theme="dark" */
[data-theme="dark"] {
--primary: #ffb300;
--primary-hover: #ffc107;
--primary: #fee285;
--primary-hover: #fcc50b;
--primary-focus: rgba(255, 179, 0, 0.25);
--primary-inverse: rgba(0, 0, 0, 0.75);
--background-color: #f5f4f3;
--main-text-color: #475569;
--water: #44b9da;
--leaf: #7b990a;
--flower: #ffaeae;
}
/* Amber (Common styles) */
:root {
@ -37,8 +52,11 @@
.khoj-configure {
display: grid;
grid-template-columns: 1fr;
padding: 0 24px;
font-family: roboto, karma, segoe ui, sans-serif;
font-weight: 300;
}
.khoj-footer,
.khoj-header {
display: grid;
grid-auto-flow: column;
@ -46,6 +64,9 @@
padding: 16px 0;
margin: 0 0 16px 0;
}
.khoj-footer {
margin: 16px 0 0 0;
}
nav.khoj-nav {
display: grid;
@ -64,7 +85,7 @@ a.khoj-logo {
}
.khoj-nav a {
color: #333;
color: var(--main-text-color);
text-decoration: none;
font-size: 20px;
font-weight: normal;
@ -85,22 +106,88 @@ img.khoj-logo {
justify-self: center;
}
a.khoj-banner {
/* Dropdown in navigation menu*/
#khoj-nav-menu-container {
display: flex;
align-items: center;
}
.khoj-nav-dropdown-content {
display: block;
grid-auto-flow: row;
position: absolute;
background-color: var(--background-color);
min-width: 160px;
box-shadow: 0px 8px 16px 0px rgba(0,0,0,0.2);
right: 15vw;
top: 64px;
z-index: 1;
opacity: 0;
transition: opacity 0.1s ease-in-out;
pointer-events: none;
text-align: left;
}
.khoj-nav-dropdown-content.show {
opacity: 1;
pointer-events: auto;
}
.khoj-nav-dropdown-content a {
color: black;
padding: 12px 16px;
text-decoration: none;
display: block;
}
.khoj-nav-dropdown-content a:hover {
background-color: var(--primary-hover);
}
.khoj-nav-username {
padding: 12px 16px;
text-decoration: none;
display: block;
font-weight: bold;
}
.circle {
border-radius: 50%;
border: 3px dotted var(--main-text-color);
width: 40px;
height: 40px;
padding: 3px;
cursor: pointer;
}
.circle:hover {
background-color: var(--primary-hover);
}
.user-initial {
background-color: var(--background-color);
color: black;
display: grid;
justify-content: center;
align-items: center;
font-size: 20px;
box-sizing: unset;
width: 40px;
height: 40px;
}
.subscribed {
border: 3px solid var(--primary-hover);
}
p.khoj-banner {
font-size: medium;
margin: 0;
padding: 10px;
@media screen and (max-width: 700px) {
.khoj-nav-dropdown-content {
display: block;
grid-auto-flow: row;
position: absolute;
background-color: var(--background-color);
min-width: 160px;
box-shadow: 0px 8px 16px 0px rgba(0,0,0,0.2);
right: 10px;
z-index: 1;
opacity: 0;
transition: opacity 0.1s ease-in-out;
pointer-events: none;
}
}
p#khoj-banner {
display: inline;
}
@media only screen and (max-width: 600px) {
@media only screen and (max-width: 700px) {
div.khoj-header {
display: grid;
grid-auto-flow: column;

31
src/khoj/interface/web/assets/utils.js Normal file
View file

@ -0,0 +1,31 @@
// Toggle the navigation menu
function toggleMenu() {
var menu = document.getElementById("khoj-nav-menu");
menu.classList.toggle("show");
}
// Close the dropdown menu if the user clicks outside of it
document.addEventListener('click', function(event) {
let menu = document.getElementById("khoj-nav-menu");
let menuContainer = document.getElementById("khoj-nav-menu-container");
let isClickOnMenu = menuContainer.contains(event.target) || menuContainer === event.target;
if (isClickOnMenu === false && menu.classList.contains("show")) {
menu.classList.remove("show");
}
});
console.log(`%c %s`, "font-family:monospace", `
__ __ __ __ ______ __ _____ __
/\\ \\/ / /\\ \\_\\ \\ /\\ __ \\ /\\ \\ /\\ __ \\ /\\ \\
\\ \\ _"-. \\ \\ __ \\ \\ \\ \\/\\ \\ _\\_\\ \\ \\ \\ __ \\ \\ \\ \\
\\ \\_\\ \\_\\ \\ \\_\\ \\_\\ \\ \\_____\\ /\\_____\\ \\ \\_\\ \\_\\ \\ \\_\\
\\/_/\\/_/ \\/_/\\/_/ \\/_____/ \\/_____/ \\/_/\\/_/ \\/_/
Greetings traveller,
I am ✨Khoj✨, your open-source, personal AI copilot.
See my source code at https://github.com/khoj-ai/khoj
Read my operating manual at https://docs.khoj.dev
`);

172
src/khoj/interface/web/base_config.html
View file

@ -8,19 +8,15 @@
<link rel="stylesheet" href="/static/assets/pico.min.css">
<link rel="stylesheet" href="/static/assets/khoj.css">
</head>
<script type="text/javascript" src="/static/assets/utils.js"></script>
<body class="khoj-configure">
<div class="khoj-header-wrapper">
<div class="filler"></div>
<div class="khoj-header">
<a class="khoj-logo" href="https://khoj.dev" target="_blank">
<img class="khoj-logo" src="/static/assets/icons/khoj-logo-sideways-500.png" alt="Khoj"></img>
</a>
<nav class="khoj-nav">
<a class="khoj-nav" href="/chat">Chat</a>
<a class="khoj-nav" href="/">Search</a>
<a class="khoj-nav khoj-nav-selected" href="/config">Settings</a>
</nav>
</div>
<!--Add Header Logo and Nav Pane-->
{% import 'utils.html' as utils %}
{{ utils.heading_pane(user_photo, username, is_active, has_documents) }}
<div class="filler"></div>
</div>
<div class=”content”>
@ -28,6 +24,9 @@
{% endblock %}
</div>
</body>
<script>
document.getElementById("settings-nav").classList.add("khoj-nav-selected");
</script>
<style>
html, body {
width: 100%;
@ -38,10 +37,19 @@
img.khoj-logo {
max-width: none!important;
}
div.khoj-header-wrapper{
div.khoj-header-wrapper {
display: grid;
grid-template-columns: 1fr min(70vw, 100%) 1fr;
}
.circle {
width: 52px;
height: 52px;
}
.user-initial {
width: 40px;
height: 40px;
}
.page {
display: grid;
grid-auto-flow: row;
@ -52,9 +60,33 @@
justify-self: center;
}
div.section-manage-files,
div.api-settings {
display: grid;
grid-template-columns: 1fr;
justify-items: start;
gap: 8px;
padding: 24px 24px;
background: var(--background-color);
border: 1px solid rgb(229, 229, 229);
border-radius: 4px;
box-shadow: 0px 1px 3px 0px rgba(0,0,0,0.1),0px 1px 2px -1px rgba(0,0,0,0.8);
}
div.section.general-settings {
justify-self: center;
div.section-manage-files {
width: 640px;
}
div.api-settings {
grid-template-rows: 1fr 1fr auto;
}
#api-settings-card-description {
margin: 8px 0 0 0;
}
#api-settings-keys-table {
margin-bottom: 16px;
}
div.instructions {
@ -77,21 +109,20 @@
display: grid;
grid-template-rows: repeat(3, 1fr);
gap: 8px;
padding: 24px 16px;
padding: 24px 16px 8px;
width: 320px;
height: 180px;
background: white;
background: var(--background-color);
border: 1px solid rgb(229, 229, 229);
border-radius: 4px;
box-shadow: 0px 1px 3px 0px rgba(0,0,0,0.1),0px 1px 2px -1px rgba(0,0,0,0.1);
box-shadow: 0px 1px 3px 0px rgba(0,0,0,0.1),0px 1px 2px -1px rgba(0,0,0,0.8);
overflow: hidden;
}
div.finalize-buttons {
display: grid;
gap: 8px;
padding: 24px 16px;
padding: 32px 0px 0px;
width: 320px;
background: white;
border-radius: 4px;
overflow: hidden;
}
@ -131,10 +162,13 @@
color: grey;
font-size: 16px;
}
.card-button-row {
.card-description-row {
padding-top: 4px;
}
.card-action-row {
display: grid;
grid-template-columns: auto;
text-align: right;
grid-auto-flow: row;
justify-content: left;
}
.card-button {
border: none;
@ -159,7 +193,7 @@
}
button.card-button {
color: rgb(255, 136, 136);
color: var(--flower);
background: transparent;
font-size: 16px;
cursor: pointer;
@ -170,8 +204,43 @@
text-align: left;
}
button.card-button.happy {
color: rgb(0, 146, 0);
button.remove-file-button:hover {
background-color: rgb(255 235 235);
border-radius: 3px;
border: none;
color: var(--flower);
padding: 4px;
cursor: pointer;
}
button.remove-file-button {
background-color: rgb(253 214 214);
border-radius: 3px;
border: none;
color: var(--flower);
padding: 4px;
width: 32px;
margin-bottom: 0px
}
div.file-element {
display: grid;
grid-template-columns: 1fr 5fr 1fr;
border: 1px solid rgb(229, 229, 229);
border-radius: 4px;
box-shadow: 0px 1px 3px 0px rgba(0,0,0,0.1),0px 1px 2px -1px rgba(0,0,0,0.8);
padding: 4px 0;
margin-bottom: 8px;
justify-items: center;
align-items: center;
}
div.remove-button-container {
text-align: right;
}
.card-button.happy {
color: var(--leaf);
}
img.configured-icon {
@ -205,22 +274,51 @@
100% { transform: rotate(360deg); }
}
#status {
padding-top: 32px;
}
div.finalize-actions {
grid-auto-flow: column;
grid-gap: 24px;
}
@media screen and (max-width: 600px) {
button#logout {
font-size: 16px;
cursor: pointer;
margin: 0;
padding: 0;
height: 32px;
}
select#chat-models {
margin-bottom: 0;
padding: 8px;
}
div.api-settings {
width: 640px;
}
img.api-key-action:hover {
cursor: pointer;
}
a {
color: #3b82f6;
text-decoration: none;
}
@media screen and (max-width: 700px) {
.section-cards {
grid-template-columns: 1fr;
}
}
@media only screen and (max-width: 600px) {
@media only screen and (max-width: 700px) {
body {
display: grid;
grid-template-columns: 1fr;
grid-template-rows: 1fr auto auto auto minmax(80px, 100%);
grid-template-rows: 1fr repeat(4, auto);
}
body > * {
grid-column: 1;
@ -242,6 +340,24 @@
width: 320px;
}
div.khoj-header-wrapper {
grid-template-columns: auto;
}
div.section-manage-files,
div.api-settings {
width: auto;
}
#status {
padding-top: 12px;
}
div.finalize-actions {
padding: 12px 0 0;
}
div.finalize-buttons {
padding: 0;
}
}
</style>
</html>

360
src/khoj/interface/web/chat.html
View file

@ -8,7 +8,18 @@
<link rel="manifest" href="/static/khoj_chat.webmanifest">
<link rel="stylesheet" href="/static/assets/khoj.css">
</head>
<script type="text/javascript" src="/static/assets/utils.js"></script>
<script>
let welcome_message = `
Hi, I am Khoj, your open, personal AI 👋🏽. I can help:
• 🧠 Answer general knowledge questions
• 💡 Be a sounding board for your ideas
• 📜 Chat with your notes & documents
Download the <a class='inline-chat-link' href='https://khoj.dev/downloads'>🖥️ Desktop app</a> to chat with your computer docs.
To get started, just start typing below. You can also type / to see a list of commands.
`.trim()
let chatOptions = [];
function copyProgrammaticOutput(event) {
// Remove the first 4 characters which are the "Copy" button
@ -32,32 +43,101 @@
let escaped_ref = reference.replaceAll('"', '&quot;');
// Generate HTML for Chat Reference
return `<sup><abbr title="${escaped_ref}" tabindex="0">${index}</abbr></sup>`;
let short_ref = escaped_ref.slice(0, 140);
short_ref = short_ref.length < escaped_ref.length ? short_ref + "..." : short_ref;
let referenceButton = document.createElement('button');
referenceButton.innerHTML = short_ref;
referenceButton.id = `ref-${index}`;
referenceButton.classList.add("reference-button");
referenceButton.classList.add("collapsed");
referenceButton.tabIndex = 0;
// Add event listener to toggle full reference on click
referenceButton.addEventListener('click', function() {
console.log(`Toggling ref-${index}`)
if (this.classList.contains("collapsed")) {
this.classList.remove("collapsed");
this.classList.add("expanded");
this.innerHTML = escaped_ref;
} else {
this.classList.add("collapsed");
this.classList.remove("expanded");
this.innerHTML = short_ref;
}
});
return referenceButton;
}
function renderMessage(message, by, dt=null) {
function renderMessage(message, by, dt=null, annotations=null) {
let message_time = formatDate(dt ?? new Date());
let by_name = by == "khoj" ? "🏮 Khoj" : "🤔 You";
let formattedMessage = formatHTMLMessage(message);
// Generate HTML for Chat Message and Append to Chat Body
document.getElementById("chat-body").innerHTML += `
<div data-meta="${by_name} at ${message_time}" class="chat-message ${by}">
<div class="chat-message-text ${by}">${formattedMessage}</div>
</div>
`;
let chatBody = document.getElementById("chat-body");
// Create a new div for the chat message
let chatMessage = document.createElement('div');
chatMessage.className = `chat-message ${by}`;
chatMessage.dataset.meta = `${by_name} at ${message_time}`;
// Create a new div for the chat message text and append it to the chat message
let chatMessageText = document.createElement('div');
chatMessageText.className = `chat-message-text ${by}`;
chatMessageText.innerHTML = formattedMessage;
chatMessage.appendChild(chatMessageText);
// Append annotations div to the chat message
if (annotations) {
chatMessageText.appendChild(annotations);
}
// Append chat message div to chat body
chatBody.appendChild(chatMessage);
// Scroll to bottom of chat-body element
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
chatBody.scrollTop = chatBody.scrollHeight;
}
function renderMessageWithReference(message, by, context=null, dt=null) {
let references = '';
if (context) {
references = context
.map((reference, index) => generateReference(reference, index))
.join("<sup>,</sup>");
if (context == null || context.length == 0) {
renderMessage(message, by, dt);
return;
}
renderMessage(message+references, by, dt);
let references = document.createElement('div');
let referenceExpandButton = document.createElement('button');
referenceExpandButton.classList.add("reference-expand-button");
let expandButtonText = context.length == 1 ? "1 reference" : `${context.length} references`;
referenceExpandButton.innerHTML = expandButtonText;
references.appendChild(referenceExpandButton);
let referenceSection = document.createElement('div');
referenceSection.classList.add("reference-section");
referenceSection.classList.add("collapsed");
referenceExpandButton.addEventListener('click', function() {
if (referenceSection.classList.contains("collapsed")) {
referenceSection.classList.remove("collapsed");
referenceSection.classList.add("expanded");
} else {
referenceSection.classList.add("collapsed");
referenceSection.classList.remove("expanded");
}
});
references.classList.add("references");
if (context) {
for (let index in context) {
let reference = context[index];
let polishedReference = generateReference(reference, index);
referenceSection.appendChild(polishedReference);
}
}
references.appendChild(referenceSection);
renderMessage(message, by, dt, references);
}
function formatHTMLMessage(htmlMessage) {
@ -66,6 +146,8 @@
// Replace any ** with <b> and __ with <u>
newHTML = newHTML.replace(/\*\*([\s\S]*?)\*\*/g, '<b>$1</b>');
newHTML = newHTML.replace(/__([\s\S]*?)__/g, '<u>$1</u>');
// Remove any text between <s>[INST] and </s> tags. These are spurious instructions for the AI chat model.
newHTML = newHTML.replace(/<s>\[INST\].+(<\/s>)?/g, '');
return newHTML;
}
@ -115,6 +197,7 @@
.then(response => {
const reader = response.body.getReader();
const decoder = new TextDecoder();
let references = null;
function readStream() {
reader.read().then(({ done, value }) => {
@ -122,7 +205,11 @@
// Evaluate the contents of new_response_text.innerHTML after all the data has been streamed
const currentHTML = newResponseText.innerHTML;
newResponseText.innerHTML = formatHTMLMessage(currentHTML);
if (references != null) {
newResponseText.appendChild(references);
}
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
document.getElementById("chat-input").removeAttribute("disabled");
return;
}
@ -135,11 +222,36 @@
const rawReference = chunk.split("### compiled references:")[1];
const rawReferenceAsJson = JSON.parse(rawReference);
let polishedReference = rawReferenceAsJson.map((reference, index) => generateReference(reference, index))
.join("<sup>,</sup>");
references = document.createElement('div');
references.classList.add("references");
newResponseText.innerHTML += polishedReference;
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
let referenceExpandButton = document.createElement('button');
referenceExpandButton.classList.add("reference-expand-button");
let expandButtonText = rawReferenceAsJson.length == 1 ? "1 reference" : `${rawReferenceAsJson.length} references`;
referenceExpandButton.innerHTML = expandButtonText;
references.appendChild(referenceExpandButton);
let referenceSection = document.createElement('div');
referenceSection.classList.add("reference-section");
referenceSection.classList.add("collapsed");
referenceExpandButton.addEventListener('click', function() {
if (referenceSection.classList.contains("collapsed")) {
referenceSection.classList.remove("collapsed");
referenceSection.classList.add("expanded");
} else {
referenceSection.classList.add("collapsed");
referenceSection.classList.remove("expanded");
}
});
rawReferenceAsJson.forEach((reference, index) => {
let polishedReference = generateReference(reference, index);
referenceSection.appendChild(polishedReference);
});
references.appendChild(referenceSection);
readStream();
} else {
// Display response from Khoj
@ -156,12 +268,12 @@
});
}
readStream();
document.getElementById("chat-input").removeAttribute("disabled");
});
}
function incrementalChat(event) {
if (!event.shiftKey && event.key === 'Enter') {
event.preventDefault();
chat();
}
}
@ -215,14 +327,14 @@
.then(data => {
if (data.detail) {
// If the server returns a 500 error with detail, render a setup hint.
renderMessage("Hi 👋🏾, to get started you have two options:<ol><li><b>Use OpenAI</b>: <ol><li>Get your <a class='inline-chat-link' href='https://platform.openai.com/account/api-keys'>OpenAI API key</a></li><li>Save it in the Khoj <a class='inline-chat-link' href='/config/processor/conversation/openai'>chat settings</a></li><li>Click Configure on the Khoj <a class='inline-chat-link' href='/config'>settings page</a></li></ol></li><li><b>Enable offline chat</b>: <ol><li>Go to the Khoj <a class='inline-chat-link' href='/config'>settings page</a> and enable offline chat</li></ol></li></ol>", "khoj");
renderMessage("Hi 👋🏾, to start chatting add available chat models options via <a class='inline-chat-link' href='/server/admin'>the Django Admin panel</a> on the Server", "khoj");
// Disable chat input field and update placeholder text
document.getElementById("chat-input").setAttribute("disabled", "disabled");
document.getElementById("chat-input").setAttribute("placeholder", "Configure Khoj to enable chat");
} else {
// Set welcome message on load
renderMessage("Hey 👋🏾, what's up?", "khoj");
renderMessage(welcome_message, "khoj");
}
return data.response;
})
@ -233,6 +345,7 @@
});
})
.catch(err => {
console.log(err);
return;
});
@ -257,37 +370,12 @@
}
</script>
<body>
<div id="khoj-banner-container" class="khoj-banner-container">
{% if demo %}
<!-- Banner linking to https://khoj.dev -->
<a class="khoj-banner" href="https://khoj.dev" target="_blank">
<p id="khoj-banner" class="khoj-banner">
Enroll in Khoj cloud to get your own assistant
</p>
</a>
<input type="text" id="khoj-banner-email" placeholder="email" class="khoj-banner-email"></input>
<button id="khoj-banner-submit" class="khoj-banner-button">Submit</button>
{% endif %}
<div id="khoj-empty-container" class="khoj-empty-container">
</div>
<!--Add Header Logo and Nav Pane-->
<div class="khoj-header">
{% if demo %}
<a class="khoj-logo" href="https://khoj.dev" target="_blank">
<img class="khoj-logo" src="/static/assets/icons/khoj-logo-sideways-500.png" alt="Khoj"></img>
</a>
{% else %}
<a class="khoj-logo" href="/">
<img class="khoj-logo" src="/static/assets/icons/khoj-logo-sideways-500.png" alt="Khoj"></img>
</a>
{% endif %}
<nav class="khoj-nav">
<a class="khoj-nav khoj-nav-selected" href="/chat">Chat</a>
<a class="khoj-nav" href="/">Search</a>
{% if not demo %}
<a class="khoj-nav" href="/config">Settings</a>
{% endif %}
</nav>
</div>
{% import 'utils.html' as utils %}
{{ utils.heading_pane(user_photo, username, is_active, has_documents) }}
<!-- Chat Body -->
<div id="chat-body"></div>
@ -295,11 +383,12 @@
<!-- Chat Footer -->
<div id="chat-footer">
<div id="chat-tooltip" style="display: none;"></div>
<textarea id="chat-input" class="option" oninput="onChatInput()" onkeyup=incrementalChat(event) autofocus="autofocus" placeholder="Type / to see a list of commands, or just type your questions and hit enter.">
</textarea>
<textarea id="chat-input" class="option" oninput="onChatInput()" onkeydown=incrementalChat(event) autofocus="autofocus" placeholder="Type / to see a list of commands, or just type your questions and hit enter."></textarea>
</div>
</body>
<script>
document.getElementById("chat-nav").classList.add("khoj-nav-selected");
</script>
<style>
html, body {
height: 100%;
@ -309,8 +398,8 @@
}
body {
display: grid;
background: #fff;
color: #475569;
background: var(--background-color);
color: var(--main-text-color);
text-align: center;
font-family: roboto, karma, segoe ui, sans-serif;
font-size: 20px;
@ -321,6 +410,86 @@
padding: 10px;
margin: 10px;
}
div.collapsed {
display: none;
}
div.expanded {
display: block;
}
div.references {
padding-top: 8px;
}
div.reference {
display: grid;
grid-template-rows: auto;
grid-auto-flow: row;
grid-column-gap: 10px;
grid-row-gap: 10px;
margin: 10px;
}
div.expanded.reference-section {
display: grid;
grid-template-rows: auto;
grid-auto-flow: row;
grid-column-gap: 10px;
grid-row-gap: 10px;
margin: 10px;
}
button.reference-button {
background: var(--background-color);
color: var(--main-text-color);
border: 1px solid var(--main-text-color);
border-radius: 5px;
padding: 5px;
font-size: 14px;
font-weight: 300;
line-height: 1.5em;
cursor: pointer;
transition: background 0.2s ease-in-out;
text-align: left;
max-height: 75px;
transition: max-height 0.3s ease-in-out;
overflow: hidden;
}
button.reference-button.expanded {
max-height: none;
}
button.reference-button::before {
content: "▶";
margin-right: 5px;
display: inline-block;
transition: transform 0.3s ease-in-out;
}
button.reference-button:active:before,
button.reference-button[aria-expanded="true"]::before {
transform: rotate(90deg);
}
button.reference-expand-button {
background: var(--background-color);
color: var(--main-text-color);
border: 1px dotted var(--main-text-color);
border-radius: 5px;
padding: 5px;
font-size: 14px;
font-weight: 300;
line-height: 1.5em;
cursor: pointer;
transition: background 0.4s ease-in-out;
text-align: left;
}
button.reference-expand-button:hover {
background: var(--primary-hover);
}
#chat-body {
font-size: medium;
margin: 0px;
@ -332,7 +501,7 @@
content: attr(data-meta);
display: block;
font-size: x-small;
color: #475569;
color: var(--main-text-color);
margin: -8px 4px 0 -5px;
}
/* move message by khoj to left */
@ -402,7 +571,7 @@
top: 91%;
right: -2px;
border: 10px solid transparent;
border-left-color: #475569;
border-left-color: var(--main-text-color);
border-right: 0;
margin-top: -10px;
transform: rotate(-60deg)
@ -418,7 +587,7 @@
#chat-footer > * {
padding: 15px;
border-radius: 5px;
border: 1px solid #475569;
border: 1px solid var(--main-text-color);
background: #f9fafc
}
.option:hover {
@ -451,9 +620,9 @@
}
a.inline-chat-link {
color: #475569;
color: var(--main-text-color);
text-decoration: none;
border-bottom: 1px dotted #475569;
border-bottom: 1px dotted var(--main-text-color);
}
@media (pointer: coarse), (hover: none) {
@ -479,7 +648,7 @@
padding: 2px 4px;
}
}
@media only screen and (max-width: 600px) {
@media only screen and (max-width: 700px) {
body {
grid-template-columns: 1fr;
grid-template-rows: auto auto minmax(80px, 100%) auto;
@ -492,14 +661,8 @@
margin: 4px;
grid-template-columns: auto;
}
a.khoj-banner {
display: block;
}
p.khoj-banner {
padding: 0;
}
}
@media only screen and (min-width: 600px) {
@media only screen and (min-width: 700px) {
body {
grid-template-columns: auto min(70vw, 100%) auto;
grid-template-rows: auto auto minmax(80px, 100%) auto;
@ -509,14 +672,6 @@
}
}
div.khoj-banner-container {
background: linear-gradient(-45deg, #FFC107, #FF9800, #FF5722, #FF9800, #FFC107);
background-size: 400% 400%;
animation: gradient 15s ease infinite;
text-align: center;
padding: 10px;
}
div#chat-tooltip {
text-align: left;
font-size: medium;
@ -538,19 +693,7 @@
text-align: center;
}
button#khoj-banner-submit,
input#khoj-banner-email {
padding: 10px;
border-radius: 5px;
border: 1px solid #475569;
background: #f9fafc;
}
button#khoj-banner-submit:hover,
input#khoj-banner-email:hover {
box-shadow: 0 0 11px #aaa;
}
div.khoj-banner-container-hidden {
div.khoj-empty-container {
margin: 0px;
padding: 0px;
}
@ -570,39 +713,4 @@
white-space: pre-wrap;
}
</style>
<script>
if ("{{demo}}" === "False") {
document.getElementById("khoj-banner-container").classList.remove("khoj-banner-container");
document.getElementById("khoj-banner-container").classList.add("khoj-banner-container-hidden");
}
var khojBannerSubmit = document.getElementById("khoj-banner-submit");
khojBannerSubmit?.addEventListener("click", function(event) {
event.preventDefault();
var email = document.getElementById("khoj-banner-email").value;
fetch("https://app.khoj.dev/beta/users/", {
method: "POST",
body: JSON.stringify({
email: email
}),
headers: {
"Content-Type": "application/json"
}
}).then(function(response) {
return response.json();
}).then(function(data) {
console.log(data);
if (data.user != null) {
document.getElementById("khoj-banner").innerHTML = "Thanks for signing up. We'll be in touch soon! 🚀";
document.getElementById("khoj-banner-submit").remove();
} else {
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
}
}).catch(function(error) {
console.log(error);
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
});
});
</script>
</html>

669
src/khoj/interface/web/config.html
View file

@ -3,217 +3,118 @@
<div class="page">
<div class="section">
<h2 class="section-title">Plugins</h2>
<h2 class="section-title">Content</h2>
<div class="section-cards">
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/computer.png" alt="Computer">
<h3 id="card-title-computer" class="card-title">
Files
<img id="configured-icon-computer"
style="display: {% if not current_model_state.computer %}none{% endif %}"
class="configured-icon"
src="/static/assets/icons/confirm-icon.svg"
alt="Configured">
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Manage files from your computer</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content-source/computer">
{% if current_model_state.computer %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
<div id="clear-computer" class="card-action-row"
style="display: {% if not current_model_state.computer %}none{% endif %}">
<button class="card-button" onclick="clearContentType('computer')">
Disable
</button>
</div>
</div>
</div>
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/github.svg" alt="Github">
<h3 class="card-title">
Github
{% if current_config.content_type.github %}
{% if current_model_state.github == False %}
<img id="misconfigured-icon-github" class="configured-icon" src="/static/assets/icons/question-mark-icon.svg" alt="Not Configured" title="Embeddings have not been generated yet for this content type. Either the configuration is invalid, or you just need to click Configure.">
{% else %}
<img id="configured-icon-github" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
{% endif %}
<img id="configured-icon-github"
class="configured-icon"
src="/static/assets/icons/confirm-icon.svg"
alt="Configured"
style="display: {% if not current_model_state.github %}none{% endif %}">
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Set repositories to index</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content_type/github">
{% if current_config.content_type.github %}
<a class="card-button" href="/config/content-source/github">
{% if current_model_state.github %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.content_type.github %}
<div id="clear-github" class="card-action-row">
<div id="clear-github"
class="card-action-row"
style="display: {% if not current_model_state.github %}none{% endif %}">
<button class="card-button" onclick="clearContentType('github')">
Disable
</button>
</div>
{% endif %}
</div>
</div>
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/notion.svg" alt="Notion">
<h3 class="card-title">
Notion
{% if current_config.content_type.notion %}
{% if current_model_state.notion == False %}
<img id="misconfigured-icon-notion" class="configured-icon" src="/static/assets/icons/question-mark-icon.svg" alt="Not Configured" title="Embeddings have not been generated yet for this content type. Either the configuration is invalid, or you just need to click Configure.">
{% else %}
<img id="configured-icon-notion" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
{% endif %}
<img id="configured-icon-notion"
class="configured-icon"
src="/static/assets/icons/confirm-icon.svg"
alt="Configured"
style="display: {% if not current_model_state.notion %}none{% endif %}">
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Configure your settings from Notion</p>
<p class="card-description">Sync your Notion pages</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content_type/notion">
{% if current_config.content_type.content %}
<a class="card-button" href="/config/content-source/notion">
{% if current_model_state.notion %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.content_type.notion %}
<div id="clear-notion" class="card-action-row">
<div id="clear-notion"
class="card-action-row"
style="display: {% if not current_model_state.notion %}none{% endif %}">
<button class="card-button" onclick="clearContentType('notion')">
Disable
</button>
</div>
{% endif %}
</div>
</div>
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/markdown.svg" alt="markdown">
<h3 class="card-title">
Markdown
{% if current_config.content_type.markdown %}
{% if current_model_state.markdown == False%}
<img id="misconfigured-icon-markdown" class="configured-icon" src="/static/assets/icons/question-mark-icon.svg" alt="Not Configured" title="Embeddings have not been generated yet for this content type. Either the configuration is invalid, or you just need to click Configure.">
{% else %}
<img id="configured-icon-markdown" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
{% endif %}
</h3>
</div>
<div class="general-settings section">
<div id="status" style="display: none;"></div>
</div>
<div class="section finalize-actions general-settings">
<div class="section-cards">
<div class="finalize-buttons">
<button id="configure" type="submit" title="Update index with the latest changes">💾 Save All</button>
</div>
<div class="card-description-row">
<p class="card-description">Set markdown files to index</p>
<div class="finalize-buttons">
<button id="reinitialize" type="submit" title="Regenerate index from scratch">🔄 Reinitialize</button>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content_type/markdown">
{% if current_config.content_type.markdown %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.content_type.markdown %}
<div id="clear-markdown" class="card-action-row">
<button class="card-button" onclick="clearContentType('markdown')">
Disable
</button>
</div>
{% endif %}
</div>
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/org.svg" alt="org">
<h3 class="card-title">
Org
{% if current_config.content_type.org %}
{% if current_model_state.org == False %}
<img id="misconfigured-icon-org" class="configured-icon" src="/static/assets/icons/question-mark-icon.svg" alt="Not Configured" title="Embeddings have not been generated yet for this content type. Either the configuration is invalid, or you just need to click Configure.">
{% else %}
<img id="configured-icon-org" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
{% endif %}
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Set org files to index</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content_type/org">
{% if current_config.content_type.org %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.content_type.org %}
<div id="clear-org" class="card-action-row">
<button class="card-button" onclick="clearContentType('org')">
Disable
</button>
</div>
{% endif %}
</div>
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/pdf.svg" alt="PDF">
<h3 class="card-title">
PDF
{% if current_config.content_type.pdf %}
{% if current_model_state.pdf == False %}
<img id="misconfigured-icon-pdf" class="configured-icon" src="/static/assets/icons/question-mark-icon.svg" alt="Not Configured" title="Embeddings have not been generated yet for this content type. Either the configuration is invalid, or you need to click Configure.">
{% else %}
<img id="configured-icon-pdf" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
{% endif %}
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Set PDF files to index</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content_type/pdf">
{% if current_config.content_type.pdf %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.content_type.pdf %}
<div id="clear-pdf" class="card-action-row">
<button class="card-button" onclick="clearContentType('pdf')">
Disable
</button>
</div>
{% endif %}
</div>
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/plaintext.svg" alt="Plaintext">
<h3 class="card-title">
Plaintext
{% if current_config.content_type.plaintext %}
{% if current_model_state.plaintext == False %}
<img id="misconfigured-icon-plaintext" class="configured-icon" src="/static/assets/icons/question-mark-icon.svg" alt="Not Configured" title="Embeddings have not been generated yet for this content type. Either the configuration is invalid, or you need to click Configure.">
{% else %}
<img id="configured-icon-plaintext" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
{% endif %}
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Set Plaintext files to index</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content_type/plaintext">
{% if current_config.content_type.plaintext %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.content_type.plaintext %}
<div id="clear-plaintext" class="card-action-row">
<button class="card-button" onclick="clearContentType('plaintext')">
Disable
</button>
</div>
{% endif %}
</div>
</div>
</div>
@ -223,214 +124,213 @@
<div class="section-cards">
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/openai-logomark.svg" alt="Chat">
<img class="card-icon" src="/static/assets/icons/chat.svg" alt="Chat">
<h3 class="card-title">
Chat
{% if current_config.processor and current_config.processor.conversation.openai %}
{% if current_model_state.conversation_openai == False %}
<img id="misconfigured-icon-conversation-processor" class="configured-icon" src="/static/assets/icons/question-mark-icon.svg" alt="Not Configured" title="The OpenAI configuration did not work as expected.">
{% else %}
<img id="configured-icon-conversation-processor" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
{% endif %}
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Setup online chat using OpenAI</p>
<select id="chat-models">
{% for option in conversation_options %}
<option value="{{ option.id }}" {% if option.id == selected_conversation_config %}selected{% endif %}>{{ option.chat_model }}</option>
{% endfor %}
</select>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/processor/conversation/openai">
{% if current_config.processor and current_config.processor.conversation.openai %}
Update
{% else %}
Setup
{% endif %}
<button id="save-model" class="card-button happy" onclick="updateChatModel()">
Save
</button>
</div>
</div>
</div>
</div>
<div class="section">
<h2 class="section-title">Clients</h2>
<div class="api-settings">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/key.svg" alt="API Key">
<h3 class="card-title">API Keys</h3>
</div>
<div class="card-description-row">
<p id="api-settings-card-description" class="card-description">Manage access from your client apps to Khoj</p>
</div>
<table id="api-settings-keys-table">
<thead>
<tr>
<th scope="col">Name</th>
<th scope="col">Key</th>
<th scope="col">Actions</th>
</tr>
</thead>
<tbody id="api-key-list"></tbody>
</table>
<div class="card-action-row">
<button class="card-button happy" id="generate-api-key" onclick="generateAPIKey()">
Generate API Key
</button>
</div>
</div>
</div>
{% if billing_enabled %}
<div class="section">
<h2 class="section-title">Billing</h2>
<div class="section-cards">
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/credit-card.png" alt="Credit Card">
<h3 class="card-title">
<span>Subscription</span>
<img id="configured-icon-subscription"
style="display: {% if subscription_state == 'trial' or subscription_state == 'expired' %}none{% endif %}"
class="configured-icon"
src="/static/assets/icons/confirm-icon.svg"
alt="Configured">
</h3>
</div>
<div class="card-description-row">
<p id="trial-description"
class="card-description"
style="display: {% if subscription_state != 'trial' %}none{% endif %}">
Subscribe to Khoj Cloud
</p>
<p id="unsubscribe-description"
class="card-description"
style="display: {% if subscription_state != 'subscribed' %}none{% endif %}">
You are <b>subscribed</b> to Khoj Cloud. Subscription will <b>renew</b> on <b>{{ subscription_renewal_date }}</b>
</p>
<p id="resubscribe-description"
class="card-description"
style="display: {% if subscription_state != 'unsubscribed' %}none{% endif %}">
You are <b>subscribed</b> to Khoj Cloud. Subscription will <b>expire</b> on <b>{{ subscription_renewal_date }}</b>
</p>
<p id="expire-description"
class="card-description"
style="display: {% if subscription_state != 'expired' %}none{% endif %}">
Subscribe to Khoj Cloud. Subscription <b>expired</b> on <b>{{ subscription_renewal_date }}</b>
</p>
</div>
<div class="card-action-row">
<button id="unsubscribe-button"
class="card-button"
onclick="unsubscribe()"
style="display: {% if subscription_state != 'subscribed' %}none{% endif %};">
Unsubscribe
</button>
<button id="resubscribe-button"
class="card-button happy"
onclick="resubscribe()"
style="display: {% if subscription_state != 'unsubscribed' %}none{% endif %};">
Resubscribe
</button>
<a id="subscribe-button"
class="card-button happy"
href="{{ khoj_cloud_subscription_url }}?prefilled_email={{ username }}"
style="display: {% if subscription_state == 'subscribed' or subscription_state == 'unsubscribed' %}none{% endif %};">
Subscribe
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.processor and current_config.processor.conversation.openai %}
<div id="clear-conversation" class="card-action-row">
<button class="card-button" onclick="clearConversationProcessor()">
Disable
</button>
</div>
{% endif %}
</div>
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/chat.svg" alt="Chat">
<h3 class="card-title">
Offline Chat
<img id="configured-icon-conversation-enable-offline-chat" class="configured-icon {% if current_config.processor and current_config.processor.conversation and current_config.processor.conversation.offline_chat.enable_offline_chat and current_model_state.conversation_gpt4all %}enabled{% else %}disabled{% endif %}" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% if current_config.processor and current_config.processor.conversation and current_config.processor.conversation.offline_chat.enable_offline_chat and not current_model_state.conversation_gpt4all %}
<img id="misconfigured-icon-conversation-enable-offline-chat" class="configured-icon" src="/static/assets/icons/question-mark-icon.svg" alt="Not Configured" title="The model was not downloaded as expected.">
{% endif %}
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Setup offline chat</p>
</div>
<div id="clear-enable-offline-chat" class="card-action-row {% if current_config.processor and current_config.processor.conversation and current_config.processor.conversation.offline_chat.enable_offline_chat %}enabled{% else %}disabled{% endif %}">
<button class="card-button" onclick="toggleEnableLocalLLLM(false)">
Disable
</button>
</div>
<div id="set-enable-offline-chat" class="card-action-row {% if current_config.processor and current_config.processor.conversation and current_config.processor.conversation.offline_chat.enable_offline_chat %}disabled{% else %}enabled{% endif %}">
<button class="card-button happy" onclick="toggleEnableLocalLLLM(true)">
Enable
</button>
</div>
<div id="toggle-enable-offline-chat" class="card-action-row disabled">
<div class="loader"></div>
</div>
</div>
</div>
</div>
<div class="section general-settings">
<div id="results-count" title="Number of items to show in search and use for chat response">
<label for="results-count-slider">Results Count: <span id="results-count-value">5</span></label>
<input type="range" id="results-count-slider" name="results-count-slider" min="1" max="10" step="1" value="5">
</div>
<div id="status" style="display: none;"></div>
</div>
<div class="section finalize-actions general-settings">
<div class="section-cards">
<div class="finalize-buttons">
<button id="configure" type="submit" title="Update index with the latest changes">⚙️ Configure</button>
</div>
<div class="finalize-buttons">
<button id="reinitialize" type="submit" title="Regenerate index from scratch">🔄 Reinitialize</button>
</div>
</div>
</div>
{% endif %}
<div class="section"></div>
</div>
<script>
function clearContentType(content_type) {
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/delete/config/data/content_type/' + content_type, {
function updateChatModel() {
const chatModel = document.getElementById("chat-models").value;
const saveModelButton = document.getElementById("save-model");
saveModelButton.disabled = true;
saveModelButton.innerHTML = "Saving...";
fetch('/api/config/data/conversation/model?id=' + chatModel, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken
}
})
.then(response => response.json())
.then(data => {
if (data.status == "ok") {
var contentTypeClearButton = document.getElementById("clear-" + content_type);
contentTypeClearButton.style.display = "none";
var configuredIcon = document.getElementById("configured-icon-" + content_type);
if (configuredIcon) {
configuredIcon.style.display = "none";
}
var misconfiguredIcon = document.getElementById("misconfigured-icon-" + content_type);
if (misconfiguredIcon) {
misconfiguredIcon.style.display = "none";
}
saveModelButton.innerHTML = "Save";
saveModelButton.disabled = false;
} else {
saveModelButton.innerHTML = "Error";
saveModelButton.disabled = false;
}
})
};
function toggleEnableLocalLLLM(enable) {
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
var toggleEnableLocalLLLMButton = document.getElementById("toggle-enable-offline-chat");
var featuresHintText = document.getElementById("features-hint-text");
toggleEnableLocalLLLMButton.classList.remove("disabled");
toggleEnableLocalLLLMButton.classList.add("enabled");
function clearContentType(content_source) {
if (enable) {
featuresHintText.style.display = "block";
featuresHintText.innerHTML = "An open source model is being downloaded in the background. Hang tight, this may take a few minutes ⏳.";
featuresHintText.classList.add("show");
}
fetch('/api/config/data/processor/conversation/offline_chat' + '?enable_offline_chat=' + enable, {
method: 'POST',
fetch('/api/config/data/content-source/' + content_source, {
method: 'DELETE',
headers: {
'Content-Type': 'application/json',
}
})
.then(response => response.json())
.then(data => {
if (data.status == "ok") {
document.getElementById("configured-icon-" + content_source).style.display = "none";
document.getElementById("clear-" + content_source).style.display = "none";
} else {
document.getElementById("configured-icon-" + content_source).style.display = "";
document.getElementById("clear-" + content_source).style.display = "";
}
})
};
function unsubscribe() {
fetch('/api/subscription?operation=cancel&email={{username}}', {
method: 'PATCH',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken
},
})
.then(response => response.json())
.then(data => {
if (data.status == "ok") {
// Toggle the Enabled/Disabled UI based on the action/response.
var enableLocalLLLMButton = document.getElementById("set-enable-offline-chat");
var disableLocalLLLMButton = document.getElementById("clear-enable-offline-chat");
var configuredIcon = document.getElementById("configured-icon-conversation-enable-offline-chat");
var toggleEnableLocalLLLMButton = document.getElementById("toggle-enable-offline-chat");
if (data.success) {
document.getElementById("unsubscribe-description").style.display = "none";
document.getElementById("unsubscribe-button").style.display = "none";
toggleEnableLocalLLLMButton.classList.remove("enabled");
toggleEnableLocalLLLMButton.classList.add("disabled");
document.getElementById("resubscribe-description").style.display = "";
document.getElementById("resubscribe-button").style.display = "";
if (enable) {
enableLocalLLLMButton.classList.add("disabled");
enableLocalLLLMButton.classList.remove("enabled");
configuredIcon.classList.add("enabled");
configuredIcon.classList.remove("disabled");
disableLocalLLLMButton.classList.remove("disabled");
disableLocalLLLMButton.classList.add("enabled");
} else {
enableLocalLLLMButton.classList.remove("disabled");
enableLocalLLLMButton.classList.add("enabled");
configuredIcon.classList.remove("enabled");
configuredIcon.classList.add("disabled");
disableLocalLLLMButton.classList.add("disabled");
disableLocalLLLMButton.classList.remove("enabled");
}
featuresHintText.classList.remove("show");
featuresHintText.innerHTML = "";
}
}
})
}
function clearConversationProcessor() {
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/delete/config/data/processor/conversation/openai', {
method: 'POST',
function resubscribe() {
fetch('/api/subscription?operation=resubscribe&email={{username}}', {
method: 'PATCH',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken
}
},
})
.then(response => response.json())
.then(data => {
if (data.status == "ok") {
var conversationClearButton = document.getElementById("clear-conversation");
conversationClearButton.style.display = "none";
if (data.success) {
document.getElementById("resubscribe-description").style.display = "none";
document.getElementById("resubscribe-button").style.display = "none";
var configuredIcon = document.getElementById("configured-icon-conversation-processor");
if (configuredIcon) {
configuredIcon.style.display = "none";
}
var misconfiguredIcon = document.getElementById("misconfigured-icon-conversation-processor");
if (misconfiguredIcon) {
misconfiguredIcon.style.display = "none";
}
document.getElementById("unsubscribe-description").style.display = "";
document.getElementById("unsubscribe-button").style.display = "";
}
})
};
}
var configure = document.getElementById("configure");
configure.addEventListener("click", function(event) {
event.preventDefault();
updateIndex(
force=false,
successText="Configured successfully!",
successText="Saved!",
errorText="Unable to configure. Raise issue on Khoj <a href='https://github.com/khoj-ai/khoj/issues'>Github</a> or <a href='https://discord.gg/BDgyabRM6e'>Discord</a>.",
button=configure,
loadingText="Configuring...",
emoji="⚙️");
loadingText="Saving...",
emoji="💾");
});
var reinitialize = document.getElementById("reinitialize");
@ -438,7 +338,7 @@
event.preventDefault();
updateIndex(
force=true,
successText="Reinitialized successfully!",
successText="Reinitialized!",
errorText="Unable to reinitialize. Raise issue on Khoj <a href='https://github.com/khoj-ai/khoj/issues'>Github</a> or <a href='https://discord.gg/BDgyabRM6e'>Discord</a>.",
button=reinitialize,
loadingText="Reinitializing...",
@ -447,6 +347,7 @@
function updateIndex(force, successText, errorText, button, loadingText, emoji) {
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
const original_html = button.innerHTML;
button.disabled = true;
button.innerHTML = emoji + " " + loadingText;
fetch('/api/update?&client=web&force=' + force, {
@ -458,14 +359,17 @@
})
.then(response => response.json())
.then(data => {
console.log('Success:', data);
if (data.detail != null) {
throw new Error(data.detail);
}
document.getElementById("status").innerHTML = emoji + " " + successText;
document.getElementById("status").style.display = "block";
document.getElementById("status").style.display = "none";
button.disabled = false;
button.innerHTML = '✅ Done!';
button.innerHTML = `✅ ${successText}`;
setTimeout(function() {
button.innerHTML = original_html;
}, 2000);
})
.catch((error) => {
console.error('Error:', error);
@ -473,27 +377,116 @@
document.getElementById("status").style.display = "block";
button.disabled = false;
button.innerHTML = '⚠️ Unsuccessful';
setTimeout(function() {
button.innerHTML = original_html;
}, 2000);
});
content_sources = ["computer", "github", "notion"];
content_sources.forEach(content_source => {
fetch(`/api/config/data/${content_source}`, {
method: 'GET',
headers: {
'Content-Type': 'application/json',
}
})
.then(response => response.json())
.then(data => {
if (data.length > 0) {
document.getElementById("configured-icon-" + content_source).style.display = "";
document.getElementById("clear-" + content_source).style.display = "";
} else {
document.getElementById("configured-icon-" + content_source).style.display = "none";
document.getElementById("clear-" + content_source).style.display = "none";
}
});
});
}
// Setup the results count slider
const resultsCountSlider = document.getElementById('results-count-slider');
const resultsCountValue = document.getElementById('results-count-value');
function generateAPIKey() {
const apiKeyList = document.getElementById("api-key-list");
fetch('/auth/token', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
})
.then(response => response.json())
.then(tokenObj => {
apiKeyList.innerHTML += generateTokenRow(tokenObj);
});
}
// Set the initial value of the slider
resultsCountValue.textContent = resultsCountSlider.value;
function copyAPIKey(token) {
// Copy API key to clipboard
navigator.clipboard.writeText(token);
// Flash the API key copied message
const copyApiKeyButton = document.getElementById(`api-key-${token}`);
original_html = copyApiKeyButton.innerHTML
setTimeout(function() {
copyApiKeyButton.innerHTML = "✅ Copied!";
setTimeout(function() {
copyApiKeyButton.innerHTML = original_html;
}, 1000);
}, 100);
}
// Store the slider value in localStorage when it changes
resultsCountSlider.addEventListener('input', () => {
resultsCountValue.textContent = resultsCountSlider.value;
localStorage.setItem('khojResultsCount', resultsCountSlider.value);
});
function deleteAPIKey(token) {
const apiKeyList = document.getElementById("api-key-list");
fetch(`/auth/token?token=${token}`, {
method: 'DELETE',
})
.then(response => {
if (response.ok) {
const apiKeyItem = document.getElementById(`api-key-item-${token}`);
apiKeyList.removeChild(apiKeyItem);
}
});
}
// Get the slider value from localStorage on page load
const storedResultsCount = localStorage.getItem('khojResultsCount');
if (storedResultsCount) {
resultsCountSlider.value = storedResultsCount;
resultsCountValue.textContent = storedResultsCount;
function generateTokenRow(tokenObj) {
let token = tokenObj.token;
let tokenName = tokenObj.name;
let truncatedToken = token.slice(0, 4) + "..." + token.slice(-4);
let tokenId = `${tokenName}-${truncatedToken}`;
return `
<tr id="api-key-item-${token}">
<td><b>${tokenName}</b></td>
<td id="api-key-${token}">${truncatedToken}</td>
<td>
<img onclick="copyAPIKey('${token}')" class="configured-icon api-key-action enabled" src="/static/assets/icons/copy-solid.svg" alt="Copy API Key" title="Copy API Key">
<img onclick="deleteAPIKey('${token}')" class="configured-icon api-key-action enabled" src="/static/assets/icons/trash-solid.svg" alt="Delete API Key" title="Delete API Key">
</td>
</tr>
`;
}
function listApiKeys() {
const apiKeyList = document.getElementById("api-key-list");
fetch('/auth/token')
.then(response => response.json())
.then(tokens => {
apiKeyList.innerHTML = tokens.map(generateTokenRow).join("");
});
}
// List user's API keys on page load
listApiKeys();
function removeFile(path) {
fetch('/api/config/data/file?filename=' + path, {
method: 'DELETE',
headers: {
'Content-Type': 'application/json',
}
})
.then(response => response.json())
.then(data => {
if (data.status == "ok") {
getAllFilenames();
}
})
}
</script>
{% endblock %}

130
src/khoj/interface/web/content_source_computer_input.html Normal file
View file

@ -0,0 +1,130 @@
{% extends "base_config.html" %}
{% block content %}
<div class="page">
<div class="section">
<h2 class="section-title">
<img class="card-icon" src="/static/assets/icons/computer.png" alt="files">
<span class="card-title-text">Files</span>
<div class="instructions">
<p class="card-description">Manage files from your computer</p>
<p id="get-desktop-client" class="card-description">Download the <a href="https://khoj.dev/downloads">Khoj Desktop app</a> to sync documents from your computer</p>
</div>
</h2>
<div class="section-manage-files">
<div id="delete-all-files" class="delete-all-files">
<button id="delete-all-files" type="submit" title="Remove all computer files from Khoj">🗑️ Delete all</button>
</div>
<div class="indexed-files">
</div>
</div>
</div>
</div>
<style>
#desktop-client {
font-weight: normal;
}
.indexed-files {
width: 100%;
}
.content-name {
font-size: smaller;
}
</style>
<script>
function removeFile(path) {
fetch('/api/config/data/file?filename=' + path, {
method: 'DELETE',
headers: {
'Content-Type': 'application/json',
}
})
.then(response => response.ok ? response.json() : Promise.reject(response))
.then(data => {
if (data.status == "ok") {
getAllComputerFilenames();
}
})
}
// Get all currently indexed files
function getAllComputerFilenames() {
fetch('/api/config/data/computer')
.then(response => response.json())
.then(data => {
var indexedFiles = document.getElementsByClassName("indexed-files")[0];
indexedFiles.innerHTML = "";
if (data.length == 0) {
document.getElementById("delete-all-files").style.display = "none";
indexedFiles.innerHTML = "<div class='card-description'>No documents synced with Khoj</div>";
} else {
document.getElementById("get-desktop-client").style.display = "none";
document.getElementById("delete-all-files").style.display = "block";
}
for (var filename of data) {
let fileElement = document.createElement("div");
fileElement.classList.add("file-element");
let fileExtension = filename.split('.').pop();
if (fileExtension === "org")
image_name = "org.svg"
else if (fileExtension === "pdf")
image_name = "pdf.svg"
else if (fileExtension === "markdown" || fileExtension === "md")
image_name = "markdown.svg"
else
image_name = "plaintext.svg"
let fileIconElement = document.createElement("img");
fileIconElement.classList.add("card-icon");
fileIconElement.src = `/static/assets/icons/${image_name}`;
fileIconElement.alt = "File";
fileElement.appendChild(fileIconElement);
let fileNameElement = document.createElement("div");
fileNameElement.classList.add("content-name");
fileNameElement.innerHTML = filename;
fileElement.appendChild(fileNameElement);
let buttonContainer = document.createElement("div");
buttonContainer.classList.add("remove-button-container");
let removeFileButton = document.createElement("button");
removeFileButton.classList.add("remove-file-button");
removeFileButton.innerHTML = "🗑️";
removeFileButton.addEventListener("click", ((filename) => {
return () => {
removeFile(filename);
};
})(filename));
buttonContainer.appendChild(removeFileButton);
fileElement.appendChild(buttonContainer);
indexedFiles.appendChild(fileElement);
}
})
.catch((error) => {
console.error('Error:', error);
});
}
// Get all currently indexed files on page load
getAllComputerFilenames();
let deleteAllComputerFilesButton = document.getElementById("delete-all-files");
deleteAllComputerFilesButton.addEventListener("click", function(event) {
event.preventDefault();
fetch('/api/config/data/content-source/computer', {
method: 'DELETE',
headers: {
'Content-Type': 'application/json',
}
})
.then(response => response.json())
.then(data => {
if (data.status == "ok") {
getAllComputerFilenames();
}
})
});
</script>
{% endblock %}

71
src/khoj/interface/web/content_type_github_input.html → src/khoj/interface/web/content_source_github_input.html
View file

@ -38,24 +38,6 @@
{% endfor %}
</div>
<button type="button" id="add-repository-button">Add Repository</button>
<table style="display: none;" >
<tr>
<td>
<label for="compressed-jsonl">Compressed JSONL (Output)</label>
</td>
<td>
<input type="text" id="compressed-jsonl" name="compressed-jsonl" value="{{ current_config['compressed_jsonl'] }}">
</td>
</tr>
<tr>
<td>
<label for="embeddings-file">Embeddings File (Output)</label>
</td>
<td>
<input type="text" id="embeddings-file" name="embeddings-file" value="{{ current_config['embeddings_file'] }}">
</td>
</tr>
</table>
<div class="section">
<div id="success" style="display: none;"></div>
<button id="submit" type="submit">Save</button>
@ -64,6 +46,9 @@
</div>
</div>
<style>
td {
padding: 10px 0;
}
div.repo {
width: 100%;
height: 100%;
@ -107,8 +92,6 @@
submit.addEventListener("click", function(event) {
event.preventDefault();
const compressed_jsonl = document.getElementById("compressed-jsonl").value;
const embeddings_file = document.getElementById("embeddings-file").value;
const pat_token = document.getElementById("pat-token").value;
if (pat_token == "") {
@ -144,8 +127,13 @@
return;
}
const submitButton = document.getElementById("submit");
submitButton.disabled = true;
submitButton.innerHTML = "Saving...";
// Save Github config on server
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/config/data/content_type/github', {
fetch('/api/config/data/content-source/github', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
@ -154,20 +142,43 @@
body: JSON.stringify({
"pat_token": pat_token,
"repos": repos,
"compressed_jsonl": compressed_jsonl,
"embeddings_file": embeddings_file,
})
})
.then(response => response.json())
.then(data => { data["status"] === "ok" ? data : Promise.reject(data) })
.catch(error => {
document.getElementById("success").innerHTML = "⚠️ Failed to save Github settings.";
document.getElementById("success").style.display = "block";
submitButton.innerHTML = "⚠️ Failed to save settings";
setTimeout(function() {
submitButton.innerHTML = "Save";
submitButton.disabled = false;
}, 2000);
return;
});
// Index Github content on server
fetch('/api/update?t=github')
.then(response => response.json())
.then(data => { data["status"] == "ok" ? data : Promise.reject(data) })
.then(data => {
if (data["status"] == "ok") {
document.getElementById("success").innerHTML = "✅ Successfully updated. Go to your <a href='/config'>settings page</a> to complete setup.";
document.getElementById("success").style.display = "block";
} else {
document.getElementById("success").innerHTML = "⚠️ Failed to update settings.";
document.getElementById("success").style.display = "block";
}
document.getElementById("success").style.display = "none";
submitButton.innerHTML = "✅ Successfully updated";
setTimeout(function() {
submitButton.innerHTML = "Save";
submitButton.disabled = false;
}, 2000);
})
.catch(error => {
document.getElementById("success").innerHTML = "⚠️ Failed to save Github content.";
document.getElementById("success").style.display = "block";
submitButton.innerHTML = "⚠️ Failed to save content";
setTimeout(function() {
submitButton.innerHTML = "Save";
submitButton.disabled = false;
}, 2000);
});
});
</script>
{% endblock %}

67
src/khoj/interface/web/content_type_notion_input.html → src/khoj/interface/web/content_source_notion_input.html
View file

@ -20,24 +20,6 @@
</td>
</tr>
</table>
<table style="display: none;" >
<tr>
<td>
<label for="compressed-jsonl">Compressed JSONL (Output)</label>
</td>
<td>
<input type="text" id="compressed-jsonl" name="compressed-jsonl" value="{{ current_config['compressed_jsonl'] }}">
</td>
</tr>
<tr>
<td>
<label for="embeddings-file">Embeddings File (Output)</label>
</td>
<td>
<input type="text" id="embeddings-file" name="embeddings-file" value="{{ current_config['embeddings_file'] }}">
</td>
</tr>
</table>
<div class="section">
<div id="success" style="display: none;"></div>
<button id="submit" type="submit">Save</button>
@ -51,8 +33,6 @@
submit.addEventListener("click", function(event) {
event.preventDefault();
const compressed_jsonl = document.getElementById("compressed-jsonl").value;
const embeddings_file = document.getElementById("embeddings-file").value;
const token = document.getElementById("token").value;
if (token == "") {
@ -61,8 +41,13 @@
return;
}
const submitButton = document.getElementById("submit");
submitButton.disabled = true;
submitButton.innerHTML = "Saving...";
// Save Notion config on server
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/config/data/content_type/notion', {
fetch('/api/config/data/content-source/notion', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
@ -70,20 +55,42 @@
},
body: JSON.stringify({
"token": token,
"compressed_jsonl": compressed_jsonl,
"embeddings_file": embeddings_file,
})
})
.then(response => response.json())
.then(data => { data["status"] === "ok" ? data : Promise.reject(data) })
.catch(error => {
document.getElementById("success").innerHTML = "⚠️ Failed to save Notion settings.";
document.getElementById("success").style.display = "block";
submitButton.innerHTML = "⚠️ Failed to save settings";
setTimeout(function() {
submitButton.innerHTML = "Save";
submitButton.disabled = false;
}, 2000);
return;
});
// Index Notion content on server
fetch('/api/update?t=notion')
.then(response => response.json())
.then(data => { data["status"] == "ok" ? data : Promise.reject(data) })
.then(data => {
if (data["status"] == "ok") {
document.getElementById("success").innerHTML = "✅ Successfully updated. Go to your <a href='/config'>settings page</a> to complete setup.";
document.getElementById("success").style.display = "block";
} else {
document.getElementById("success").innerHTML = "⚠️ Failed to update settings.";
document.getElementById("success").style.display = "block";
}
document.getElementById("success").style.display = "none";
submitButton.innerHTML = "✅ Successfully updated";
setTimeout(function() {
submitButton.innerHTML = "Save";
submitButton.disabled = false;
}, 2000);
})
.catch(error => {
document.getElementById("success").innerHTML = "⚠️ Failed to save Notion content.";
document.getElementById("success").style.display = "block";
submitButton.innerHTML = "⚠️ Failed to save content";
setTimeout(function() {
submitButton.innerHTML = "Save";
submitButton.disabled = false;
}, 2000);
});
});
</script>
{% endblock %}

189
src/khoj/interface/web/content_type_input.html
View file

@ -1,189 +0,0 @@
{% extends "base_config.html" %}
{% block content %}
<div class="page">
<div class="section">
<h2 class="section-title">
<img class="card-icon" src="/static/assets/icons/{{ content_type }}.svg" alt="{{ content_type|capitalize }}">
<span class="card-title-text">{{ content_type|capitalize }}</span>
</h2>
<form id="config-form">
<table>
<tr>
<td>
<label for="input-files" title="Add a {{content_type}} file for Khoj to index">Files</label>
</td>
<td id="input-files-cell">
{% if current_config['input_files'] is none %}
<input type="text" id="input-files" name="input-files" placeholder="~\Documents\notes.{{content_type}}">
{% else %}
{% for input_file in current_config['input_files'] %}
<input type="text" id="input-files" name="input-files" value="{{ input_file }}" placeholder="~\Documents\notes.{{content_type}}">
{% endfor %}
{% endif %}
</td>
<td>
<button type="button" id="input-files-button">Add</button>
</td>
</tr>
<tr>
<td>
<label for="input-filter" title="Add a folder with {{content_type}} files for Khoj to index">Folders</label>
</td>
<td id="input-filter-cell">
{% if current_config['input_filter'] is none %}
<input type="text" id="input-filter" name="input-filter" placeholder="~/Documents/{{content_type}}">
{% else %}
{% for input_filter in current_config['input_filter'] %}
<input type="text" id="input-filter" name="input-filter" placeholder="~/Documents/{{content_type}}" value="{{ input_filter }}">
{% endfor %}
{% endif %}
</td>
<td>
<button type="button" id="input-filter-button">Add</button>
</td>
</tr>
</table>
<table style="display: none;" >
<tr>
<td>
<label for="compressed-jsonl">Compressed JSONL (Output)</label>
</td>
<td>
<input type="text" id="compressed-jsonl" name="compressed-jsonl" value="{{ current_config['compressed_jsonl'] }}">
</td>
</tr>
<tr>
<td>
<label for="embeddings-file">Embeddings File (Output)</label>
</td>
<td>
<input type="text" id="embeddings-file" name="embeddings-file" value="{{ current_config['embeddings_file'] }}">
</td>
</tr>
<tr>
<td>
<label for="index-heading-entries">Index Heading Entries</label>
</td>
<td>
<input type="text" id="index-heading-entries" name="index-heading-entries" value="{{ current_config['index_heading_entries'] }}">
</td>
</tr>
</table>
<div class="section">
<div id="success" style="display: none;" ></div>
<button id="submit" type="submit">Save</button>
</div>
</form>
</div>
</div>
<script>
function addButtonEventListener(fieldName) {
var button = document.getElementById(fieldName + "-button");
button.addEventListener("click", function(event) {
var cell = document.getElementById(fieldName + "-cell");
var newInput = document.createElement("input");
newInput.setAttribute("type", "text");
newInput.setAttribute("name", fieldName);
cell.appendChild(newInput);
})
}
addButtonEventListener("input-files");
addButtonEventListener("input-filter");
function getValidInputNodes(nodes) {
var validNodes = [];
for (var i = 0; i < nodes.length; i++) {
const nodeValue = nodes[i].value;
if (nodeValue === "" || nodeValue === null || nodeValue === undefined || nodeValue === "None") {
continue;
}
validNodes.push(nodes[i]);
}
return validNodes;
}
submit.addEventListener("click", function(event) {
event.preventDefault();
let globFormat = "**/*"
let suffixes = [];
if ('{{content_type}}' == "markdown")
suffixes = [".md", ".markdown"]
else if ('{{content_type}}' == "org")
suffixes = [".org"]
else if ('{{content_type}}' === "pdf")
suffixes = [".pdf"]
else if ('{{content_type}}' === "plaintext")
suffixes = ['.*']
let globs = suffixes.map(x => `${globFormat}${x}`)
var inputFileNodes = document.getElementsByName("input-files");
var inputFiles = getValidInputNodes(inputFileNodes).map(node => node.value);
var inputFilterNodes = document.getElementsByName("input-filter");
var inputFilter = [];
var nodes = getValidInputNodes(inputFilterNodes);
// A regex that checks for globs in the path. If they exist,
// we are going to just not add our own globing. If they don't,
// then we will assume globbing should be done.
const glob_regex = /([*?\[\]])/;
if (nodes.length > 0) {
for (var i = 0; i < nodes.length; i++) {
for (var j = 0; j < globs.length; j++) {
if (glob_regex.test(nodes[i].value)) {
inputFilter.push(nodes[i].value);
} else {
inputFilter.push(nodes[i].value + globs[j]);
}
}
}
}
if (inputFiles.length === 0 && inputFilter.length === 0) {
alert("You must specify at least one input file or input filter.");
return;
}
if (inputFiles.length == 0) {
inputFiles = null;
}
if (inputFilter.length == 0) {
inputFilter = null;
}
var compressed_jsonl = document.getElementById("compressed-jsonl").value;
var embeddings_file = document.getElementById("embeddings-file").value;
var index_heading_entries = document.getElementById("index-heading-entries").value;
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/config/data/content_type/{{ content_type }}', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken
},
body: JSON.stringify({
"input_files": inputFiles,
"input_filter": inputFilter,
"compressed_jsonl": compressed_jsonl,
"embeddings_file": embeddings_file,
"index_heading_entries": index_heading_entries
})
})
.then(response => response.json())
.then(data => {
if (data["status"] == "ok") {
document.getElementById("success").innerHTML = "✅ Successfully updated. Go to your <a href='/config'>settings page</a> to complete setup.";
document.getElementById("success").style.display = "block";
} else {
document.getElementById("success").innerHTML = "⚠️ Failed to update settings.";
document.getElementById("success").style.display = "block";
}
})
});
</script>
{% endblock %}

129
src/khoj/interface/web/login.html Normal file
View file

@ -0,0 +1,129 @@
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0 maximum-scale=1.0">
<title>Khoj - Login</title>
<link rel="icon" type="image/png" sizes="128x128" href="/static/assets/icons/favicon-128x128.png">
<link rel="manifest" href="/static/khoj.webmanifest">
<link rel="stylesheet" href="/static/assets/khoj.css">
</head>
<body>
<div class="khoj-header"></div>
<!-- Login Modal -->
<div id="login-modal">
<img class="khoj-logo" src="/static/assets/icons/favicon-128x128.png" alt="Khoj"></img>
<div class="login-modal-title">Log in to Khoj</div>
<!-- Sign Up/Login with Google OAuth -->
<div
class="g_id_signin"
data-shape="circle"
data-text="continue_with"
data-logo_alignment="center"
data-size="large"
data-type="standard">
</div>
<div id="g_id_onload"
data-client_id="{{ google_client_id }}"
data-ux_mode="redirect"
data-login_uri="{{ redirect_uri }}"
data-auto-select="true">
</div>
</div>
<div class="khoj-footer"></div>
</div>
</body>
<style>
@media only screen and (max-width: 700px) {
body {
display: grid;
grid-template-columns: 1fr;
grid-template-rows: 1fr auto 1fr;
font-size: small!important;
}
body > * {
grid-column: 1;
}
}
@media only screen and (min-width: 700px) {
body {
display: grid;
grid-template-columns: 1fr min(70vw, 100%) 1fr;
grid-template-rows: 1fr auto 1fr;
}
body > * {
grid-column: 2;
}
}
body {
padding: 0px;
margin: 0px;
height: 100%;
background: var(--background-color);
color: var(--main-text-color);
font-family: roboto, karma, segoe ui, sans-serif;
font-size: 20px;
font-weight: 300;
line-height: 1.5em;
}
body > * {
padding: 10px;
margin: 10px;
}
@keyframes gradient {
0% {
background-position: 0% 50%;
}
50% {
background-position: 100% 50%;
}
100% {
background-position: 0% 50%;
}
}
a.khoj-logo {
text-align: center;
justify-self: center;
}
div#login-modal {
display: grid;
grid-template-columns: 1fr;
grid-template-rows: 1fr auto auto 1fr;
gap: 32px;
min-height: 300px;
background: var(--background-color);
margin-left: 25%;
margin-right: 25%;
}
div.g_id_signin {
margin: 0 auto;
display: block;
}
div.login-modal-title {
text-align: center;
line-height: 28px;
font-size: 24px;
font-weight: 500;
}
@media only screen and (max-width: 700px) {
div#login-modal {
margin-left: 10%;
margin-right: 10%;
}
}
</style>
<script src="https://accounts.google.com/gsi/client" async defer></script>
</html>

131
src/khoj/interface/web/index.html → src/khoj/interface/web/search.html
View file

@ -10,6 +10,7 @@
</head>
<script type="text/javascript" src="/static/assets/org.min.js"></script>
<script type="text/javascript" src="/static/assets/markdown-it.min.js"></script>
<script type="text/javascript" src="/static/assets/utils.js"></script>
<script>
function render_image(item) {
@ -94,6 +95,15 @@
}).join("\n");
}
function render_xml(query, data) {
return data.map(function (item) {
return `<div class="results-xml">` +
`<b><a href="${item.additional.file}">${item.additional.heading}</a></b>` +
`<xml>${item.entry}</xml>` +
`</div>`
}).join("\n");
}
function render_multiple(query, data, type) {
let html = "";
data.forEach(item => {
@ -113,6 +123,8 @@
html += `<div class="results-notion">` + `<b><a href="${item.additional.file}">${item.additional.heading}</a></b>` + `<p>${item.entry}</p>` + `</div>`;
} else if (item.additional.file.endsWith(".html")) {
html += render_html(query, [item]);
} else if (item.additional.file.endsWith(".xml")) {
html += render_xml(query, [item])
} else {
html += `<div class="results-plugin">` + `<b><a href="${item.additional.file}">${item.additional.heading}</a></b>` + `<p>${item.entry}</p>` + `</div>`;
}
@ -170,10 +182,13 @@
// Execute Search and Render Results
url = createRequestUrl(query, type, results_count || 5, rerank);
fetch(url)
fetch(url, {
headers: {
"Content-Type": "application/json"
}
})
.then(response => response.json())
.then(data => {
console.log(data);
document.getElementById("results").innerHTML = render_results(data, query, type);
});
}
@ -195,8 +210,8 @@
fetch("/api/config/types")
.then(response => response.json())
.then(enabled_types => {
// Show warning if no content types are enabled
if (enabled_types.detail) {
// Show warning if no content types are enabled, or just one ("all")
if (enabled_types[0] === "all" && enabled_types.length === 1) {
document.getElementById("results").innerHTML = "<div id='results-error'>To use Khoj search, setup your content plugins on the Khoj <a class='inline-chat-link' href='/config'>settings page</a>.</div>";
document.getElementById("query").setAttribute("disabled", "disabled");
document.getElementById("query").setAttribute("placeholder", "Configure Khoj to enable search");
@ -254,37 +269,9 @@
</script>
<body>
{% if demo %}
<!-- Banner linking to https://khoj.dev -->
<div class="khoj-banner-container">
<a class="khoj-banner" href="https://khoj.dev" target="_blank">
<p id="khoj-banner" class="khoj-banner">
Enroll in Khoj cloud to get your own assistant
</p>
</a>
<input type="text" id="khoj-banner-email" placeholder="email" class="khoj-banner-email"></input>
<button id="khoj-banner-submit" class="khoj-banner-button">Submit</button>
</div>
{% endif %}
<!--Add Header Logo and Nav Pane-->
<div class="khoj-header">
{% if demo %}
<a class="khoj-logo" href="https://khoj.dev" target="_blank">
<img class="khoj-logo" src="/static/assets/icons/khoj-logo-sideways-500.png" alt="Khoj"></img>
</a>
{% else %}
<a class="khoj-logo" href="/">
<img class="khoj-logo" src="/static/assets/icons/khoj-logo-sideways-500.png" alt="Khoj"></img>
</a>
{% endif %}
<nav class="khoj-nav">
<a class="khoj-nav" href="/chat">Chat</a>
<a class="khoj-nav khoj-nav-selected" href="/">Search</a>
{% if not demo %}
<a class="khoj-nav" href="/config">Settings</a>
{% endif %}
</nav>
</div>
{% import 'utils.html' as utils %}
{{ utils.heading_pane(user_photo, username, is_active, has_documents) }}
<!--Add Text Box To Enter Query, Trigger Incremental Search OnChange -->
<input type="text" id="query" class="option" onkeyup=incrementalSearch(event) autofocus="autofocus" placeholder="Search your knowledge base using natural language">
@ -297,9 +284,12 @@
<!-- Section to Render Results -->
<div id="results"></div>
</body>
<script>
document.getElementById("search-nav").classList.add("khoj-nav-selected");
</script>
<style>
@media only screen and (max-width: 600px) {
@media only screen and (max-width: 700px) {
body {
display: grid;
grid-template-columns: 1fr;
@ -310,7 +300,7 @@
grid-column: 1;
}
}
@media only screen and (min-width: 600px) {
@media only screen and (min-width: 700px) {
body {
display: grid;
grid-template-columns: 1fr min(70vw, 100%) 1fr;
@ -324,8 +314,8 @@
body {
padding: 0px;
margin: 0px;
background: #fff;
color: #475569;
background: var(--background-color);
color: var(--main-text-color);
font-family: roboto, karma, segoe ui, sans-serif;
font-size: 20px;
font-weight: 300;
@ -343,7 +333,7 @@
#options > * {
padding: 15px;
border-radius: 5px;
border: 1px solid #475569;
border: 1px solid var(--main-text-color);
background: #f9fafc
}
.option:hover {
@ -371,7 +361,7 @@
.image {
width: 20vw;
border-radius: 10px;
border: 1px solid #475569;
border: 1px solid var(--main-text-color);
}
#json {
white-space: pre-wrap;
@ -414,7 +404,7 @@
padding: 3.5px 3.5px 0;
margin-right: 5px;
border-radius: 5px;
border: 1px solid #475569;
border: 1px solid var(--main-text-color);
background-color: #ef4444;
font-size: small;
}
@ -457,14 +447,6 @@
max-width: 90%;
}
div.khoj-banner-container {
background: linear-gradient(-45deg, #FFC107, #FF9800, #FF5722, #FF9800, #FFC107);
background-size: 400% 400%;
animation: gradient 15s ease infinite;
text-align: center;
padding: 10px;
}
@keyframes gradient {
0% {
background-position: 0% 50%;
@ -481,57 +463,6 @@
text-align: center;
}
button#khoj-banner-submit,
input#khoj-banner-email {
padding: 10px;
border-radius: 5px;
border: 1px solid #475569;
background: #f9fafc;
}
button#khoj-banner-submit:hover,
input#khoj-banner-email:hover {
box-shadow: 0 0 11px #aaa;
}
@media only screen and (max-width: 600px) {
a.khoj-banner {
display: block;
}
p.khoj-banner {
padding: 0;
}
}
</style>
<script>
var khojBannerSubmit = document.getElementById("khoj-banner-submit");
khojBannerSubmit?.addEventListener("click", function(event) {
event.preventDefault();
var email = document.getElementById("khoj-banner-email").value;
fetch("https://app.khoj.dev/beta/users/", {
method: "POST",
body: JSON.stringify({
email: email
}),
headers: {
"Content-Type": "application/json"
}
}).then(function(response) {
return response.json();
}).then(function(data) {
console.log(data);
if (data.user != null) {
document.getElementById("khoj-banner").innerHTML = "Thanks for signing up. We'll be in touch soon! 🚀";
document.getElementById("khoj-banner-submit").remove();
} else {
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
}
}).catch(function(error) {
console.log(error);
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
});
});
</script>
</html>

34
src/khoj/interface/web/utils.html Normal file
View file

@ -0,0 +1,34 @@
{% macro heading_pane(user_photo, username, is_active, has_documents) -%}
<div class="khoj-header">
<a class="khoj-logo" href="/" target="_blank">
<img class="khoj-logo" src="/static/assets/icons/khoj-logo-sideways-500.png" alt="Khoj"></img>
</a>
<nav class="khoj-nav">
<a id="chat-nav" class="khoj-nav" href="/chat">💬 Chat</a>
{% if has_documents %}
<a id="search-nav" class="khoj-nav" href="/search">🔎 Search</a>
{% endif %}
<!-- Dropdown Menu -->
<div id="khoj-nav-menu-container" class="khoj-nav dropdown">
{% if user_photo and user_photo != "None" %}
{% if is_active %}
<img id="profile-picture" class="circle subscribed" src="{{ user_photo }}" alt="{{ username[0].upper() }}" onclick="toggleMenu()" referrerpolicy="no-referrer">
{% else %}
<img id="profile-picture" class="circle" src="{{ user_photo }}" alt="{{ username[0].upper() }}" onclick="toggleMenu()" referrerpolicy="no-referrer">
{% endif %}
{% else %}
{% if is_active %}
<div id="profile-picture" class="circle user-initial subscribed" alt="{{ username[0].upper() }}" onclick="toggleMenu()">{{ username[0].upper() }}</div>
{% else %}
<div id="profile-picture" class="circle user-initial" alt="{{ username[0].upper() }}" onclick="toggleMenu()">{{ username[0].upper() }}</div>
{% endif %}
{% endif %}
<div id="khoj-nav-menu" class="khoj-nav-dropdown-content">
<div class="khoj-nav-username"> {{ username }} </div>
<a id="settings-nav" class="khoj-nav" href="/config">⚙️ Settings</a>
<a class="khoj-nav" href="/auth/logout">🔑 Logout</a>
</div>
</div>
</nav>
</div>
{%- endmacro %}

64
src/khoj/main.py
View file

@ -3,11 +3,6 @@ import os
import sys
import locale
if sys.stdout is None:
sys.stdout = open(os.devnull, "w")
if sys.stderr is None:
sys.stderr = open(os.devnull, "w")
import logging
import threading
import warnings
@ -19,19 +14,32 @@ warnings.filterwarnings("ignore", message=r"legacy way to download files from th
# External Packages
import uvicorn
import django
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from rich.logging import RichHandler
import schedule
# Internal Packages
from khoj.configure import configure_routes, initialize_server
from khoj.utils import state
from khoj.utils.cli import cli
from django.core.asgi import get_asgi_application
from django.core.management import call_command
# Initialize Django
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.settings")
django.setup()
# Initialize Django Database
call_command("migrate", "--noinput")
# Initialize Django Static Files
call_command("collectstatic", "--noinput")
# Initialize the Application Server
app = FastAPI()
# Get Django Application
django_app = get_asgi_application()
# Add CORS middleware
app.add_middleware(
CORSMiddleware,
@ -44,6 +52,12 @@ app.add_middleware(
# Set Locale
locale.setlocale(locale.LC_ALL, "")
# Internal Packages. We do this after setting up Django so that Django features are accessible to the app.
from khoj.configure import configure_routes, initialize_server, configure_middleware
from khoj.utils import state
from khoj.utils.cli import cli
from khoj.utils.initialization import initialization
# Setup Logger
rich_handler = RichHandler(rich_tracebacks=True)
rich_handler.setFormatter(fmt=logging.Formatter(fmt="%(message)s", datefmt="[%X]"))
@ -52,7 +66,7 @@ logging.basicConfig(handlers=[rich_handler])
logger = logging.getLogger("khoj")
def run():
def run(should_start_server=True):
# Turn Tokenizers Parallelism Off. App does not support it.
os.environ["TOKENIZERS_PARALLELISM"] = "false"
@ -61,8 +75,7 @@ def run():
args = cli(state.cli_args)
set_state(args)
# Create app directory, if it doesn't exist
state.config_file.parent.mkdir(parents=True, exist_ok=True)
logger.info(f"🚒 Initializing Khoj v{state.khoj_version}")
# Set Logging Level
if args.verbose == 0:
@ -70,6 +83,11 @@ def run():
elif args.verbose >= 1:
logger.setLevel(logging.DEBUG)
initialization()
# Create app directory, if it doesn't exist
state.config_file.parent.mkdir(parents=True, exist_ok=True)
# Set Log File
fh = logging.FileHandler(state.config_file.parent / "khoj.log", encoding="utf-8")
fh.setLevel(logging.DEBUG)
@ -82,8 +100,22 @@ def run():
# Start Server
configure_routes(app)
initialize_server(args.config, required=False)
start_server(app, host=args.host, port=args.port, socket=args.socket)
# Mount Django and Static Files
app.mount("/server", django_app, name="server")
static_dir = "static"
if not os.path.exists(static_dir):
os.mkdir(static_dir)
app.mount(f"/{static_dir}", StaticFiles(directory=static_dir), name=static_dir)
# Configure Middleware
configure_middleware(app)
initialize_server(args.config)
# If the server is started through gunicorn (external to the script), don't start the server
if should_start_server:
start_server(app, host=args.host, port=args.port, socket=args.socket)
def set_state(args):
@ -92,7 +124,7 @@ def set_state(args):
state.verbose = args.verbose
state.host = args.host
state.port = args.port
state.demo = args.demo
state.anonymous_mode = args.anonymous_mode
state.khoj_version = version("khoj-assistant")
state.chat_on_gpu = args.chat_on_gpu
@ -115,3 +147,5 @@ def poll_task_scheduler():
if __name__ == "__main__":
run()
else:
run(should_start_server=False)

139
src/khoj/migrations/migrate_server_pg.py Normal file
View file

@ -0,0 +1,139 @@
"""
The application config currently looks like this:
app:
should-log-telemetry: true
content-type:
...
processor:
conversation:
conversation-logfile: ~/.khoj/processor/conversation/conversation_logs.json
max-prompt-size: null
offline-chat:
chat-model: mistral-7b-instruct-v0.1.Q4_0.gguf
enable-offline-chat: false
openai:
api-key: sk-blah
chat-model: gpt-3.5-turbo
tokenizer: null
search-type:
asymmetric:
cross-encoder: cross-encoder/ms-marco-MiniLM-L-6-v2
encoder: sentence-transformers/multi-qa-MiniLM-L6-cos-v1
encoder-type: null
model-directory: /Users/si/.khoj/search/asymmetric
image:
encoder: sentence-transformers/clip-ViT-B-32
encoder-type: null
model-directory: /Users/si/.khoj/search/image
symmetric:
cross-encoder: cross-encoder/ms-marco-MiniLM-L-6-v2
encoder: sentence-transformers/all-MiniLM-L6-v2
encoder-type: null
model-directory: ~/.khoj/search/symmetric
version: 0.14.0
The new version will looks like this:
app:
should-log-telemetry: true
processor:
conversation:
offline-chat:
enabled: false
openai:
api-key: sk-blah
chat-model-options:
- chat-model: gpt-3.5-turbo
tokenizer: null
type: openai
- chat-model: mistral-7b-instruct-v0.1.Q4_0.gguf
tokenizer: null
type: offline
search-type:
asymmetric:
cross-encoder: cross-encoder/ms-marco-MiniLM-L-6-v2
encoder: sentence-transformers/multi-qa-MiniLM-L6-cos-v1
version: 0.15.0
"""
import logging
from packaging import version
from khoj.utils.yaml import load_config_from_file, save_config_to_file
from database.models import (
OpenAIProcessorConversationConfig,
OfflineChatProcessorConversationConfig,
ChatModelOptions,
SearchModelConfig,
)
logger = logging.getLogger(__name__)
def migrate_server_pg(args):
schema_version = "0.15.0"
raw_config = load_config_from_file(args.config_file)
previous_version = raw_config.get("version")
if previous_version is None or version.parse(previous_version) < version.parse(schema_version):
logger.info(
f"Migrating configuration used for version {previous_version} to latest version for server with postgres in {args.version_no}"
)
raw_config["version"] = schema_version
if raw_config is None:
return args
if "search-type" in raw_config and raw_config["search-type"]:
if "asymmetric" in raw_config["search-type"]:
# Delete all existing search models
SearchModelConfig.objects.filter(model_type=SearchModelConfig.ModelType.TEXT).delete()
# Create new search model from existing Khoj YAML config
asymmetric_search = raw_config["search-type"]["asymmetric"]
SearchModelConfig.objects.create(
name="default",
model_type=SearchModelConfig.ModelType.TEXT,
bi_encoder=asymmetric_search.get("encoder"),
cross_encoder=asymmetric_search.get("cross-encoder"),
)
if "processor" in raw_config and raw_config["processor"] and "conversation" in raw_config["processor"]:
processor_conversation = raw_config["processor"]["conversation"]
if "offline-chat" in raw_config["processor"]["conversation"]:
offline_chat = raw_config["processor"]["conversation"]["offline-chat"]
OfflineChatProcessorConversationConfig.objects.create(
enabled=offline_chat.get("enable-offline-chat"),
)
ChatModelOptions.objects.create(
chat_model=offline_chat.get("chat-model"),
tokenizer=processor_conversation.get("tokenizer"),
max_prompt_size=processor_conversation.get("max-prompt-size"),
model_type=ChatModelOptions.ModelType.OFFLINE,
)
if (
"openai" in raw_config["processor"]["conversation"]
and raw_config["processor"]["conversation"]["openai"]
):
openai = raw_config["processor"]["conversation"]["openai"]
if openai.get("api-key") is None:
logger.error("OpenAI API Key is not set. Will not be migrating OpenAI config.")
else:
if openai.get("chat-model") is None:
openai["chat-model"] = "gpt-3.5-turbo"
OpenAIProcessorConversationConfig.objects.create(
api_key=openai.get("api-key"),
)
ChatModelOptions.objects.create(
chat_model=openai.get("chat-model"),
tokenizer=processor_conversation.get("tokenizer"),
max_prompt_size=processor_conversation.get("max-prompt-size"),
model_type=ChatModelOptions.ModelType.OPENAI,
)
save_config_to_file(raw_config, args.config_file)
return args

18
src/khoj/processor/conversation/gpt4all/chat_model.py
View file

@ -55,10 +55,10 @@ def extract_questions_offline(
last_year = datetime.now().year - 1
last_christmas_date = f"{last_year}-12-25"
next_christmas_date = f"{datetime.now().year}-12-25"
system_prompt = prompts.extract_questions_system_prompt_llamav2.format(
message=(prompts.system_prompt_message_extract_questions_llamav2)
system_prompt = prompts.system_prompt_extract_questions_gpt4all.format(
message=(prompts.system_prompt_message_extract_questions_gpt4all)
)
example_questions = prompts.extract_questions_llamav2_sample.format(
example_questions = prompts.extract_questions_gpt4all_sample.format(
query=text,
chat_history=chat_history,
current_date=current_date,
@ -150,14 +150,14 @@ def converse_offline(
elif conversation_command == ConversationCommand.General or is_none_or_empty(compiled_references_message):
conversation_primer = user_query
else:
conversation_primer = prompts.notes_conversation_llamav2.format(
conversation_primer = prompts.notes_conversation_gpt4all.format(
query=user_query, references=compiled_references_message
)
# Setup Prompt with Primer or Conversation History
messages = generate_chatml_messages_with_context(
conversation_primer,
prompts.system_prompt_message_llamav2,
prompts.system_prompt_message_gpt4all,
conversation_log,
model_name=model,
max_prompt_size=max_prompt_size,
@ -183,16 +183,16 @@ def llm_thread(g, messages: List[ChatMessage], model: Any):
conversation_history = messages[1:-1]
formatted_messages = [
prompts.chat_history_llamav2_from_assistant.format(message=message.content)
prompts.khoj_message_gpt4all.format(message=message.content)
if message.role == "assistant"
else prompts.chat_history_llamav2_from_user.format(message=message.content)
else prompts.user_message_gpt4all.format(message=message.content)
for message in conversation_history
]
stop_words = ["<s>"]
chat_history = "".join(formatted_messages)
templated_system_message = prompts.system_prompt_llamav2.format(message=system_message.content)
templated_user_message = prompts.general_conversation_llamav2.format(query=user_message.content)
templated_system_message = prompts.system_prompt_gpt4all.format(message=system_message.content)
templated_user_message = prompts.user_message_gpt4all.format(message=user_message.content)
prompted_message = templated_system_message + chat_history + templated_user_message
state.chat_lock.acquire()

31
src/khoj/processor/conversation/gpt4all/utils.py
View file

@ -2,24 +2,33 @@ import logging
from khoj.utils import state
logger = logging.getLogger(__name__)
def download_model(model_name: str):
try:
from gpt4all import GPT4All
import gpt4all
except ModuleNotFoundError as e:
logger.info("There was an error importing GPT4All. Please run pip install gpt4all in order to install it.")
raise e
# Use GPU for Chat Model, if available
try:
device = "gpu" if state.chat_on_gpu else "cpu"
model = GPT4All(model_name=model_name, device=device)
logger.debug(f"Loaded {model_name} chat model to {device.upper()}")
except ValueError:
model = GPT4All(model_name=model_name)
logger.debug(f"Loaded {model_name} chat model to CPU.")
# Download the chat model
chat_model_config = gpt4all.GPT4All.retrieve_model(model_name=model_name, allow_download=True)
return model
# Decide whether to load model to GPU or CPU
try:
# Try load chat model to GPU if:
# 1. Loading chat model to GPU isn't disabled via CLI and
# 2. Machine has GPU
# 3. GPU has enough free memory to load the chat model
device = (
"gpu" if state.chat_on_gpu and gpt4all.pyllmodel.LLModel().list_gpu(chat_model_config["path"]) else "cpu"
)
except ValueError:
device = "cpu"
# Now load the downloaded chat model onto appropriate device
chat_model = gpt4all.GPT4All(model_name=model_name, device=device, allow_download=False)
logger.debug(f"Loaded chat model to {device.upper()}.")
return chat_model

30
src/khoj/processor/conversation/openai/gpt.py
View file

@ -20,27 +20,6 @@ from khoj.utils.helpers import ConversationCommand, is_none_or_empty
logger = logging.getLogger(__name__)
def summarize(session, model, api_key=None, temperature=0.5, max_tokens=200):
"""
Summarize conversation session using the specified OpenAI chat model
"""
messages = [ChatMessage(content=prompts.summarize_chat.format(), role="system")] + session
# Get Response from GPT
logger.debug(f"Prompt for GPT: {messages}")
response = completion_with_backoff(
messages=messages,
model_name=model,
temperature=temperature,
max_tokens=max_tokens,
model_kwargs={"stop": ['"""'], "frequency_penalty": 0.2},
openai_api_key=api_key,
)
# Extract, Clean Message from GPT's Response
return str(response.content).replace("\n\n", "")
def extract_questions(
text,
model: Optional[str] = "gpt-4",
@ -131,16 +110,14 @@ def converse(
completion_func(chat_response=prompts.no_notes_found.format())
return iter([prompts.no_notes_found.format()])
elif conversation_command == ConversationCommand.General or is_none_or_empty(compiled_references):
conversation_primer = prompts.general_conversation.format(current_date=current_date, query=user_query)
conversation_primer = prompts.general_conversation.format(query=user_query)
else:
conversation_primer = prompts.notes_conversation.format(
current_date=current_date, query=user_query, references=compiled_references
)
conversation_primer = prompts.notes_conversation.format(query=user_query, references=compiled_references)
# Setup Prompt with Primer or Conversation History
messages = generate_chatml_messages_with_context(
conversation_primer,
prompts.personality.format(),
prompts.personality.format(current_date=current_date),
conversation_log,
model,
max_prompt_size,
@ -157,4 +134,5 @@ def converse(
temperature=temperature,
openai_api_key=api_key,
completion_func=completion_func,
model_kwargs={"stop": ["Notes:\n["]},
)

7
src/khoj/processor/conversation/openai/utils.py
View file

@ -69,15 +69,15 @@ def completion_with_backoff(**kwargs):
reraise=True,
)
def chat_completion_with_backoff(
messages, compiled_references, model_name, temperature, openai_api_key=None, completion_func=None
messages, compiled_references, model_name, temperature, openai_api_key=None, completion_func=None, model_kwargs=None
):
g = ThreadedGenerator(compiled_references, completion_func=completion_func)
t = Thread(target=llm_thread, args=(g, messages, model_name, temperature, openai_api_key))
t = Thread(target=llm_thread, args=(g, messages, model_name, temperature, openai_api_key, model_kwargs))
t.start()
return g
def llm_thread(g, messages, model_name, temperature, openai_api_key=None):
def llm_thread(g, messages, model_name, temperature, openai_api_key=None, model_kwargs=None):
callback_handler = StreamingChatCallbackHandler(g)
chat = ChatOpenAI(
streaming=True,
@ -86,6 +86,7 @@ def llm_thread(g, messages, model_name, temperature, openai_api_key=None):
model_name=model_name, # type: ignore
temperature=temperature,
openai_api_key=openai_api_key or os.getenv("OPENAI_API_KEY"),
model_kwargs=model_kwargs,
request_timeout=20,
max_retries=1,
client=None,

74
src/khoj/processor/conversation/prompts.py
View file

@ -4,30 +4,44 @@ from langchain.prompts import PromptTemplate
## Personality
## --
personality = PromptTemplate.from_template("You are Khoj, a smart, inquisitive and helpful personal assistant.")
personality = PromptTemplate.from_template(
"""
You are Khoj, a smart, inquisitive and helpful personal assistant.
Use your general knowledge and the past conversation with the user as context to inform your responses.
You were created by Khoj Inc. with the following capabilities:
- You *CAN REMEMBER ALL NOTES and PERSONAL INFORMATION FOREVER* that the user ever shares with you.
- You cannot set reminders.
- Say "I don't know" or "I don't understand" if you don't know what to say or if you don't know the answer to a question.
- Ask crisp follow-up questions to get additional context, when the answer cannot be inferred from the provided notes or past conversations.
- Sometimes the user will share personal information that needs to be remembered, like an account ID or a residential address. These can be acknowledged with a simple "Got it" or "Okay".
Note: More information about you, the company or other Khoj apps can be found at https://khoj.dev.
Today is {current_date} in UTC.
""".strip()
)
## General Conversation
## --
general_conversation = PromptTemplate.from_template(
"""
Using your general knowledge and our past conversations as context, answer the following question.
Current Date: {current_date}
Question: {query}
{query}
""".strip()
)
no_notes_found = PromptTemplate.from_template(
"""
I'm sorry, I couldn't find any relevant notes to respond to your message.
""".strip()
)
system_prompt_message_llamav2 = f"""You are Khoj, a smart, inquisitive and helpful personal assistant.
## Conversation Prompts for GPT4All Models
## --
system_prompt_message_gpt4all = f"""You are Khoj, a smart, inquisitive and helpful personal assistant.
Using your general knowledge and our past conversations as context, answer the following question.
If you do not know the answer, say 'I don't know.'"""
system_prompt_message_extract_questions_llamav2 = f"""You are Khoj, a kind and intelligent personal assistant. When the user asks you a question, you ask follow-up questions to clarify the necessary information you need in order to answer from the user's perspective.
system_prompt_message_extract_questions_gpt4all = f"""You are Khoj, a kind and intelligent personal assistant. When the user asks you a question, you ask follow-up questions to clarify the necessary information you need in order to answer from the user's perspective.
- Write the question as if you can search for the answer on the user's personal notes.
- Try to be as specific as possible. Instead of saying "they" or "it" or "he", use the name of the person or thing you are referring to. For example, instead of saying "Which store did they go to?", say "Which store did Alice and Bob go to?".
- Add as much context from the previous questions and notes as required into your search queries.
@ -35,61 +49,47 @@ system_prompt_message_extract_questions_llamav2 = f"""You are Khoj, a kind and i
What follow-up questions, if any, will you need to ask to answer the user's question?
"""
system_prompt_llamav2 = PromptTemplate.from_template(
system_prompt_gpt4all = PromptTemplate.from_template(
"""
<s>[INST] <<SYS>>
{message}
<</SYS>>Hi there! [/INST] Hello! How can I help you today? </s>"""
)
extract_questions_system_prompt_llamav2 = PromptTemplate.from_template(
system_prompt_extract_questions_gpt4all = PromptTemplate.from_template(
"""
<s>[INST] <<SYS>>
{message}
<</SYS>>[/INST]</s>"""
)
general_conversation_llamav2 = PromptTemplate.from_template(
"""
<s>[INST] {query} [/INST]
""".strip()
)
chat_history_llamav2_from_user = PromptTemplate.from_template(
user_message_gpt4all = PromptTemplate.from_template(
"""
<s>[INST] {message} [/INST]
""".strip()
)
chat_history_llamav2_from_assistant = PromptTemplate.from_template(
khoj_message_gpt4all = PromptTemplate.from_template(
"""
{message}</s>
""".strip()
)
conversation_llamav2 = PromptTemplate.from_template(
"""
<s>[INST] {query} [/INST]
""".strip()
)
## Notes Conversation
## --
notes_conversation = PromptTemplate.from_template(
"""
Using my personal notes and our past conversations as context, answer the following question.
Ask crisp follow-up questions to get additional context, when the answer cannot be inferred from the provided notes or past conversations.
These questions should end with a question mark.
Current Date: {current_date}
Use my personal notes and our past conversations to inform your response.
Ask crisp follow-up questions to get additional context, when a helpful response cannot be provided from the provided notes or past conversations.
Notes:
{references}
Question: {query}
Query: {query}
""".strip()
)
notes_conversation_llamav2 = PromptTemplate.from_template(
notes_conversation_gpt4all = PromptTemplate.from_template(
"""
User's Notes:
{references}
@ -98,13 +98,6 @@ Question: {query}
)
## Summarize Chat
## --
summarize_chat = PromptTemplate.from_template(
f"{personality.format()} Summarize the conversation from your first person perspective"
)
## Summarize Notes
## --
summarize_notes = PromptTemplate.from_template(
@ -132,7 +125,10 @@ Question: {user_query}
Answer (in second person):"""
)
extract_questions_llamav2_sample = PromptTemplate.from_template(
## Extract Questions
## --
extract_questions_gpt4all_sample = PromptTemplate.from_template(
"""
<s>[INST] <<SYS>>Current Date: {current_date}<</SYS>> [/INST]</s>
<s>[INST] How was my trip to Cambodia? [/INST]
@ -157,8 +153,6 @@ Use these notes from the user's previous conversations to provide a response:
)
## Extract Questions
## --
extract_questions = PromptTemplate.from_template(
"""
You are Khoj, an extremely smart and helpful search assistant with the ability to retrieve information from the user's notes.
@ -255,7 +249,7 @@ help_message = PromptTemplate.from_template(
**/default**: Chat using your knowledge base and Khoj's general knowledge for context.
**/help**: Show this help message.
You are using the **{model}** model.
You are using the **{model}** model on the **{device}**.
**version**: {version}
""".strip()
)

32
src/khoj/processor/embeddings.py Normal file
View file

@ -0,0 +1,32 @@
from typing import List
from sentence_transformers import SentenceTransformer, CrossEncoder
from torch import nn
from khoj.utils.helpers import get_device
from khoj.utils.rawconfig import SearchResponse
class EmbeddingsModel:
def __init__(self, model_name: str = "thenlper/gte-small"):
self.encode_kwargs = {"normalize_embeddings": True}
self.model_kwargs = {"device": get_device()}
self.model_name = model_name
self.embeddings_model = SentenceTransformer(self.model_name, **self.model_kwargs)
def embed_query(self, query):
return self.embeddings_model.encode([query], show_progress_bar=False, **self.encode_kwargs)[0]
def embed_documents(self, docs):
return self.embeddings_model.encode(docs, show_progress_bar=True, **self.encode_kwargs).tolist()
class CrossEncoderModel:
def __init__(self, model_name: str = "cross-encoder/ms-marco-MiniLM-L-6-v2"):
self.model_name = model_name
self.cross_encoder_model = CrossEncoder(model_name=self.model_name, device=get_device())
def predict(self, query, hits: List[SearchResponse], key: str = "compiled"):
cross_inp = [[query, hit.additional[key]] for hit in hits]
cross_scores = self.cross_encoder_model.predict(cross_inp, activation_fct=nn.Sigmoid())
return cross_scores

75
src/khoj/processor/github/github_to_jsonl.py → src/khoj/processor/github/github_to_entries.py
View file

@ -2,7 +2,7 @@
import logging
import time
from datetime import datetime
from typing import Dict, List, Union
from typing import Dict, List, Union, Tuple
# External Packages
import requests
@ -10,20 +10,32 @@ import requests
# Internal Packages
from khoj.utils.helpers import timer
from khoj.utils.rawconfig import Entry, GithubContentConfig, GithubRepoConfig
from khoj.processor.markdown.markdown_to_jsonl import MarkdownToJsonl
from khoj.processor.org_mode.org_to_jsonl import OrgToJsonl
from khoj.processor.text_to_jsonl import TextToJsonl
from khoj.utils.jsonl import compress_jsonl_data
from khoj.utils.rawconfig import Entry
from khoj.processor.markdown.markdown_to_entries import MarkdownToEntries
from khoj.processor.org_mode.org_to_entries import OrgToEntries
from khoj.processor.text_to_entries import TextToEntries
from database.models import Entry as DbEntry, GithubConfig, KhojUser
logger = logging.getLogger(__name__)
class GithubToJsonl(TextToJsonl):
def __init__(self, config: GithubContentConfig):
class GithubToEntries(TextToEntries):
def __init__(self, config: GithubConfig):
super().__init__(config)
self.config = config
raw_repos = config.githubrepoconfig.all()
repos = []
for repo in raw_repos:
repos.append(
GithubRepoConfig(
name=repo.name,
owner=repo.owner,
branch=repo.branch,
)
)
self.config = GithubContentConfig(
pat_token=config.pat_token,
repos=repos,
)
self.session = requests.Session()
self.session.headers.update({"Authorization": f"token {self.config.pat_token}"})
@ -37,7 +49,9 @@ class GithubToJsonl(TextToJsonl):
else:
return
def process(self, previous_entries=[], files=None, full_corpus=True):
def process(
self, files: dict[str, str] = None, full_corpus: bool = True, user: KhojUser = None, regenerate: bool = False
) -> Tuple[int, int]:
if self.config.pat_token is None or self.config.pat_token == "":
logger.error(f"Github PAT token is not set. Skipping github content")
raise ValueError("Github PAT token is not set. Skipping github content")
@ -45,7 +59,7 @@ class GithubToJsonl(TextToJsonl):
for repo in self.config.repos:
current_entries += self.process_repo(repo)
return self.update_entries_with_ids(current_entries, previous_entries)
return self.update_entries_with_ids(current_entries, user=user)
def process_repo(self, repo: GithubRepoConfig):
repo_url = f"https://api.github.com/repos/{repo.owner}/{repo.name}"
@ -63,43 +77,42 @@ class GithubToJsonl(TextToJsonl):
current_entries = []
with timer(f"Extract markdown entries from github repo {repo_shorthand}", logger):
current_entries = MarkdownToJsonl.convert_markdown_entries_to_maps(
*GithubToJsonl.extract_markdown_entries(markdown_files)
current_entries = MarkdownToEntries.convert_markdown_entries_to_maps(
*GithubToEntries.extract_markdown_entries(markdown_files)
)
with timer(f"Extract org entries from github repo {repo_shorthand}", logger):
current_entries += OrgToJsonl.convert_org_nodes_to_entries(*GithubToJsonl.extract_org_entries(org_files))
current_entries += OrgToEntries.convert_org_nodes_to_entries(
*GithubToEntries.extract_org_entries(org_files)
)
with timer(f"Extract commit messages from github repo {repo_shorthand}", logger):
current_entries += self.convert_commits_to_entries(self.get_commits(repo_url), repo)
with timer(f"Extract issues from github repo {repo_shorthand}", logger):
issue_entries = GithubToJsonl.convert_issues_to_entries(
*GithubToJsonl.extract_github_issues(self.get_issues(repo_url))
issue_entries = GithubToEntries.convert_issues_to_entries(
*GithubToEntries.extract_github_issues(self.get_issues(repo_url))
)
current_entries += issue_entries
with timer(f"Split entries by max token size supported by model {repo_shorthand}", logger):
current_entries = TextToJsonl.split_entries_by_max_tokens(current_entries, max_tokens=256)
current_entries = TextToEntries.split_entries_by_max_tokens(current_entries, max_tokens=256)
return current_entries
def update_entries_with_ids(self, current_entries, previous_entries):
def update_entries_with_ids(self, current_entries, user: KhojUser = None):
# Identify, mark and merge any new entries with previous entries
with timer("Identify new or updated entries", logger):
entries_with_ids = TextToJsonl.mark_entries_for_update(
current_entries, previous_entries, key="compiled", logger=logger
num_new_embeddings, num_deleted_embeddings = self.update_embeddings(
current_entries,
DbEntry.EntryType.GITHUB,
DbEntry.EntrySource.GITHUB,
key="compiled",
logger=logger,
user=user,
)
with timer("Write github entries to JSONL file", logger):
# Process Each Entry from All Notes Files
entries = list(map(lambda entry: entry[1], entries_with_ids))
jsonl_data = MarkdownToJsonl.convert_markdown_maps_to_jsonl(entries)
# Compress JSONL formatted Data
compress_jsonl_data(jsonl_data, self.config.compressed_jsonl)
return entries_with_ids
return num_new_embeddings, num_deleted_embeddings
def get_files(self, repo_url: str, repo: GithubRepoConfig):
# Get the contents of the repository
@ -274,7 +287,7 @@ class GithubToJsonl(TextToJsonl):
entries = []
entry_to_file_map = []
for doc in markdown_files:
entries, entry_to_file_map = MarkdownToJsonl.process_single_markdown_file(
entries, entry_to_file_map = MarkdownToEntries.process_single_markdown_file(
doc["content"], doc["path"], entries, entry_to_file_map
)
return entries, dict(entry_to_file_map)
@ -285,7 +298,7 @@ class GithubToJsonl(TextToJsonl):
entry_to_file_map = []
for doc in org_files:
entries, entry_to_file_map = OrgToJsonl.process_single_org_file(
entries, entry_to_file_map = OrgToEntries.process_single_org_file(
doc["content"], doc["path"], entries, entry_to_file_map
)
return entries, dict(entry_to_file_map)

91
src/khoj/processor/jsonl/jsonl_to_jsonl.py
View file

@ -1,91 +0,0 @@
# Standard Packages
import glob
import logging
from pathlib import Path
from typing import List
# Internal Packages
from khoj.processor.text_to_jsonl import TextToJsonl
from khoj.utils.helpers import get_absolute_path, timer
from khoj.utils.jsonl import load_jsonl, compress_jsonl_data
from khoj.utils.rawconfig import Entry
logger = logging.getLogger(__name__)
class JsonlToJsonl(TextToJsonl):
# Define Functions
def process(self, previous_entries=[], files: dict[str, str] = {}, full_corpus: bool = True):
# Extract required fields from config
input_jsonl_files, input_jsonl_filter, output_file = (
self.config.input_files,
self.config.input_filter,
self.config.compressed_jsonl,
)
# Get Jsonl Input Files to Process
all_input_jsonl_files = JsonlToJsonl.get_jsonl_files(input_jsonl_files, input_jsonl_filter)
# Extract Entries from specified jsonl files
with timer("Parse entries from jsonl files", logger):
input_jsons = JsonlToJsonl.extract_jsonl_entries(all_input_jsonl_files)
current_entries = list(map(Entry.from_dict, input_jsons))
# Split entries by max tokens supported by model
with timer("Split entries by max token size supported by model", logger):
current_entries = self.split_entries_by_max_tokens(current_entries, max_tokens=256)
# Identify, mark and merge any new entries with previous entries
with timer("Identify new or updated entries", logger):
entries_with_ids = TextToJsonl.mark_entries_for_update(
current_entries, previous_entries, key="compiled", logger=logger
)
with timer("Write entries to JSONL file", logger):
# Process Each Entry from All Notes Files
entries = list(map(lambda entry: entry[1], entries_with_ids))
jsonl_data = JsonlToJsonl.convert_entries_to_jsonl(entries)
# Compress JSONL formatted Data
compress_jsonl_data(jsonl_data, output_file)
return entries_with_ids
@staticmethod
def get_jsonl_files(jsonl_files=None, jsonl_file_filters=None):
"Get all jsonl files to process"
absolute_jsonl_files, filtered_jsonl_files = set(), set()
if jsonl_files:
absolute_jsonl_files = {get_absolute_path(jsonl_file) for jsonl_file in jsonl_files}
if jsonl_file_filters:
filtered_jsonl_files = {
filtered_file
for jsonl_file_filter in jsonl_file_filters
for filtered_file in glob.glob(get_absolute_path(jsonl_file_filter), recursive=True)
}
all_jsonl_files = sorted(absolute_jsonl_files | filtered_jsonl_files)
files_with_non_jsonl_extensions = {
jsonl_file for jsonl_file in all_jsonl_files if not jsonl_file.endswith(".jsonl")
}
if any(files_with_non_jsonl_extensions):
print(f"[Warning] There maybe non jsonl files in the input set: {files_with_non_jsonl_extensions}")
logger.debug(f"Processing files: {all_jsonl_files}")
return all_jsonl_files
@staticmethod
def extract_jsonl_entries(jsonl_files):
"Extract entries from specified jsonl files"
entries = []
for jsonl_file in jsonl_files:
entries.extend(load_jsonl(Path(jsonl_file)))
return entries
@staticmethod
def convert_entries_to_jsonl(entries: List[Entry]):
"Convert each entry to JSON and collate as JSONL"
return "".join([f"{entry.to_json()}\n" for entry in entries])

48
src/khoj/processor/markdown/markdown_to_jsonl.py → src/khoj/processor/markdown/markdown_to_entries.py
View file

@ -3,29 +3,28 @@ import logging
import re
import urllib3
from pathlib import Path
from typing import List
from typing import Tuple, List
# Internal Packages
from khoj.processor.text_to_jsonl import TextToJsonl
from khoj.processor.text_to_entries import TextToEntries
from khoj.utils.helpers import timer
from khoj.utils.constants import empty_escape_sequences
from khoj.utils.jsonl import compress_jsonl_data
from khoj.utils.rawconfig import Entry, TextContentConfig
from khoj.utils.rawconfig import Entry
from database.models import Entry as DbEntry, KhojUser
logger = logging.getLogger(__name__)
class MarkdownToJsonl(TextToJsonl):
def __init__(self, config: TextContentConfig):
super().__init__(config)
self.config = config
class MarkdownToEntries(TextToEntries):
def __init__(self):
super().__init__()
# Define Functions
def process(self, previous_entries=[], files=None, full_corpus: bool = True):
def process(
self, files: dict[str, str] = None, full_corpus: bool = True, user: KhojUser = None, regenerate: bool = False
) -> Tuple[int, int]:
# Extract required fields from config
output_file = self.config.compressed_jsonl
if not full_corpus:
deletion_file_names = set([file for file in files if files[file] == ""])
files_to_process = set(files) - deletion_file_names
@ -35,8 +34,8 @@ class MarkdownToJsonl(TextToJsonl):
# Extract Entries from specified Markdown files
with timer("Parse entries from Markdown files into dictionaries", logger):
current_entries = MarkdownToJsonl.convert_markdown_entries_to_maps(
*MarkdownToJsonl.extract_markdown_entries(files)
current_entries = MarkdownToEntries.convert_markdown_entries_to_maps(
*MarkdownToEntries.extract_markdown_entries(files)
)
# Split entries by max tokens supported by model
@ -45,19 +44,18 @@ class MarkdownToJsonl(TextToJsonl):
# Identify, mark and merge any new entries with previous entries
with timer("Identify new or updated entries", logger):
entries_with_ids = TextToJsonl.mark_entries_for_update(
current_entries, previous_entries, key="compiled", logger=logger, deletion_filenames=deletion_file_names
num_new_embeddings, num_deleted_embeddings = self.update_embeddings(
current_entries,
DbEntry.EntryType.MARKDOWN,
DbEntry.EntrySource.COMPUTER,
"compiled",
logger,
deletion_file_names,
user,
regenerate=regenerate,
)
with timer("Write markdown entries to JSONL file", logger):
# Process Each Entry from All Notes Files
entries = list(map(lambda entry: entry[1], entries_with_ids))
jsonl_data = MarkdownToJsonl.convert_markdown_maps_to_jsonl(entries)
# Compress JSONL formatted Data
compress_jsonl_data(jsonl_data, output_file)
return entries_with_ids
return num_new_embeddings, num_deleted_embeddings
@staticmethod
def extract_markdown_entries(markdown_files):
@ -70,7 +68,7 @@ class MarkdownToJsonl(TextToJsonl):
for markdown_file in markdown_files:
try:
markdown_content = markdown_files[markdown_file]
entries, entry_to_file_map = MarkdownToJsonl.process_single_markdown_file(
entries, entry_to_file_map = MarkdownToEntries.process_single_markdown_file(
markdown_content, markdown_file, entries, entry_to_file_map
)
except Exception as e:

Some files were not shown because too many files have changed in this diff Show more