Commit graph

146 commits

Author SHA1 Message Date
Debanjum
4bc5c1357a Upgrade server, documentation dependencies. Spell fix docker-compose.yml 2024-12-10 15:47:47 -08:00
Debanjum
354dc12b3b
Style the Admin Panel with a modern theme and Khoj branding ()
Overview
- The default django admin panel UI looks pretty dated and didn't
  have any Khoj specific branding
- Used the Unfold Django admin panel theme for a modern look
- Used the Khoj logo and name in Admin panel title, headings, favicons

Details:
All models shown on Admin panel need to inherit from unfold's
ModelAdmin to get styling applied. So

- Make all models on Admin panel inherit from unfold's ModelAdmin
- Subclassed UserAdmin to inherit from unfold's ModelAdmin
- Deregistered the unused Auth Group model from the Admin panel
  We can add it back when its actually used. Avoid confusion for now
- Explicitly register DjangoJobExecution on admin panel and again make
  it inherit from the unfold.admin.ModelAdmin
2024-12-04 23:53:43 -08:00
Debanjum
8c120a5139 Fallback to json5 loader if json.loads cannot parse complex json str
JSON5 spec is more flexible, try to load using a fast json5 parser if
the stricter json.loads from the standard library can't load the
raw complex json string into a python dictionary/list
2024-11-26 21:17:00 -08:00
Debanjum
96904e0769 Add script to evaluate khoj on Google's FRAMES benchmark
Google's FRAMES benchmark evaluates multi-step retrieval and reasoning
capabilities of an agent.

The script uses Gemini as an LLM Judge to evaluate Khoj responses to
the FRAMES benchmark prompts against the ground truth provided by it.
2024-11-02 04:57:42 -07:00
Debanjum
31b5fde163 Only enable prompt tracer if git python is installed 2024-11-02 02:07:02 -07:00
sabaimran
8d1b1bc78e Move the git python dependency into top level dependencies 2024-11-01 22:51:00 -07:00
Debanjum
b3a63017b5 Support setting seed for reproducible LLM response generation
Anthropic models do not support seed. But offline, gemini and openai
models do. Use these to debug and test Khoj via KHOJ_LLM_SEED env var
2024-10-30 14:00:21 -07:00
Debanjum Singh Solanky
10c8fd3b2a Save conversation traces to git for visualization 2024-10-26 04:59:19 -07:00
sabaimran
db959a504d Fix the version of pymupdf to avert build errors 2024-10-21 12:56:51 -07:00
Debanjum Singh Solanky
7ebfc24a96 Upgrade Django version used by Khoj server 2024-10-17 11:58:52 -07:00
Debanjum Singh Solanky
9b10b3e7a1 Remove unused langchain openai server dependency 2024-09-29 04:06:35 -07:00
Debanjum Singh Solanky
077b88bafa Make RapidOCR dependency optional as flaky requirements
RapidOCR depends on OpenCV which by default requires a bunch of GUI
paramters. This system package dependency set (like libgl1) is flaky

Making the RapidOCR dependency optional should allow khoj to be more
resilient to setup/dependency failures

Trade-off is that OCR for documents may not always be available and
it'll require looking at server logs to find out when this happens
2024-09-19 15:10:31 -07:00
Alexander Matyasko
9570933506
Support Google's Gemini model series ()
* Add functions to chat with Google's gemini model series
  * Gracefully close thread when there's an exception in the gemini llm thread
* Use enums for verifying the chat model option type
* Add a migration to add the gemini chat model type to the db model
* Fix chat model selection verification and math prompt tuning
* Fix extract questions method with gemini. Enforce json response in extract questions.
* Add standard stop sequence for Gemini chat response generation

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
2024-09-12 18:17:55 -07:00
Debanjum Singh Solanky
72fbbc092c Upgrade Django, FastAPI, Uvicorn packages
- Update Django to 5.0.8
- Update Uvicorn to 0.30.6
- Update FastAPI minimum versions to 0.110.0
2024-09-09 10:40:53 -07:00
sabaimran
7b8b3a66ae Revert django version to previous patch 2024-08-23 11:12:41 -07:00
Debanjum Singh Solanky
a60baa55fb Upgrade Django, a Khoj server dependency, to version 5.0.8 2024-08-20 12:32:00 -07:00
Debanjum Singh Solanky
acdc3f9470 Unwrap any json in md code block, when parsing chat actor responses
This is a more robust way to extract json output requested from
gemma-2 (2B, 9B) models which tend to return json in md codeblocks.

Other models should remain unaffected by this change.

Also removed request to not wrap json in codeblocks from prompts. As
code is doing the unwrapping automatically now, when present
2024-08-16 14:16:29 -05:00
Debanjum Singh Solanky
1cdfa8087c Update Khoj tagline to "Your Second Brain" 2024-08-05 02:27:05 +05:30
Debanjum Singh Solanky
53eabe0c06 Support Gemma 2 for Offline Chat
- Pass system message as the first user chat message as Gemma 2
  doesn't support system messages
- Use gemma-2 chat format
- Pass chat model name to generic, extract questions chat actors
  Used to figure out chat template to use for model
  For generic chat actor argument was anyway available but not being
  passed, which is confusing
2024-07-18 03:09:38 +05:30
Debanjum Singh Solanky
583fa3c188 Migrate the pypi package to khoj project name. Update references
- Deprecate khoj-assistant pypi package. Use more accurate and
  succinct pypi project name, khoj
- Update references to sye khoj pypi package in docs and code instead
  of the legacy khoj-assistant pypi package
- Update pypi workflow to publish to both khoj, khoj-assistant for now
- Update stale python 3.9 support mentioned in our pyproject. Can't
  support python 3.9 as depend on latest django which support >=3.10
2024-07-17 10:41:16 +05:30
Debanjum Singh Solanky
02658ad4fd Upgrade Django version 2024-07-11 16:35:10 +05:30
sabaimran
260aa61818 Remove tests for python3.9 2024-07-09 12:28:11 +05:30
sabaimran
4471c1e37f Apply mitigations for piling up open connections
- Because we're using a FastAPI api framework with a Django ORM, we're running into some interesting conditions around connection pooling and clean-up. We're ending up with a large pile-up of open, stale connections to the DB recurringly when the server has been running for a while. To mitigate this problem, given starlette and django run in different python threads, add a middleware that will go and call the connection clean up method in each of the threads.
2024-07-09 12:22:58 +05:30
Debanjum Singh Solanky
a353d883a0 Make it optional to set the encoder, cross-encoder configs via admin UI
Upgrade sentence-transformer, add einops dependency for some sentence
transformer models like nomic
2024-07-05 16:09:30 +05:30
Debanjum Singh Solanky
0d04018622 Install pydantic with optional email validator package
Otherwise Khoj fails on startup. Not sure why, must be new changes to
pydantic?
2024-06-24 16:12:20 +05:30
Debanjum Singh Solanky
22f6db0a6b Upgrade RapidOCR and enable for Python 3.12. Fix PDF OCR test 2024-06-22 16:01:55 +05:30
Debanjum Singh Solanky
55a23eae25 Upgrade pillow to fix pytest workflow failure 2024-06-22 15:17:43 +05:30
Raghav Tirumale
bd3b590153
Support Indexing Docx Files ()
* Add support for indexing docx files and associated unit tests

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
2024-06-20 11:18:01 +05:30
sabaimran
a57e1e7a14 Fix langchain, tenacity versions 2024-06-17 14:52:11 +05:30
sabaimran
ce9c14f894 Fix more packages related to langchain in the pyproject.toml 2024-06-17 14:38:05 +05:30
Debanjum Singh Solanky
179c70dba8 Upgrade Khoj llama-cpp, django and jinja dependencies 2024-06-04 09:05:53 +05:30
sabaimran
4aac84e1c1 Pin rsesend verison in pyproject.toml 2024-05-30 07:05:11 +05:30
sabaimran
01cdc54ad0
Add support for Anthropic models ()
* Add support for chatting with Anthropic's suite of models

- Had to use a custom class because there was enough nuance with how the anthropic SDK works that it would be better to simply separate out the logic. The extract questions flow needed modification of the system prompt in order to work as intended with the haiku model
2024-05-26 22:50:34 +05:30
sabaimran
0b7910d4af Pin th elangchain-community version explicitly 2024-05-21 05:26:17 -05:00
sabaimran
2b8e5a86cc Update version for resent library in pyproject.toml 2024-05-09 13:43:27 -07:00
sabaimran
eb65532386 Use Django ap scheduler in place of the sqlalchemy one 2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
230d160602 Improve rendering task scheduled settings view and message
- Render crontime string in natural language in message & settings UI
- Show more fields in tasks web config UI
- Add link to the tasks settings page in task scheduled chat response
- Improve task variables names
  Rename executing_query to query_to_run. scheduling_query to
  scheduling_request
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
3ce06a938c Render scheduled task response as html to improve readability in email 2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
c17dbbeb92 Render next run time in user timezone in config, chat UIs
- Pass timezone string from ipapi to khoj via clients
  - Pass this data from web, desktop and obsidian clients to server
- Use user tz to render next run time of scheduled task in user tz
2024-05-01 08:30:10 +05:30
Debanjum Singh Solanky
c11742f443 Add chat actor to schedule run query for user at specified times
- Detect when user intends to schedule a task, aka reminder
  Add new output mode: reminder. Add example of selecting the reminder
  output mode
- Extract schedule time (as cron timestring) and inferred query to run
  from user message
- Use APScheduler to call chat with inferred query at scheduled time
- Handle reminder scheduling from both websocket and http chat requests

- Support constructing scheduled task using chat history as context
  Pass chat history to scheduled query generator for improved context
  for scheduled task generation
2024-05-01 08:28:59 +05:30
Debanjum
17a06f152c
Support Llama 3 and Improve Offline Chat Actors ()
- Add support for Llama 3 in Khoj offline mode
- Make chat actors generate valid json with more local models
- Fix offline chat actor tests
2024-04-25 14:00:56 +05:30
Debanjum Singh Solanky
89ef23de50 Upgrade gunicorn and make it only a production dependency 2024-04-24 11:28:55 +05:30
Debanjum Singh Solanky
a2e4e4bede Add support for Llama 3 in Khoj offline mode
- Improve extract question prompts to explicitly request JSON list
- Use llama-3 chat format if HF repo_id mentions llama-3. The
  llama-cpp-python logic for detecting when to use llama-3 chat format
  isn't robust enough currently
2024-04-24 09:40:00 +05:30
sabaimran
46210695b6 pin version of huggingface hub explicitly to ensure relevant constants are present. Closes 2024-04-17 01:09:36 +05:30
Debanjum Singh Solanky
4977b55106 Use offline chat prompt config to set context window of loaded chat model
Previously you couldn't configure the n_ctx of the loaded offline chat
model. This made it hard to use good offline chat model (which these
days also have larger context) on machines with lower VRAM
2024-04-14 02:35:36 +05:30
Debanjum
9a48f72041
Index more text file types from Desktop, Github ()
### Index more text file types 
- Index all text, code files in Github repos. Not just md, org files
- Send more text file types from Desktop app and improve indexing them
- Identify file type by content & allow server to index all text files

### Deprecate Github Indexing Features
- Stop indexing commits, issues and issue comments in a Github repo
- Skip indexing Github repo on hitting Github API rate limit

### Fixes and Improvements
- **Fix indexing files in sub-folders from Desktop app**
- Standardize structure of text to entries to match other entry processors
2024-04-12 00:08:29 +05:30
sabaimran
3fe94a67b0
Send welcome emails when a new user signs up ()
* Don't trigger any re-indexing on server initailization

* Integrate Resend to send welcome emails when a new user signs up

- Only send if this is the first time they've signed in
- Configure welcome email with basic styling, as more complex designs don't work and style tag did not work
2024-04-10 19:57:33 +05:30
Debanjum Singh Solanky
89915dcb4c Identify file type by content & allow server to index all text files
- Use Magika's AI for a tiny, portable and better file type
  identification system
- Existing file type identification tools like `file' and `magic'
  require system level packages, that may not be installed by default
  on all operating systems (e.g `file' command on Windows)
2024-04-09 20:19:39 +05:30
Debanjum Singh Solanky
14fbf594b2 Support using Python 3.12 with Khoj
- RapidOCR for indexing image PDFs doesn't currently support python 3.12.
  It's an optional dependency anyway, so only install it if python < 3.12
- Run unit tests with python version 3.12 as well

Resolves 
2024-04-07 11:23:44 +05:30
sabaimran
47fc7e1ce6 Rebase with matser 2024-04-02 16:16:06 +05:30