Commit graph

4014 commits

Author SHA1 Message Date
sabaimran
f71e4969d3 Skip summarize while it's broken, and snip some other parts of the workflow while under construction 2024-10-09 16:40:06 -07:00
sabaimran
f7e6f99a32 add typing for extract document references 2024-10-09 16:05:34 -07:00
sabaimran
6960fb097c update types of prev iterations response 2024-10-09 16:04:39 -07:00
sabaimran
4978360852 Fix type of previous_iterations 2024-10-09 16:02:41 -07:00
sabaimran
46ef205a75 Add additional type annotations for compiled_references et al 2024-10-09 16:01:52 -07:00
sabaimran
4fbaef10e9 Correct usage of the summarize function 2024-10-09 15:58:05 -07:00
sabaimran
c91678078d Correct the usage of query passed to summarize function 2024-10-09 15:55:55 -07:00
sabaimran
f867d5ed72 Working prototype of meta-level chain of reasoning and execution
- Create a more dynamic reasoning agent that can evaluate information and understand what it doesn't know, making moves to get that information
- Lots of hacks and code that needs to be reversed later on before submission
2024-10-09 15:54:25 -07:00
Debanjum
00546c1a63
Fix link to llama-cpp-python setup docs 2024-10-09 01:30:33 -07:00
Debanjum Singh Solanky
05fb0f14d3 Use user chat models for train of thought when no server chat settings
Update chat actors to use user's chat model for train of thought. This
requires passing the user info as argument to all the chat actors.

Whether the user is subscribed or not can be inferred from the user
info being passed, so it doesn't need to be passed as a separate
argument to chat actor functions

Let send_message_to_model function infer chat model instead of passing
it as an argument from some chat actors. Better if this logic can be
done in a single place.
2024-10-09 00:07:08 -07:00
Debanjum Singh Solanky
ec0c79217f Do not set server chat settings on first run
Server chat settings can be set for advanced self-hosted or multi-user
cloud setups. They are not necessary anymore as we fallback to use the
users chat model for train of thought now
2024-10-09 00:07:08 -07:00
Debanjum Singh Solanky
a9009ea774 Default to use user chat model if server chat settings not defined
Fallback to use user chat model for train of thought if server chat
settings not defined.

This simplifies switching chat models for single-user, self-hosted
setups by just changing the chat model on the user settings page.

Server chat settings, when set, controls the default user chat model
and the chat model that is used for Khoj's train of thought.

Previously a self-hosted user had to update both the server chat
settings in the admin panel and their own user chat model in the user
settings panel to explicitly switch to a different chat model (i.e to
switch to a new model for both train of thought & response generation)

You can still set server chat settings to use a different chat
model for train of thought and response generation. But this is only
necessary for advanced self-hosted or cloud hosted setups of Khoj.
2024-10-09 00:07:08 -07:00
Debanjum Singh Solanky
9a056383e0 Reduce size of start chat and edit buttons on agent card in web app 2024-10-09 00:00:32 -07:00
Debanjum Singh Solanky
dc7f22f76c Mention no. of docs in agents knowledge base in its badge hover text 2024-10-08 23:51:00 -07:00
Debanjum Singh Solanky
13fb22f7e7 Update agent form data shown in edit card after save operaton on web app
Previously you had to refresh the page to see the updated data on
reopening the agents edit card after a save operation.

Now you see the latest saved agent data on reopening the agents edit
card. This should avoid confusion on whether the data was saved
correctly
2024-10-08 23:26:04 -07:00
Debanjum Singh Solanky
dd770cf1b9 Start chat with public and protected agents when shared via link 2024-10-08 22:10:07 -07:00
Debanjum Singh Solanky
80212c50fd Use default agent in others chats with an agent if agent made private
If a public or protected agent is made private. Other users who were
having conversation with that agent will have to carry on their
conversation using default agent instead
2024-10-08 22:08:38 -07:00
Debanjum Singh Solanky
d628f89ce9 Prefetch agents related database models 2024-10-08 21:59:15 -07:00
Debanjum Singh Solanky
8de67c5d4d Fallback to use general command if no tool selected by agent 2024-10-08 19:48:02 -07:00
Debanjum Singh Solanky
b80c4bcfdd Improve agent command descriptions 2024-10-08 19:47:51 -07:00
Debanjum Singh Solanky
67d0e59eac Pass chat history to the summarize chat actor 2024-10-08 18:44:52 -07:00
Debanjum Singh Solanky
7e3090060b Encourage Gemini to output more verbose responses 2024-10-08 18:41:43 -07:00
Debanjum Singh Solanky
bbbdba3093 Time embedding model load for better visibility into app startup time
Loading the embeddings model, even locally seems to be taking much
longer. Use timer to track visibility into embedding, cross-encoder
model load times
2024-10-08 18:41:43 -07:00
Debanjum Singh Solanky
516472a8d5 Switch default tokenizer to tiktoken as more widely used
The tiktoken BPE based tokenizers seem more widely used these days.

Fallback to gpt-4o tiktoken tokenizer to count tokens for context
stuffing
2024-10-08 18:41:43 -07:00
Debanjum Singh Solanky
2b8f7f3efb Reuse a single func to format conversation for Gemini
This deduplicates code and prevents logic from deviating across gemini
chat actors
2024-10-08 18:41:42 -07:00
Debanjum Singh Solanky
452e360175 Do not use max prompt size to limit Gemini max output tokens
We should start disambiguating the the max input from output size. Max
prompt size should only be used for the max input context to an LLM.

If required max_output_tokens should be set as a separate new field
2024-10-08 15:30:08 -07:00
Debanjum Singh Solanky
bdc36fec5d Remove unnecessary whitespace indent from personality context 2024-10-08 15:30:08 -07:00
sabaimran
3daa3c003d When tool selection is not done successfully with an agent, return all agent tools as options 2024-10-08 15:03:58 -07:00
sabaimran
ad716ca58d Delete associated entries with an agent when it is deleted 2024-10-08 15:00:21 -07:00
sabaimran
f7fc6dbdc8 Limit agent creation and modification to subscribed users 2024-10-08 14:59:57 -07:00
sabaimran
c7638a783e Dynamically update added files when upload in agent creation 2024-10-07 21:54:11 -07:00
sabaimran
e10a0571ff Only check the prompt safety if the agent is not private 2024-10-07 21:42:14 -07:00
sabaimran
f700d5bddb Add summarization capability with agent knowledge base 2024-10-07 21:20:23 -07:00
sabaimran
df3dc33e96 Show reference icon and domain side by side 2024-10-07 20:28:48 -07:00
sabaimran
59e55f981f Reset agent to default when continuing with deceased agent 2024-10-07 20:28:33 -07:00
sabaimran
874776024a Handle chat history rendering when agent is deceased 2024-10-07 20:28:10 -07:00
sabaimran
f232c2b059 Allow user to chat with agent knowledge base if general mode 2024-10-07 19:55:33 -07:00
sabaimran
c00654ae58 Update default agent settings 2024-10-07 18:11:24 -07:00
sabaimran
3d0e183bea Add more log lines when encountering rate limiting 2024-10-07 14:36:12 -07:00
sabaimran
e4a8a69bc8 Add a subtle check mark when the copy button is selected 2024-10-07 09:41:03 -07:00
sabaimran
405c047c0c
Include agent personality through subtasks and support custom agents ()
Currently, the personality of the agent is only included in the final response that it returns to the user. Historically, this was because models were quite bad at navigating the additional context of personality, and there was a bias towards having more control over certain operations (e.g., tool selection, question extraction).

Going forward, it should be more approachable to have prompts included in the sub tasks that Khoj runs in order to response to a given query. Make this possible in this PR. This also sets us up for agent creation becoming available soon.

Create custom agents in 

Agents are useful insofar as you can personalize them to fulfill specific subtasks you need to accomplish. In this PR, we add support for using custom agents that can be configured with a custom system prompt (aka persona) and knowledge base (from your own indexed documents). Once created, private agents can be accessible only to the creator, and protected agents can be accessible via a direct link.

Custom tool selection for agents in 

Expose the functionality to select which tools a given agent has access to. By default, they have all. Can limit both information sources and output modes.
Add new tools to the agent modification form
2024-10-07 00:21:55 -07:00
sabaimran
c0193744f5 Update readme.md 2024-10-06 12:26:53 -07:00
sabaimran
d4ffeca90a Fix notion indexing with manually set token 2024-10-05 09:13:16 -07:00
sabaimran
29a422b6bc Remove the single dollar sign delimeters from katex rendering 2024-10-04 12:24:19 -07:00
Debanjum Singh Solanky
e217cb5840 Suggest notification type automation on Automation page of web app 2024-10-03 16:36:23 -07:00
Debanjum Singh Solanky
f626b34436 Update automation docs with screenshots 2024-10-03 16:36:05 -07:00
sabaimran
27c7e54695 Release Khoj version 1.24.1 2024-10-03 13:21:10 -07:00
Debanjum
4a1cb50da3
Make Online Search Location Aware ()
## Overview
Add user country code as context for doing online search with serper.dev API.
This should find more user relevant results from online searches by Khoj

## Details
### Major
- Default to using system clock to infer user timezone on js clients
- Infer country from timezone when only timezone received by chat API
- Localize online search results to user country when location available

### Minor
- Add `__str__` func to `LocationData` class to deduplicate location string generation
2024-10-03 12:33:47 -07:00
sabaimran
cb4052e333 Bump up rate limit for subscribed users and add an option to create new conversation in the POST request 2024-10-03 12:31:58 -07:00
sabaimran
7a5cd06162
Improve the login page ()
* Init version of improved login page
* Use split screen view, add a gradient
2024-10-02 14:26:46 -07:00