Commit graph

3198 commits

Author SHA1 Message Date
sabaimran
467de76fc1 Improve the image diagramming prompts and response parsing 2024-11-20 18:59:40 -08:00
Debanjum
2203236e4c Update desktop app dependencies 2024-11-20 13:05:55 -08:00
Debanjum
6f1adcfe67
Track Usage Metrics in Chat API. Track Running Cost, Accuracy in Evals (#985)
- Track, return cost and usage metrics in chat api response
  Track input, output token usage and cost of interactions with 
  openai, anthropic and google chat models for each call to the khoj chat api
- Collect, display and store costs & accuracy of eval run currently in progress
  This provides more insight into eval runs during execution 
  instead of having to wait until the eval run completes.
2024-11-20 12:59:44 -08:00
Debanjum
bbd24f1e98 Improve dropdown menus on web app setting page with scroll & min-width
- Previously when settings list became long the dropdown height would
  overflow screen height. Now it's max height is clamped and  y-scroll
- Previously the dropdown content would take width of content. This
  would mean the menu could sometimes be less wide than the button. It
  felt strange. Now dropdown content is at least width of parent button
2024-11-20 12:27:13 -08:00
Debanjum
c53c3db96b Track, return cost and usage metrics in chat api response
- Track input, output token usage and cost for interactions
  via chat api with openai, anthropic and google chat models

- Get usage metadata from OpenAI using stream_options
- Handle openai proxies that do not support passing usage in response

- Add new usage, end response  events returned by chat api.
  - This can be optionally consumed by clients at a later point
  - Update streaming clients to mark message as completed after new
    end response event, not after end llm response event
- Ensure usage data from final response generation step is included
  - Pass usage data after llm response complete. This allows gathering
    token usage and cost for the final response generation step across
    streaming and non-streaming modes
2024-11-20 12:17:58 -08:00
Debanjum
80df3bb8c4 Enable prompt tracing only when PROMPTRACE_DIR env var set
Better to decouple prompt tracing from debug mode or verbosity level
and require explicit, independent config to enable prompt tracing
2024-11-20 11:54:02 -08:00
Debanjum
9ab76ccaf1 Skip adding agent to chat metadata when chat unset to avoids null ref 2024-11-19 21:10:23 -08:00
Debanjum
4da0499cd7 Stream responses by openai's o1 model series, as api now supports it
Previously o1 models did not support streaming responses via API. Now
they seem to do
2024-11-19 21:10:23 -08:00
Debanjum
7bdc9590dd Fix handling sources, output in chat actor when is automated task
Remove unnecessary ```python prefix removal. It isn't being triggered
in json deserialize path.
2024-11-19 13:49:27 -08:00
Debanjum
0e7d611a80 Remove ```python codeblock prefix from raw json before deserialize 2024-11-19 12:53:52 -08:00
Debanjum
001c13ef43 Upgrade web app package dependencies 2024-11-19 12:53:52 -08:00
sabaimran
5134d49d71 Release Khoj version 1.30.1 2024-11-18 17:30:33 -08:00
sabaimran
8bdd0b26d3 And a connections clean up decorator to all scheduled tasks 2024-11-18 17:19:36 -08:00
Debanjum
817601872f Update default offline models enabled 2024-11-18 16:38:17 -08:00
Debanjum
653127bf1d Improve data source, output mode selection
- Set output mode to single string. Specify output schema in prompt
  - Both thesee should encourage model to select only 1 output mode
    instead of encouraging it in prompt too many times
  - Output schema should also improve schema following in general
- Standardize variable, func name of io selector for readability
- Fix chat actors to test the io selector chat actor
- Make chat actor return sources, output separately for better
  disambiguation, at least during tests, for now
2024-11-18 15:11:37 -08:00
Debanjum
e3fd51d14b Pass user arg to create title from query in new automation flow 2024-11-18 12:58:10 -08:00
Debanjum
9e74de9b4f Improve serializing conversation JSON to print messages on console
- Handle chatml message.content with non-json serializable data like
  WebP image binary data used by Gemini models
2024-11-18 12:57:05 -08:00
sabaimran
3f70d2f685 Add more graceful exception handling when tool selection doesn't work 2024-11-18 09:34:49 -08:00
sabaimran
f75085dc7a Release Khoj version 1.30.0 2024-11-17 21:36:22 -08:00
sabaimran
c72813ba67
Merge pull request #981 from rznzippy/bugfix/980/database-connections-leakage
Fix database connections leakage (#980)
2024-11-17 21:01:06 -08:00
sabaimran
7d50c6590d
Merge pull request #977 from khoj-ai/features/improve-tool-selection
- JSON extract from LLMs is pretty decent now, so get the input tools and output modes all in one go. It'll help the model think through the full cycle of what it wants to do to handle the request holistically.
- Make slight improvements to tool selection indicators
2024-11-17 20:08:19 -08:00
Debanjum
48567fd468 Do not erase partial message when generation stopped via button on web app
Previously, we'd replace the generated message with an error message
when message generation stopped via stop button on chat page of web app.
So the partially generated message (which could be useful) gets lost.

This change just stops generation, while keeping the generated
response so any useful information from the partially generated
message can be retrieved.
2024-11-17 16:29:18 -08:00
Debanjum
285006d6c9 Sync chat models in Khoj with OpenAI proxies (e.g Ollama) on startup
- Allows managing chat models in the OpenAI proxy service like Ollama.
- Removes need to manually add, remove chat models from Khoj Admin Panel
  for these OpenAI compatible API services when enabled.
- Khoj still mantains the chat models configs within Khoj, so they can
  be configured via the Khoj admin panel as usual.
2024-11-17 15:34:36 -08:00
Debanjum
d6eece63f4 Use Jina API Key of Jina web scraper if configured in DB
Previously Jina search didn't API key. Now that it does need API key,
we should re-use the API key set in the Jina web scraper config,
otherwise fallback to using JINA_API_KEY from environment variable, if
either is present.

Resolves #978
2024-11-17 15:34:14 -08:00
sabaimran
6531f24ca0 Further improvements for descriptions to LLM on modes, code, diagram, image. 2024-11-17 13:23:57 -08:00
sabaimran
0eba6ce315 When diagram generation fails, save to conversation log
- Update tool name when choosing tools to execute
2024-11-17 13:23:12 -08:00
sabaimran
7e662a05f8 Merge branch 'master' of github.com:khoj-ai/khoj into features/improve-tool-selection 2024-11-17 12:26:55 -08:00
Ilya Khrustalev
00b1af8f99 Fix database connections leakage (#980) 2024-11-17 19:15:05 +01:00
Debanjum
69ef6829c1 Simplify integrating Ollama, OpenAI proxies with Khoj on first run
- Integrate with Ollama or other openai compatible APIs by simply
  setting `OPENAI_API_BASE' environment variable in docker-compose etc.
- Update docs on integrating with Ollama, openai proxies on first run
- Auto populate all chat models supported by openai compatible APIs
- Auto set vision enabled for all commercial models

- Minor
  - Add huggingface cache to khoj_models volume. This is where chat
  models and (now) sentence transformer models are stored by default
  - Reduce verbosity of yarn install of web app. Otherwise hit docker
  log size limit & stops showing remaining logs after web app install
  - Suggest `ollama pull <model_name>` to start it in background
2024-11-17 02:08:20 -08:00
Debanjum
2366fa08b9 Update default vision supported & anthropic chat models on first run
- Update to latest initialize with new claude 3.5 sonnet and haiku models
- Update to set vision enabled for google and anthropic models by
  default. Previously we didn't support but we've supported this for a
  month or two now
2024-11-17 02:08:20 -08:00
Debanjum
23ab258d78 Improve user conversation config details on Admin panel
Show user email and chat model that is associated with the user
conversation config
2024-11-17 02:08:20 -08:00
Debanjum
fc45aceecf Delete unused favicon ico in old web app directory 2024-11-17 02:08:20 -08:00
Debanjum
a16fc3ade8 Only add /research prefix when no slash command in message on web app
- Explictly adding a slash command is a higher priority intent than
  research mode being enabled in the background. Respect that for a
  more intuitive UX flow.

- Explicit slash commands do not currently work in research mode.
  You've to turn research mode off to use other slash commands. This
  is strange, unnecessary given intent priority is clear.
2024-11-17 02:08:20 -08:00
sabaimran
a1b4587b34 Remove extract_images flag from PDF loader 2024-11-15 21:46:35 -08:00
sabaimran
e3f1ea9dee Improve tool, output mode selection process
- JSON extract from LLMs is pretty decent now, so get the input tools and output modes all in one go. It'll help the model think through the full cycle of what it wants to do to handle the request holistically.
- Make slight improvements to tool selection indicators
2024-11-15 13:53:53 -08:00
sabaimran
c1a5b32ebf Do not start server when importing the main.py file, unless gunicorn
- Add more graceful shutdown when closing bg scheduler thread
2024-11-14 17:36:51 -08:00
sabaimran
be3ee5ec9f Add cool new suggestion cards for math, diagramming 2024-11-14 17:36:51 -08:00
Debanjum
8e009f48ce Show tool call error in next iteration. Allow rerun if model requests.
Previously errors would get eaten up but the model wouldn't see
anything. And the model wouldn't be allowed re-run the same query-tool
combination in the next iteration.

This update should give it insight into why it didn't get a result. So
it can make an informed (hopefully better) decision on what to do next.
And re-run the previous query if appropriate.
2024-11-13 22:50:14 -08:00
Debanjum
604da90fa8 Wrap try/catch around online search in research mode like other tools
Previously when call to online search API etc. failed, it'd error out
of response to query in research mode. Khoj should skip tool use that
iteration but continue to try respond.
2024-11-13 16:46:09 -08:00
Debanjum
8851b5f78a Standardize chat message truncation and serialization before print
Previously chatml messages were just strings, now they can be list of
strings or list of dicts as well.

- Use json seriallization to manage their variations and truncate them
  before printing for context.
- Put logic in single function for use across chat models
2024-11-13 16:30:17 -08:00
Debanjum
15b0cfa3dd Improve structured message truncation in logger
Previously chatml messages were just strings.
Since gemini, anthropic models always have messages as list of
strings, truncate those strings instead of the list of message content
2024-11-13 14:32:22 -08:00
Debanjum
153ae8bea9 Cut binary, long output files from code result for context efficiency
Removing binary data and truncating large data in output files
generated by code runs should improve speed and cost of research mode
runs with large or binary output files.

Previously binary data in code results was passed around in iteration
context during research mode. This made the context inefficient
because models have limited efficiency and reasoning capabilities over
b64 encoded image (and other binary) data and would hit context limits
leading to unnecessary truncation of other useful context

Also remove image data when logging output of code execution
2024-11-13 14:32:22 -08:00
sabaimran
4a1b1e8b9a Add support for interrupting messages after they've been sent. 2024-11-12 22:22:45 -08:00
sabaimran
d607ad7a27 Release Khoj version 1.29.1 2024-11-12 10:32:56 -08:00
sabaimran
8ec1764e42 Handle size calculation more gracefully for converted documents, depending on type 2024-11-12 02:00:29 -08:00
sabaimran
b6714c202f Increase the title character limit to 500 for conversations 2024-11-12 01:51:19 -08:00
sabaimran
f05e64cf8c Release Khoj version 1.29.0 2024-11-11 21:46:25 -08:00
sabaimran
47d3c8c235 Remove email query parameter from subscription patch api 2024-11-11 21:39:49 -08:00
sabaimran
d7027109a5 And null handling for response output_files in code output 2024-11-11 21:14:56 -08:00
sabaimran
d68243a3fb Revert clean_json logic temporarily. Eventually, we should do better validation here to extract markdown-formatted json. 2024-11-11 21:05:17 -08:00
sabaimran
1cab6c081f Add better error handling for diagram output, and fix chat history construct
- Make the `clean_json` method more robust as well
2024-11-11 20:44:19 -08:00
sabaimran
7bd2f83f97 Wrap test in suggestionCard 2024-11-11 20:12:46 -08:00
Debanjum
48862a8400
Enable Passing External Documents for Analysis in Code Sandbox (#960)
- Allow passing user files as input into code sandbox for analysis
- Update prompt to give more example of complex, multi-line code
- Simplify logic for model. Run one program at a time, 
  instead of allowing model to run multiple programs in parallel
- Show Code generated charts and docs in Reference pane of web app and make them downloaded
2024-11-11 19:37:17 -08:00
Debanjum
5078ac0ce2 Await on conversation save when generate conversation title via API 2024-11-11 19:17:39 -08:00
Debanjum
e1d0015248 Allow disabling Khoj telemetry via KHOJ_TELEMETRY_DISABLE env var 2024-11-11 19:17:39 -08:00
Debanjum
a52500d289 Show generated code artifacts before notes and online references 2024-11-11 18:00:22 -08:00
Debanjum
218eed83cd Show output file not code on hover. Remove reference card title border 2024-11-11 18:00:22 -08:00
Debanjum
b970cfd4b3 Align styling of reference panel card across code, docs, web results
- Add a border below heading
- Show code snippet in pre block
- Overflow-x when reference side panel open to allow seeing whole text
  via x-scroll
- Align header, body position of reference cards with each other
- Only show filename in doc reference cards at message bottom.
  Show full file path in hover and reference side panel
2024-11-11 18:00:22 -08:00
Debanjum
8e9f4262a9 Render code output files with code references in reference section
- Improve rendering code reference with better icons, smaller text and
  different line clamps for better visibility
- Show code output files as sub card of code card in reference section
- Allow downloading files generated by code instead of rendering it in
  chat message directly
- Show executed code before online references in reference panel
2024-11-11 18:00:22 -08:00
Debanjum
92c1efe6ee Fixes to render & save code context with non text based output modes
- Fix to render code generated chart with images, excalidraw diagrams
- Fix to save code context to chat history in image, diagram output modes
- Fix bug in image markdown being wrapped twice in markdown syntax
- Render newline in code references shown on chat page of web app
  Previously newlines weren't getting rendered. This made the code
  executed by Khoj hard to read in references. This changes fixes that.

  `dangerouslySetInnerHTML' usage is justified as rendered code
  snippet is being sanitized by DOMPurify before rendering.
2024-11-11 18:00:22 -08:00
Debanjum
af0215765c Decode code text output files from b64 to str to ease client processing 2024-11-11 18:00:22 -08:00
Debanjum
7b39f2014a Enable analysing user documents in code sandbox and other improvements
- Run one program at a time, instead of allowing model to pass
  multiple programs to run in parallel to simplify logic for model
- Update prompt to give more example of complex, multi-line code
- Allow passing user files as input into code sandbox for analysis

- Log code execution timer at info level to evaluate execution latencies
  in production

- Type the generated code for easier processing by caller functions
2024-11-11 17:59:37 -08:00
sabaimran
dc109559d4 Research mode gray when off, colored when on 2024-11-11 16:35:07 -08:00
sabaimran
cdda9c2e73 Improve text wrapping for attached files and preview context
For the research mode toggle, make it not fill when it's off
2024-11-11 13:32:10 -08:00
sabaimran
dd36303bb7 Fix sending file attachments in save_to_conversation method
- When files attached but upload fails, don't update the state variables
- Make removing null characters in pdf extraction more space efficient
2024-11-11 12:53:06 -08:00
Debanjum
536fe994be Remove unused db adapter methods, like for fact checker data store 2024-11-11 12:22:34 -08:00
Debanjum
10bca6fa8f Convert required user param check into decorator. Use with more adapters 2024-11-11 12:22:32 -08:00
Debanjum
ff5c10c221 Do not CRUD on entries, files & conversations in DB for null user
Increase defense-in-depth by reducing paths to create, read, update or
delete entries, files and conversations in DB when user is unset.
2024-11-11 12:20:07 -08:00
sabaimran
27fa39353e Make custom agent creation flow available to everyone
- For private agents, add guardrails to prevent against any misuse or violation of terms of service.
2024-11-11 11:54:59 -08:00
sabaimran
b563f46a2e
Merge pull request #957 from khoj-ai/features/include-full-file-in-convo-with-filter
Support including file attachments in the chat message

Now that models have much larger context windows, we can reasonably include full texts of certain files in the messages. Do this when an explicit file filter is set in a conversation. Do so in a separate user message in order to mitigate any confusion in the operation.

Pipe the relevant attached_files context through all methods calling into models.

This breaks certain prior behaviors. We will no longer automatically be processing/generating embeddings on the backend and adding documents to the "brain". You'll have to go to settings and go through the upload documents flow there in order to add docs to the brain (i.e., have search include them during question / response).
2024-11-11 11:34:42 -08:00
sabaimran
2bb2ff27a4 Rename attached_files to query_files. Update relevant backend and client-side code. 2024-11-11 11:21:26 -08:00
sabaimran
47937d5148 Merge branch 'features/include-full-file-in-convo-with-filter' of github.com:khoj-ai/khoj into features/include-full-file-in-convo-with-filter 2024-11-11 09:34:08 -08:00
sabaimran
ae4eb96d48 Consolidate file name to icon mapping 2024-11-11 09:34:04 -08:00
Debanjum
7954f39633 Use accept param to file input to indicate supported file types in web app
Remove unused total size calculations in chat input
2024-11-11 04:06:17 -08:00
Debanjum
4223b355dc Use python stdlib methods to write pdf, docx to temp files for loaders
Use python standard method tempfile.NamedTemporaryFile to write,
delete temporary files safely.
2024-11-11 03:24:50 -08:00
Debanjum
fd15fc1e59 Move construct chat history back to it's original position in file
Keep function where it original was allows tracking diffs and change
history more easily
2024-11-11 03:24:50 -08:00
Debanjum
35d6c792e4 Show snippet of truncated messages in debug logs to avoid log flooding 2024-11-11 02:30:38 -08:00
sabaimran
8805e731fd Merge branch 'master' of github.com:khoj-ai/khoj into features/include-full-file-in-convo-with-filter 2024-11-10 19:24:11 -08:00
sabaimran
a5e2b9e745 Exit early when running an automation if the conversation for the automation does not exist. 2024-11-10 19:22:21 -08:00
sabaimran
55200be4fa Apply agent color fill to the toggle both in off and on states 2024-11-10 19:16:43 -08:00
Debanjum
7468f6a6ed Deduplicate online references returned by chat API to clients
This will ensure only unique online references are shown in all
clients.

The duplication issue was exacerbated in research mode as even with
different online search queries, you can get previously seen results.

This change does a global deduplication across all online results seen
across research iterations before returning them in client reponse.
2024-11-10 16:10:32 -08:00
Debanjum
137687ee49 Deduplicate searches in normal mode & across research iterations
- Deduplicate online, doc search queries across research iterations.
  This avoids running previously run online, doc searches again and
  dedupes online, doc context seen by model to generate response.
- Deduplicate online search queries generated by chat model for each
  user query.
- Do not pass online, docs, code context separately when generate
  response in research mode. These are already collected in the meta
  research passed with the user query
- Improve formatting of context passed to generate research response
  - Use xml tags to delimit context. Pass per iteration queries in each
    iteration result
  - Put user query before meta research results in user message passed
    for generating response

This deduplications will improve speed, cost & quality of research mode
2024-11-10 16:10:32 -08:00
Debanjum
306f7a2132 Show error in picking next tool to researcher llm in next iteration
Previously the whole research mode response would fail if the pick
next tool call to chat model failed. Now instead of it completely
failing, the researcher actor is told to try again in next iteration.

This allows for a more graceful degradation in answering a research
question even if a (few?) calls to the chat model fail.
2024-11-10 14:52:02 -08:00
Debanjum
eb492f3025 Only keep webpage content requested, even if Jina API gets more data
Jina search API returns content of all webpages in search results.
Previously code wouldn't remove content beyond max_webpages_to_read
limit set. Now, webpage content in organic results aree explicitly
removed beyond the requested max_webpage_to_read limit.

This should align behavior of online results from Jina with other
online search providers. And restrict llm context to a reasonable size
when using Jina for online search.
2024-11-10 14:51:16 -08:00
Debanjum
8ef7892c5e Exclude non-dictionary doc context from chat history sent to chat models
This fixes chat with old chat sessions. Fixes issue with old Whatsapp
users can't chat with Khoj because chat history doc context was
stored as a list earlier
2024-11-10 14:51:16 -08:00
Debanjum
d892ab3174 Fix handling of command rate limit and improve rate limit messages
Command rate limit wouldn't be shown to user as server wouldn't be
able to handle HTTP exception in the middle of streaming.

Catch exception and render it as LLM response message instead for
visibility into command rate limiting to user on client

Log rate limmit messages for all rate limit events on server as info
messages

Convert exception messages into first person responses by Khoj to
prevent breaking the third wall and provide more details on wht
happened and possible ways to resolve them.
2024-11-10 14:51:16 -08:00
Debanjum
80ee35b9b1 Wrap messages in web, obsidian UI to stay within screen when long links
Wrap long links etc. in chat messages and train of thought lists on
web app app and obsidian plugin by breaking them into newlines by word
2024-11-10 14:49:51 -08:00
sabaimran
170d959feb Handle offline messages differently, as they don't respond well to the structured messages 2024-11-09 19:52:46 -08:00
sabaimran
2c543bedd7 Add typing to the constructed messages listed 2024-11-09 19:40:27 -08:00
sabaimran
79b15e4594 Only add images when they're present and vision enabled 2024-11-09 19:37:30 -08:00
sabaimran
bd55028115 Fix randint import from random when creating filenames for tmp 2024-11-09 19:17:18 -08:00
sabaimran
92b6b3ef7b Add attached files to latest structured message in chat ml format 2024-11-09 19:17:00 -08:00
sabaimran
835fa80a4b Allow docx conversion in the chatFunction.ts 2024-11-09 18:51:00 -08:00
sabaimran
459318be13 And random suffixes to decreases any clash probability when writing tmp files to disc 2024-11-09 18:46:34 -08:00
sabaimran
dbf0c26247 Remove _summary_ description in function descriptions 2024-11-09 18:42:42 -08:00
sabaimran
e5ac076fc4 Move construct_chat_history method back to conversation.utils.py 2024-11-09 18:27:46 -08:00
sabaimran
bc95a99fb4 Make tracer the last input parameter for all the relevant chat helper methods 2024-11-09 18:22:46 -08:00
sabaimran
ceb29eae74 Add phone number verification and remove telemetry update call from place where authentication middleware isn't yet installed (in the middleware itself). 2024-11-09 12:25:36 -08:00
sabaimran
3badb27744 Remove stored uploaded files after they're processed. 2024-11-08 23:28:02 -08:00
sabaimran
78630603f4 Delete the fact checker application 2024-11-08 17:27:42 -08:00
sabaimran
807687a0ac Automatically generate titles for conversations from history 2024-11-08 16:02:34 -08:00
sabaimran
7159b0b735 Enforce limits on file size when converting to text 2024-11-08 15:27:28 -08:00
sabaimran
4695174149 Add support for file preview in the chat input area (before message sent) 2024-11-08 15:12:48 -08:00
sabaimran
ad46b0e718 Label pages when extract text from pdf, docs content. Fix scroll area in doc preview. 2024-11-08 14:53:20 -08:00
sabaimran
ee062d1c48 Fix parsing for PDFs via content indexing API 2024-11-07 18:17:29 -08:00
sabaimran
623a97a9ee Merge branch 'master' of github.com:khoj-ai/khoj into features/include-full-file-in-convo-with-filter 2024-11-07 17:18:23 -08:00
sabaimran
33498d876b Simplify the share chat page. Don't need it to maintain its own conversation history
- When chatting on a shared page, fork and redirect to a new conversation page
2024-11-07 17:14:11 -08:00
sabaimran
4b8be55958 Convert UUID to string when forking a conversation 2024-11-07 17:13:04 -08:00
sabaimran
9bbe27fe36 Set default value of attached files to empty list 2024-11-07 17:12:45 -08:00
sabaimran
3a51996f64 Process attached files in the chat history and add them to the chat message 2024-11-07 16:06:58 -08:00
sabaimran
a89160e2f7 Add support for converting an attached doc and chatting with it
- Document is first converted in the chatinputarea, then sent to the chat component. From there, it's sent in the chat API body and then processed by the backend
- We couldn't directly use a UploadFile type in the backend API because we'd have to convert the api type to a multipart form. This would require other client side migrations without uniform benefit, which is why we do it in this two-phase process. This also gives us capacity to repurpose the moe generic interface down the road.
2024-11-07 16:06:37 -08:00
sabaimran
e521853895 Remove unnecessary console.log statements 2024-11-07 16:03:31 -08:00
sabaimran
92c3b9c502 Add function to get an icon from a file type 2024-11-07 16:02:53 -08:00
sabaimran
140c67f6b5 Remove focus ring from the text area component 2024-11-07 16:02:02 -08:00
sabaimran
b8ed98530f Accept attached files in the chat API
- weave through all subsequent subcalls to models, where relevant, and save to conversation log
2024-11-07 16:01:48 -08:00
sabaimran
ecc81e06a7 Add separate methods for docx and pdf files to just convert files to raw text, before further processing 2024-11-07 16:01:08 -08:00
sabaimran
394035136d Add an api that gets a document, and converts it to just text 2024-11-07 16:00:10 -08:00
sabaimran
3b1e8462cd Include attach files in calls to extract questions 2024-11-07 15:59:15 -08:00
sabaimran
de73cbc610 Add support for relaying attached files through backend calls to models 2024-11-07 15:58:52 -08:00
Debanjum
05a93fcbed v-align attach, send buttons with chat input text area on web app
Otherwise, those buttons look off-center when images are attached to
the chat input area
2024-11-05 17:10:53 -08:00
sabaimran
a0480d5f6c use fill weight for the toggle right (enabled state) for research mode 2024-11-04 22:01:09 -08:00
sabaimran
dc26da0a12 Add uploaded files in the conversation file filter for a new convo 2024-11-04 22:00:47 -08:00
Debanjum
b51ee644aa Fix escaping filename when normalizing in org node parser 2024-11-04 20:24:57 -08:00
Debanjum
5724d16a6f Fix passing images to anthropic chat models to extract questions 2024-11-04 20:24:57 -08:00
sabaimran
7543360210 Merge branch 'master' of github.com:khoj-ai/khoj into features/include-full-file-in-convo-with-filter 2024-11-04 16:55:48 -08:00
sabaimran
b6145df3be Handle file retrieval when agent is None 2024-11-04 16:55:22 -08:00
sabaimran
3dc9139cee Add additional handling for when file_object comes back empty 2024-11-04 16:53:07 -08:00
sabaimran
a27b8d3e54 Remove summarize condition for only 1 file filter 2024-11-04 16:51:37 -08:00
sabaimran
362bdebd02 Add methods for reading full files by name and including context
Now that models have much larger context windows, we can reasonably include full texts of certain files in the messages. Do this when an explicit file filter is set in a conversation. Do so in a separate user message in order to mitigate any confusion in the operation.

Pipe the relevant attached_files context through all methods calling into models.

We'll want to limit the file sizes for which this is used and provide more helpful UI indicators that this sort of behavior is taking place.
2024-11-04 16:37:13 -08:00
sabaimran
e3ca52b7cb Use .get() to get text accompanying image url, instead of subindexing 2024-11-04 16:09:16 -08:00
sabaimran
1e89baca7b Deprecate the UserSearchModelConfig and remove all references
- The server has moved to a model of standardization for the embeddings generation workflow. Remove references to the support for differentiated models.
- The migration script fo ra new model needs to be updated to accommodate full regeneration.
2024-11-04 12:24:41 -08:00
sabaimran
99c1d2831a Release Khoj version 1.28.3 2024-11-02 12:23:11 -07:00
sabaimran
075b4ecf15 Call subscription_to_state with sync_to_async wrapper when getting user subscription state
- This is needed in case the renewal_date is not set and we need to reset it for the user
2024-11-02 12:22:35 -07:00
sabaimran
ec44cbe1e7 Release Khoj version 1.28.2 2024-11-02 07:53:51 -07:00
Debanjum
31b5fde163 Only enable prompt tracer if git python is installed 2024-11-02 02:07:02 -07:00
sabaimran
5b18dc96e0 Release Khoj version 1.28.1 2024-11-01 22:51:51 -07:00
Debanjum
e85dd59295 Release Khoj version 1.28.0 2024-11-01 19:06:59 -07:00
Debanjum
14e453039d Add prompt tracing, agent personality to infer webpage urls chat actor 2024-11-01 18:12:50 -07:00
Debanjum
ab321dc518 Expect query before tool in response to give think space in research prompt 2024-11-01 17:51:41 -07:00
Debanjum
1a83bbcc94 Clean API chat router. Move FeedbackData response type to router helper 2024-11-01 17:51:41 -07:00
sabaimran
e6eb87bbb5 Merge branch 'improve-debug-reasoning-and-other-misc-fixes' of github.com:khoj-ai/khoj into improve-debug-reasoning-and-other-misc-fixes 2024-11-01 16:48:39 -07:00
sabaimran
a213b593e8 Limit the number of urls the webscraper can extract for scraping 2024-11-01 16:48:36 -07:00
sabaimran
327fcb8f62 create defiltered query after conversation command is extracted 2024-11-01 16:48:03 -07:00
sabaimran
b79a9ec36d Clarify description of the code evaluation environment: not for document creation 2024-11-01 16:47:27 -07:00
Debanjum
9c7b36dc69 Use standard per minute rate limits across user types 2024-11-01 16:16:06 -07:00
Debanjum
ac21b10dd5 Simplify logic to get default search model. Remove unused import 2024-11-01 15:14:00 -07:00
sabaimran
2b35790165 Merge branch 'master' of github.com:khoj-ai/khoj into improve-debug-reasoning-and-other-misc-fixes 2024-11-01 14:51:26 -07:00
sabaimran
baa939f4ce When running code, strip any code delimiters. Disable application json type specification in Gemini request. 2024-11-01 13:47:39 -07:00
sabaimran
8fd2fe162f Determine if research mode is enabled by checking the conversation commands and 'linting' them in the selection phase 2024-11-01 13:12:34 -07:00
sabaimran
cead1598b9 Don't reset research mode after completing research execution 2024-11-01 13:00:11 -07:00
Debanjum
c1c779a7ef Do not yaml format raw code results in context for LLM. It's confusing 2024-11-01 12:45:26 -07:00
sabaimran
b3dad1f393 Standardize rate limits to 1/6 ratio 2024-11-01 12:21:09 -07:00
Debanjum
cd75151431 Do not allow auto selecting research mode as tool for now.
You are required to manually turning it on. This takes longer and
should be a high intent activity initiated by user
2024-11-01 12:07:52 -07:00
Debanjum
0b0cfb35e6 Simplify in research mode check in api_chat.
- Dedent code for readability
- Use better name for in research mode check
- Continue to remove inferred summarize command when multiple files in
  file filter even when not in research mode
- Continue to show select information source train of thought.
  It was removed by mistake earlier
2024-11-01 12:07:08 -07:00
Debanjum
73750ef286 Merge branch 'master' into features/advanced-reasoning 2024-11-01 11:42:01 -07:00
sabaimran
1fc280db35 Handle case where infer_webpage_url returns no valid urls 2024-11-01 11:41:32 -07:00
Debanjum
1c920273dd
Add Prompt Tracer to Visualize, Analyze and Debug Khoj's Train of Thought (#951)
## Overview
Use git to capture prompt traces of khoj's train of thought. View, analyze and debug them using your favorite git client (e.g vscode, magit).

- Each commit captures an interaction with an LLM
  The commit writes the query, response and system message each to a separate file in the repo.
  The commit message captures the chat model, Khoj version and other metadata
- Each conversation turn can have multiple interactions with an LLM (e.g Khoj's train of thought)
- Each new conversation turn forks from and merges back into its conversation branch
- Each new conversation branches from the user branch
- Each new user branches from root commit on the main branch

## Usage
1. Set `KHOJ_DEBUG=true` or start khoj in very verbose mode with `khoj -vv` to turn on prompt tracing
2. Chat with Khoj as usual 
3. Open the promptrace git repo to view the generated prompt traces using your favorite git porcelain. 
   The Khoj prompt trace git repo is created at `/tmp/khoj_promptrace` by default. You can configure the prompt trace directory by setting the `PROMPTRACE_DIR`environment variable.

## Implementation
- Add utility functions to capture prompt traces using git (via `gitpython`)
- Make each model provider in Khoj commit their LLM interactions with promptrace
- Weave chat metadata from chat API through all chat actors and commit it to the prompt trace
2024-11-01 11:33:54 -07:00
sabaimran
33d36ee58c Add experimental notice to research mode tooltip 2024-11-01 11:00:27 -07:00
sabaimran
0145b2a366 Set usage limits on the research mode 2024-11-01 10:29:33 -07:00
sabaimran
3ea94ac972 Only include inferred-queries in chat history when present 2024-10-31 22:01:41 -07:00
sabaimran
149cbe1019 Use bottom anchor for the commandbar popover 2024-10-31 20:40:38 -07:00
sabaimran
21858acccc Remove conversation command always in query, filter out inferred queries that were not with selected tool when going through tool selection iterations 2024-10-31 20:27:38 -07:00
sabaimran
19241805ee Merge branch 'master' of github.com:khoj-ai/khoj into improve-debug-reasoning-and-other-misc-fixes 2024-10-31 18:20:23 -07:00
Debanjum
302bd51d17 Improve online chat actor prompt for research and normal mode
- Match the online query generator prompt to match the formatting of
  extract questions
- Separate iteration results by newline
- Improve webpage and online tool descriptions
2024-10-31 18:17:12 -07:00
Debanjum
52163fe299 Improve research planner prompt to reduce looping 2024-10-31 18:17:01 -07:00
sabaimran
7ebf999688 Merge branch 'master' of github.com:khoj-ai/khoj into features/advanced-reasoning 2024-10-31 18:15:13 -07:00
sabaimran
159ea44883 Remove frame references in the diagramming prompts 2024-10-31 18:14:51 -07:00
Debanjum
89597aefe9 Json dump contents in prompt tracer to make structure discernable 2024-10-31 18:08:42 -07:00
Debanjum
5b15176e20 Only add /research prefix in research mode if not already in user query 2024-10-31 18:08:42 -07:00
sabaimran
559601dd0a Do not exit if/else loop in research loop when notes not found 2024-10-31 13:51:10 -07:00
sabaimran
a13760640c Only show trash can when turnId is present 2024-10-31 13:19:16 -07:00
Debanjum
adca6cbe9d Merge branch 'master' into add-prompt-tracer-for-observability 2024-10-31 02:28:34 -07:00
Debanjum
e17dc9f7b5 Put train of thought ui before Khoj response on web app 2024-10-31 02:24:53 -07:00
Debanjum
e8e6ead39f Fix deleting new messages generated after conversation load 2024-10-30 20:56:38 -07:00
Debanjum
cb90abc660 Resolve train of thought component needs unique key id error on web app 2024-10-30 14:00:21 -07:00
Debanjum
ca5a6831b6 Add ability to delete messages from the web app 2024-10-30 14:00:21 -07:00
Debanjum
ba15686682 Store turn id with each chat message. Expose API to delete chat turn
Each chat turn is a user query, khoj response message pair
2024-10-30 14:00:21 -07:00
Debanjum
f64f5b3b6e Handle add/delete file filter operation on non-existent conversation 2024-10-30 14:00:21 -07:00
Debanjum
b3a63017b5 Support setting seed for reproducible LLM response generation
Anthropic models do not support seed. But offline, gemini and openai
models do. Use these to debug and test Khoj via KHOJ_LLM_SEED env var
2024-10-30 14:00:21 -07:00
Debanjum
d44e68ba01 Improve handling embedding model config from admin interface
- Allow server to start if loading embedding model fails with an error.
  This allows fixing the embedding model config via admin panel.

  Previously server failed to start if embedding model was configured
  incorrectly. This prevented fixing the model config via admin panel.

- Convert boolean string in config json to actual booleans when passed
  via admin panel as json before passing to model, query configs

- Only create default model if no search model configured by admin.
  Return first created search model if its been configured by admin.
2024-10-30 14:00:21 -07:00
Debanjum
358a6ce95d Defer turning cursor color to selected agents color for later
Capability exists but idea needs to be investigated further
2024-10-30 14:00:21 -07:00
Debanjum
2ac840e3f2 Make cursor in chat input take on selected agent color 2024-10-30 14:00:21 -07:00
Debanjum
1448b8b3fc Use 3rd person for user in research prompt to reduce person confusion
Models were getting a bit confused about who is search for who's
information. Using third person to explicitly call out on who's behalf
these searches are running seems to perform better across
models (gemini's, gpt etc.), even if the role of the message is user.
2024-10-30 13:49:48 -07:00
Debanjum
b8c6989677 Separate example from actual question in extract question prompt 2024-10-30 13:49:48 -07:00
Debanjum
86ffd7a7a2 Handle \n, dedupe json cleaning into single function for reusability
Use placeholder for newline in json object values until json parsed
and values extracted. This is useful when research mode models outputs
multi-line codeblocks in queries etc.
2024-10-30 13:49:48 -07:00
Debanjum
83ca820abe Encourage Anthropic models to output json object using { prefill
Anthropic API doesn't have ability to enforce response with valid json
object, unlike all the other model types.

While the model will usually adhere to json output instructions.

This step is meant to more strongly encourage it to just output json
object when response_type of json_object is requested.
2024-10-30 13:49:48 -07:00
Debanjum
dc8e89b5de Pass tool AIs iteration history as chat history for better context
Separate conversation history with user from the conversation history
between the tool AIs and the researcher AI.

Tools AIs don't need top level conversation history, that context is
meant for the researcher AI.

The invoked tool AIs need previous attempts at using the tool in this
research runs iteration history to better tune their next run.

Or at least that is the hypothesis to break the models looping.
2024-10-30 13:49:48 -07:00
Debanjum
d865994062 Rename code tool arg previous_iteration_history' to context' 2024-10-30 13:49:48 -07:00
Debanjum
06aeca2670 Make researcher, docs search AIs ask more diverse retrieval questions
Models weren't generating a diverse enough set of questions. They'd do
minor variations on the original query. What is required is asking
queries from a bunch of different lenses to retrieve the requisite
information.

This prompt updates shows the AIs the breadth of questions to by
example and instruction. Seem like performance improved based on vibes
2024-10-30 13:49:48 -07:00
Debanjum
01881dc7a2 Revert "Make extract question prompt in 1st person wrt user as its a user message"
This reverts commit 6d3602798aa1b95a30c557576fd4f93ddef2ae76.
2024-10-30 13:49:48 -07:00
Debanjum
3e695df198 Make extract question prompt in 1st person wrt user as its a user message
Divide Example from Actual chat history section in prompt
2024-10-30 13:49:48 -07:00
Debanjum
a3751d6a04 Make extract relevant information system prompt work for any document
Previously it was too strongly tuned for extracting information from
only webpages. This shouldn't be necessary
2024-10-30 13:49:48 -07:00
Debanjum
a39e747d07 Improve passing user name in pick next research tool prompt 2024-10-30 13:49:48 -07:00
Debanjum
deff512baa Improve research mode prompts to reduce looping, increase webpage reads 2024-10-30 13:49:48 -07:00
Debanjum
d3184ae39a Simplify storing and displaying document results in research mode
- Mention count of notes and files disovered
- Store query associated with each compiled reference retrieved for
  easier referencing
2024-10-30 13:49:48 -07:00
Debanjum
8bd94bf855 Do not use a message branch if no msg id provided to prompt tracer 2024-10-30 13:49:48 -07:00
sabaimran
b63fbc5345 Add a simple badget to the dropdown menu that shows subscription status 2024-10-30 13:00:16 -07:00
sabaimran
82f3d79064 Merge branch 'master' of github.com:khoj-ai/khoj into features/advanced-reasoning 2024-10-30 11:32:10 -07:00
sabaimran
2b2564257e Handle subscription case where it's set to trial, but renewal_date is not set. set the renewal_date for LENGTH_OF_FREE_TRIAL days from subscription creation. 2024-10-30 11:05:31 -07:00
Debanjum
9935d4db0b Do not use a message branch if no msg id provided to prompt tracer 2024-10-28 17:50:27 -07:00