Commit graph

4116 commits

Author SHA1 Message Date
Debanjum Singh Solanky
78bac4ae05 Add migration script to convert PNG to WebP references in database 2024-04-13 19:06:28 +05:30
Debanjum Singh Solanky
c6e8443631 Update clients to support rendering webp images inline
This is for self-hosted scenarios where AWS S3 uploads is not enabled
2024-04-13 13:11:18 +05:30
Debanjum Singh Solanky
d21f22ffa1 Store Khoj generated images as webp instead of png for faster loading 2024-04-13 13:03:32 +05:30
Debanjum Singh Solanky
b820daf38f Makes logs less noisy
- Show telemetry enabled/disabled state on init, not every 2 minutes
- Convert no docs synced logs to debug level instead of warning
  Having synced docs isn't as important to use Khoj now, unlike before
2024-04-13 11:22:58 +05:30
Debanjum Singh Solanky
b8bc6bee83 Always remove loading animation on Desktop app if can't login to server 2024-04-13 11:02:44 +05:30
Debanjum Singh Solanky
382507051f Fix get_user_photo to only return photo, not user name from DB 2024-04-13 11:02:30 +05:30
sabaimran
f06ec485cb Fix redirect url process for login flow, existing user 2024-04-12 17:10:05 +05:30
sabaimran
87b9a93fa1 Update assertion line to match new logic 2024-04-12 13:09:19 +05:30
sabaimran
b86e68a29d Make it easier to view agents in the admin page 2024-04-12 13:02:22 +05:30
sabaimran
e58bd0e485 Remove mbox file from list of files expected to be included 2024-04-12 12:55:22 +05:30
sabaimran
6634d603a8 Add links for contributors to use in the readme 2024-04-12 12:49:12 +05:30
sabaimran
1377a44a1a Suppress debug logs from uvicorn.error to avoid clutter from websockets
- If application is not in DEBUG_MODE
2024-04-12 12:12:16 +05:30
Debanjum Singh Solanky
89b8ec3546 Release Khoj version 1.10.2 2024-04-12 11:53:32 +05:30
Debanjum Singh Solanky
50b4788a91 Remove chat loading animation in login required state on Desktop app 2024-04-12 11:50:54 +05:30
Debanjum Singh Solanky
b3f4794d91 Remove the unnecessary async/await func chains on Desktop app 2024-04-12 11:49:25 +05:30
Debanjum Singh Solanky
1e30a072d4 Just use file ext to identify indexable files to fix Desktop app install
- Magika on Desktop app was too bloated (100Mb to 250Mb) and broke
  install for some reason. Not sure why it was causing the app install
  to fail but do not have time to currently investigate

- Just use file extensions whitelist it's good enough for now. Let
  server handle the deeper identification of file type
2024-04-12 11:16:07 +05:30
Debanjum Singh Solanky
5c7797dbca Only check content type if file extension cannot identify text file 2024-04-12 03:40:42 +05:30
Debanjum Singh Solanky
7d2ef728e6 Fix identifying pdf files on server
Introduced bug in previous commit that would stop indexing PDF files
as trying to check content_group instead of mime_type is application/pdf
2024-04-12 03:07:46 +05:30
Debanjum Singh Solanky
07f8fb5c5b Release Khoj version 1.10.1 2024-04-12 02:18:07 +05:30
Debanjum Singh Solanky
a7d9102c33 Make identifying text, code files with Magika more robust on server
Use identified content group rather than mime_type to find text files.
2024-04-12 02:12:26 +05:30
Debanjum Singh Solanky
60337086f9 Release Khoj version 1.10.0 2024-04-12 01:01:02 +05:30
Debanjum Singh Solanky
34c3f70203 Index only files with valid text extension in folders synced by Desktop app
This maintains consistent set of indexable files from Desktop app,
whether indexing via file or folder filters
2024-04-12 00:59:54 +05:30
Debanjum
9a48f72041
Index more text file types from Desktop, Github (#692)
### Index more text file types 
- Index all text, code files in Github repos. Not just md, org files
- Send more text file types from Desktop app and improve indexing them
- Identify file type by content & allow server to index all text files

### Deprecate Github Indexing Features
- Stop indexing commits, issues and issue comments in a Github repo
- Skip indexing Github repo on hitting Github API rate limit

### Fixes and Improvements
- **Fix indexing files in sub-folders from Desktop app**
- Standardize structure of text to entries to match other entry processors
2024-04-12 00:08:29 +05:30
Debanjum Singh Solanky
0819b83d0b Fix constructing status update strings for intermediate chat steps 2024-04-11 20:31:32 +05:30
Debanjum Singh Solanky
d15b9bc272 Tell doc search actor to not generate online queries for doc search
This can pick up irrelevant details from notes
2024-04-11 19:49:41 +05:30
Debanjum Singh Solanky
15a78b19ad Improve Inferred Document Search Query Extraction from GPT
Using stop_words = "\n" was preventing JSON responses with newlines in
them
2024-04-11 19:24:04 +05:30
Debanjum Singh Solanky
653681967e Show inferred document search queries in intermediate chat step on Web app 2024-04-11 19:24:04 +05:30
Debanjum Singh Solanky
997741119a Show better intermediate steps when responding to chat via web socket
- Show internet search, webpage read, image query, image generation steps
- Standardize, improve rendering of the intermediate steps on the web app

Benefits:
1. Improved transparency, allow users to see what Khoj is doing behind
   the scenes and modify their query patterns to improve response quality
2. Reduced websocket connection keep alive timeouts for long running steps
2024-04-11 18:04:40 +05:30
sabaimran
fae7900f19 Remove more 2024-04-11 00:27:44 +05:30
sabaimran
5d1dd3e2b7 If resend not enabled, don't send the welcome email 2024-04-10 23:52:42 +05:30
sabaimran
d2f9c43c8e Use datetime.timezone.utc instead of datetime.utc 2024-04-10 23:07:43 +05:30
Debanjum Singh Solanky
f2dc9709b7 Use Magika to more robustly identify text files to send for indexing
- `file-type' doesn't handle mis-labelled files or files without
   extensions well

- Only show supported file types in file selector dialog on Desktop app
  Use Magika to get list of text file extensions. Combine with other
  supported extensions to get complete list of supported file extensions.
  Use it to limit selectable files in the File Open dialog.

  Note: Folder selector will index text files with no extensions as well
2024-04-10 22:44:24 +05:30
sabaimran
3fe94a67b0
Send welcome emails when a new user signs up (#691)
* Don't trigger any re-indexing on server initailization

* Integrate Resend to send welcome emails when a new user signs up

- Only send if this is the first time they've signed in
- Configure welcome email with basic styling, as more complex designs don't work and style tag did not work
2024-04-10 19:57:33 +05:30
Debanjum
6d153022f6
Improve nav pane, chat session UI on Desktop, Web app (#693)
### Enable copying chat messages. Improve copy button behavior and styling
  - Add button to copy chat messages on Desktop, Web apps
  - Improve copy button's icon, hover color & click animation in Desktop, Web apps

### Improve Navigation, Chat Session Panes on Desktop, Web apps
  - Dynamically generate navigation menu based on user info from server
  - Create API endpoint to get authenticated user information
  - Collapse navigation tabs into icons on mobile. Add spacing to them
  - Add Chat navigation tab back to top pane on Web app
  - Use proper icons for Search, Chat and Agents tab on navigation pane
  
### Miscellaneous Improvements
  - Make current chat expand to full width when session panel collapsed on Desktop App
  - Add chat session loading spinner to Desktop App (same as Web app)
   
### Fixes 
  - Show title bar in Khoj desktop app on Windows to simplify close, minimize etc.
  - Only render first run setup message once if error or server not running
  - Fix showing Search navigation tab from Agent pages on web client
2024-04-10 19:54:12 +05:30
Debanjum Singh Solanky
48d249db9e Center the nav item text and user profile initial icons 2024-04-10 19:38:43 +05:30
Debanjum Singh Solanky
60f6a1c6f1 Use svg icons in nav pane to standardize styling on Web, Desktop apps
Emojis varied based on device. svg icons standardize icon styles of
the web, desktop apps
2024-04-10 19:38:43 +05:30
Debanjum Singh Solanky
cccea484e4 Pass username, location context in system prompt instead of chat message
The username and location in system prompt should disambiguate user
context from user's actual message for the chat model.

It doesn't need to be told to not mention the context or acknowledge
the context instructions in it's response, as it understands that this
information is just context and not part of the user's actual message.
2024-04-10 15:05:33 +05:30
Debanjum Singh Solanky
804c04f7b9 Do not render copy message button on every Khoj thinking step
Only render copy chat message button once, after message text is rendered
2024-04-10 14:48:36 +05:30
sabaimran
bb15c9605d Add a sitemap plugin 2024-04-10 14:35:04 +05:30
sabaimran
a4afada746 Remove client-side timeouts for the khoj socket 2024-04-10 13:35:25 +05:30
Debanjum Singh Solanky
cadeaac769 Align conversation sessions side panel on Desktop app with Web app
- Move new conversation button to right of "Conversation" title
- Reduce size of chat message loading ellipsis animation
- Add loading animation for chat session
2024-04-10 10:34:36 +05:30
Debanjum Singh Solanky
1c3d129e08 Add button to copy chat messages on Desktop client 2024-04-10 10:34:36 +05:30
Debanjum Singh Solanky
0a5a91619e Improve copy button's icon, hover color & click animation in Desktop UI 2024-04-10 10:34:36 +05:30
Debanjum Singh Solanky
184873213c Add button to copy chat messages on Web client 2024-04-10 10:34:36 +05:30
Debanjum Singh Solanky
f56522cb8e Improve copy button's icon, hover color & click animation in Web UI 2024-04-10 10:34:36 +05:30
Debanjum Singh Solanky
8ff3890ba8 Dynamically generate navigation menu based on user info from server 2024-04-10 10:34:36 +05:30
Debanjum Singh Solanky
94c69eb8e3 Create API endpoint to get authenticated user information
This help clients render UI with user information
2024-04-09 21:04:44 +05:30
Debanjum Singh Solanky
377e979800 Make current chat expand to full width when session panel collapsed
This behavior also matches web client behavior on chat session panel
collapse
2024-04-09 21:04:44 +05:30
Debanjum Singh Solanky
913dcdfbcd Only render first run setup message once if error or server not running 2024-04-09 21:04:44 +05:30
Debanjum Singh Solanky
3b630841bd s/aget_all_filenames_by_source/get_all_filenames_by_source as sync func 2024-04-09 21:04:44 +05:30