Commit graph

4149 commits

Author SHA1 Message Date
Debanjum Singh Solanky
082fe937b9 Reduce Windows App Size by Removing Unused Libraries under Torch/Lib
Tighten the duplicate library removal code in Khoj.spec
2022-08-18 11:28:28 +03:00
Debanjum
b78ee317ae
Reduce Debian, Mac App Size. Remove unused libraries under Torch/Lib
## Changes
- On **Debian**
  - `libtorch_cuda.so` (1Gb) and `libtorch_cpu.so` (700Mb) are large shared libs
  - They are available at package root and under `torch/lib` directory in the package
  - We remove the unused, duplicate libraries from under `torch/lib` as only the top level libraries are used
- On **Mac**
  - Remove `libtorch_{cpu,python}.dylib` under `torch/lib` directory from the Mac app.

## App Size Reduction
  - **Debian amd64** app size by **42%** from **1.6Gb to 920Mb**  
  - **Mac arm64** app by **15%** from **190Mb to 160Mb**
  - **Mac amd64** app by **33%** from **340Mb to 230Mb**

## Reference
- [Release Workflow Run](https://github.com/debanjum/khoj/actions/runs/2878104171) after changes
- [Release Workflow Run](https://github.com/debanjum/khoj/actions/runs/2869906116) before changes
2022-08-17 20:48:56 +00:00
Debanjum Singh Solanky
d25ddb93f7 Fix missing closing bracket from SOURCE_DATE_EPOCH def in release.yml 2022-08-17 23:17:27 +03:00
Debanjum Singh Solanky
9ee02b0804 Add --noconfirm in call to pyinstaller from Github release workflow
Added just for safety, workflow works fine without it too
2022-08-17 23:04:26 +03:00
Debanjum Singh Solanky
7cf345a138 Exclude unused mac libs under torch/lib. Reduce Mac app size by 30Mb
libtorch_cuda only seems to be imported in Linux. Which is why the
size of the Mac, Windows apps are 700Mb smaller than the Debian app size.

Guessing this is because libtorch_cuda only works on Linux machines?

Anyway, removing libtorch_{cpu,python}.dylib under torch/lib from the
Mac app reduces it's size from 190Mb to 160Mb. 15% reduction isn't too bad
2022-08-17 22:59:01 +03:00
Debanjum Singh Solanky
0273be0232 Exclude unused libs under torch/lib. Reduce Debian package size by 700Mb
- libtorch_cuda.so (1Gb) and libtorch_cpu.so (700Mb) are large shared
  libs that are available at package root and under torch/lib.
- The top level imports are used, so they unused libs are removed from
  package
- This reduces the single file package size from 1.6Gb to 920Mb
2022-08-17 22:32:58 +03:00
Debanjum Singh Solanky
5a20283202 Set Pyinstaller, Pip environment to create reproducible builds of Khoj
- Dependency Version Pinning
  - First level dependency versions have been pinned.
  - Transitive dependencies have not been specified yet

- Testing
  - The Pyinstaller build has been only minimally tested for reproducibility
  - The Khoj package generated for PyPi have not been tested for reproducibility

- References
  - https://reproducible-builds.org/docs/source-date-epoch/
  - https://pyinstaller.org/en/stable/advanced-topics.html#creating-a-reproducible-build
2022-08-17 20:09:35 +03:00
Debanjum Singh Solanky
f821b614ab Fix path to Khoj executable in Khoj.desktop for Debian package 2022-08-17 19:52:35 +03:00
Debanjum Singh Solanky
43049f761b Fix query performance numbers in Readme 2022-08-17 18:32:55 +03:00
Debanjum Singh Solanky
a37724f338 Fix Debian package permission. Set version on manual workflow trigger 2022-08-16 20:30:46 +03:00
Debanjum Singh Solanky
4d83a1d13b Bump Khoj pip version to 0.1.6 to publish pre-release builds 2022-08-16 17:37:41 +03:00
Debanjum Singh Solanky
3d0f979475 Make the .deb package use version in Debian package versioning format 2022-08-16 17:35:14 +03:00
Debanjum Singh Solanky
3000b91297 Fix and Update Readme. Delete old Demo from Repository 2022-08-16 16:57:56 +03:00
Debanjum Singh Solanky
c4fd661909 Move the experimental /chat API to under /beta/chat 2022-08-16 16:36:15 +03:00
Debanjum
a482c2a8b0
Update Demo with Install, Configure and Search shown
Demo contains:
- Setup using pip
- Configure using Desktop GUI
- Search via Web and Emacs interfaces
2022-08-15 16:15:43 -07:00
Debanjum
7fc8672666
Improve Desktop GUI and Documentation
- Improve Documentation
  - 7866add Add Interface Screenshots to Docs
  - 8ad3482 Update Readme Instructions to use Desktop GUI to configure App
- Fix Markdown Search Bug on Backend
  - b891347 Fix condition in router to trigger markdown search
- Improve Desktop GUI
  - 67ab40b Regenerate embeddings everytime user clicks configure in Desktop GUI
  - 7f479b0 Improve Displaying Error to User on Khoj window in Desktop GUI
  - 873bb9d Do not force the Khoj window to always be on top. It's needlessly annoying
  - 9bc4fd5 Set Web Interface URL from loaded state in Desktop GUIs. Not hard-coded
2022-08-15 23:01:37 +00:00
Debanjum Singh Solanky
35ce511709 Add Interface Screenshots to Readme 2022-08-16 01:59:29 +03:00
Debanjum Singh Solanky
342c72b156 Update Readme Instructions to use Desktop GUI to configure App
- Configure app using the configure screen during first run.
  No config file args required to be passed via CLI by users

- Update instructions to copy khoj_sample.yml to default app configure
  file location and edit that. Instead of editing the khoj_sample.yml
  directly in source
2022-08-16 01:59:29 +03:00
Debanjum Singh Solanky
b8913476ba Fix if condition in router to trigger markdown search 2022-08-16 00:37:16 +03:00
Debanjum Singh Solanky
9bc4fd539e Set Web Interface URL from loaded state in Desktop GUIs. Not hard-coded 2022-08-16 00:37:16 +03:00
Debanjum Singh Solanky
7f479b0104 Improve Displaying Error to User on Khoj window in Desktop GUI
- Show a helpful error message in the GUI to the user, instead of the
  crashing if loading config fails, for e.g if file wasn't found
- Collate GUI errors into an ErrorType enum class
- Remove previous error messages before showing the new one
2022-08-16 00:37:16 +03:00
Debanjum Singh Solanky
873bb9dd97 Do not force the Khoj window to always be on top. It's needlessly annoying 2022-08-16 00:37:16 +03:00
Debanjum Singh Solanky
67ab40bb01 Regenerate embeddings everytime user clicks configure in Desktop GUI
Previously if the embeddings were already there only the khoj.yml
config file would get updated. The embeddings would remain old.

1. This results in a stale app state where the config doesn't
   match the embeddings

2. Currently the user cannot update their config from the config
   screen. They'd have to use a combination of config screen and web
   interface>regenerate button to trigger it or delete their ~/.khoj dir

This commit should resolve the above issues
2022-08-16 00:37:16 +03:00
Debanjum Singh Solanky
2647e6bab4 Display re-ranked results triggered via keybinding in khoj.el
- Prevent immediate overwrite of re-ranked results by
  incremental-search without rerank triggered via post-command-hook.

- This triggers right after the reranking results are rendered, so
  user never ends up seeing them
2022-08-15 18:41:12 +03:00
Debanjum Singh Solanky
7421ef2724 Fix path of app artifacts to attach to release via release pipeline 2022-08-15 17:41:08 +03:00
Debanjum Singh Solanky
c21ab4316c Fix publish of pip package to PyPi on git tag 2022-08-15 06:55:00 +03:00
Debanjum Singh Solanky
237e207304 Create separate app artifacts for each operating system 2022-08-15 06:39:45 +03:00
Debanjum Singh Solanky
a91d2df300 Simplify Emacs interface to only rerank results on explicit command 2022-08-15 06:20:13 +03:00
Debanjum Singh Solanky
e846829a2e Reset Khoj.el version to align with Khoj package version 2022-08-15 06:20:13 +03:00
Debanjum Singh Solanky
4308f51d2c Fix workflow_dispatch trigger name used in github release workflow 2022-08-15 06:09:54 +03:00
Debanjum
a26f7f4716
Merge pull request #66 from debanjum/use-venv-to-reduce-debian-app-size
Improve Release Workflow, Automatically Publish Artifacts to Release
2022-08-15 03:04:18 +00:00
Debanjum Singh Solanky
14710da962 Add version, arch to app name. Publish artifacts to release
- Allow manual trigger of workflow
2022-08-15 06:01:47 +03:00
Debanjum Singh Solanky
2142cba627 Simplify upload artifacts to single action 2022-08-15 05:11:37 +03:00
Debanjum
778fb724f9
Create Github Workflow to Wrap Khoj into OS Specific Apps
- Why
  - Simplify Installation of Khoj by providing OS specific apps
- What
  - Create Github workflow to wrap Khoj into
    - [X] A Mac .app Disk Image
    - [X] A Windows .exe app
    - [X] A Linux .deb package

Improves #56 
Closes #64, #65
2022-08-14 07:28:42 +03:00
Debanjum Singh Solanky
a44cf39ab9 Limit release workflow to only run on creation of git tags
This reverts commit 7eb3dc71de.
2022-08-14 07:27:33 +03:00
Debanjum Singh Solanky
7eb3dc71de Revert "Revert release workflow to only run on creation of git tags"
This reverts commit 34cfc0382b.
2022-08-14 06:49:01 +03:00
Debanjum Singh Solanky
62711408b0 Run Install Deps step in bash shell to avoid powershell on windows 2022-08-14 06:47:58 +03:00
Debanjum Singh Solanky
34cfc0382b Revert release workflow to only run on creation of git tags 2022-08-14 06:45:17 +03:00
Debanjum Singh Solanky
357b584142 Re-enable windows, macos in matrix of release PR 2022-08-14 06:41:54 +03:00
Debanjum Singh Solanky
19294c788a Install libxcb on ubuntu as well 2022-08-14 06:30:00 +03:00
Debanjum Singh Solanky
4fd590addd Install libegl1 as root 2022-08-14 06:28:12 +03:00
Debanjum Singh Solanky
f3be84d8b9 Fix fpm command and install libegl1 on linux runner 2022-08-14 06:24:16 +03:00
Debanjum Singh Solanky
c422967276 Setup Ruby properly to add FPM Ruby package to PATH 2022-08-14 06:08:18 +03:00
Debanjum Singh Solanky
8696032b40 Set permission of the single Khoj app file generated by Pyinstaller 2022-08-14 05:46:52 +03:00
Debanjum Singh Solanky
47bac4f61e Only run ubuntu in OS matrix. Speeds verifying debian package creation 2022-08-14 05:27:02 +03:00
Debanjum Singh Solanky
7ce9c3ecce Re-enable running workflow via PR temporarily 2022-08-14 05:08:59 +03:00
Debanjum Singh Solanky
fed0b591af Package Khoj as Debian app in Github Release Workflow 2022-08-14 05:07:58 +03:00
Debanjum Singh Solanky
d815e0d115 Run Release Workflow on pushing git tag. Add release badge to Readme 2022-08-14 04:21:11 +03:00
Debanjum Singh Solanky
3bc4d75ba9 Consolidate app creation using pyinstaller into a single step 2022-08-14 04:10:26 +03:00
Debanjum Singh Solanky
0a0a8d43e0 Install Mac specific DMG creator only on the Mac Runner 2022-08-14 03:49:40 +03:00