sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-18 10:37:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 23
3381 commits 9 branches 114 tags 135 MiB
Commit graph

157 commits

Author SHA1 Message Date
sabaimran
8b2af0b5ef
Add support for our first Local LLM 🤖🏠 (#330) 
* Add support for gpt4all's falcon model as an additional conversation processor
- Update the UI pages to allow the user to point to the new endpoints for GPT
- Update the internal schemas to support both GPT4 models and OpenAI
- Add unit tests benchmarking some of the Falcon performance
* Add exc_info to include stack trace in error logs for text processors
* Pull shared functions into utils.py to be used across gpt4 and gpt
* Add migration for new processor conversation schema
* Skip GPT4All actor tests due to typing issues
* Fix Obsidian processor configuration in auto-configure flow
* Rename enable_local_llm to enable_offline_chat
 2023-07-26 16:27:08 -07:00
Debanjum Singh Solanky
c28755ccd2 Fix diff blocks, links, remove footnotes & rearrange sections in docs 
Extract performance into separate sectin into shoving it under search
Create page for web interface
 2023-07-21 00:58:30 -07:00
sabaimran
c2249eadb2
Add a Github workflow that allows you to build dev versions of Desktop applications (#309) 
* Add a Github workflow that allows you to build dev versions of Desktop applications
* Add pull_request trigger for testing
* Fix errant open quote in Package Khoj App step
* Nix the release step, since this isn't associated with any tags
- Set retention period for uploaded artifacts to 1 day
* Remove pull_request trigger - limit to manual triggers and pushes to master
 2023-07-13 22:11:39 -07:00
Debanjum Singh Solanky
f664a74e77 Update Khoj server to run on non standard port, 42110 instead of 8000 
Resolves #295
 2023-07-10 21:27:58 -07:00
Debanjum Singh Solanky
2fbc609233 Add content write permission to jobs in github release workflow 2023-07-01 06:23:45 -07:00
Debanjum Singh Solanky
51dfa48e2b Have Khoj support Python 3.11 as Pytorch supports it now 
- Previously Khoj could only support Python upto 3.10 due to pytorch.
  But lots of folks had python 3.11 installed by default on their machines.

  This required installing python 3.10 and dealing with virtual envs.

  With Torch >= 2.0.1 now able to support python 3.11, at least one
  class of installation troubles for Khoj should drop. See
  https://github.com/pytorch/pytorch/issues/86566 for reference

- Preliminary testing indicates using the new torch 2.x may reduce
  search time by 25% (from 80ms to 60ms on Mac M1)

- Update Docs to not require mentioning python <=3.10 required
- Update Github test workflow to run khoj tests with python 3.11 too
 2023-06-29 15:13:26 -07:00
Debanjum Singh Solanky
69d4fa6525 Rename project links across repo from debanjum/khoj to khoj-ai/khoj 2023-06-21 00:13:21 -07:00
Debanjum Singh Solanky
8617cf1389 Push telemetry to Posthog to grok Khoj usage 2023-06-05 22:47:49 +05:30
Debanjum Singh Solanky
55d72231b3 Generate docker image for telemetry server using Github workflow 2023-05-17 16:08:21 +05:30
Debanjum Singh Solanky
35aa06067f Release Khoj version 0.6.0 
Upload styles.css via release workflow
 2023-03-31 18:13:16 +07:00
Debanjum Singh Solanky
e9ca04af0d Require dash, org to run ERT tests for khoj.el 2023-03-23 01:46:26 +04:00
Debanjum Singh Solanky
ed177db2be Emojify step names in workflows. Stop publishing to TestPyPi from PR 2023-03-01 10:56:39 -06:00
Debanjum Singh Solanky
47c2cc63e1 Automate uploading Obsidian artifacts to new releases 2023-02-17 19:57:44 -06:00
Debanjum Singh Solanky
a8940462c4 Automate khoj python package versioning using hatch-vcs and Git tags 2023-02-17 18:19:01 -06:00
Debanjum Singh Solanky
fd0a2f55f8 Run mypy checks in test workflow and on push (via pre-commit) 
- Run mypy on git push (not every commit) but for all files
  - Running it on pre-commit, doesn't make sense as mypy wants to look
    at all files, not just diff files
  - But this is too time consuming to run every commit, so run on push

- Update development section documentation on installing, manually
  running pre-commit for validation that includes running mypy checks
 2023-02-17 16:08:56 -06:00
Debanjum Singh Solanky
051f0e3fb5 Add, configure and run pre-commit locally and in test workflow 2023-02-17 13:31:36 -06:00
Debanjum Singh Solanky
6130fddf45 Install pytest as optional dev dependency of app in test workflow 2023-02-17 10:11:57 -06:00
Debanjum Singh Solanky
7a9a811874 Fix authors, homepage URL in pyproject.toml and workflow triggers 2023-02-16 03:19:56 -06:00
Debanjum Singh Solanky
dcb86c2d3e Build khoj python package using hatchling, pyproject.toml 
- Why
  - pyprojects.toml is the python standards compliant config format
    - allows collating python tooling configs into single standard file
  - hatch(-ling) is a new lightweight build system for python packages

- Detailed Changes
  - Replace setup.py, setuptools with pyproject.toml, hatchling for
    khoj python config and build
  - move pytest into optional development dependencies
  - add more links to khoj in the project urls section
  - add topic classifiers and keywords to find khoj package

  - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup
  - Update pypi workflow to set python package version in pyproject.toml
 2023-02-16 02:37:32 -06:00
Debanjum Singh Solanky
a403def19e Fix workflow to publish Khoj python package to PyPi 2023-02-14 22:19:21 -06:00
Debanjum Singh Solanky
af6d65a909 Create tagged Docker image on new tag/release 2023-02-14 20:04:06 -06:00
Debanjum Singh Solanky
25e06f26c0 Improve workflow to publish khoj to pypi 
- Use emoji's to improve visual indicator of action step
- Rename to pypi instead of the more ambiguous publish name
  Publish could mean publish docker image, publish to pypi, MELPA or
  Obsidian plugin
- Update workflow badge, link pypi badge to khoj pypi package page
- Use pypa official github action to upload package to (test) pypi
  instead of doing it manually using twine
- Upload python package artifact for easier access for testing.
  As uploading to testpypi doesn't work for PRs by others from forked repos
 2023-02-14 20:03:35 -06:00
Debanjum Singh Solanky
bc7477ea3e Move Emacs, Obsidian plugin code out from under src/khoj directory 
- What
  - The Emacs and Obsidian interfaces stay in their original
    directories under src/
  - src/khoj now only contains code meant for pypi packaging

- Benefits
  - This avoids having to update khoj MELPA, Obsidian plugin config as
    the Emacs, Obsidian code is under their original directories
  - It separates the code in src/khoj meant for python packaging from
    code for external interfaces like Emacs and Obsidian
 2023-02-14 15:44:22 -06:00
Debanjum Singh Solanky
f83cf4ebc6 Check wheel contents in workflow before publishing it to PyPI 2023-02-14 15:20:44 -06:00
Debanjum Singh Solanky
25a749ca1d Use the src/ layout to fix packaging Khoj for PyPi 
- Why
  The khoj pypi packages should be installed in `khoj' directory.
  Previously it was being installed into `src' directory, which is a
  generic top level directory name that is discouraged from being used

- Changes
 - move src/* to src/khoj/*
 - update `setup.py' to `find_packages' in `src' instead of project root
 - rename imports to form `from khoj.*' in complete project
 - update `constants.web_directory' path to use `khoj' directory
 - rename root logger to `khoj' in `main.py'
 - fix image_search tests to use the newly rename `khoj' logger
 - update config, docs, workflows to reference new path `src/khoj'
 2023-02-14 15:19:06 -06:00
Debanjum Singh Solanky
cc31cd070d Enable the publish workflow for PRs created in the main repo 
The publish workflow was previously disabled for PRs in commit
d1945c5ba8
 2023-02-14 13:51:31 -06:00
Debanjum Singh Solanky
d1945c5ba8 Do not run publish workflow for PRs as forks do not have auth token 2023-02-12 07:31:24 -06:00
Debanjum Singh Solanky
55e4fa9719 Fix indentation in workflow yaml for testing khoj backend 2023-02-07 02:59:46 -03:00
Debanjum Singh Solanky
c11f7b47e4 Update workflow to run backend tests for all supported python versions 2023-02-06 21:05:34 -03:00
Debanjum Singh Solanky
f9fb58aec3 Automate khoj.el testing using Github workflow 
Install transient.el dependency as it is not available by default
before Emacs 28.1
 2023-01-23 20:33:47 -03:00
Debanjum Singh Solanky
9f0bd0a361 Add Github workflow for khoj.el build and quality checks 
Add khoj.el build badge to khoj.el Readme
 2023-01-19 20:13:19 -03:00
Debanjum Singh Solanky
123b077c68 Use apt update before apt install in test workflow on Github 2023-01-11 16:51:16 -03:00
Debanjum Singh Solanky
1dc1472c55 In publish workflow, make twine upload verbose to troubleshoot 2023-01-05 12:56:46 -03:00
Debanjum Singh Solanky
3d1199540c Update the publish workflow to also run on any tag push 2023-01-04 20:47:23 -03:00
Debanjum Singh Solanky
4842daca5f Run releases workflow on pushing any tag. 'v' prefix not required 
Obsidian for some reason cannot pick up plugin assets from releases
made with prefixed tags
 2023-01-04 20:27:56 -03:00
Saba
a53094ec92 Add workflow dispatch support in build.yml 
- To support dispatch, set the image label based on the branch name
- Master build should still be tagged with latest to get benefit of the standard production Docker label
 2022-09-15 20:28:41 +03:00
Debanjum Singh Solanky
e4d40e4d4d Update setup.py version, Readme. Remove faulty release badge for now 2022-09-07 14:51:03 +03:00
Debanjum Singh Solanky
7fe3e844d2 Fix setup of Reproducible Build environment in publish workflow 
- Note: Reproducible builds have not been validated.
  This is just preliminary work to get there.
  Further testing and fixes maybe required
 2022-08-19 21:00:12 +03:00
Debanjum Singh Solanky
60dacf3f2c Show splash screen on app start. Only supported on Windows, Linux 2022-08-19 19:16:10 +03:00
Debanjum Singh Solanky
d25ddb93f7 Fix missing closing bracket from SOURCE_DATE_EPOCH def in release.yml 2022-08-17 23:17:27 +03:00
Debanjum Singh Solanky
9ee02b0804 Add --noconfirm in call to pyinstaller from Github release workflow 
Added just for safety, workflow works fine without it too
 2022-08-17 23:04:26 +03:00
Debanjum Singh Solanky
5a20283202 Set Pyinstaller, Pip environment to create reproducible builds of Khoj 
- Dependency Version Pinning
  - First level dependency versions have been pinned.
  - Transitive dependencies have not been specified yet

- Testing
  - The Pyinstaller build has been only minimally tested for reproducibility
  - The Khoj package generated for PyPi have not been tested for reproducibility

- References
  - https://reproducible-builds.org/docs/source-date-epoch/
  - https://pyinstaller.org/en/stable/advanced-topics.html#creating-a-reproducible-build
 2022-08-17 20:09:35 +03:00
Debanjum Singh Solanky
a37724f338 Fix Debian package permission. Set version on manual workflow trigger 2022-08-16 20:30:46 +03:00
Debanjum Singh Solanky
3d0f979475 Make the .deb package use version in Debian package versioning format 2022-08-16 17:35:14 +03:00
Debanjum Singh Solanky
7421ef2724 Fix path of app artifacts to attach to release via release pipeline 2022-08-15 17:41:08 +03:00
Debanjum Singh Solanky
c21ab4316c Fix publish of pip package to PyPi on git tag 2022-08-15 06:55:00 +03:00
Debanjum Singh Solanky
237e207304 Create separate app artifacts for each operating system 2022-08-15 06:39:45 +03:00
Debanjum Singh Solanky
4308f51d2c Fix workflow_dispatch trigger name used in github release workflow 2022-08-15 06:09:54 +03:00
Debanjum Singh Solanky
14710da962 Add version, arch to app name. Publish artifacts to release 
- Allow manual trigger of workflow
 2022-08-15 06:01:47 +03:00
Debanjum Singh Solanky
2142cba627 Simplify upload artifacts to single action 2022-08-15 05:11:37 +03:00
Debanjum Singh Solanky
a44cf39ab9 Limit release workflow to only run on creation of git tags 
This reverts commit 7eb3dc71de.
 2022-08-14 07:27:33 +03:00
Debanjum Singh Solanky
7eb3dc71de Revert "Revert release workflow to only run on creation of git tags" 
This reverts commit 34cfc0382b.
 2022-08-14 06:49:01 +03:00
Debanjum Singh Solanky
62711408b0 Run Install Deps step in bash shell to avoid powershell on windows 2022-08-14 06:47:58 +03:00
Debanjum Singh Solanky
34cfc0382b Revert release workflow to only run on creation of git tags 2022-08-14 06:45:17 +03:00
Debanjum Singh Solanky
357b584142 Re-enable windows, macos in matrix of release PR 2022-08-14 06:41:54 +03:00
Debanjum Singh Solanky
19294c788a Install libxcb on ubuntu as well 2022-08-14 06:30:00 +03:00
Debanjum Singh Solanky
4fd590addd Install libegl1 as root 2022-08-14 06:28:12 +03:00
Debanjum Singh Solanky
f3be84d8b9 Fix fpm command and install libegl1 on linux runner 2022-08-14 06:24:16 +03:00
Debanjum Singh Solanky
c422967276 Setup Ruby properly to add FPM Ruby package to PATH 2022-08-14 06:08:18 +03:00
Debanjum Singh Solanky
8696032b40 Set permission of the single Khoj app file generated by Pyinstaller 2022-08-14 05:46:52 +03:00
Debanjum Singh Solanky
47bac4f61e Only run ubuntu in OS matrix. Speeds verifying debian package creation 2022-08-14 05:27:02 +03:00
Debanjum Singh Solanky
7ce9c3ecce Re-enable running workflow via PR temporarily 2022-08-14 05:08:59 +03:00
Debanjum Singh Solanky
fed0b591af Package Khoj as Debian app in Github Release Workflow 2022-08-14 05:07:58 +03:00
Debanjum Singh Solanky
d815e0d115 Run Release Workflow on pushing git tag. Add release badge to Readme 2022-08-14 04:21:11 +03:00
Debanjum Singh Solanky
3bc4d75ba9 Consolidate app creation using pyinstaller into a single step 2022-08-14 04:10:26 +03:00
Debanjum Singh Solanky
0a0a8d43e0 Install Mac specific DMG creator only on the Mac Runner 2022-08-14 03:49:40 +03:00
Debanjum Singh Solanky
7466665499 Add runs-on property to specify OS from matrix to run job on 2022-08-14 03:43:07 +03:00
Debanjum Singh Solanky
a10a2ea3ad Run job matrix to wrap Khoj into app installers for Debian, Windows too 2022-08-14 03:39:45 +03:00
Debanjum Singh Solanky
22a9aefaef Publish Mac App Disk Image generated by release workflow as artifact 2022-08-14 03:28:41 +03:00
Debanjum Singh Solanky
dc553546bc Remove empty environment arguments in release workflow 2022-08-14 03:04:21 +03:00
Debanjum Singh Solanky
46a8257783 Just make Github runner build app for Mac x86 64 bit architecture 2022-08-14 02:54:14 +03:00
Debanjum Singh Solanky
8acc9e49d5 Put env variable on separate line. Remove unneeded workflow triggers 2022-08-14 02:40:43 +03:00
Debanjum Singh Solanky
65f3d0ff3b Trigger github workflow on PR 2022-08-14 02:38:20 +03:00
Debanjum Singh Solanky
b9b0278c99 Try create universal Mac App binary as Github Mac Runner is x86_64 2022-08-14 02:29:03 +03:00
Debanjum Singh Solanky
8cbdf7319e Create Github workflow to wrap Khoj into a Mac .App Disk Image 2022-08-14 01:53:37 +03:00
Debanjum Singh Solanky
da5284bbda Install libegl to fix libegl1.so import error in Github tests workflow 2022-08-11 22:57:05 +03:00
Debanjum Singh Solanky
50575b283c Reorder publish actions by order of importance in publish workflow 2022-08-04 04:43:05 +03:00
Debanjum Singh Solanky
c3ae3cb021 Publish to (Test) PyPi with PEP440 Compliant Version Numbers 
- Use .devN for publish to testpypi on push to PR
- Use aN to publish to pypi on push to master
 2022-08-04 04:29:15 +03:00
Debanjum Singh Solanky
89044c70bd Publish to Test PyPI on Push to PR or Master Branch 2022-08-04 03:59:38 +03:00
Debanjum Singh Solanky
4bc5ac2420 Add pre-release version to package and upload to TestPyPI 
Update version to form {current-version}-alpha.{unix-timestamp} in setup.py
 2022-08-04 03:21:33 +03:00
Debanjum Singh Solanky
b2beff6a0e Only run build workflow on publish to master. Not on pull request 2022-08-04 02:20:05 +03:00
Debanjum Singh Solanky
7a353066dd Improve name, stages in Github Workflows for Publish and Test 2022-08-03 22:37:16 +03:00
Debanjum Singh Solanky
f55eaf0a53 Remove comma in command to pip install multiple packages 2022-08-03 22:01:18 +03:00
Debanjum Singh Solanky
ee3eddb044 Create Workflow to Publish Application to (test) PyPI 2022-08-03 21:57:27 +03:00
Debanjum Singh Solanky
34296031c4 Name test workflow. Check if exiftool is even required for pytests 2022-08-03 21:50:20 +03:00
Debanjum Singh Solanky
5d391fdcbe Just use Python version 3.10 in test Github Workflow 2022-08-03 20:25:15 +03:00
Debanjum Singh Solanky
d42203d29e Fix --upgrade flag to pip install command in Test Github Workflow 2022-08-03 20:20:34 +03:00
Debanjum Singh Solanky
f9a10ad5ee Use pip (not conda) to install khoj, run pytest in Github Workflow 2022-08-03 20:00:37 +03:00
Debanjum Singh Solanky
c96bc03f58 Trigger github build, test on updates to setup.py 2022-08-03 11:56:40 +03:00
Debanjum Singh Solanky
0797e9cef8 Run github workflows only when relevant paths are modified 2022-08-01 02:30:23 +03:00
Saba
7bb35ccc7e Run build on PR 2022-07-04 18:09:47 -04:00
Saba
0f88abd219 Allocate 8GB of memory to docker container. Adjust path to Dockerfile in Github action 2022-07-04 14:01:59 -04:00
Debanjum Singh Solanky
f08591c880 Set PORT arg when building docker image in the build workflow 2022-01-29 18:11:47 -05:00
Debanjum Singh Solanky
359f25b0a4 Rename publish workflow to build. Add badge to the workflow on Readme 2022-01-29 18:11:47 -05:00
Debanjum Singh Solanky
4add348e3c Remove context from path to Dockerfile in Github build, push action 2022-01-29 17:16:12 -05:00
Debanjum Singh Solanky
fa685dc37f Create Github workflow to build, publish docker container to registry 
- Rename the build workflow to test workflow
 2022-01-29 17:08:19 -05:00
Debanjum Singh Solanky
7c773d29ef Update github workflow to use environment.yml under config/ directory 2022-01-29 03:43:34 -05:00
Debanjum Singh Solanky
d91f645198 Try cache conda build step using online docs 
Refer:
 - https://github.com/marketplace/actions/setup-miniconda
 2021-12-15 11:48:14 +05:30
sabaimran
930bdb5403
Use ubuntu-latest as machine for Github Actions 2021-12-11 12:54:32 -05:00
sabaimran
386534c236
Remove apt-get update from build.yml 2021-12-11 12:52:13 -05:00