- Instead of building the package locally like before
The issue started since moving to dynamic git based versioning with hatch-vcs
This should reduce image size of docker builds too
- Also move to ubuntu image since pyqt6 builds available on it, so do
not need to build it locally for image
- This s
- Remove unneeded type ignore for mps with the latest mypy
- Stop excluding PyQT desktop GUI code from MyPy checks
- Do not warn about unused ignores. Some issue with mypy giving
different errors in different environments (venv, system and pre-commit)
- Run mypy on git push (not every commit) but for all files
- Running it on pre-commit, doesn't make sense as mypy wants to look
at all files, not just diff files
- But this is too time consuming to run every commit, so run on push
- Update development section documentation on installing, manually
running pre-commit for validation that includes running mypy checks
- Why
- pyprojects.toml is the python standards compliant config format
- allows collating python tooling configs into single standard file
- hatch(-ling) is a new lightweight build system for python packages
- Detailed Changes
- Replace setup.py, setuptools with pyproject.toml, hatchling for
khoj python config and build
- move pytest into optional development dependencies
- add more links to khoj in the project urls section
- add topic classifiers and keywords to find khoj package
- Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup
- Update pypi workflow to set python package version in pyproject.toml
- Use Rich to render uvicorn, fastAPI logs as well
The previous CustomFormatter only worked on khoj logs
- Improve rendering stacktrace on errors using Rich
- Use emoji's to improve visual indicator of action step
- Rename to pypi instead of the more ambiguous publish name
Publish could mean publish docker image, publish to pypi, MELPA or
Obsidian plugin
- Update workflow badge, link pypi badge to khoj pypi package page
- Use pypa official github action to upload package to (test) pypi
instead of doing it manually using twine
- Upload python package artifact for easier access for testing.
As uploading to testpypi doesn't work for PRs by others from forked repos
- What
- The Emacs and Obsidian interfaces stay in their original
directories under src/
- src/khoj now only contains code meant for pypi packaging
- Benefits
- This avoids having to update khoj MELPA, Obsidian plugin config as
the Emacs, Obsidian code is under their original directories
- It separates the code in src/khoj meant for python packaging from
code for external interfaces like Emacs and Obsidian
- Why
The khoj pypi packages should be installed in `khoj' directory.
Previously it was being installed into `src' directory, which is a
generic top level directory name that is discouraged from being used
- Changes
- move src/* to src/khoj/*
- update `setup.py' to `find_packages' in `src' instead of project root
- rename imports to form `from khoj.*' in complete project
- update `constants.web_directory' path to use `khoj' directory
- rename root logger to `khoj' in `main.py'
- fix image_search tests to use the newly rename `khoj' logger
- update config, docs, workflows to reference new path `src/khoj'
- By default the obsidian plugin automatically configures the khoj
backend to index the current vault
- For more complex scenarios, users can manage their ~/.khoj/khoj.yml
manually by toggling the auto-configure setting off in the khoj
plugin settings
Resolves#156
- setup.py best practise recommends only specifying core dependencies,
not dependencies of core dependencies in it
- Latest sentence-transformer (version 2.2.2) correctly installs its
huggingface_hub dependency. Else application fails to start
### Background
1. Obsidian stores markdown notes as `utf8`[1]
2. By default, the python `open` command uses the OS locale encoding[2]
### Issue
Based on above background, if the OS locale encoding isn't `utf8` it causes the `UnicodeDecodeError: <locale_encoding> codec can't decode byte` error
### Fix
- Read markdown files as `utf8`
The Obsidian plugin is the main use-case for markdown files in khoj currently and that stores md files as `utf8`.
Do not assume utf8 for other content types like org-mode, beancount for now.
- Fail if error in reading file as utf8, instead of ignoring errors.
Would rather have user realize that their files are not going to get indexed correctly.
[1]: https://forum.obsidian.md/t/better-handle-md-files-not-stored-in-utf8-format/13524/3
[2]: https://docs.python.org/3/library/functions.html#open
- Khoj supports indexing subdirectories but the khoj docker config
wasn't updated to support the same
- This should also allow khoj docker users to index multiple separate
directory trees by mounting them into separate sub folders within
/data/<content-type>/.
For e.g /data/org/dir1, /data/org/dir2 etc in khoj_docker.yml
- Background
1. Obsidian stores markdown notes as utf8[1]
2. By default, the python `open' command uses the OS locale encoding[2]
This was causing the `UnicodeDecodeError: <locale_encoding> codec can't decode byte' error
- Fix
- Read markdown files as utf8
The Obsidian plugin is the main use-case for markdown files in
khoj currently and that stores md files as utf8.
Do not assume utf8 for other content types like org-mode, beancount for now.
- Fail if error in reading file as utf8, instead of ignoring errors.
Would rather have user realize that their files are not going to
get indexed correctly.
[1]: https://forum.obsidian.md/t/better-handle-md-files-not-stored-in-utf8-format/13524/3
[2]: https://docs.python.org/3/library/functions.html#open
