Configure app routes after configuring server.
Import API routers after search type is dynamically populated.
Allow API to recognize the dynamically populated plugin search types
as valid type query param.
Enable searching for plugin type content.
- Instead of building the package locally like before
The issue started since moving to dynamic git based versioning with hatch-vcs
This should reduce image size of docker builds too
- Also move to ubuntu image since pyqt6 builds available on it, so do
not need to build it locally for image
- This s
- Remove unneeded type ignore for mps with the latest mypy
- Stop excluding PyQT desktop GUI code from MyPy checks
- Do not warn about unused ignores. Some issue with mypy giving
different errors in different environments (venv, system and pre-commit)
- Run mypy on git push (not every commit) but for all files
- Running it on pre-commit, doesn't make sense as mypy wants to look
at all files, not just diff files
- But this is too time consuming to run every commit, so run on push
- Update development section documentation on installing, manually
running pre-commit for validation that includes running mypy checks
- Why
- pyprojects.toml is the python standards compliant config format
- allows collating python tooling configs into single standard file
- hatch(-ling) is a new lightweight build system for python packages
- Detailed Changes
- Replace setup.py, setuptools with pyproject.toml, hatchling for
khoj python config and build
- move pytest into optional development dependencies
- add more links to khoj in the project urls section
- add topic classifiers and keywords to find khoj package
- Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup
- Update pypi workflow to set python package version in pyproject.toml
- Use Rich to render uvicorn, fastAPI logs as well
The previous CustomFormatter only worked on khoj logs
- Improve rendering stacktrace on errors using Rich
- Use emoji's to improve visual indicator of action step
- Rename to pypi instead of the more ambiguous publish name
Publish could mean publish docker image, publish to pypi, MELPA or
Obsidian plugin
- Update workflow badge, link pypi badge to khoj pypi package page
- Use pypa official github action to upload package to (test) pypi
instead of doing it manually using twine
- Upload python package artifact for easier access for testing.
As uploading to testpypi doesn't work for PRs by others from forked repos
- What
- The Emacs and Obsidian interfaces stay in their original
directories under src/
- src/khoj now only contains code meant for pypi packaging
- Benefits
- This avoids having to update khoj MELPA, Obsidian plugin config as
the Emacs, Obsidian code is under their original directories
- It separates the code in src/khoj meant for python packaging from
code for external interfaces like Emacs and Obsidian
- Why
The khoj pypi packages should be installed in `khoj' directory.
Previously it was being installed into `src' directory, which is a
generic top level directory name that is discouraged from being used
- Changes
- move src/* to src/khoj/*
- update `setup.py' to `find_packages' in `src' instead of project root
- rename imports to form `from khoj.*' in complete project
- update `constants.web_directory' path to use `khoj' directory
- rename root logger to `khoj' in `main.py'
- fix image_search tests to use the newly rename `khoj' logger
- update config, docs, workflows to reference new path `src/khoj'
- By default the obsidian plugin automatically configures the khoj
backend to index the current vault
- For more complex scenarios, users can manage their ~/.khoj/khoj.yml
manually by toggling the auto-configure setting off in the khoj
plugin settings
Resolves#156
- setup.py best practise recommends only specifying core dependencies,
not dependencies of core dependencies in it
- Latest sentence-transformer (version 2.2.2) correctly installs its
huggingface_hub dependency. Else application fails to start
### Background
1. Obsidian stores markdown notes as `utf8`[1]
2. By default, the python `open` command uses the OS locale encoding[2]
### Issue
Based on above background, if the OS locale encoding isn't `utf8` it causes the `UnicodeDecodeError: <locale_encoding> codec can't decode byte` error
### Fix
- Read markdown files as `utf8`
The Obsidian plugin is the main use-case for markdown files in khoj currently and that stores md files as `utf8`.
Do not assume utf8 for other content types like org-mode, beancount for now.
- Fail if error in reading file as utf8, instead of ignoring errors.
Would rather have user realize that their files are not going to get indexed correctly.
[1]: https://forum.obsidian.md/t/better-handle-md-files-not-stored-in-utf8-format/13524/3
[2]: https://docs.python.org/3/library/functions.html#open
