sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-23 15:38:55 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 5
khoj/pyproject.toml

131 lines
3.3 KiB
TOML
Raw Normal View History

Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
[build-system]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-18 00:08:05 +01:00
requires = ["hatchling", "hatch-vcs"]
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
build-backend = "hatchling.build"
[project]
name = "khoj-assistant"
Sync desktop app package.json with other Khoj clients metadata - Make `bump_version.sh' script set version for the Khoj desktop app too - Sync Khoj desktop app authors, license, description and version with the other interfaces and server - Update description in packages metadata to match project subtitle on Github
2023-10-14 05:22:33 +02:00
description = "An AI copilot for your Second Brain"
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
readme = "README.md"
license = "GPL-3.0-or-later"
Have Khoj support Python 3.11 as Pytorch supports it now - Previously Khoj could only support Python upto 3.10 due to pytorch. But lots of folks had python 3.11 installed by default on their machines. This required installing python 3.10 and dealing with virtual envs. With Torch >= 2.0.1 now able to support python 3.11, at least one class of installation troubles for Khoj should drop. See https://github.com/pytorch/pytorch/issues/86566 for reference - Preliminary testing indicates using the new torch 2.x may reduce search time by 25% (from 80ms to 60ms on Mac M1) - Update Docs to not require mentioning python <=3.10 required - Update Github test workflow to run khoj tests with python 3.11 too
2023-06-29 23:54:51 +02:00
requires-python = ">=3.8"
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
authors = [
Fix authors, homepage URL in pyproject.toml and workflow triggers
2023-02-16 10:12:04 +01:00
{ name = "Debanjum Singh Solanky, Saba Imran" },
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
keywords = [
"search",
"semantic-search",
"productivity",
"NLP",
"AI",
"org-mode",
"markdown",
"images",
Wire up PDF to jsonl processor to Khoj server layer (API, config) - Specify PDF content to index via khoj.yml - Index PDF content on app start, reconfigure - Expose PDF as a search type via API
2023-06-01 06:25:48 +02:00
"pdf",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
classifiers = [
"Development Status :: 4 - Beta",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
Have Khoj support Python 3.11 as Pytorch supports it now - Previously Khoj could only support Python upto 3.10 due to pytorch. But lots of folks had python 3.11 installed by default on their machines. This required installing python 3.10 and dealing with virtual envs. With Torch >= 2.0.1 now able to support python 3.11, at least one class of installation troubles for Khoj should drop. See https://github.com/pytorch/pytorch/issues/86566 for reference - Preliminary testing indicates using the new torch 2.x may reduce search time by 25% (from 80ms to 60ms on Mac M1) - Update Docs to not require mentioning python <=3.10 required - Update Github test workflow to run khoj tests with python 3.11 too
2023-06-29 23:54:51 +02:00
"Programming Language :: Python :: 3.11",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"Topic :: Internet :: WWW/HTTP :: Indexing/Search",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Topic :: Scientific/Engineering :: Human Machine Interfaces",
"Topic :: Text Processing :: Linguistic",
]
dependencies = [
Fix plaintext HTML parsing and rendering (#464) * Store conversation command options in an Enum * Move to slash commands instead of using @ to specify general commands * Calculate conversation command once & pass it as arg to child funcs * Add /notes command to respond using only knowledge base as context This prevents the chat model to try respond using it's general world knowledge only without any references pulled from the indexed knowledge base * Test general and notes slash commands in openai chat director tests --------- Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
2023-08-27 20:24:30 +02:00
"bs4 >= 0.0.1",
Loosen dateparser dependency to get python3.10 wheel for regex package This should reduce chances of installation errors due to regex package being built from source for python3.11 Previously, the regex dependency of dateparser = 1.1.1 didn't have a wheel for python 3.11. This would trigger building the regex package from scratch which would fail for a lot of folks
2023-08-07 07:48:40 +02:00
"dateparser >= 1.1.1",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"defusedxml == 0.7.1",
"fastapi == 0.77.1",
Use multi-part form to receive files to index on server - This uses existing HTTP affordance to process files - Better handling of binary file formats as removes need to url encode/decode - Less memory utilization than streaming json as files get automatically written to disk once memory utilization exceeds preset limits - No manual parsing of raw files streams required
2023-10-12 02:14:15 +02:00
"python-multipart >= 0.0.5",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"jinja2 == 3.1.2",
Restrict openai package version below 1.0.0 to avoid breaking changes
2023-10-10 04:30:58 +02:00
"openai >= 0.27.0, < 1.0.0",
Retain desktop builds for 3 days to allow user tests Upgrade minimum tiktoken version to work for encoding gpt4
2023-08-09 08:00:40 +02:00
"tiktoken >= 0.3.2",
Add retry logic to OpenAI API queries to increase Chat tenacity - Move completion and chat_completion into helper methods under utils.py - Add retry with exponential backoff on OpenAI exceptions using tenacity package. This is officially suggested and used by other popular GPT based libraries
2023-03-25 06:36:51 +01:00
"tenacity >= 8.2.2",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"pillow == 9.3.0",
Upgrade minimum supported pydantic version
2023-07-03 21:22:56 +02:00
"pydantic >= 1.10.10",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"pyyaml == 6.0",
"rich >= 13.3.1",
"schedule == 1.1.0",
"sentence-transformers == 2.2.2",
Fix the minimum version of the transformers library required to address #404
2023-08-02 18:10:14 +02:00
"transformers >= 4.28.0",
[Multi-User Part 5]: Add a production Docker file and use a gunicorn configuration with it (#514) - Add a productionized setup for the Khoj server using `gunicorn` with multiple workers for handling requests - Add a new Dockerfile meant for production config at `ghcr.io/khoj-ai/khoj:prod`; the existing Docker config should remain the same
2023-10-26 22:15:31 +02:00
"torch == 2.0.1",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"uvicorn == 0.17.6",
Fix bug in /chat endpoint for general and update depdendencies
2023-08-28 23:12:04 +02:00
"aiohttp == 3.8.5",
Extract prompts as LangChain Prompt Templates into a separate module Improves code modularity, cleanliness. Reduces bloat in GPT.py module
2023-05-28 14:15:31 +02:00
"langchain >= 0.0.187",
Use Github Rest API to index Markdown files in Github Repository The Llama_Hub Github plugin is fairly limited. The Github Rest API is well supported and can easily be extended to index commit messages, issues, discussions, PRs etc.
2023-06-17 10:39:57 +02:00
"requests >= 2.26.0",
Add beautiful as dependency to pass pytests
2023-06-24 00:10:09 +02:00
"bs4 >= 0.0.1",
Fix anyio package version to avoid backwards compatibility issue with start_blocking_portal method
2023-08-31 23:14:13 +02:00
"anyio == 3.7.1",
[Multi-User Part 1]: Enable storage of settings for plaintext files based on user account (#498) - Partition configuration for indexing local data based on user accounts - Store indexed data in an underlying postgres db using the `pgvector` extension - Add migrations for all relevant user data and embeddings generation. Very little performance optimization has been done for the lookup time - Apply filters using SQL queries - Start removing many server-level configuration settings - Configure GitHub test actions to run during any PR. Update the test action to run in a containerized environment with a DB. - Update the Docker image and docker-compose.yml to work with the new application design
2023-10-26 18:42:29 +02:00
"pymupdf >= 1.23.5",
[Multi-User]: Part 0 - Add support for logging in with Google (#487) * Add concept of user authentication to the request session via GoogleUser
2023-10-15 04:39:13 +02:00
"django == 4.2.5",
"authlib == 1.2.1",
Resolve issues with GPT4All and fix prompt for yesterday extract questions date filter (#483) - GPT4All integration had ceased working with 0.1.7 specification. Update to use 1.0.12. At a later date, we should also use first party support for llama v2 via gpt4all - Update the system prompt for the extract_questions flow to add start and end date to the yesterday date filter example. - Update all setup data in conftest.py to use new client-server indexing pattern
2023-09-18 23:41:26 +02:00
"gpt4all == 1.0.12; platform_system == 'Linux' and platform_machine == 'x86_64'",
"gpt4all == 1.0.12; platform_system == 'Windows' or platform_system == 'Darwin'",
[Multi-User]: Part 0 - Add support for logging in with Google (#487) * Add concept of user authentication to the request session via GoogleUser
2023-10-15 04:39:13 +02:00
"itsdangerous == 2.1.2",
"httpx == 0.25.0",
[Multi-User Part 1]: Enable storage of settings for plaintext files based on user account (#498) - Partition configuration for indexing local data based on user accounts - Store indexed data in an underlying postgres db using the `pgvector` extension - Add migrations for all relevant user data and embeddings generation. Very little performance optimization has been done for the lookup time - Apply filters using SQL queries - Start removing many server-level configuration settings - Configure GitHub test actions to run during any PR. Update the test action to run in a containerized environment with a DB. - Update the Docker image and docker-compose.yml to work with the new application design
2023-10-26 18:42:29 +02:00
"pgvector == 0.2.3",
"psycopg2-binary == 2.9.9",
[Multi-User Part 2]: Add login pages and gate access to application behind login wall (#503) - Make most routes conditional on authentication *if anonymous mode is not enabled*. If anonymous mode is enabled, it scaffolds a default user and uses that for all application interactions. - Add a basic login page and add routes for redirecting the user if logged in
2023-10-26 19:17:29 +02:00
"google-auth == 2.23.3",
"python-multipart == 0.0.6",
[Multi-User Part 5]: Add a production Docker file and use a gunicorn configuration with it (#514) - Add a productionized setup for the Khoj server using `gunicorn` with multiple workers for handling requests - Add a new Dockerfile meant for production config at `ghcr.io/khoj-ai/khoj:prod`; the existing Docker config should remain the same
2023-10-26 22:15:31 +02:00
"gunicorn == 21.2.0",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-18 00:08:05 +01:00
dynamic = ["version"]
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
[project.urls]
Rename project links across repo from debanjum/khoj to khoj-ai/khoj
2023-06-21 09:13:21 +02:00
Homepage = "https://github.com/khoj-ai/khoj#readme"
Issues = "https://github.com/khoj-ai/khoj/issues"
Discussions = "https://github.com/khoj-ai/khoj/discussions"
Releases = "https://github.com/khoj-ai/khoj/releases"
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
[project.scripts]
khoj = "khoj.main:run"
[project.optional-dependencies]
test = [
Add, configure and run pre-commit locally and in test workflow
2023-02-17 19:07:59 +01:00
"pytest >= 7.1.2",
Add trio, move freezegun, factory-boy to project test dependencies
2023-06-29 07:04:34 +02:00
"freezegun >= 1.2.0",
"factory-boy >= 3.2.1",
"trio >= 0.22.0",
Run pytests in parallel on multiple CPU cores using pytest-xdist for speed
2023-10-13 01:19:48 +02:00
"pytest-xdist",
Add, configure and run pre-commit locally and in test workflow
2023-02-17 19:07:59 +01:00
]
dev = [
"khoj-assistant[test]",
"mypy >= 1.0.1",
"black >= 23.1.0",
"pre-commit >= 3.0.4",
[Multi-User Part 1]: Enable storage of settings for plaintext files based on user account (#498) - Partition configuration for indexing local data based on user accounts - Store indexed data in an underlying postgres db using the `pgvector` extension - Add migrations for all relevant user data and embeddings generation. Very little performance optimization has been done for the lookup time - Apply filters using SQL queries - Start removing many server-level configuration settings - Configure GitHub test actions to run during any PR. Update the test action to run in a containerized environment with a DB. - Update the Docker image and docker-compose.yml to work with the new application design
2023-10-26 18:42:29 +02:00
"pytest-django == 4.5.2",
"pytest-asyncio == 0.21.1",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-18 00:08:05 +01:00
[tool.hatch.version]
source = "vcs"
raw-options.local_scheme = "no-local-version" # PEP440 compliant version for PyPi
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
[tool.hatch.build.targets.sdist]
include = ["src/khoj"]
[tool.hatch.build.targets.wheel]
Move mypy config into pyproject.toml. Ignore 2 remaining mypy issues
2023-02-16 10:33:08 +01:00
packages = ["src/khoj"]
[tool.mypy]
files = "src/khoj"
pretty = true
strict_optional = false
install_types = true
Add, configure and run pre-commit locally and in test workflow
2023-02-17 19:07:59 +01:00
ignore_missing_imports = true
Move mypy config into pyproject.toml. Ignore 2 remaining mypy issues
2023-02-16 10:33:08 +01:00
non_interactive = true
show_error_codes = true
Fix or ignore MyPy issues in PyQt desktop GUI code - Remove unneeded type ignore for mps with the latest mypy - Stop excluding PyQT desktop GUI code from MyPy checks - Do not warn about unused ignores. Some issue with mypy giving different errors in different environments (venv, system and pre-commit)
2023-02-17 23:11:17 +01:00
warn_unused_ignores = false
Use Black to format Khoj server code and tests
2023-02-17 17:04:26 +01:00
[tool.black]
Add, configure and run pre-commit locally and in test workflow
2023-02-17 19:07:59 +01:00
line-length = 120
Configure tests to mark chat quality tests & filter unhelpful warnings - Mark chat quality tests, register custom mark for chat quality - Filter unhelpful deprecation warnings from within dateparser library - Error if tests use unregistered marks
2023-03-15 01:56:14 +01:00
[tool.pytest.ini_options]
addopts = "--strict-markers"
markers = [
"chatquality: Evaluate chatbot capabilities and quality",
]
Reference in a new issue Copy permalink