sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-18 02:27:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 23
khoj/pyproject.toml

116 lines
2.8 KiB
TOML
Raw Normal View History

Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
[build-system]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-17 23:08:05 +00:00
requires = ["hatchling", "hatch-vcs"]
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
build-backend = "hatchling.build"
[project]
name = "khoj-assistant"
Fix Khoj subtitle in desktop entry, pyproject, cli and Obsidian Readme
2023-07-02 03:24:46 +00:00
description = "An AI personal assistant for your Digital Brain"
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
readme = "README.md"
license = "GPL-3.0-or-later"
Have Khoj support Python 3.11 as Pytorch supports it now - Previously Khoj could only support Python upto 3.10 due to pytorch. But lots of folks had python 3.11 installed by default on their machines. This required installing python 3.10 and dealing with virtual envs. With Torch >= 2.0.1 now able to support python 3.11, at least one class of installation troubles for Khoj should drop. See https://github.com/pytorch/pytorch/issues/86566 for reference - Preliminary testing indicates using the new torch 2.x may reduce search time by 25% (from 80ms to 60ms on Mac M1) - Update Docs to not require mentioning python <=3.10 required - Update Github test workflow to run khoj tests with python 3.11 too
2023-06-29 21:54:51 +00:00
requires-python = ">=3.8"
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
authors = [
Fix authors, homepage URL in pyproject.toml and workflow triggers
2023-02-16 09:12:04 +00:00
{ name = "Debanjum Singh Solanky, Saba Imran" },
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
]
keywords = [
"search",
"semantic-search",
"productivity",
"NLP",
"AI",
"org-mode",
"markdown",
"images",
Wire up PDF to jsonl processor to Khoj server layer (API, config) - Specify PDF content to index via khoj.yml - Index PDF content on app start, reconfigure - Expose PDF as a search type via API
2023-06-01 04:25:48 +00:00
"pdf",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
]
classifiers = [
"Development Status :: 4 - Beta",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
Have Khoj support Python 3.11 as Pytorch supports it now - Previously Khoj could only support Python upto 3.10 due to pytorch. But lots of folks had python 3.11 installed by default on their machines. This required installing python 3.10 and dealing with virtual envs. With Torch >= 2.0.1 now able to support python 3.11, at least one class of installation troubles for Khoj should drop. See https://github.com/pytorch/pytorch/issues/86566 for reference - Preliminary testing indicates using the new torch 2.x may reduce search time by 25% (from 80ms to 60ms on Mac M1) - Update Docs to not require mentioning python <=3.10 required - Update Github test workflow to run khoj tests with python 3.11 too
2023-06-29 21:54:51 +00:00
"Programming Language :: Python :: 3.11",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
"Topic :: Internet :: WWW/HTTP :: Indexing/Search",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Topic :: Scientific/Engineering :: Human Machine Interfaces",
"Topic :: Text Processing :: Linguistic",
]
dependencies = [
"dateparser == 1.1.1",
"defusedxml == 0.7.1",
"fastapi == 0.77.1",
"jinja2 == 3.1.2",
Improve and simplify Khoj Chat using ChatGPT - Set context by either including last 2 chat messages from active session or past 2 conversation summaries from conversation logs - Set personality in system message - Place personality system message before last completed back & forth This may stop ChatGPT forgetting its personality as conversation progresses given: - The conditioning based on system role messages is light - If system message is too far back in conversation history, the model may forget its personality conditioning - If system message at end of conversation, the model can think its the start of a new conversation - Inserting the system message before last completed back & forth should prevent ChatGPT from assuming its the start of a new conversation while not losing personality conditioning from the system message - Simplfy the Khoj Chat API to for now just answer from users notes instead of trying to infer other potential interaction types. - This is the default expected behavior from the feature anyway - Use the compiled text of the top 2 search results for context - Benefits of using ChatGPT - Better model - 1/10th the price - No hand rolled prompt required to make GPT provide more chatty, assistant type responses
2023-03-04 17:01:49 +00:00
"openai >= 0.27.0",
Truncate message logs to below max supported prompt size by model - Use tiktoken to count tokens for chat models - Make conversation turns to add to prompt configurable via method argument to generate_chatml_messages_with_context method
2023-03-24 21:37:55 +00:00
"tiktoken >= 0.3.0",
Add retry logic to OpenAI API queries to increase Chat tenacity - Move completion and chat_completion into helper methods under utils.py - Add retry with exponential backoff on OpenAI exceptions using tenacity package. This is officially suggested and used by other popular GPT based libraries
2023-03-25 05:36:51 +00:00
"tenacity >= 8.2.2",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
"pillow == 9.3.0",
Upgrade minimum supported pydantic version
2023-07-03 19:22:56 +00:00
"pydantic >= 1.10.10",
Migrate from PyQT6 to PySide6
2023-07-12 01:43:44 +00:00
"pyside6 >= 6.5.1",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
"pyyaml == 6.0",
"rich >= 13.3.1",
"schedule == 1.1.0",
"sentence-transformers == 2.2.2",
Have Khoj support Python 3.11 as Pytorch supports it now - Previously Khoj could only support Python upto 3.10 due to pytorch. But lots of folks had python 3.11 installed by default on their machines. This required installing python 3.10 and dealing with virtual envs. With Torch >= 2.0.1 now able to support python 3.11, at least one class of installation troubles for Khoj should drop. See https://github.com/pytorch/pytorch/issues/86566 for reference - Preliminary testing indicates using the new torch 2.x may reduce search time by 25% (from 80ms to 60ms on Mac M1) - Update Docs to not require mentioning python <=3.10 required - Update Github test workflow to run khoj tests with python 3.11 too
2023-06-29 21:54:51 +00:00
"torch >= 2.0.1",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
"uvicorn == 0.17.6",
Use aiohttp version 3.8.4 as 4.x breaks docker image build
2023-03-25 22:10:53 +00:00
"aiohttp == 3.8.4",
Extract prompts as LangChain Prompt Templates into a separate module Improves code modularity, cleanliness. Reduces bloat in GPT.py module
2023-05-28 12:15:31 +00:00
"langchain >= 0.0.187",
Create PDF to JSONL processor using PyPDF and LangChain Switch `pydantic' to >= 1.9.1 else `langchain.document_loaders' starts throwing typing error for python 3.8, 3.9
2023-06-01 04:01:18 +00:00
"pypdf >= 3.9.0",
Use Github Rest API to index Markdown files in Github Repository The Llama_Hub Github plugin is fairly limited. The Github Rest API is well supported and can easily be extended to index commit messages, issues, discussions, PRs etc.
2023-06-17 08:39:57 +00:00
"requests >= 2.26.0",
Add beautiful as dependency to pass pytests
2023-06-23 22:10:09 +00:00
"bs4 >= 0.0.1",
Add support for our first Local LLM 🤖🏠 (#330) * Add support for gpt4all's falcon model as an additional conversation processor - Update the UI pages to allow the user to point to the new endpoints for GPT - Update the internal schemas to support both GPT4 models and OpenAI - Add unit tests benchmarking some of the Falcon performance * Add exc_info to include stack trace in error logs for text processors * Pull shared functions into utils.py to be used across gpt4 and gpt * Add migration for new processor conversation schema * Skip GPT4All actor tests due to typing issues * Fix Obsidian processor configuration in auto-configure flow * Rename enable_local_llm to enable_offline_chat
2023-07-26 23:27:08 +00:00
"gpt4all==1.0.5",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-17 23:08:05 +00:00
dynamic = ["version"]
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
[project.urls]
Rename project links across repo from debanjum/khoj to khoj-ai/khoj
2023-06-21 07:13:21 +00:00
Homepage = "https://github.com/khoj-ai/khoj#readme"
Issues = "https://github.com/khoj-ai/khoj/issues"
Discussions = "https://github.com/khoj-ai/khoj/discussions"
Releases = "https://github.com/khoj-ai/khoj/releases"
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
[project.scripts]
khoj = "khoj.main:run"
[project.optional-dependencies]
test = [
Add, configure and run pre-commit locally and in test workflow
2023-02-17 18:07:59 +00:00
"pytest >= 7.1.2",
Add trio, move freezegun, factory-boy to project test dependencies
2023-06-29 05:04:34 +00:00
"freezegun >= 1.2.0",
"factory-boy >= 3.2.1",
"trio >= 0.22.0",
Add, configure and run pre-commit locally and in test workflow
2023-02-17 18:07:59 +00:00
]
dev = [
"khoj-assistant[test]",
"mypy >= 1.0.1",
"black >= 23.1.0",
"pre-commit >= 3.0.4",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-17 23:08:05 +00:00
[tool.hatch.version]
source = "vcs"
raw-options.local_scheme = "no-local-version" # PEP440 compliant version for PyPi
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 01:02:30 +00:00
[tool.hatch.build.targets.sdist]
include = ["src/khoj"]
[tool.hatch.build.targets.wheel]
Move mypy config into pyproject.toml. Ignore 2 remaining mypy issues
2023-02-16 09:33:08 +00:00
packages = ["src/khoj"]
[tool.mypy]
files = "src/khoj"
pretty = true
strict_optional = false
install_types = true
Add, configure and run pre-commit locally and in test workflow
2023-02-17 18:07:59 +00:00
ignore_missing_imports = true
Move mypy config into pyproject.toml. Ignore 2 remaining mypy issues
2023-02-16 09:33:08 +00:00
non_interactive = true
show_error_codes = true
Fix or ignore MyPy issues in PyQt desktop GUI code - Remove unneeded type ignore for mps with the latest mypy - Stop excluding PyQT desktop GUI code from MyPy checks - Do not warn about unused ignores. Some issue with mypy giving different errors in different environments (venv, system and pre-commit)
2023-02-17 22:11:17 +00:00
warn_unused_ignores = false
Use Black to format Khoj server code and tests
2023-02-17 16:04:26 +00:00
[tool.black]
Add, configure and run pre-commit locally and in test workflow
2023-02-17 18:07:59 +00:00
line-length = 120
Configure tests to mark chat quality tests & filter unhelpful warnings - Mark chat quality tests, register custom mark for chat quality - Filter unhelpful deprecation warnings from within dateparser library - Error if tests use unregistered marks
2023-03-15 00:56:14 +00:00
[tool.pytest.ini_options]
addopts = "--strict-markers"
markers = [
"chatquality: Evaluate chatbot capabilities and quality",
]
Reference in a new issue Copy permalink