sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-23 23:48:56 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 16
khoj/pyproject.toml

118 lines
2.8 KiB
TOML
Raw Normal View History

Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
[build-system]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-18 00:08:05 +01:00
requires = ["hatchling", "hatch-vcs"]
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
build-backend = "hatchling.build"
[project]
name = "khoj-assistant"
Fix Khoj subtitle in desktop entry, pyproject, cli and Obsidian Readme
2023-07-02 05:24:46 +02:00
description = "An AI personal assistant for your Digital Brain"
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
readme = "README.md"
license = "GPL-3.0-or-later"
Have Khoj support Python 3.11 as Pytorch supports it now - Previously Khoj could only support Python upto 3.10 due to pytorch. But lots of folks had python 3.11 installed by default on their machines. This required installing python 3.10 and dealing with virtual envs. With Torch >= 2.0.1 now able to support python 3.11, at least one class of installation troubles for Khoj should drop. See https://github.com/pytorch/pytorch/issues/86566 for reference - Preliminary testing indicates using the new torch 2.x may reduce search time by 25% (from 80ms to 60ms on Mac M1) - Update Docs to not require mentioning python <=3.10 required - Update Github test workflow to run khoj tests with python 3.11 too
2023-06-29 23:54:51 +02:00
requires-python = ">=3.8"
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
authors = [
Fix authors, homepage URL in pyproject.toml and workflow triggers
2023-02-16 10:12:04 +01:00
{ name = "Debanjum Singh Solanky, Saba Imran" },
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
keywords = [
"search",
"semantic-search",
"productivity",
"NLP",
"AI",
"org-mode",
"markdown",
"images",
Wire up PDF to jsonl processor to Khoj server layer (API, config) - Specify PDF content to index via khoj.yml - Index PDF content on app start, reconfigure - Expose PDF as a search type via API
2023-06-01 06:25:48 +02:00
"pdf",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
classifiers = [
"Development Status :: 4 - Beta",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
Have Khoj support Python 3.11 as Pytorch supports it now - Previously Khoj could only support Python upto 3.10 due to pytorch. But lots of folks had python 3.11 installed by default on their machines. This required installing python 3.10 and dealing with virtual envs. With Torch >= 2.0.1 now able to support python 3.11, at least one class of installation troubles for Khoj should drop. See https://github.com/pytorch/pytorch/issues/86566 for reference - Preliminary testing indicates using the new torch 2.x may reduce search time by 25% (from 80ms to 60ms on Mac M1) - Update Docs to not require mentioning python <=3.10 required - Update Github test workflow to run khoj tests with python 3.11 too
2023-06-29 23:54:51 +02:00
"Programming Language :: Python :: 3.11",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"Topic :: Internet :: WWW/HTTP :: Indexing/Search",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Topic :: Scientific/Engineering :: Human Machine Interfaces",
"Topic :: Text Processing :: Linguistic",
]
dependencies = [
Fix plaintext HTML parsing and rendering (#464) * Store conversation command options in an Enum * Move to slash commands instead of using @ to specify general commands * Calculate conversation command once & pass it as arg to child funcs * Add /notes command to respond using only knowledge base as context This prevents the chat model to try respond using it's general world knowledge only without any references pulled from the indexed knowledge base * Test general and notes slash commands in openai chat director tests --------- Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
2023-08-27 20:24:30 +02:00
"bs4 >= 0.0.1",
Loosen dateparser dependency to get python3.10 wheel for regex package This should reduce chances of installation errors due to regex package being built from source for python3.11 Previously, the regex dependency of dateparser = 1.1.1 didn't have a wheel for python 3.11. This would trigger building the regex package from scratch which would fail for a lot of folks
2023-08-07 07:48:40 +02:00
"dateparser >= 1.1.1",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"defusedxml == 0.7.1",
"fastapi == 0.77.1",
"jinja2 == 3.1.2",
Improve and simplify Khoj Chat using ChatGPT - Set context by either including last 2 chat messages from active session or past 2 conversation summaries from conversation logs - Set personality in system message - Place personality system message before last completed back & forth This may stop ChatGPT forgetting its personality as conversation progresses given: - The conditioning based on system role messages is light - If system message is too far back in conversation history, the model may forget its personality conditioning - If system message at end of conversation, the model can think its the start of a new conversation - Inserting the system message before last completed back & forth should prevent ChatGPT from assuming its the start of a new conversation while not losing personality conditioning from the system message - Simplfy the Khoj Chat API to for now just answer from users notes instead of trying to infer other potential interaction types. - This is the default expected behavior from the feature anyway - Use the compiled text of the top 2 search results for context - Benefits of using ChatGPT - Better model - 1/10th the price - No hand rolled prompt required to make GPT provide more chatty, assistant type responses
2023-03-04 18:01:49 +01:00
"openai >= 0.27.0",
Retain desktop builds for 3 days to allow user tests Upgrade minimum tiktoken version to work for encoding gpt4
2023-08-09 08:00:40 +02:00
"tiktoken >= 0.3.2",
Add retry logic to OpenAI API queries to increase Chat tenacity - Move completion and chat_completion into helper methods under utils.py - Add retry with exponential backoff on OpenAI exceptions using tenacity package. This is officially suggested and used by other popular GPT based libraries
2023-03-25 06:36:51 +01:00
"tenacity >= 8.2.2",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"pillow == 9.3.0",
Upgrade minimum supported pydantic version
2023-07-03 21:22:56 +02:00
"pydantic >= 1.10.10",
Migrate from PyQT6 to PySide6
2023-07-12 03:43:44 +02:00
"pyside6 >= 6.5.1",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"pyyaml == 6.0",
"rich >= 13.3.1",
"schedule == 1.1.0",
"sentence-transformers == 2.2.2",
Fix the minimum version of the transformers library required to address #404
2023-08-02 18:10:14 +02:00
"transformers >= 4.28.0",
Have Khoj support Python 3.11 as Pytorch supports it now - Previously Khoj could only support Python upto 3.10 due to pytorch. But lots of folks had python 3.11 installed by default on their machines. This required installing python 3.10 and dealing with virtual envs. With Torch >= 2.0.1 now able to support python 3.11, at least one class of installation troubles for Khoj should drop. See https://github.com/pytorch/pytorch/issues/86566 for reference - Preliminary testing indicates using the new torch 2.x may reduce search time by 25% (from 80ms to 60ms on Mac M1) - Update Docs to not require mentioning python <=3.10 required - Update Github test workflow to run khoj tests with python 3.11 too
2023-06-29 23:54:51 +02:00
"torch >= 2.0.1",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"uvicorn == 0.17.6",
Fix bug in /chat endpoint for general and update depdendencies
2023-08-28 23:12:04 +02:00
"aiohttp == 3.8.5",
Extract prompts as LangChain Prompt Templates into a separate module Improves code modularity, cleanliness. Reduces bloat in GPT.py module
2023-05-28 14:15:31 +02:00
"langchain >= 0.0.187",
Use Github Rest API to index Markdown files in Github Repository The Llama_Hub Github plugin is fairly limited. The Github Rest API is well supported and can easily be extended to index commit messages, issues, discussions, PRs etc.
2023-06-17 10:39:57 +02:00
"requests >= 2.26.0",
Add beautiful as dependency to pass pytests
2023-06-24 00:10:09 +02:00
"bs4 >= 0.0.1",
Replace Falcon 🦅 model with Llama V2 🦙 for offline chat (#352) * Working example with LlamaV2 running locally on my machine - Download from huggingface - Plug in to GPT4All - Update prompts to fit the llama format * Add appropriate prompts for extracting questions based on a query based on llama format * Rename Falcon to Llama and make some improvements to the extract_questions flow * Do further tuning to extract question prompts and unit tests * Disable extracting questions dynamically from Llama, as results are still unreliable
2023-07-28 05:51:20 +02:00
"gpt4all >= 1.0.7",
Fix anyio package version to avoid backwards compatibility issue with start_blocking_portal method
2023-08-31 23:14:13 +02:00
"anyio == 3.7.1",
Add front-end Electron application for Khoj local file syncing (#473) * Initial version - setup a file-push architecture for generating embeddings with Khoj * Use state.host and state.port for configuring the URL for the indexer * Fix parsing of PDF files * Read markdown files from streamed data and update unit tests * On application startup, load in embeddings from configurations files, rather than regenerating the corpus based on file system * Init: refactor indexer/batch endpoint to support a generic file ingestion format * Add features to better support indexing from files sent by the desktop client * Initial commit with Electron application - Adds electron app * Add import for pymupdf, remove import for pypdf * Allow user to configure khoj host URL * Remove search type configuration from index.html * Use v1 path for current indexer routes
2023-09-06 21:04:18 +02:00
"pymupdf >= 1.23.3",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-18 00:08:05 +01:00
dynamic = ["version"]
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
[project.urls]
Rename project links across repo from debanjum/khoj to khoj-ai/khoj
2023-06-21 09:13:21 +02:00
Homepage = "https://github.com/khoj-ai/khoj#readme"
Issues = "https://github.com/khoj-ai/khoj/issues"
Discussions = "https://github.com/khoj-ai/khoj/discussions"
Releases = "https://github.com/khoj-ai/khoj/releases"
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
[project.scripts]
khoj = "khoj.main:run"
[project.optional-dependencies]
test = [
Add, configure and run pre-commit locally and in test workflow
2023-02-17 19:07:59 +01:00
"pytest >= 7.1.2",
Add trio, move freezegun, factory-boy to project test dependencies
2023-06-29 07:04:34 +02:00
"freezegun >= 1.2.0",
"factory-boy >= 3.2.1",
"trio >= 0.22.0",
Add, configure and run pre-commit locally and in test workflow
2023-02-17 19:07:59 +01:00
]
dev = [
"khoj-assistant[test]",
"mypy >= 1.0.1",
"black >= 23.1.0",
"pre-commit >= 3.0.4",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-18 00:08:05 +01:00
[tool.hatch.version]
source = "vcs"
raw-options.local_scheme = "no-local-version" # PEP440 compliant version for PyPi
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
[tool.hatch.build.targets.sdist]
include = ["src/khoj"]
[tool.hatch.build.targets.wheel]
Move mypy config into pyproject.toml. Ignore 2 remaining mypy issues
2023-02-16 10:33:08 +01:00
packages = ["src/khoj"]
[tool.mypy]
files = "src/khoj"
pretty = true
strict_optional = false
install_types = true
Add, configure and run pre-commit locally and in test workflow
2023-02-17 19:07:59 +01:00
ignore_missing_imports = true
Move mypy config into pyproject.toml. Ignore 2 remaining mypy issues
2023-02-16 10:33:08 +01:00
non_interactive = true
show_error_codes = true
Fix or ignore MyPy issues in PyQt desktop GUI code - Remove unneeded type ignore for mps with the latest mypy - Stop excluding PyQT desktop GUI code from MyPy checks - Do not warn about unused ignores. Some issue with mypy giving different errors in different environments (venv, system and pre-commit)
2023-02-17 23:11:17 +01:00
warn_unused_ignores = false
Use Black to format Khoj server code and tests
2023-02-17 17:04:26 +01:00
[tool.black]
Add, configure and run pre-commit locally and in test workflow
2023-02-17 19:07:59 +01:00
line-length = 120
Configure tests to mark chat quality tests & filter unhelpful warnings - Mark chat quality tests, register custom mark for chat quality - Filter unhelpful deprecation warnings from within dateparser library - Error if tests use unregistered marks
2023-03-15 01:56:14 +01:00
[tool.pytest.ini_options]
addopts = "--strict-markers"
markers = [
"chatquality: Evaluate chatbot capabilities and quality",
]
Reference in a new issue Copy permalink