sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-30 10:53:02 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
khoj/pyproject.toml

111 lines
2.6 KiB
TOML
Raw Normal View History

Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
[build-system]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-18 00:08:05 +01:00
requires = ["hatchling", "hatch-vcs"]
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
build-backend = "hatchling.build"
[project]
name = "khoj-assistant"
description = "A natural language search engine for your personal notes, transactions and images"
readme = "README.md"
license = "GPL-3.0-or-later"
requires-python = ">=3.8, <3.11"
authors = [
Fix authors, homepage URL in pyproject.toml and workflow triggers
2023-02-16 10:12:04 +01:00
{ name = "Debanjum Singh Solanky, Saba Imran" },
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
keywords = [
"search",
"semantic-search",
"productivity",
"NLP",
"AI",
"org-mode",
"markdown",
"beancount",
"images",
Wire up PDF to jsonl processor to Khoj server layer (API, config) - Specify PDF content to index via khoj.yml - Index PDF content on app start, reconfigure - Expose PDF as a search type via API
2023-06-01 06:25:48 +02:00
"pdf",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
classifiers = [
"Development Status :: 4 - Beta",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Internet :: WWW/HTTP :: Indexing/Search",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Topic :: Scientific/Engineering :: Human Machine Interfaces",
"Topic :: Text Processing :: Linguistic",
]
dependencies = [
"dateparser == 1.1.1",
"defusedxml == 0.7.1",
"fastapi == 0.77.1",
"jinja2 == 3.1.2",
Improve and simplify Khoj Chat using ChatGPT - Set context by either including last 2 chat messages from active session or past 2 conversation summaries from conversation logs - Set personality in system message - Place personality system message before last completed back & forth This may stop ChatGPT forgetting its personality as conversation progresses given: - The conditioning based on system role messages is light - If system message is too far back in conversation history, the model may forget its personality conditioning - If system message at end of conversation, the model can think its the start of a new conversation - Inserting the system message before last completed back & forth should prevent ChatGPT from assuming its the start of a new conversation while not losing personality conditioning from the system message - Simplfy the Khoj Chat API to for now just answer from users notes instead of trying to infer other potential interaction types. - This is the default expected behavior from the feature anyway - Use the compiled text of the top 2 search results for context - Benefits of using ChatGPT - Better model - 1/10th the price - No hand rolled prompt required to make GPT provide more chatty, assistant type responses
2023-03-04 18:01:49 +01:00
"openai >= 0.27.0",
Truncate message logs to below max supported prompt size by model - Use tiktoken to count tokens for chat models - Make conversation turns to add to prompt configurable via method argument to generate_chatml_messages_with_context method
2023-03-24 22:37:55 +01:00
"tiktoken >= 0.3.0",
Add retry logic to OpenAI API queries to increase Chat tenacity - Move completion and chat_completion into helper methods under utils.py - Add retry with exponential backoff on OpenAI exceptions using tenacity package. This is officially suggested and used by other popular GPT based libraries
2023-03-25 06:36:51 +01:00
"tenacity >= 8.2.2",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"pillow == 9.3.0",
Create PDF to JSONL processor using PyPDF and LangChain Switch `pydantic' to >= 1.9.1 else `langchain.document_loaders' starts throwing typing error for python 3.8, 3.9
2023-06-01 06:01:18 +02:00
"pydantic >= 1.9.1",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
"pyqt6 == 6.3.1",
"pyyaml == 6.0",
"rich >= 13.3.1",
"schedule == 1.1.0",
"sentence-transformers == 2.2.2",
"torch == 1.13.1",
"uvicorn == 0.17.6",
Use aiohttp version 3.8.4 as 4.x breaks docker image build
2023-03-25 23:10:53 +01:00
"aiohttp == 3.8.4",
Extract prompts as LangChain Prompt Templates into a separate module Improves code modularity, cleanliness. Reduces bloat in GPT.py module
2023-05-28 14:15:31 +02:00
"langchain >= 0.0.187",
Create PDF to JSONL processor using PyPDF and LangChain Switch `pydantic' to >= 1.9.1 else `langchain.document_loaders' starts throwing typing error for python 3.8, 3.9
2023-06-01 06:01:18 +02:00
"pypdf >= 3.9.0",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-18 00:08:05 +01:00
dynamic = ["version"]
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
[project.urls]
Fix authors, homepage URL in pyproject.toml and workflow triggers
2023-02-16 10:12:04 +01:00
Homepage = "https://github.com/debanjum/khoj#readme"
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
Issues = "https://github.com/debanjum/khoj/issues"
Discussions = "https://github.com/debanjum/khoj/discussions"
Releases = "https://github.com/debanjum/khoj/releases"
[project.scripts]
khoj = "khoj.main:run"
[project.optional-dependencies]
test = [
Add, configure and run pre-commit locally and in test workflow
2023-02-17 19:07:59 +01:00
"pytest >= 7.1.2",
]
dev = [
"khoj-assistant[test]",
"mypy >= 1.0.1",
"black >= 23.1.0",
"pre-commit >= 3.0.4",
Test Chat Director Capabilities: Answer from notes, chat history etc - Chat directors are broad agents. - Chat directors orchestrate narrow actor agents to synthesize final response for the user - Agents are Prompts + ML Model - Test Chat Director Capabilities 1. [X] Answer from retrieved notes 2. [X] Answer from chat history 3. [X] Answer general questions 4. [X] Carry out multi-turn conversation 5. [X] Say don't know when answer not in provided context 6. [X] Answers that require current date awareness This test is expected to fail as the chat is not capable of doing this without the Search actor. But the test allows assessing chat quality 7. [X] Date-aware aggregation across multiple different notes This test is expected to fail as the chat is not capable of doing this without the Search actor. But the test allows assessing chat quality 8. [X] Ask clarification questions if no unambiguous answer in provided context 9. [X] Retrieve answer from chat history beyond lookback window This test is expected to fail as the chat director is not capable of searching chat history yet. But the test allows assessing chat quality 10. [X] Retrieve context for answer using multiple independent searches on knowledge base This test is expected to fail as the chat is not capable of doing this without the Search actor. But the test allows assessing chat quality
2023-03-15 21:52:59 +01:00
"freezegun >= 1.2.0",
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
]
Automate khoj python package versioning using hatch-vcs and Git tags
2023-02-18 00:08:05 +01:00
[tool.hatch.version]
source = "vcs"
raw-options.local_scheme = "no-local-version" # PEP440 compliant version for PyPi
Build khoj python package using hatchling, pyproject.toml - Why - pyprojects.toml is the python standards compliant config format - allows collating python tooling configs into single standard file - hatch(-ling) is a new lightweight build system for python packages - Detailed Changes - Replace setup.py, setuptools with pyproject.toml, hatchling for khoj python config and build - move pytest into optional development dependencies - add more links to khoj in the project urls section - add topic classifiers and keywords to find khoj package - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup - Update pypi workflow to set python package version in pyproject.toml
2023-02-16 02:02:30 +01:00
[tool.hatch.build.targets.sdist]
include = ["src/khoj"]
[tool.hatch.build.targets.wheel]
Move mypy config into pyproject.toml. Ignore 2 remaining mypy issues
2023-02-16 10:33:08 +01:00
packages = ["src/khoj"]
[tool.mypy]
files = "src/khoj"
pretty = true
strict_optional = false
install_types = true
Add, configure and run pre-commit locally and in test workflow
2023-02-17 19:07:59 +01:00
ignore_missing_imports = true
Move mypy config into pyproject.toml. Ignore 2 remaining mypy issues
2023-02-16 10:33:08 +01:00
non_interactive = true
show_error_codes = true
Fix or ignore MyPy issues in PyQt desktop GUI code - Remove unneeded type ignore for mps with the latest mypy - Stop excluding PyQT desktop GUI code from MyPy checks - Do not warn about unused ignores. Some issue with mypy giving different errors in different environments (venv, system and pre-commit)
2023-02-17 23:11:17 +01:00
warn_unused_ignores = false
Use Black to format Khoj server code and tests
2023-02-17 17:04:26 +01:00
[tool.black]
Add, configure and run pre-commit locally and in test workflow
2023-02-17 19:07:59 +01:00
line-length = 120
Configure tests to mark chat quality tests & filter unhelpful warnings - Mark chat quality tests, register custom mark for chat quality - Filter unhelpful deprecation warnings from within dateparser library - Error if tests use unregistered marks
2023-03-15 01:56:14 +01:00
[tool.pytest.ini_options]
addopts = "--strict-markers"
markers = [
"chatquality: Evaluate chatbot capabilities and quality",
]
Reference in a new issue Copy permalink