Mirror of khoj from Github
Find a file
Debanjum Singh Solanky 86575b2946 Chunk text in preference order of para, sentence, word, character
- Previous simplistic chunking strategy of splitting text by space
  didn't capture notes with newlines, no spaces. For e.g in #620

- New strategy will try chunk the text at more natural points like
  paragraph, sentence, word first. If none of those work it'll split
  at character to fit within max token limit

- Drop long words while preserving original delimiters

Resolves #620
2024-04-04 02:41:55 +05:30
.github Update stale Khoj pypi package metadata 2024-03-29 00:06:55 +05:30
documentation Fix docs showing how to setup llama-cpp with Khoj 2024-03-31 15:36:40 +05:30
scripts Fix bump_version.sh to commit, clean-up after desktop app version bump 2023-12-22 21:42:03 +05:30
src Chunk text in preference order of para, sentence, word, character 2024-04-04 02:41:55 +05:30
tests Chunk text in preference order of para, sentence, word, character 2024-04-04 02:41:55 +05:30
.dockerignore Use pypi khoj to fix docker builds and dockerize github workflow 2023-02-19 01:57:01 -06:00
.gitattributes Exclude tests data file from programming stats on Github 2023-08-28 11:00:52 -07:00
.gitignore [Multi-User Part 5]: Add a production Docker file and use a gunicorn configuration with it (#514) 2023-10-26 13:15:31 -07:00
.pre-commit-config.yaml Add isort to the pre-commit configuration and apply it to the whole project (#595) 2023-12-28 18:04:02 +05:30
docker-compose.yml Set default value of KHOJ_DEBUG to False in the docker-compose file 2024-03-01 21:51:13 +05:30
Dockerfile Remove unused git dependency from Docker images 2024-02-16 17:41:09 +05:30
gunicorn-config.py Configure production setup for moving to single worker model 2024-03-30 10:35:55 +05:30
LICENSE Change license to GNU AGPLv3 from GNU GPLv3 2023-11-16 11:14:06 -08:00
manifest.json Release Khoj version 1.8.0 2024-03-31 00:06:15 +05:30
prod.Dockerfile Configure production setup for moving to single worker model 2024-03-30 10:35:55 +05:30
pyproject.toml Rebase with matser 2024-04-02 16:16:06 +05:30
pytest.ini Move the django app into the src/khoj folder for better organization and functionality 2023-11-21 10:56:04 -08:00
README.md Add num online for Discord badge 2024-03-10 17:48:30 +05:30
versions.json Release Khoj version 1.8.0 2024-03-31 00:06:15 +05:30

Khoj Logo

test dockerize pypi Discord

An AI personal assistant for your digital brain


Khoj is an AI application to search and chat with your notes and documents.
It is open-source, self-hostable and accessible on Desktop, Emacs, Obsidian, Web and Whatsapp.
It works with pdf, markdown, org-mode, notion files and github repositories.
It can paint, search the internet and understand speech.


🔎 Search 💬 Chat
Quickly retrieve relevant documents using natural language Get answers and create content from your existing knowledge base
Does not need internet Can be configured to work without internet

Contributors

Cheers to our awesome contributors! 🎉

Made with contrib.rocks.