sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-18 02:27:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 23
khoj/config/sample_config.yml

53 lines
1.7 KiB
YAML
Raw Normal View History

Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
content-type:
Merge sample_config, docker_sample_config yml into a single sample_config.yml - Update readme to indicate how to update the new sample_config to run on test data
2022-01-29 06:32:12 +00:00
# The /data/folder/ prefix to the folders is here because this is
# the directory to which the local files are copied in the docker-compose.
# If changing, the docker-compose volumes should also be changed to match.
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
org:
Merge sample_config, docker_sample_config yml into a single sample_config.yml - Update readme to indicate how to update the new sample_config to run on test data
2022-01-29 06:32:12 +00:00
input-files: null
input-filter: "/data/notes/*.org"
Minor fix to notes jsonl file extension in sample_config.yml
2022-01-29 09:13:36 +00:00
compressed-jsonl: "/data/embeddings/notes.jsonl.gz"
Mount embeddings to /data/embeddings for directory naming consistency - Keeps directory paths consistent between host and container volumes - Consistency simplifies documentation and updates required to setup sample_config.yml for local installation
2022-01-29 08:24:02 +00:00
embeddings-file: "/data/embeddings/note_embeddings.pt"
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
Create test markdown files. Use them in sample config, docker-compose
2022-07-21 17:54:30 +00:00
markdown:
input-files: null
input-filter: "/data/markdown/*.md"
compressed-jsonl: "/data/embeddings/markdown.jsonl.gz"
embeddings-file: "/data/embeddings/markdown_embeddings.pt"
Update sample config to add minimal config for ledger, image search
2021-08-23 04:54:49 +00:00
ledger:
Add all configurables to sample_config. Add test music, ledger data - Get ledger sample from github.com/debanjum/company-ledger - Get music sample from github.com/debanjum/org-music
2021-10-02 23:11:27 +00:00
input-files: null
Merge sample_config, docker_sample_config yml into a single sample_config.yml - Update readme to indicate how to update the new sample_config to run on test data
2022-01-29 06:32:12 +00:00
input-filter: /data/ledger/*.beancount
Mount embeddings to /data/embeddings for directory naming consistency - Keeps directory paths consistent between host and container volumes - Consistency simplifies documentation and updates required to setup sample_config.yml for local installation
2022-01-29 08:24:02 +00:00
compressed-jsonl: /data/embeddings/transactions.jsonl.gz
embeddings-file: /data/embeddings/transaction_embeddings.pt
Update sample config to add minimal config for ledger, image search
2021-08-23 04:54:49 +00:00
image:
Allow indexing multiple image directories for image search
2022-07-19 22:54:03 +00:00
input-directories: ["/data/images/"]
Mount embeddings to /data/embeddings for directory naming consistency - Keeps directory paths consistent between host and container volumes - Consistency simplifies documentation and updates required to setup sample_config.yml for local installation
2022-01-29 08:24:02 +00:00
embeddings-file: "/data/embeddings/image_embeddings.pt"
Make image batch size to encode configurable via config.yml
2021-09-16 17:51:39 +00:00
batch-size: 50
Merge sample_config, docker_sample_config yml into a single sample_config.yml - Update readme to indicate how to update the new sample_config to run on test data
2022-01-29 06:32:12 +00:00
use-xmp-metadata: true
Add all configurables to sample_config. Add test music, ledger data - Get ledger sample from github.com/debanjum/company-ledger - Get music sample from github.com/debanjum/org-music
2021-10-02 23:11:27 +00:00
music:
Merge sample_config, docker_sample_config yml into a single sample_config.yml - Update readme to indicate how to update the new sample_config to run on test data
2022-01-29 06:32:12 +00:00
input-files: ["/data/music/music.org"]
Add all configurables to sample_config. Add test music, ledger data - Get ledger sample from github.com/debanjum/company-ledger - Get music sample from github.com/debanjum/org-music
2021-10-02 23:11:27 +00:00
input-filter: null
Mount embeddings to /data/embeddings for directory naming consistency - Keeps directory paths consistent between host and container volumes - Consistency simplifies documentation and updates required to setup sample_config.yml for local installation
2022-01-29 08:24:02 +00:00
compressed-jsonl: "/data/embeddings/songs.jsonl.gz"
embeddings-file: "/data/embeddings/song_embeddings.pt"
Update sample config to add minimal config for ledger, image search
2021-08-23 04:54:49 +00:00
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
search-type:
Add Search Config for Symmetric Model. Save Model to Disk
2022-01-14 21:46:56 +00:00
symmetric:
Use a more accurate model for symmetric semantic search - The all-MiniLM-L6-v2 is more accurate - The exact previous model isn't benchmarked but based on the performance of the closest model to it. Seems like the new model maybe similar in speed and size - On very preliminary evaluation of the model, the new model seems faster, with pretty decent results
2022-07-18 16:16:40 +00:00
encoder: "sentence-transformers/all-MiniLM-L6-v2"
Add Search Config for Symmetric Model. Save Model to Disk
2022-01-14 21:46:56 +00:00
cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
Merge sample_config, docker_sample_config yml into a single sample_config.yml - Update readme to indicate how to update the new sample_config to run on test data
2022-01-29 06:32:12 +00:00
model_directory: "/data/models/symmetric"
Add Search Config for Symmetric Model. Save Model to Disk
2022-01-14 21:46:56 +00:00
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
asymmetric:
Use a better model for asymmetric semantic search - The multi-qa-MiniLM-L6-cos-v1 is more extensively benchmarked[1] - It has the right mix of model query speed, size and performance on benchmarks - On hugging face it has way more downloads and likes than the msmarco model[2] - On very preliminary evaluation of the model - It doubles the encoding speed of all entries (down from ~8min to 4mins) - It gave more entries that stay relevant to the query (3/5 vs 1/5 earlier) [1]: https://www.sbert.net/docs/pretrained_models.html [2]: https://huggingface.co/sentence-transformers
2022-07-18 16:00:19 +00:00
encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1"
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
Merge sample_config, docker_sample_config yml into a single sample_config.yml - Update readme to indicate how to update the new sample_config to run on test data
2022-01-29 06:32:12 +00:00
model_directory: "/data/models/asymmetric"
Update sample config to add minimal config for ledger, image search
2021-08-23 04:54:49 +00:00
image:
encoder: "clip-ViT-B-32"
Merge sample_config, docker_sample_config yml into a single sample_config.yml - Update readme to indicate how to update the new sample_config to run on test data
2022-01-29 06:32:12 +00:00
model_directory: "/data/models/image_encoder"
Make conversation processor configurable
2021-11-26 19:56:26 +00:00
processor:
conversation:
openai-api-key: null
Mount embeddings to /data/embeddings for directory naming consistency - Keeps directory paths consistent between host and container volumes - Consistency simplifies documentation and updates required to setup sample_config.yml for local installation
2022-01-29 08:24:02 +00:00
conversation-logfile: "/data/embeddings/conversation_logs.json"
Reference in a new issue Copy permalink