sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-18 02:27:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 23
khoj/config/khoj_sample.yml

25 lines
743 B
YAML
Raw Normal View History

Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
content-type:
Refactor Khoj - Panchayat integration to represent minimal changes required + name the new endpoint Panchayat
2022-08-27 15:05:30 +00:00
panchayat:
input-files: null
input-filter: "/home/saba/projects/panchayat/panchayat/instance/*.yaml"
compressed-jsonl: "./khoj/embeddings/panchyat.jsonl.gz"
embeddings-file: "./khoj/embeddings/panchayat_embeddings.pt"
Update sample config to add minimal config for ledger, image search
2021-08-23 04:54:49 +00:00
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
search-type:
Add Search Config for Symmetric Model. Save Model to Disk
2022-01-14 21:46:56 +00:00
symmetric:
Use a more accurate model for symmetric semantic search - The all-MiniLM-L6-v2 is more accurate - The exact previous model isn't benchmarked but based on the performance of the closest model to it. Seems like the new model maybe similar in speed and size - On very preliminary evaluation of the model, the new model seems faster, with pretty decent results
2022-07-18 16:16:40 +00:00
encoder: "sentence-transformers/all-MiniLM-L6-v2"
Add Search Config for Symmetric Model. Save Model to Disk
2022-01-14 21:46:56 +00:00
cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
model_directory: "~/.khoj/search/symmetric/"
Add Search Config for Symmetric Model. Save Model to Disk
2022-01-14 21:46:56 +00:00
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
asymmetric:
Use a better model for asymmetric semantic search - The multi-qa-MiniLM-L6-cos-v1 is more extensively benchmarked[1] - It has the right mix of model query speed, size and performance on benchmarks - On hugging face it has way more downloads and likes than the msmarco model[2] - On very preliminary evaluation of the model - It doubles the encoding speed of all entries (down from ~8min to 4mins) - It gave more entries that stay relevant to the query (3/5 vs 1/5 earlier) [1]: https://www.sbert.net/docs/pretrained_models.html [2]: https://huggingface.co/sentence-transformers
2022-07-18 16:00:19 +00:00
encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1"
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
model_directory: "~/.khoj/search/asymmetric/"
Update sample config to add minimal config for ledger, image search
2021-08-23 04:54:49 +00:00
image:
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
encoder: "sentence-transformers/clip-ViT-B-32"
model_directory: "~/.khoj/search/image/"
Make conversation processor configurable
2021-11-26 19:56:26 +00:00
processor:
Reference in a new issue Copy permalink