sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-18 10:37:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 23
khoj/config/khoj_sample.yml

58 lines
2.5 KiB
YAML
Raw Normal View History

Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
content-type:
org:
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
input-files: # ["/path/to/org-file.org"] REQUIRED IF input-filter IS NOT SET OR
Quote all values passed to input-filter fields in sample yaml files
2023-01-09 01:40:18 +00:00
input-filter: # ["/path/to/org/*.org"] REQUIRED IF input-files IS NOT SET
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
compressed-jsonl: "~/.khoj/content/org/org.jsonl.gz"
embeddings-file: "~/.khoj/content/org/org_embeddings.pt"
Add index_heading_entries flag to default and sample khoj configs
2022-09-11 14:30:02 +00:00
index_heading_entries: false # Set to true to index entries with empty body
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
Create test markdown files. Use them in sample config, docker-compose
2022-07-21 17:54:30 +00:00
markdown:
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
input-files: # ["/path/to/markdown-file.md"] REQUIRED IF input-filter IS NOT SET OR
Set input-filter to list in khoj_docker.yml and khoj_sample.yml `input-filter' was converted to a list a while back but the sample khoj configs were not updated to reflect this. This change fixes that
2023-01-09 00:08:00 +00:00
input-filter: # ["/path/to/markdown/*.md"] REQUIRED IF input-files IS NOT SET
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
compressed-jsonl: "~/.khoj/content/markdown/markdown.jsonl.gz"
embeddings-file: "~/.khoj/content/markdown/markdown_embeddings.pt"
Create test markdown files. Use them in sample config, docker-compose
2022-07-21 17:54:30 +00:00
Update sample config to add minimal config for ledger, image search
2021-08-23 04:54:49 +00:00
ledger:
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
input-files: # ["/path/to/ledger-file.beancount"] REQUIRED IF input-filter is not set OR
Quote all values passed to input-filter fields in sample yaml files
2023-01-09 01:40:18 +00:00
input-filter: # ["/path/to/ledger/*.beancount"] REQUIRED IF input-files is not set
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
compressed-jsonl: "~/.khoj/content/ledger/ledger.jsonl.gz"
embeddings-file: "~/.khoj/content/ledger/ledger_embeddings.pt"
Update sample config to add minimal config for ledger, image search
2021-08-23 04:54:49 +00:00
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
image:
Quote all values passed to input-filter fields in sample yaml files
2023-01-09 01:40:18 +00:00
input-directories: # ["/path/to/images/"] REQUIRED IF input-filter IS NOT SET OR
input-filter: # ["/path/to/images/*.jpg"] REQUIRED IF input-directories IS NOT SET
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
embeddings-file: "~/.khoj/content/image/image_embeddings.pt"
batch-size: 50
Use of XMP metadata in Khoj Image Search is broken. Disable by default - CLIP Image score and XMP metadata score are not combining well. When combined they give non sensical results. Enable only once figure how best to combine the two. - Show scores with higher precision for image search - Image search scores seem to be mostly be between 0.2 - 0.3 for some reason - Higher precision scores make it easier to understand the quality of returned results perceived by the model itself
2022-08-18 20:10:33 +00:00
use-xmp-metadata: false
Add all configurables to sample_config. Add test music, ledger data - Get ledger sample from github.com/debanjum/company-ledger - Get music sample from github.com/debanjum/org-music
2021-10-02 23:11:27 +00:00
music:
Quote all values passed to input-filter fields in sample yaml files
2023-01-09 01:40:18 +00:00
input-files: # ["/path/to/music-file.org"] REQUIRED IF input-filter IS NOT SET OR
input-filter: # ["/path/to/music/*.org"] REQUIRED IF input-files IS NOT SET
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
compressed-jsonl: "~/.khoj/content/music/music.jsonl.gz"
embeddings-file: "~/.khoj/content/music/music_embeddings.pt"
Update sample config to add minimal config for ledger, image search
2021-08-23 04:54:49 +00:00
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
search-type:
Add Search Config for Symmetric Model. Save Model to Disk
2022-01-14 21:46:56 +00:00
symmetric:
Use a more accurate model for symmetric semantic search - The all-MiniLM-L6-v2 is more accurate - The exact previous model isn't benchmarked but based on the performance of the closest model to it. Seems like the new model maybe similar in speed and size - On very preliminary evaluation of the model, the new model seems faster, with pretty decent results
2022-07-18 16:16:40 +00:00
encoder: "sentence-transformers/all-MiniLM-L6-v2"
Add Search Config for Symmetric Model. Save Model to Disk
2022-01-14 21:46:56 +00:00
cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
Add encoder-type field to the search-type sections in khoj_sample.yml
2023-01-09 01:07:13 +00:00
encoder-type: sentence_transformers.SentenceTransformer
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
model_directory: "~/.khoj/search/symmetric/"
Add Search Config for Symmetric Model. Save Model to Disk
2022-01-14 21:46:56 +00:00
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
asymmetric:
Use a better model for asymmetric semantic search - The multi-qa-MiniLM-L6-cos-v1 is more extensively benchmarked[1] - It has the right mix of model query speed, size and performance on benchmarks - On hugging face it has way more downloads and likes than the msmarco model[2] - On very preliminary evaluation of the model - It doubles the encoding speed of all entries (down from ~8min to 4mins) - It gave more entries that stay relevant to the query (3/5 vs 1/5 earlier) [1]: https://www.sbert.net/docs/pretrained_models.html [2]: https://huggingface.co/sentence-transformers
2022-07-18 16:00:19 +00:00
encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1"
Use YAML file to allow user to configure application. Add tests - YAML Config - Can specify all params[1] earlier being passed via cmd args in config YAML - Can now also configure sentence-transformer models to use etc for search - [1] Config params - org files - compressed entries file config path - embeddings file config path - Include sample_config.yaml - Include sample .org file from this repos readmes - CLI - Configuration Priority: Config via cmd > Config via YAML > Default Config - Test CLI, include test config.yml for the tests - Set default type to None unless set via query param to API Run notes search if search_enabled, also if type is None (default) Prepares for running queries on all search types unless type specified in API query param - Update Readme
2021-08-22 01:47:55 +00:00
cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
Add encoder-type field to the search-type sections in khoj_sample.yml
2023-01-09 01:07:13 +00:00
encoder-type: sentence_transformers.SentenceTransformer
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
model_directory: "~/.khoj/search/asymmetric/"
Update sample config to add minimal config for ledger, image search
2021-08-23 04:54:49 +00:00
image:
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
encoder: "sentence-transformers/clip-ViT-B-32"
Add encoder-type field to the search-type sections in khoj_sample.yml
2023-01-09 01:07:13 +00:00
encoder-type: sentence_transformers.SentenceTransformer
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
model_directory: "~/.khoj/search/image/"
Make conversation processor configurable
2021-11-26 19:56:26 +00:00
processor:
conversation:
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
openai-api-key: # "YOUR_OPENAI_API_KEY"
Make OpenAI conversation model configurable via khoj.yml - Default to using `text-davinci-003' if conversation model not explicitly configured by user. Stop using the older `davinci' and `davinci-instruct' models - Use `model' instead of `engine' as parameter. Usage of `engine' parameter in OpenAI API is deprecated
2023-01-09 03:08:03 +00:00
model: "text-davinci-003"
Add chat-model param in sample config yml and document (#341) * add chat-model config param to docs * add chat-model param to sample config yml
2023-07-22 23:53:08 +00:00
chat-model: "gpt-3.5-turbo"
Create khoj_sample file with all configurable fields in one place - Reason - Simplifies code. No merge_dict required - 1 place for user to see all configurables, defaults and required values - Details - Remove default_config from code. Set defaults in khoj_sample.yml itself - Keep fields required to be set by user as empty in khoj_sample to YAML - Set defaults for fields not requiring configuration by user
2022-08-04 19:57:35 +00:00
conversation-logfile: "~/.khoj/processor/conversation/conversation_logs.json"
Reference in a new issue Copy permalink