khoj/docs/advanced.md


## Advanced Usage

### Search across Different Languages (Self-Hosting)
To search for notes in multiple, different languages, you can use a [multi-lingual model](https://www.sbert.net/docs/pretrained_models.html#multi-lingual-models).<br />
For example, the [paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) supports [50+ languages](https://www.sbert.net/docs/pretrained_models.html#:~:text=we%20used%20the%20following%2050%2B%20languages), has good search quality and speed. To use it:
1. Manually update the search config in server's admin settings page. Go to [the search config](http://localhost:42110/server/admin/database/searchmodelconfig/). Either create a new one, if none exists, or update the existing one. Set the bi_encoder to `sentence-transformers/multi-qa-MiniLM-L6-cos-v1` and the cross_encoder to `cross-encoder/ms-marco-MiniLM-L-6-v2`.
2. Regenerate your content index from all the relevant clients. This step is very important, as you'll need to re-encode all your content with the new model.

### Query Filters

Use structured query syntax to filter entries from your knowledge based used by search results or chat responses.

- **Word Filter**: Get entries that include/exclude a specified term
  - Entries that contain term_to_include: `+"term_to_include"`
  - Entries that contain term_to_exclude: `-"term_to_exclude"`
- **Date Filter**: Get entries containing dates in YYYY-MM-DD format from specified date (range)
  - Entries from April 1st 1984: `dt:"1984-04-01"`
  - Entries after March 31st 1984: `dt>="1984-04-01"`
  - Entries before April 2nd 1984 : `dt<="1984-04-01"`
- **File Filter**: Get entries from a specified file
  - Entries from incoming.org file: `file:"incoming.org"`
- Combined Example
  - `what is the meaning of life? file:"1984.org" dt>="1984-01-01" dt<="1985-01-01" -"big" -"brother"`
  - Adds all filters to the natural language query. It should return entries
    - from the file *1984.org*
    - containing dates from the year *1984*
    - excluding words *"big"* and *"brother"*
    - that best match the natural language query *"what is the meaning of life?"*
📝 Add a documentation base for Khoj! (#333) * Add docs for more organized, accessible information detailing Khoj setup * Delete duplicated files * Add a coverpage without enabling it. Add logo and theme * Remove obsidian README.md * Add plausible script to index.html via docsify 2023-07-21 07:34:25 +02:00
			`## Advanced Usage`
Update references to all documentation to reflect instructions for managed service - By default assume the audience of this website is people looking to understand the featuer offering of Khoj, and then people who are looking to self-host 2023-11-17 00:26:03 +01:00
			`### Search across Different Languages (Self-Hosting)`
Fix diff blocks, links, remove footnotes & rearrange sections in docs Extract performance into separate sectin into shoving it under search Create page for web interface 2023-07-21 09:05:44 +02:00			`To search for notes in multiple, different languages, you can use a [multi-lingual model](https://www.sbert.net/docs/pretrained_models.html#multi-lingual-models).<br />`
			`For example, the [paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) supports [50+ languages](https://www.sbert.net/docs/pretrained_models.html#:~:text=we%20used%20the%20following%2050%2B%20languages), has good search quality and speed. To use it:`
Update references to all documentation to reflect instructions for managed service - By default assume the audience of this website is people looking to understand the featuer offering of Khoj, and then people who are looking to self-host 2023-11-17 00:26:03 +01:00			1. Manually update the search config in server's admin settings page. Go to [the search config](http://localhost:42110/server/admin/database/searchmodelconfig/). Either create a new one, if none exists, or update the existing one. Set the bi_encoder to `sentence-transformers/multi-qa-MiniLM-L6-cos-v1` and the cross_encoder to `cross-encoder/ms-marco-MiniLM-L-6-v2`.
			`2. Regenerate your content index from all the relevant clients. This step is very important, as you'll need to re-encode all your content with the new model.`
Document slash command and query filter in docs for chat and search 2023-08-28 19:43:17 +02:00
			`### Query Filters`

			`Use structured query syntax to filter entries from your knowledge based used by search results or chat responses.`

			`- Word Filter: Get entries that include/exclude a specified term`
			- Entries that contain term_to_include: `+"term_to_include"`
			- Entries that contain term_to_exclude: `-"term_to_exclude"`
			`- Date Filter: Get entries containing dates in YYYY-MM-DD format from specified date (range)`
			- Entries from April 1st 1984: `dt:"1984-04-01"`
			- Entries after March 31st 1984: `dt>="1984-04-01"`
			- Entries before April 2nd 1984 : `dt<="1984-04-01"`
			`- File Filter: Get entries from a specified file`
			- Entries from incoming.org file: `file:"incoming.org"`
			`- Combined Example`
			- `what is the meaning of life? file:"1984.org" dt>="1984-01-01" dt<="1985-01-01" -"big" -"brother"`
			`- Adds all filters to the natural language query. It should return entries`
			`- from the file 1984.org`
			`- containing dates from the year 1984`
			`- excluding words "big" and "brother"`
			`- that best match the natural language query "what is the meaning of life?"`