Commit graph

74 commits

Author SHA1 Message Date
Debanjum Singh Solanky
110e3df0b7 Set default config in the constant module. Use from there to configure app
- Avoid having to pass the khoj_sample.yml data file into pip, native apps
- Packaging data files into python packages is annoying.
  - There's `MANIFEST.in`, `data_files` and `package_data` in setup.py
  - Bdist, wheel, generated source tarball use different set of these fields
    and put the data files in different locations
  - Rather just code the default config into a constant. Avoid
    pointless file reads as well this way
2022-08-12 02:18:46 +03:00
Debanjum Singh Solanky
fad2f3a2e7 Resolve config_file to absolute right at start on parsing args in cli
- Assume path is absolute in yaml util module while saving, loading file
  - This follows same convention as jsonl. Which just operates on
    passed file path, assuming it is of appropriate form.
    Responsibility to put it in appropriate form is on the caller, for now
2022-08-12 01:34:08 +03:00
Debanjum Singh Solanky
44fe70513a Handle situation where default config directory or file does not exist
- Include khoj_sample.yml in pip package to load default config from
- Create khoj config directory if it doesn't exist
- Load config from khoj_sample.yml if khoj.yml config doesn't exist
2022-08-12 01:17:34 +03:00
Debanjum Singh Solanky
6af2d6bb6d Add Flag to Start App without Native GUI 2022-08-11 20:59:57 +03:00
Debanjum Singh Solanky
1ff049599f Show current config on config screen. Load default config if config unset
- Track current (saved/loaded) config separate from the new config (to
  be written) when user clicks Start

- Fallback to using default config when no config for the specific
  content type or processor is specified in khoj.yml
  - Earlier were only loading default config on first run, not after

- Create Child CheckBox, LineEdit classes for Processor Widgets
  - Create ProcessorType, similar to SearchType
  - Track ProcessorType the widgets are associated with
  - Simplify update, save, load of config based on type
2022-08-11 19:11:25 +03:00
Debanjum Singh Solanky
23e06f483d Do not emit type tags when dumping config YAML to file 2022-08-11 19:08:36 +03:00
Debanjum Singh Solanky
34018c7d4b Store args passed from commandline at app start in global app state 2022-08-11 00:11:35 +03:00
Debanjum Singh Solanky
62eb66b8ca Rename load_config_from_file to more descriptive parse_config_from_file 2022-08-10 22:28:51 +03:00
Debanjum Singh Solanky
328cc00439 Create global constant to store app root directory 2022-08-10 20:09:03 +03:00
Debanjum Singh Solanky
d2c7b28172 Extract code to load config from YAML file into new utils.yaml module 2022-08-10 20:07:44 +03:00
Debanjum Singh Solanky
a588a8e21f Make config_file an optional argument. It can be generated on FRE
- Make config_file an optional arg. It defaults to default khoj config dir
- Return args.config as None if no config_file explicitly passed by user
- Parent can use args.config = None as signal to trigger first run experience
2022-08-09 17:02:02 +03:00
Debanjum Singh Solanky
21af122447 Clean up unused methods, module imports. Add comments 2022-08-09 16:59:38 +03:00
Debanjum Singh Solanky
7b04978f52 Put global state variables into separate state module
- Variables storing app, device state aren't constants.
  Do not mix with actual constants like empty_escape_sequence, web_directory
2022-08-06 03:13:18 +03:00
Debanjum Singh Solanky
b04c84721b Extract configure and routers from main.py into separate modules
- Main.py was becoming too big to manage. It had both
  controllers/routers and component configurations (search, processors)
  in it

- Now that the native app GUI code is also getting added to the main
  path, good time to split/modularize/clean main.py

- Put global state into a separate file to share across modules
2022-08-06 02:39:18 +03:00
Debanjum Singh Solanky
675e821d95 Make embeddings, jsonl paths absolute. Create directories if non-existent 2022-08-05 02:57:59 +03:00
Debanjum Singh Solanky
ca5a8bd113 Make config file a positional argument, as it is required
- Test invalid config file path throws. Remove redundant cli test

- Simplify cli parser code
  - Do not need to explicitly check if args.config_file set.
    argparser checks for positional arguments automatically

- Use standard semantics for cli args
  - All positional args are required. Non positional args are optional

- Improve command line --help description
2022-08-05 01:09:40 +03:00
Debanjum Singh Solanky
1374065092 Mark all required fields for config. Throw if no input_* field specified
- Add custom validator to throw if neither input_filter or
  input_<files|directories> are specified

- Set field expecting paths to type Path

- Now that default_config isn't used in code. We can update
  fields in rawconfig to specify whether they're required or not.
  This lets pydantic validate config file and throw appropriate error
2022-08-05 01:08:48 +03:00
Debanjum Singh Solanky
f78d6ae754 Create khoj_sample file with all configurable fields in one place
- Reason
  - Simplifies code. No merge_dict required
  - 1 place for user to see all configurables, defaults and required values

- Details
  - Remove default_config from code. Set defaults in khoj_sample.yml itself
  - Keep fields required to be set by user as empty in khoj_sample to YAML
  - Set defaults for fields not requiring configuration by user
2022-08-05 01:08:33 +03:00
Debanjum Singh Solanky
3abf3e5ee0 Update merge_dicts to recursively merge the dictionaries
Previously it was only merging dictionary at the first/top level
2022-08-04 22:46:20 +03:00
Debanjum Singh Solanky
17c38b526a Default config for each search types to None
- Setting up default compressed-jsonl, embeddings-file was only required
  for org search_type, while org-files and org-filter were allowed to be
  passed as command line argument
- This avoided having to set compressed-jsonl and embeddings-file via
  command line argument as well for org search type
- Now that all search types are only configurable via config file, We
  can default all search types to None. The default config for the
  rest of the search types wasn't being used anyway
2022-07-31 22:23:57 +03:00
Debanjum Singh Solanky
b83021a723 Improve code readability of merge_dicts helper method 2022-07-31 22:07:56 +03:00
Debanjum Singh Solanky
38aede68f2 Only configure org via config file for consistency across search types
- Previously org-files were configurable via cmdline args.
  Where as none of the other search types are
- This is an artifact of how the application grew
- It can be removed for better consistency and
  equal preference given all search types
2022-07-31 22:02:03 +03:00
Debanjum Singh Solanky
65fea7681a Rename notes search type to org search, now that markdown notes supported 2022-07-21 22:09:44 +04:00
Debanjum Singh Solanky
d4d7dbaca6 Support Natural Search on Markdown Files
- Reason:
  Allow natural search on markdown based notes, documentation,
  websites etc

- Details:
  - Create markdown processor to extract Markdown entries (identified by
    Heading) into standard jsonl format required by text_search
  - Update API, Configs to support interfacing with new markdown type
  - Update Emacs, Web clients to support interfacing with new markdown
    type via API
  - Update Readme to mentiond markdown is also supported

Closes #35
2022-07-21 22:07:05 +04:00
Debanjum Singh Solanky
0602d018c0 Merge Symmetric, Asymmetric Search Types into a single Text Search Type
- The code for both the text search types were mostly the same
  It was earlier done this way for expedience while experimenting
- The minor differences were reconciled and merged into a single
  text_search type
- This simplifies the app and making it easier to process other
  text types
2022-07-21 21:19:52 +04:00
Debanjum Singh Solanky
0917f1574d Consolidate jsonl helper methods in a single file under utils module 2022-07-21 03:30:13 +04:00
Debanjum Singh Solanky
de726c4b6c Minor fixes to unused installer utility script 2022-07-21 03:30:13 +04:00
Debanjum Singh Solanky
5aad297286 Reuse logic to extract entries across symmetric, asymmetric search
Now that the logic to compile entries is in the processor layer, the
extract_entries method is standard across (text) search_types

Extract the load_jsonl method as a utility helper method.
Use it in (a)symmetric search types
2022-07-21 02:53:18 +04:00
Debanjum Singh Solanky
6c9ffdba57 Allow indexing multiple image directories for image search 2022-07-20 02:56:01 +04:00
Debanjum Singh Solanky
732b2d287f Give the project a short, less generic name. Rename it to Khoj
- Semantic Search was just a placeholder used to test the idea out
  Didn't want to get into naming at that point of time
2022-07-19 18:26:16 +04:00
Debanjum Singh Solanky
989526ae54 Use a more accurate model for symmetric semantic search
- The all-MiniLM-L6-v2 is more accurate
  - The exact previous model isn't benchmarked but based on the
    performance of the closest model to it. Seems like the new model
    maybe similar in speed and size

- On very preliminary evaluation of the model, the new model seems
  faster, with pretty decent results
2022-07-18 20:27:26 +04:00
Debanjum Singh Solanky
4a90972e38 Use a better model for asymmetric semantic search
- The multi-qa-MiniLM-L6-cos-v1 is more extensively benchmarked[1]
- It has the right mix of model query speed, size and performance on benchmarks
- On hugging face it has way more downloads and likes than the msmarco model[2]
- On very preliminary evaluation of the model
  - It doubles the encoding speed of all entries (down from ~8min to 4mins)
  - It gave more entries that stay relevant to the query (3/5 vs 1/5 earlier)

[1]: https://www.sbert.net/docs/pretrained_models.html
[2]: https://huggingface.co/sentence-transformers
2022-07-18 20:27:26 +04:00
Debanjum Singh Solanky
f5d6d1e752 Tiny style fix to separate functions by 2 newlines 2022-06-29 23:47:17 +04:00
Debanjum Singh Solanky
3d8a07f252 Extract empty line escape sequences var into constants file for reuse 2022-02-27 19:01:49 -05:00
Saba
33bc62dc19 Fix type of use_xmp_metadata to be bool, rather than str 2022-01-24 21:53:26 -05:00
Debanjum Singh Solanky
179153dc5a Rename RawConfig Types for Consistency
- Naming convention - [ContentType][ConfigType]Config
  - Where [ConfigType] ~ Content, Search, Processor
  - Where [ContentType] ~ Text, Image, Asymmetric, Symmetric, Conversation

- Current Configs:
  - Content:
    - Org Notes
    - Org Music
    - Image
    - Ledger/Beancount

  - Search:
     - Asymmetric
     - Symmetric
     - Image

  - Processor:
    - Conversation
2022-01-14 20:54:38 -05:00
Debanjum Singh Solanky
c64e0c2965 Load model from HuggingFace if model_directory unset in config YAML
- Do not save/load the model to/from disk when model_directory unset
in config.yml
- Add symmetric search default config to cli.py
2022-01-14 17:36:59 -05:00
Debanjum Singh Solanky
510faa1904 Save Image Search Model to Disk 2022-01-14 17:36:59 -05:00
Debanjum Singh Solanky
934ec233b0 Add Search Config for Symmetric Model. Save Model to Disk 2022-01-14 17:36:59 -05:00
Debanjum Singh Solanky
b63026d97c Save Asymmetric Search Model to Disk
- Improve application load time
- Remove dependence on internet to startup application and perform semantic search
2022-01-14 17:36:27 -05:00
Debanjum Singh Solanky
ea28897cdd Remove deprecated conversation_history field from config 2022-01-12 10:35:52 -05:00
Debanjum Singh Solanky
6dc2a99d35 Merge branch 'master' of github.com:debanjum/semantic-search into add-summarize-capability-to-chat-bot
- Fix openai_api_key being set in ConfigProcessorConfig
- Merge addition of config UI and config instantiation updates
2021-12-20 13:30:42 +05:30
Debanjum Singh Solanky
65da7daf1f Load, Save Conversation Session Summaries to Log. s/chat_log/chat_session
Conversation logs structure now has session info too instead of just chat info
Session info will allow loading past conversation summaries as context for AI in new conversations

{
    "session": [
    {
        "summary": <chat_session_summary>,
        "session-start": <session_start_index_in_chat_log>,
        "session-end": <session_end_index_in_chat_log>
    }],
    "chat": [
    {
        "intent": <intent-object>
        "trigger-emotion": <emotion-triggered-by-message>
        "by": <AI|Human>
        "message": <chat_message>
        "created": <message_created_date>
    }]
}
2021-12-15 10:17:07 +05:30
Saba
d65190c3ee Update unit tests, files with removing model suffix to config types 2021-12-09 08:50:38 -05:00
Saba
76e9e9da2f Update unit tests to use the new BaseModel types 2021-12-05 09:31:39 -05:00
Saba
10e4065e05 Consolidate the search config models and pass verbose as a top level flag 2021-12-04 11:43:48 -05:00
Saba
43e647835b Append Model Suffixed to config models 2021-12-04 10:51:21 -05:00
Saba
e068968b35 Update imports for raw config models in config.py 2021-12-04 10:44:55 -05:00
Saba
4d6284b0af Remove Test suffix from Config models 2021-12-04 10:44:13 -05:00
Saba
7ca4fc3453 Resolve mrege conflicts with updated processor conversation data model 2021-11-28 16:22:52 -05:00