sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-24 07:55:07 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 33
3504 commits 10 branches 106 tags 131 MiB
Commit graph

3504 commits

This branch
This branch
All branches
Author SHA1 Message Date
Debanjum Singh Solanky
c31abad0a6 Mount embeddings to /data/embeddings for directory naming consistency 
- Keeps directory paths consistent between host and container volumes
- Consistency simplifies documentation and updates required to setup
  sample_config.yml for local installation
 2022-01-29 03:24:02 -05:00
Debanjum Singh Solanky
b0067fc32e Store docker, conda, semantic-search configuration in a config directory 
- Improves organization of config files required for application
- Declutters the application root directory from configs
 2022-01-29 02:41:11 -05:00
Debanjum Singh Solanky
79c2224eaa Improve test data organization and update correspoding conftests 
- Put test data for each content type into separate directories
- Makes config.yml for docker and local host consistent
  - Prepending tests to /data in sample_config.yml makes application
    run on local host using test data
  - Allows mounting separate volume for each content type in docker-compose
- Ignore gitignore to only add tests content, not generated models or embeddings
 2022-01-29 02:03:17 -05:00
Debanjum Singh Solanky
3e889760c7 Merge sample_config, docker_sample_config yml into a single sample_config.yml 
- Update readme to indicate how to update the new sample_config to run on test data
 2022-01-29 01:32:12 -05:00
Debanjum Singh Solanky
2bc2780501 Mention the experimental /chat API interacts with OpenAI's API 2022-01-29 00:11:40 -05:00
Debanjum Singh Solanky
6ed667aed0 Add Troubleshooting Section, Minor Fixes to Readme 2022-01-29 00:11:40 -05:00
Debanjum
d943d2be80
Merge pull request #21 from debanjum/saba/dockerize 
Add Docker support to semantic-search
 2022-01-28 20:27:40 -08:00
Saba
1ba7fa66e5 Update README and default folders in docker_sample_config.yml 
- Add instruction to using Docker with README
- Use the ./tests/data folder in docker_sample_conifg.yml so it can work right away for users
 2022-01-28 23:20:50 -05:00
Saba
52e701b3c2 Simplify Dockerfile by removing multibuild 
- Install exiftool dependency directly in the miniconda image
 2022-01-24 21:54:10 -05:00
Saba
33bc62dc19 Fix type of use_xmp_metadata to be bool, rather than str 2022-01-24 21:53:26 -05:00
Saba
9fb410fc25 Clean up docker_sample_config.yml 
- Uncomment other search types
- Explain the file prefixes behavior and how it interfaces with the docker image
 2022-01-24 14:11:38 -05:00
Saba
9802023c79 Clean up docker-compose 
- Mount the local directory to /app
- Reformat the file paths to generically indicate what their purpose is
- Add comments to assist users who wasnt to modify properties themselves
 2022-01-24 14:10:18 -05:00
Saba
4ae8c15170 Clean the Dockerfile 
- Use /app as the working directory
- Clarify comment to explain why the ENTRYPOINT is constructed as it is
- Move explanations for the argument to docker-compose, where it's set
- Copy required artifacts from the first build image into the subsequent one (exiftool)
 2022-01-24 14:08:55 -05:00
Saba
66d08ab5df Rename web to server in docker-compose.yml 2022-01-24 00:14:01 -05:00
Saba
77fa8718d9 Working example with docker-compose 
Still need quite a bit of clean-up, but this adds a working docker-compose + Dockerfile setup
 2022-01-23 23:44:38 -05:00
Saba
875188dc6f Initialize working on #20 to add Docker support 
- Add a Dockerfile which uses an Ubuntu image to install relevant dependencies (exif) and uses a Miniconda image for setting up/reusing the conda environment
- Add a dummy docker-compose file
 2022-01-23 14:57:28 -05:00
sabaimran
974690939c
Merge pull request #19 from debanjum/rename-config-types-for-consistency 
Rename RawConfig Types for Consistency
 2022-01-14 21:14:08 -05:00
Debanjum Singh Solanky
179153dc5a Rename RawConfig Types for Consistency 
- Naming convention - [ContentType][ConfigType]Config
  - Where [ConfigType] ~ Content, Search, Processor
  - Where [ContentType] ~ Text, Image, Asymmetric, Symmetric, Conversation

- Current Configs:
  - Content:
    - Org Notes
    - Org Music
    - Image
    - Ledger/Beancount

  - Search:
     - Asymmetric
     - Symmetric
     - Image

  - Processor:
    - Conversation
 2022-01-14 20:54:38 -05:00
Debanjum
ed7c2901f5
Merge pull request #18 from debanjum/deb/save-models-to-disk-on-first-run 
Save Search Models to Disk on First Run

## Why
  - Improve application startup time
  - Startup application and perform semantic search even if user offline
  - Use search model config in YAML file for all search types (asymmetric, symmetric, image)

## Details
  - Load search models from disk when available
  - Use search model config specified in YAML file
  - Add search config for Symmetric Search used by Ledger/Beancount transaction search
 2022-01-14 17:30:46 -08:00
Debanjum Singh Solanky
ed144f7984 Setup Search with Search_Config to Fix Tests 
- Rename pytest fixture search_config to more appropriate
content_config
- Create search_config pytest fixture
- Use search_config where search being setup, used in tests
 2022-01-14 20:13:14 -05:00
Debanjum Singh Solanky
c64e0c2965 Load model from HuggingFace if model_directory unset in config YAML 
- Do not save/load the model to/from disk when model_directory unset
in config.yml
- Add symmetric search default config to cli.py
 2022-01-14 17:36:59 -05:00
Debanjum Singh Solanky
510faa1904 Save Image Search Model to Disk 2022-01-14 17:36:59 -05:00
Debanjum Singh Solanky
934ec233b0 Add Search Config for Symmetric Model. Save Model to Disk 2022-01-14 17:36:59 -05:00
Debanjum Singh Solanky
b63026d97c Save Asymmetric Search Model to Disk 
- Improve application load time
- Remove dependence on internet to startup application and perform semantic search
 2022-01-14 17:36:27 -05:00
Debanjum
e8146e8ebb
Merge pull request #17 from albd/patch-1 
Fix url error in README
 2022-01-13 18:09:10 -08:00
Albert Davies
2e2069f720
Fix url error in README 
Misplaced quotes
 2022-01-13 16:28:46 -08:00
Debanjum Singh Solanky
2e53fbc844 Fix the user intent extraction prompt for GPT. Clean up chatbot test 2022-01-12 10:36:01 -05:00
Debanjum Singh Solanky
ea28897cdd Remove deprecated conversation_history field from config 2022-01-12 10:35:52 -05:00
Debanjum Singh Solanky
5a686b7be9 Add logs for chat bot in verbose mode 2022-01-12 10:35:52 -05:00
Debanjum Singh Solanky
6dc2a99d35 Merge branch 'master' of github.com:debanjum/semantic-search into add-summarize-capability-to-chat-bot 
- Fix openai_api_key being set in ConfigProcessorConfig
- Merge addition of config UI and config instantiation updates
 2021-12-20 13:30:42 +05:30
Debanjum
29543d2dc3
Merge pull request #16 from debanjum/cache-conda-env-setup-to-improve-cloud-build 
Cache Conda Setup to Improve Cloud Build Time
 2021-12-18 06:58:42 -08:00
Saba
ee6aae3a40 Merge branch 'master' of github.com:debanjum/semantic-search 2021-12-16 20:36:36 -05:00
Saba
916a1ffc73 Fix formatting of REAMDE env dependencies 2021-12-16 20:36:31 -05:00
Debanjum Singh Solanky
d91f645198 Try cache conda build step using online docs 
Refer:
 - https://github.com/marketplace/actions/setup-miniconda
 2021-12-15 11:48:14 +05:30
Debanjum Singh Solanky
65da7daf1f Load, Save Conversation Session Summaries to Log. s/chat_log/chat_session 
Conversation logs structure now has session info too instead of just chat info
Session info will allow loading past conversation summaries as context for AI in new conversations

{
    "session": [
    {
        "summary": <chat_session_summary>,
        "session-start": <session_start_index_in_chat_log>,
        "session-end": <session_end_index_in_chat_log>
    }],
    "chat": [
    {
        "intent": <intent-object>
        "trigger-emotion": <emotion-triggered-by-message>
        "by": <AI|Human>
        "message": <chat_message>
        "created": <message_created_date>
    }]
}
 2021-12-15 10:17:07 +05:30
Debanjum
35b46849a2
Merge pull request #15 from debanjum/debanjum-skip-flaky-image-search-pytest 
Skip Flaky Image Search Test
 2021-12-12 02:28:15 -08:00
Debanjum
ef911aa6be
Skip Flaky Image Search Test 
Image search doesn't always return expected image path.
Should resolve remaining issues with failing cloud test. See #11
 2021-12-12 02:15:20 -08:00
Saba
d14f5d54e3 Merge branch 'master' of github.com:debanjum/semantic-search 2021-12-11 14:18:06 -05:00
Saba
4832a107d7 Merge branch 'saba/fix-unit-tests' 2021-12-11 14:17:52 -05:00
Saba
ba8dc9ed5f Update the search_config instantiated for tests in conftest 2021-12-11 14:14:31 -05:00
Saba
9ebf00e29b Add instructions for installing exiftool to README (for Ubuntu only) 2021-12-11 14:13:37 -05:00
Saba
97a6dfaa1e Use default value False for verbose parameter, and small changes 
Pass config as parameter to initialize_search, change name of API methods to handle config CRUD operations, and initalize config to FullConfig
 2021-12-11 14:13:14 -05:00
sabaimran
930bdb5403
Use ubuntu-latest as machine for Github Actions 2021-12-11 12:54:32 -05:00
sabaimran
386534c236
Remove apt-get update from build.yml 2021-12-11 12:52:13 -05:00
sabaimran
c7d88a7a63
Update build.yml to include exiftool package 2021-12-11 12:50:30 -05:00
Saba
9536358d34 Fix key error model_name issue by upgrade sentence-transformers version 
Refer to https://github.com/UKPLab/sentence-transformers/issues/1241
Also user verbose flag passed through function parameters in image_search
 2021-12-11 11:58:19 -05:00
Saba
ce7a751e6b Fix passing verbose flag down in symmetric_ledger.py 2021-12-11 11:36:32 -05:00
sabaimran
e5a19f5b0e
Merge pull request #10 from debanjum/saba/configui 
Create a UI from which a user can update their config and integrate BaseModel for the config input
 2021-12-10 22:10:49 -05:00
Saba
d65190c3ee Update unit tests, files with removing model suffix to config types 2021-12-09 08:50:38 -05:00
Debanjum Singh Solanky
0ac1e5f372 Summarize chat logs and notes returned by semantic search via /chat API 2021-12-08 02:34:07 +05:30