sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-11-23 15:38:55 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 5
3954 commits 8 branches 101 tags 130 MiB
Commit graph

3954 commits

This branch
This branch
All branches
Author SHA1 Message Date
Debanjum Singh Solanky
c31abad0a6 Mount embeddings to /data/embeddings for directory naming consistency 
- Keeps directory paths consistent between host and container volumes
- Consistency simplifies documentation and updates required to setup
  sample_config.yml for local installation
 2022-01-29 03:24:02 -05:00
Debanjum Singh Solanky
b0067fc32e Store docker, conda, semantic-search configuration in a config directory 
- Improves organization of config files required for application
- Declutters the application root directory from configs
 2022-01-29 02:41:11 -05:00
Debanjum Singh Solanky
79c2224eaa Improve test data organization and update correspoding conftests 
- Put test data for each content type into separate directories
- Makes config.yml for docker and local host consistent
  - Prepending tests to /data in sample_config.yml makes application
    run on local host using test data
  - Allows mounting separate volume for each content type in docker-compose
- Ignore gitignore to only add tests content, not generated models or embeddings
 2022-01-29 02:03:17 -05:00
Debanjum Singh Solanky
3e889760c7 Merge sample_config, docker_sample_config yml into a single sample_config.yml 
- Update readme to indicate how to update the new sample_config to run on test data
 2022-01-29 01:32:12 -05:00
Debanjum Singh Solanky
2bc2780501 Mention the experimental /chat API interacts with OpenAI's API 2022-01-29 00:11:40 -05:00
Debanjum Singh Solanky
6ed667aed0 Add Troubleshooting Section, Minor Fixes to Readme 2022-01-29 00:11:40 -05:00
Debanjum
d943d2be80
Merge pull request #21 from debanjum/saba/dockerize 
Add Docker support to semantic-search
 2022-01-28 20:27:40 -08:00
Saba
1ba7fa66e5 Update README and default folders in docker_sample_config.yml 
- Add instruction to using Docker with README
- Use the ./tests/data folder in docker_sample_conifg.yml so it can work right away for users
 2022-01-28 23:20:50 -05:00
Saba
52e701b3c2 Simplify Dockerfile by removing multibuild 
- Install exiftool dependency directly in the miniconda image
 2022-01-24 21:54:10 -05:00
Saba
33bc62dc19 Fix type of use_xmp_metadata to be bool, rather than str 2022-01-24 21:53:26 -05:00
Saba
9fb410fc25 Clean up docker_sample_config.yml 
- Uncomment other search types
- Explain the file prefixes behavior and how it interfaces with the docker image
 2022-01-24 14:11:38 -05:00
Saba
9802023c79 Clean up docker-compose 
- Mount the local directory to /app
- Reformat the file paths to generically indicate what their purpose is
- Add comments to assist users who wasnt to modify properties themselves
 2022-01-24 14:10:18 -05:00
Saba
4ae8c15170 Clean the Dockerfile 
- Use /app as the working directory
- Clarify comment to explain why the ENTRYPOINT is constructed as it is
- Move explanations for the argument to docker-compose, where it's set
- Copy required artifacts from the first build image into the subsequent one (exiftool)
 2022-01-24 14:08:55 -05:00
Saba
66d08ab5df Rename web to server in docker-compose.yml 2022-01-24 00:14:01 -05:00
Saba
77fa8718d9 Working example with docker-compose 
Still need quite a bit of clean-up, but this adds a working docker-compose + Dockerfile setup
 2022-01-23 23:44:38 -05:00
Saba
875188dc6f Initialize working on #20 to add Docker support 
- Add a Dockerfile which uses an Ubuntu image to install relevant dependencies (exif) and uses a Miniconda image for setting up/reusing the conda environment
- Add a dummy docker-compose file
 2022-01-23 14:57:28 -05:00
sabaimran
974690939c
Merge pull request #19 from debanjum/rename-config-types-for-consistency 
Rename RawConfig Types for Consistency
 2022-01-14 21:14:08 -05:00
Debanjum Singh Solanky
179153dc5a Rename RawConfig Types for Consistency 
- Naming convention - [ContentType][ConfigType]Config
  - Where [ConfigType] ~ Content, Search, Processor
  - Where [ContentType] ~ Text, Image, Asymmetric, Symmetric, Conversation

- Current Configs:
  - Content:
    - Org Notes
    - Org Music
    - Image
    - Ledger/Beancount

  - Search:
     - Asymmetric
     - Symmetric
     - Image

  - Processor:
    - Conversation
 2022-01-14 20:54:38 -05:00
Debanjum
ed7c2901f5
Merge pull request #18 from debanjum/deb/save-models-to-disk-on-first-run 
Save Search Models to Disk on First Run

## Why
  - Improve application startup time
  - Startup application and perform semantic search even if user offline
  - Use search model config in YAML file for all search types (asymmetric, symmetric, image)

## Details
  - Load search models from disk when available
  - Use search model config specified in YAML file
  - Add search config for Symmetric Search used by Ledger/Beancount transaction search
 2022-01-14 17:30:46 -08:00
Debanjum Singh Solanky
ed144f7984 Setup Search with Search_Config to Fix Tests 
- Rename pytest fixture search_config to more appropriate
content_config
- Create search_config pytest fixture
- Use search_config where search being setup, used in tests
 2022-01-14 20:13:14 -05:00
Debanjum Singh Solanky
c64e0c2965 Load model from HuggingFace if model_directory unset in config YAML 
- Do not save/load the model to/from disk when model_directory unset
in config.yml
- Add symmetric search default config to cli.py
 2022-01-14 17:36:59 -05:00
Debanjum Singh Solanky
510faa1904 Save Image Search Model to Disk 2022-01-14 17:36:59 -05:00
Debanjum Singh Solanky
934ec233b0 Add Search Config for Symmetric Model. Save Model to Disk 2022-01-14 17:36:59 -05:00
Debanjum Singh Solanky
b63026d97c Save Asymmetric Search Model to Disk 
- Improve application load time
- Remove dependence on internet to startup application and perform semantic search
 2022-01-14 17:36:27 -05:00
Debanjum
e8146e8ebb
Merge pull request #17 from albd/patch-1 
Fix url error in README
 2022-01-13 18:09:10 -08:00
Albert Davies
2e2069f720
Fix url error in README 
Misplaced quotes
 2022-01-13 16:28:46 -08:00
Debanjum Singh Solanky
2e53fbc844 Fix the user intent extraction prompt for GPT. Clean up chatbot test 2022-01-12 10:36:01 -05:00
Debanjum Singh Solanky
ea28897cdd Remove deprecated conversation_history field from config 2022-01-12 10:35:52 -05:00
Debanjum Singh Solanky
5a686b7be9 Add logs for chat bot in verbose mode 2022-01-12 10:35:52 -05:00
Debanjum Singh Solanky
6dc2a99d35 Merge branch 'master' of github.com:debanjum/semantic-search into add-summarize-capability-to-chat-bot 
- Fix openai_api_key being set in ConfigProcessorConfig
- Merge addition of config UI and config instantiation updates
 2021-12-20 13:30:42 +05:30
Debanjum
29543d2dc3
Merge pull request #16 from debanjum/cache-conda-env-setup-to-improve-cloud-build 
Cache Conda Setup to Improve Cloud Build Time
 2021-12-18 06:58:42 -08:00
Saba
ee6aae3a40 Merge branch 'master' of github.com:debanjum/semantic-search 2021-12-16 20:36:36 -05:00
Saba
916a1ffc73 Fix formatting of REAMDE env dependencies 2021-12-16 20:36:31 -05:00
Debanjum Singh Solanky
d91f645198 Try cache conda build step using online docs 
Refer:
 - https://github.com/marketplace/actions/setup-miniconda
 2021-12-15 11:48:14 +05:30
Debanjum Singh Solanky
65da7daf1f Load, Save Conversation Session Summaries to Log. s/chat_log/chat_session 
Conversation logs structure now has session info too instead of just chat info
Session info will allow loading past conversation summaries as context for AI in new conversations

{
    "session": [
    {
        "summary": <chat_session_summary>,
        "session-start": <session_start_index_in_chat_log>,
        "session-end": <session_end_index_in_chat_log>
    }],
    "chat": [
    {
        "intent": <intent-object>
        "trigger-emotion": <emotion-triggered-by-message>
        "by": <AI|Human>
        "message": <chat_message>
        "created": <message_created_date>
    }]
}
 2021-12-15 10:17:07 +05:30
Debanjum
35b46849a2
Merge pull request #15 from debanjum/debanjum-skip-flaky-image-search-pytest 
Skip Flaky Image Search Test
 2021-12-12 02:28:15 -08:00
Debanjum
ef911aa6be
Skip Flaky Image Search Test 
Image search doesn't always return expected image path.
Should resolve remaining issues with failing cloud test. See #11
 2021-12-12 02:15:20 -08:00
Saba
d14f5d54e3 Merge branch 'master' of github.com:debanjum/semantic-search 2021-12-11 14:18:06 -05:00
Saba
4832a107d7 Merge branch 'saba/fix-unit-tests' 2021-12-11 14:17:52 -05:00
Saba
ba8dc9ed5f Update the search_config instantiated for tests in conftest 2021-12-11 14:14:31 -05:00
Saba
9ebf00e29b Add instructions for installing exiftool to README (for Ubuntu only) 2021-12-11 14:13:37 -05:00
Saba
97a6dfaa1e Use default value False for verbose parameter, and small changes 
Pass config as parameter to initialize_search, change name of API methods to handle config CRUD operations, and initalize config to FullConfig
 2021-12-11 14:13:14 -05:00
sabaimran
930bdb5403
Use ubuntu-latest as machine for Github Actions 2021-12-11 12:54:32 -05:00
sabaimran
386534c236
Remove apt-get update from build.yml 2021-12-11 12:52:13 -05:00
sabaimran
c7d88a7a63
Update build.yml to include exiftool package 2021-12-11 12:50:30 -05:00
Saba
9536358d34 Fix key error model_name issue by upgrade sentence-transformers version 
Refer to https://github.com/UKPLab/sentence-transformers/issues/1241
Also user verbose flag passed through function parameters in image_search
 2021-12-11 11:58:19 -05:00
Saba
ce7a751e6b Fix passing verbose flag down in symmetric_ledger.py 2021-12-11 11:36:32 -05:00
sabaimran
e5a19f5b0e
Merge pull request #10 from debanjum/saba/configui 
Create a UI from which a user can update their config and integrate BaseModel for the config input
 2021-12-10 22:10:49 -05:00
Saba
d65190c3ee Update unit tests, files with removing model suffix to config types 2021-12-09 08:50:38 -05:00
Debanjum Singh Solanky
0ac1e5f372 Summarize chat logs and notes returned by semantic search via /chat API 2021-12-08 02:34:07 +05:30