Saba
43e647835b
Append Model Suffixed to config models
2021-12-04 10:51:21 -05:00
Saba
e068968b35
Update imports for raw config models in config.py
2021-12-04 10:44:55 -05:00
Saba
4d6284b0af
Remove Test suffix from Config models
2021-12-04 10:44:13 -05:00
Saba
7fcc8d2cef
Add null check for processor config
2021-12-04 10:11:00 -05:00
Saba
7ca4fc3453
Resolve mrege conflicts with updated processor conversation data model
2021-11-28 16:22:52 -05:00
Saba
87a6c2d716
Use parse_obj instead of parse_raw as incoming data is in dict
2021-11-28 14:34:32 -05:00
Saba
5d50487d83
Linting
...
New line at end of config.html
Remove debug print statement
2021-11-28 13:32:56 -05:00
Saba
6f466c8d99
Use global config and add a regenerate button to the config ui' && git push
2021-11-28 13:28:22 -05:00
Saba
34d1e4199c
Use alias generator when deserializing the config file
2021-11-28 13:05:48 -05:00
Saba
19b81e82f0
Write back to the raw config.yml file on update
2021-11-28 12:34:40 -05:00
Saba
8837b02de6
dump updated config to a yaml file
2021-11-28 12:26:07 -05:00
Saba
5b80b87379
Streamline None checking in initialize_search
2021-11-28 12:05:04 -05:00
Saba
bf8ae31e6a
Streamline None checking in initialize_search
2021-11-28 11:59:45 -05:00
Saba
da52433d89
Update to re-use the raw config base models in config.py as well
2021-11-28 11:57:33 -05:00
Saba
6292fe4481
Update to re-use the raw config base models in config.py as well
2021-11-28 11:57:13 -05:00
Saba
311c4b7e7b
Working API request body parsing to /post config!
2021-11-28 11:16:33 -05:00
Saba
66183cc298
Working API request body parsing to /post config!
2021-11-28 11:12:26 -05:00
Debanjum Singh Solanky
5cd920544d
Add GPT method to summarize notes and chat logs
2021-11-28 13:08:05 +05:30
Debanjum Singh Solanky
1785047ea6
Improve understand primer and load understand response as dict
2021-11-28 13:04:16 +05:30
Saba
64645c3ac1
Begin type checking/input validation effort
2021-11-27 21:47:56 -05:00
Saba
9a0264b7fc
Add a dummy POST config endpoint, integrate with editable UI
2021-11-27 20:36:03 -05:00
Saba
f3b03ea5b7
Make raw data reactive to changes
2021-11-27 19:17:15 -05:00
Debanjum Singh Solanky
67c3cd7372
Wire up GPT understand method to /chat API. Log conversation metadata too
2021-11-28 00:04:39 +05:30
Saba
3db06eee3f
Basic example of serving conifg as JSON and retriving on button click
2021-11-27 10:49:33 -05:00
Saba
3d4471e107
Merge branch 'master' of github.com:debanjum/semantic-search into saba/configui
2021-11-27 08:52:48 -05:00
Debanjum Singh Solanky
ccfb97e1a7
Wire up minimal conversation processor. Expose it over /chat API endpoint
...
Ensure conversation history persists across application restart
2021-11-27 18:12:01 +05:30
Debanjum Singh Solanky
a99b4b3434
Make conversation processor configurable
2021-11-27 18:12:01 +05:30
Debanjum Singh Solanky
d4e1120b22
Add GPT based conversation processor to understand intent and converse with user
...
- Allow conversing with user using GPT's contextually aware, generative capability
- Extract metadata, user intent from user's messages using GPT's general understanding
2021-11-27 18:12:01 +05:30
Saba
baee52648d
Set up basic ui page with no functionality
2021-11-26 14:51:11 -05:00
debanjum
46661b3057
Ensure top_k never more than total entries to run symmetric search on
2021-11-16 11:32:21 -08:00
debanjum
8c858d1a94
Reduce symmetric search results for cross-encoder to re-rank to improve search speed
2021-11-16 11:31:19 -08:00
Debanjum Singh Solanky
f3fd5ae978
Improve code comments. Do not import unused modules in asymmetric search
2021-11-17 00:58:31 +05:30
Debanjum Singh Solanky
8cf2465e8e
Ensure top_k never more than total entries to search from
2021-11-17 00:56:31 +05:30
Debanjum Singh Solanky
4d37ace3d6
Reduce search results for cross-encoder to re-rank to improve search speed
...
Search time on my notes reduced from 14s to 4s. Cross-encoder
re-ranking step takes majority time, not the cosine similarity search
2021-11-17 00:50:28 +05:30
Debanjum Singh Solanky
1832e418e5
Use raw string for regex in orgnode to fix deprecation warning
2021-10-02 17:38:31 -07:00
Debanjum Singh Solanky
f59e321419
Update CLIP model load path
2021-10-02 16:50:06 -07:00
Debanjum Singh Solanky
c47a8cdf16
Allow configuring host, port or unix socket of server via CLI
2021-10-02 16:16:33 -07:00
Debanjum Singh Solanky
516f28b082
Merge branch 'master' of github.com:debanjum/semantic-search
2021-09-30 04:17:32 -07:00
Debanjum Singh Solanky
d2905c4be6
Move tests out to project root. Use absolute import in project
...
tests/ directory in project root is more standard.
Just had to use absolute path for internal module imports to get it to
work
2021-09-30 04:12:14 -07:00
Debanjum Singh Solanky
58bb420f69
Fix image_metadata argument ordering bug. Add E2E image search test
...
- Image search test seems a little flaky
- Interchanged argument was causing inaccurate results earlier
2021-09-30 03:30:47 -07:00
Debanjum Singh Solanky
d5597442f4
Modularize Code. Wrap Search, Model Config in Classes. Add Tests
...
Details
- Rename method query_* to query in search_types for standardization
- Wrapping Config code in classes simplified mocking test config
- Reduce args beings passed to a function by passing it as single
argument wrapped in a class
- Minimize setup in main.py:__main__. Put most of it into functions
These functions can be mocked if required in tests later too
Setup Flow:
CLI_Args|Config_YAML -> (Text|Image)SearchConfig -> (Text|Image)SearchModel
2021-09-30 02:04:04 -07:00
Debanjum Singh Solanky
f4dd9cd117
Use type specific model for other search types too. Expose them via SearchModels
...
- Wrap Image, Music, Ledger search into the type of SearchModel they use
Similar to what was done for notes model by wrapping it's config
into an AsymmetricSearchModel.
- Use the uber wrapper class to expose all type specific search models
2021-09-29 21:09:42 -07:00
Debanjum Singh Solanky
352d2930ee
Use multiple threads to generate model embeddings. Other minor formating
2021-09-29 20:47:58 -07:00
Debanjum Singh Solanky
e22e0b41e3
Wrap asymmetric search model into SearchModels. Test notes search end-to-end
...
- Wrap asymmetric search model parameters into AsymmetricSearchModel class
- Create wrapper for all search type models. Put notes search model into it
- Test notes search end-to-end from client API layer to results.
Use model build on test data
2021-09-29 20:47:35 -07:00
Debanjum Singh Solanky
cde11a2331
Wrap search type enablement status in a search settings class
...
- Cleaner, more idiomatic usage of a global variable
- Simplifies mocking when testing client in pytest as setting wrapped
in object rather than a simple type. So passed around by reference
2021-09-29 19:18:33 -07:00
Debanjum Singh Solanky
81ce0cacc3
Only allow supported search types to /search, /regenerate APIs
...
- Use a SearchType to limit types that can be passed by user
- FastAPI automatically validates type passed in query param
- Available type options show up in Swagger UI, FastAPI docs
- controller code looks neater instead of doing string comparisons for type
- Test invalid, valid search types via pytest
2021-09-29 19:12:56 -07:00
Debanjum Singh Solanky
5db08c5293
Set query as heading of notes search results in Emacs Org buffer
2021-09-29 13:30:15 -07:00
Debanjum Singh Solanky
fdb60a8dcf
Set Query as Heading of Image Search Results Emacs Buffer
2021-09-16 12:30:06 -07:00
Debanjum Singh Solanky
169ddcc8c6
Make Using XMP Metadata to Enhance Image Search Optional, Configurable
...
- Break the compute embeddings method into separate methods:
compute_image_embeddings and compute_metadata_embeddings
- If image_metadata_embeddings isn't defined, do not use it to enhance
search results. Given image_metadata_embeddings wouldn't be defined
if use_xmp_metadata is False, we can avoid unnecessary addition of
args to query method
2021-09-16 12:01:05 -07:00
Debanjum Singh Solanky
a4a23d7a72
Batch encode XMP metadata from images too for image_search
2021-09-16 11:11:36 -07:00
