sij/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2024-12-04 21:03:01 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
1632 commits 12 branches 110 tags 133 MiB
Commit graph

1632 commits

This branch
This branch
All branches
Author SHA1 Message Date
Debanjum Singh Solanky
85bf15628d Use better cmdline argument names. Drop unneeded no-compress argument 
Can infer to compress or not via the output_file suffix
 2021-08-16 13:49:39 -07:00
Debanjum Singh Solanky
d9f60c00bf Warn if any input files to org-to-json are potentially non org-mode files 
That is, if the file paths in the input set don't end with .org
 2021-08-16 13:49:39 -07:00
Debanjum Singh Solanky
3aa0c30fee Use absolute file path to open files in org-to-jsonl.py, asymmetric.py 
Exit script if neither org_files, org_file_filter is present
 2021-08-16 13:49:39 -07:00
Debanjum Singh Solanky
e773611558 Remove unused jsonl_file argument from convert_org_entries_to_jsonl 2021-08-16 13:49:35 -07:00
Debanjum Singh Solanky
8b29e272d3 Standardize interface, better default args for org-to-json.py script 
- Remove non-standard, unnecessary argument for org-directory
  Pass path each file in org-files and org-files-filter argument directly
- Allow shorthand -i, -o for input files, output files
- Default to compress, unless user explicitly specifies not to
 2021-08-16 11:29:08 -07:00
Debanjum Singh Solanky
7547e90745 Minor doc updates after merging emacs package with main repository 2021-08-16 02:02:26 -07:00
Debanjum Singh Solanky
ec157ea0ff Add Emacs interface to semantic-search directly to main repository 
Too much overhead to maintain multiple repositories, especially when
the Emacs library for semantic-search is a single file.

Import Readme from the emacs-semantic-search repository too
 2021-08-16 01:27:46 -07:00
Debanjum Singh Solanky
dcf7b2d04f Remove requirements.txt for now as virtualenv setup doesn't work 
Haven't gotten it to work on Mac or Ubuntu. Remove to avoid confusion
for now. Application depends on miniconda for now
 2021-08-16 00:15:10 -07:00
Debanjum Singh Solanky
3b81fafa3e Use updated path to MiniLM bi-encoder model on hugging-face 2021-08-15 23:57:22 -07:00
Debanjum Singh Solanky
4839153086 Acknowledge ML models used for search. Simplify path used in commands 2021-08-15 23:56:18 -07:00
Debanjum Singh Solanky
c58c1d96aa Change default install directory to current, fix open file code 2021-08-15 23:01:55 -07:00
Debanjum Singh Solanky
ae15e429b5 Reduce indentation from 4 to 2 in Readme.md. 
Prevent everything looking like code blocks due to 4 space indentations
 2021-08-15 22:56:36 -07:00
Debanjum Singh Solanky
636b6195cc Add Readme, License. Update .gitignore 2021-08-15 22:52:37 -07:00
Debanjum Singh Solanky
354c541b62 Add org processor to generate compressed jsonl from org-mode files 
The corpus embeddings are generated from this compressed JSONL
using the specified transformer ML model
 2021-08-15 22:52:31 -07:00
Debanjum Singh Solanky
b74cb9a104 Move install.py to new utils dir as it's for cmdline ease of use only 2021-08-15 19:10:30 -07:00
Debanjum Singh Solanky
ec92f3e146 Move different search types into search_types directory 2021-08-15 19:09:50 -07:00
Debanjum Singh Solanky
4d681c86ec Update requirements.txt for users wanting to use pip install 2021-08-15 18:45:37 -07:00
Debanjum Singh Solanky
d75df54385 Create API interface for Semantic Search 
Use FastAPI, Uvicorn to create app with API endpoint at /search
Example Query: http://localhost:8000/?q="why sleep?"&t="notes'&n=5
 2021-08-15 18:11:48 -07:00
Debanjum Singh Solanky
e3088c8cf8 Create environment.yml to install prerequisites for app via conda 2021-08-15 17:48:38 -07:00
Debanjum Singh Solanky
660e6c3937 Add explicit filters to asymmetric search 
User can filter results to ones which include, exclude specified words
To show entries which include, exclude specific words, user should prepend
a '+', '-' before the word. E.g "+hello -bye"
 2021-08-15 17:48:38 -07:00
Debanjum Singh Solanky
91a2c598fe Resolve paths to absolute paths once. Use pathlib glob directly 2021-08-09 00:39:33 -07:00
Debanjum Singh Solanky
ca0a22f4dd Search for images similar to query image provided by the user 
Example user passes path to an image in query. e.g ~/Pictures/photo.jpg
The script should return images in images_embedding most similar to
the query image
 2021-08-09 00:21:02 -07:00
Debanjum Singh Solanky
00d0065c5b Allow user to search images via text queries 2021-08-08 23:02:30 -07:00
Debanjum Singh Solanky
181cab89d2 Ignore Title Notes i.e notes with just headings from compute 2021-08-04 21:30:09 -07:00
Debanjum Singh Solanky
d6d7b9d6a8 Make installed script executable. Minor clean-up of duplicate code 2021-08-04 18:29:20 -07:00
Debanjum Singh Solanky
2eb029a7b0 Create script to install semantic-search as a program 2021-08-02 00:29:09 -07:00
Debanjum Singh Solanky
13d5100ce6 Rename script similarity to symmetric 2021-07-31 20:37:07 -07:00
Debanjum Singh Solanky
ad7e90bec3 Modularize script, provide cmdline control, improve results rendering 2021-07-31 17:13:39 -07:00
Debanjum Singh Solanky
eb03f57917 Save, Load Embeddings to/from file to speed up script load time 2021-07-31 10:13:41 -07:00
Debanjum Singh Solanky
0914f284bb Re-rank using cross encoder to get even more relevant results 
The cross encoder re-ranked results are much better for more distant queries.
It does take more time with the cross-encoder re-ranking but it seems
worth it to get more relevant results
 2021-07-31 03:10:44 -07:00
Debanjum Singh Solanky
9864a2b551 Retrieve most relevant entries for a query using MSMarco based bi-encoder 
Returns best 3 results ranked by MSMarco based biencoder score of
query match to entries from org-mode notes
 2021-07-31 00:20:37 -07:00
debanjum
0ef5495701 Use Sentence Transformers to Encode, Query Schedule.org Headings 2021-04-04 04:53:03 -07:00