Commit graph

1618 commits

Author SHA1 Message Date
Debanjum
496806452f
Use Pip to setup Khoj in Docker, Github Workflows. Publish to PyPi Automatically
### Details
#### Github Workflows
- Make Github Workflows use `pip` instead of `conda`
- Publish Khoj as Python Package Automatically
   - Version based on [PEP440](https://peps.python.org/pep-0440/) and [Semantic Versioning](https://semver.org/) specs
   - On Push to Master: Publish a pre-release (alpha) to PyPI
   - On Push to PR: Publish a development-release on TestPyPI
   - On creating a (release) tag: Publish the latest version (Major.Minor.Patch) to PyPI
#### Docker 
   - Make `Dockerfile` use `pip` to build image for Khoj. Faster, more standard flow than `conda`
   - Only Build Docker Image on Push to Master
      - Availability of pip package negates the need for docker image for PR testing 
   - Make Docker-Compose use Docker Image published to Github Container Registry
- Default Image search to disabled. [Fix loading CLIP model](https://github.com/UKPLab/sentence-transformers/issues/1659) before re-enabling
2022-08-04 05:09:26 +03:00
Debanjum Singh Solanky
ebaf5524d1 Bump khoj patch version in setup.py
- PRs and Pushes to Master show up as newer than previous release. But
they're marked as development release or pre-release (alpha)
- Once create tag. The published pip package than becomes a final release
  in PEP440 terminology
2022-08-04 04:45:20 +03:00
Debanjum Singh Solanky
50575b283c Reorder publish actions by order of importance in publish workflow 2022-08-04 04:43:05 +03:00
Debanjum Singh Solanky
c3ae3cb021 Publish to (Test) PyPi with PEP440 Compliant Version Numbers
- Use .devN for publish to testpypi on push to PR
- Use aN to publish to pypi on push to master
2022-08-04 04:29:15 +03:00
Debanjum Singh Solanky
89044c70bd Publish to Test PyPI on Push to PR or Master Branch 2022-08-04 03:59:38 +03:00
Debanjum Singh Solanky
4bc5ac2420 Add pre-release version to package and upload to TestPyPI
Update version to form {current-version}-alpha.{unix-timestamp} in setup.py
2022-08-04 03:21:33 +03:00
Debanjum Singh Solanky
b3ebb01beb Disable image search for now as unable to load CLIP model
Loading CLIP Model from Sentence-Transformer is failing
See https://github.com/UKPLab/sentence-transformers/issues/1659 for details
2022-08-04 02:20:45 +03:00
Debanjum Singh Solanky
b2beff6a0e Only run build workflow on publish to master. Not on pull request 2022-08-04 02:20:05 +03:00
Debanjum Singh Solanky
84adf0c568 Use published docker image to run khoj service using docker-compose 2022-08-04 02:17:01 +03:00
Debanjum Singh Solanky
ee65809dc6 Update Dockerfile to use Pip instead of Conda to install application
- Reduce size of app copied to container by adding unneeded
  directories to .dockerignore. E.g Ignore docs, tests, pip build, dist

- Update MANIFEST.ini to include web, emacs interface directories
  Web directory is used by the web interface which will be exposed by
  the docker container
2022-08-04 00:14:25 +03:00
Debanjum Singh Solanky
7a353066dd Improve name, stages in Github Workflows for Publish and Test 2022-08-03 22:37:16 +03:00
Debanjum Singh Solanky
f55eaf0a53 Remove comma in command to pip install multiple packages 2022-08-03 22:01:18 +03:00
Debanjum Singh Solanky
ee3eddb044 Create Workflow to Publish Application to (test) PyPI 2022-08-03 21:57:27 +03:00
Debanjum Singh Solanky
34296031c4 Name test workflow. Check if exiftool is even required for pytests 2022-08-03 21:50:20 +03:00
Debanjum Singh Solanky
92016eb49c Update pip package to include icons, exclude docs 2022-08-03 21:49:36 +03:00
Debanjum Singh Solanky
e76930b320 Set minimum python version to 3.8 as required by dependencies 2022-08-03 21:47:53 +03:00
Debanjum Singh Solanky
5d391fdcbe Just use Python version 3.10 in test Github Workflow 2022-08-03 20:25:15 +03:00
Debanjum Singh Solanky
d42203d29e Fix --upgrade flag to pip install command in Test Github Workflow 2022-08-03 20:20:34 +03:00
Debanjum Singh Solanky
f9a10ad5ee Use pip (not conda) to install khoj, run pytest in Github Workflow 2022-08-03 20:00:37 +03:00
Debanjum Singh Solanky
71fcb1087f Add icons for web interface to render on more browsers and as PWA
Safari, Firefox for Android etc don't support SVG Favicons yet
2022-08-03 18:52:41 +03:00
Debanjum Singh Solanky
5b6b7ec123 Delete khoj network connections on incremental search teardown on Emacs interface
Currently only get into this state when debug breakpoints on backend
are keeping the connection open and user exits khoj search from Emacs
Results in a number of open connections that slow khoj down.
2022-08-03 18:52:41 +03:00
Debanjum Singh Solanky
555c1088cc Cache queries in /search controller using LRU cache
- Most concretely right now,
  it eliminates the re-rank latency hit
  on re-rank triggered on user hitting enter
  after re-rank is already done on user idle
  in the emacs interface

- Improves search latency of (incremental) search
2022-08-03 18:52:41 +03:00
Debanjum Singh Solanky
38df727ef4 Fix escape sequence usage in strings. Remove unneeded import of os
Rename /config API method to config to match it's purpose. UI is
anyway too generic, and not what it is doing
2022-08-03 18:51:55 +03:00
Debanjum Singh Solanky
a4eb55dd00 Rename khoj config yml file to follow more specific khoj*.yml pattern
- That is, sample_config.yml is renamed to khoj_sample.yml
- This makes the application config filename less generic,
  more easily identifiable with the application
- Update docs, app accordingly
2022-08-03 12:06:55 +03:00
Debanjum Singh Solanky
48681644e9 Git ignore application config.yml. This is specific to a user 2022-08-03 11:57:43 +03:00
Debanjum Singh Solanky
c96bc03f58 Trigger github build, test on updates to setup.py 2022-08-03 11:56:40 +03:00
Debanjum Singh Solanky
f642450ed9 Disable Incremental Search for Images on Web
Bug introduced in commit da118b3fed
2022-08-03 11:52:51 +03:00
Debanjum Singh Solanky
26b50471b7 Upgrade pillow to fix dependabot security advisory 2022-08-03 00:33:29 +03:00
Debanjum
1295ba90c3
Publish Khoj on PyPi
## Details

- Add `setup.py` to support pip install of khoj package
- Fixup Application to work with pip install
- Update Readme with instructions to install package via pip
- Publish Khoj as `khoj-assistant` on [PyPi](https://pypi.org/project/khoj-assistant/)
2022-08-03 00:29:49 +03:00
Debanjum Singh Solanky
b9e6273644 Include interfaces in pip package. Fix paths to web interface in app 2022-08-03 00:02:39 +03:00
Debanjum Singh Solanky
765e10bc2c Prepare Khoj for PyPi. Include Readme in dist, Fix metadata in setup.py 2022-08-02 22:53:02 +03:00
Debanjum Singh Solanky
d57bd41019 Add Instruction to Install, Run Khoj via Pip to Readme #Development 2022-08-02 21:12:27 +03:00
Debanjum Singh Solanky
1b55462fb0 Convert search_filter, conversation dir to proper modules
Add __init__.py files to their directories
2022-08-02 20:23:42 +03:00
Debanjum Singh Solanky
006a2aef57 Create setup.py to setup Khoj python application 2022-08-02 20:23:05 +03:00
Debanjum Singh Solanky
5108d45951 Wrap application startup steps into a method 2022-08-02 20:13:14 +03:00
Debanjum Singh Solanky
0ebfbb43ce Nest org, md results at level 2 on Emacs interface. Improve readability
- Makes it easier to fold/unfold, traverse and read results
- This 2 level nesting is already being used on the web interface

- Previously we were using the original nesting depth of the entry.
  This was aimed at providing more of the orginal context of the
  results. But currently this additional information does not provide
  as much, for the decreased legibility of the results
2022-08-01 04:01:18 +03:00
Debanjum
dbc7ca74c9
Minor cleanup of Readme. Move demo to docs/ directory
- Move Demo to docs/ directory to clean project root
- Minor updates to Readme
2022-08-01 02:46:44 +03:00
Debanjum Singh Solanky
bcaf86f25a Minor updates to Readme
- Help test workflows not triggered on PR with changes to docs
2022-08-01 02:42:48 +03:00
Debanjum Singh Solanky
f54f414893 Move demo video to docs/ directory to keep project root clean 2022-08-01 02:41:54 +03:00
Debanjum Singh Solanky
0797e9cef8 Run github workflows only when relevant paths are modified 2022-08-01 02:30:23 +03:00
Debanjum
06499da0c3
Align Config Web UI Code Layout, API with the rest of the application
- Put code to configure app via web interface under `src/interface/web` directory with the rest of the web interface code
- Make config web UI available under `/config` instead of at the generic `/ui` endpoint
- Rename `/config` API endpoint (that gets/sets config from/to yaml file) to `/config/data`
- Add Khoj App title, favicon to config web page
2022-08-01 01:53:21 +03:00
Debanjum Singh Solanky
1201bfddf3 Simplify name of config css from config-style.css to config.css 2022-08-01 01:34:00 +03:00
Debanjum Singh Solanky
075dba5d64 Use Khoj Title, Favicon in Config Page for Consistency 2022-08-01 01:27:14 +03:00
Debanjum Singh Solanky
56a4429f01 Move web interface to configure application into src/interface/web directory
- Improve code layout by ensuring all web interface specific code
  under the src/interface/web directory
- Rename config API to more specifi /config instead of /ui
- Rename config data GET, POST api to /config/data instead of /config
2022-08-01 00:53:42 +03:00
Debanjum
bb2ccec1ca
Populate type dropdown on the web interface with only enabled search types
- Previously we were statically populating types dropdown field in the web interface with all available search types
- This change populates the type dropdown field with only search types that are enabled/configured
- It queries the `/config` backend API to see which of the available search types are configured
2022-08-01 00:20:45 +03:00
Debanjum Singh Solanky
8b6058c879 Fix instantiating type field with value from URL query parameter
- Populate via `.then` after enabled search types in dropdown are
  populated
- Call to `/config` API is async and will usually complete after the value of type field is set from url
- So value of type field would earlier be overridden when search types
  dropdown is populated after the call to `/config` API completes
2022-08-01 00:04:50 +03:00
Debanjum Singh Solanky
7d7259bd92 Remove tests that validate configuring org using commandline arguments 2022-07-31 23:42:00 +03:00
Debanjum Singh Solanky
be253bab39 Populate type dropdown with only enabled search types in web interface
- Get /config API and check config for which available search types is
  populated. This gives us the list of enabled search types
- Dynamically populate search type field with enabled search types only
2022-07-31 23:42:00 +03:00
Debanjum Singh Solanky
0abd40aeb7 Only set query field when appropriate query param passed via URL
- Setting query value to default option when query param wasn't
  passed via URL was overriding placeholder text in query field

- We wanted placeholder text in field, not the query field to actually
  be populated by placeholder text

- This clears field when user starts typing query into the query field,
  instead of them having to manually delete the  default text populated
2022-07-31 22:29:23 +03:00
Debanjum Singh Solanky
17c38b526a Default config for each search types to None
- Setting up default compressed-jsonl, embeddings-file was only required
  for org search_type, while org-files and org-filter were allowed to be
  passed as command line argument
- This avoided having to set compressed-jsonl and embeddings-file via
  command line argument as well for org search type
- Now that all search types are only configurable via config file, We
  can default all search types to None. The default config for the
  rest of the search types wasn't being used anyway
2022-07-31 22:23:57 +03:00