Commit graph

1524 commits

Author SHA1 Message Date
Debanjum Singh Solanky
3abf3e5ee0 Update merge_dicts to recursively merge the dictionaries
Previously it was only merging dictionary at the first/top level
2022-08-04 22:46:20 +03:00
Debanjum Singh Solanky
a876b652d8 Rename khoj_sample.yml to more specific khoj_docker.yml
- Update docker-compose.yml to start container using khoj_docker.yml
- Use /data/org in input-filter for content-type > org
2022-08-04 22:42:05 +03:00
Debanjum Singh Solanky
f50f343f73 Rename org-mode test data directory to more specific org/ from notes/ 2022-08-04 22:29:57 +03:00
Debanjum Singh Solanky
61c26ba611 Only show large Khoj favicon on web interface
- Do not want browsers to use the small, grainy favicons
- Firefox for Android does use the bigger icon, when it's the only one available
- Update svg to match the 144x144 ratio just for consistency
2022-08-04 14:33:29 +03:00
Debanjum Singh Solanky
1649fa644c Autofocus on Query field in Web Interface. Improve time to query 2022-08-04 05:23:19 +03:00
Debanjum Singh Solanky
8646dc0de1 Add publish badge to Readme. Link badges to their workflow pages 2022-08-04 05:23:19 +03:00
Debanjum
496806452f
Use Pip to setup Khoj in Docker, Github Workflows. Publish to PyPi Automatically
### Details
#### Github Workflows
- Make Github Workflows use `pip` instead of `conda`
- Publish Khoj as Python Package Automatically
   - Version based on [PEP440](https://peps.python.org/pep-0440/) and [Semantic Versioning](https://semver.org/) specs
   - On Push to Master: Publish a pre-release (alpha) to PyPI
   - On Push to PR: Publish a development-release on TestPyPI
   - On creating a (release) tag: Publish the latest version (Major.Minor.Patch) to PyPI
#### Docker 
   - Make `Dockerfile` use `pip` to build image for Khoj. Faster, more standard flow than `conda`
   - Only Build Docker Image on Push to Master
      - Availability of pip package negates the need for docker image for PR testing 
   - Make Docker-Compose use Docker Image published to Github Container Registry
- Default Image search to disabled. [Fix loading CLIP model](https://github.com/UKPLab/sentence-transformers/issues/1659) before re-enabling
2022-08-04 05:09:26 +03:00
Debanjum Singh Solanky
ebaf5524d1 Bump khoj patch version in setup.py
- PRs and Pushes to Master show up as newer than previous release. But
they're marked as development release or pre-release (alpha)
- Once create tag. The published pip package than becomes a final release
  in PEP440 terminology
2022-08-04 04:45:20 +03:00
Debanjum Singh Solanky
50575b283c Reorder publish actions by order of importance in publish workflow 2022-08-04 04:43:05 +03:00
Debanjum Singh Solanky
c3ae3cb021 Publish to (Test) PyPi with PEP440 Compliant Version Numbers
- Use .devN for publish to testpypi on push to PR
- Use aN to publish to pypi on push to master
2022-08-04 04:29:15 +03:00
Debanjum Singh Solanky
89044c70bd Publish to Test PyPI on Push to PR or Master Branch 2022-08-04 03:59:38 +03:00
Debanjum Singh Solanky
4bc5ac2420 Add pre-release version to package and upload to TestPyPI
Update version to form {current-version}-alpha.{unix-timestamp} in setup.py
2022-08-04 03:21:33 +03:00
Debanjum Singh Solanky
b3ebb01beb Disable image search for now as unable to load CLIP model
Loading CLIP Model from Sentence-Transformer is failing
See https://github.com/UKPLab/sentence-transformers/issues/1659 for details
2022-08-04 02:20:45 +03:00
Debanjum Singh Solanky
b2beff6a0e Only run build workflow on publish to master. Not on pull request 2022-08-04 02:20:05 +03:00
Debanjum Singh Solanky
84adf0c568 Use published docker image to run khoj service using docker-compose 2022-08-04 02:17:01 +03:00
Debanjum Singh Solanky
ee65809dc6 Update Dockerfile to use Pip instead of Conda to install application
- Reduce size of app copied to container by adding unneeded
  directories to .dockerignore. E.g Ignore docs, tests, pip build, dist

- Update MANIFEST.ini to include web, emacs interface directories
  Web directory is used by the web interface which will be exposed by
  the docker container
2022-08-04 00:14:25 +03:00
Debanjum Singh Solanky
7a353066dd Improve name, stages in Github Workflows for Publish and Test 2022-08-03 22:37:16 +03:00
Debanjum Singh Solanky
f55eaf0a53 Remove comma in command to pip install multiple packages 2022-08-03 22:01:18 +03:00
Debanjum Singh Solanky
ee3eddb044 Create Workflow to Publish Application to (test) PyPI 2022-08-03 21:57:27 +03:00
Debanjum Singh Solanky
34296031c4 Name test workflow. Check if exiftool is even required for pytests 2022-08-03 21:50:20 +03:00
Debanjum Singh Solanky
92016eb49c Update pip package to include icons, exclude docs 2022-08-03 21:49:36 +03:00
Debanjum Singh Solanky
e76930b320 Set minimum python version to 3.8 as required by dependencies 2022-08-03 21:47:53 +03:00
Debanjum Singh Solanky
5d391fdcbe Just use Python version 3.10 in test Github Workflow 2022-08-03 20:25:15 +03:00
Debanjum Singh Solanky
d42203d29e Fix --upgrade flag to pip install command in Test Github Workflow 2022-08-03 20:20:34 +03:00
Debanjum Singh Solanky
f9a10ad5ee Use pip (not conda) to install khoj, run pytest in Github Workflow 2022-08-03 20:00:37 +03:00
Debanjum Singh Solanky
71fcb1087f Add icons for web interface to render on more browsers and as PWA
Safari, Firefox for Android etc don't support SVG Favicons yet
2022-08-03 18:52:41 +03:00
Debanjum Singh Solanky
5b6b7ec123 Delete khoj network connections on incremental search teardown on Emacs interface
Currently only get into this state when debug breakpoints on backend
are keeping the connection open and user exits khoj search from Emacs
Results in a number of open connections that slow khoj down.
2022-08-03 18:52:41 +03:00
Debanjum Singh Solanky
555c1088cc Cache queries in /search controller using LRU cache
- Most concretely right now,
  it eliminates the re-rank latency hit
  on re-rank triggered on user hitting enter
  after re-rank is already done on user idle
  in the emacs interface

- Improves search latency of (incremental) search
2022-08-03 18:52:41 +03:00
Debanjum Singh Solanky
38df727ef4 Fix escape sequence usage in strings. Remove unneeded import of os
Rename /config API method to config to match it's purpose. UI is
anyway too generic, and not what it is doing
2022-08-03 18:51:55 +03:00
Debanjum Singh Solanky
a4eb55dd00 Rename khoj config yml file to follow more specific khoj*.yml pattern
- That is, sample_config.yml is renamed to khoj_sample.yml
- This makes the application config filename less generic,
  more easily identifiable with the application
- Update docs, app accordingly
2022-08-03 12:06:55 +03:00
Debanjum Singh Solanky
48681644e9 Git ignore application config.yml. This is specific to a user 2022-08-03 11:57:43 +03:00
Debanjum Singh Solanky
c96bc03f58 Trigger github build, test on updates to setup.py 2022-08-03 11:56:40 +03:00
Debanjum Singh Solanky
f642450ed9 Disable Incremental Search for Images on Web
Bug introduced in commit da118b3fed
2022-08-03 11:52:51 +03:00
Debanjum Singh Solanky
26b50471b7 Upgrade pillow to fix dependabot security advisory 2022-08-03 00:33:29 +03:00
Debanjum
1295ba90c3
Publish Khoj on PyPi
## Details

- Add `setup.py` to support pip install of khoj package
- Fixup Application to work with pip install
- Update Readme with instructions to install package via pip
- Publish Khoj as `khoj-assistant` on [PyPi](https://pypi.org/project/khoj-assistant/)
2022-08-03 00:29:49 +03:00
Debanjum Singh Solanky
b9e6273644 Include interfaces in pip package. Fix paths to web interface in app 2022-08-03 00:02:39 +03:00
Debanjum Singh Solanky
765e10bc2c Prepare Khoj for PyPi. Include Readme in dist, Fix metadata in setup.py 2022-08-02 22:53:02 +03:00
Debanjum Singh Solanky
d57bd41019 Add Instruction to Install, Run Khoj via Pip to Readme #Development 2022-08-02 21:12:27 +03:00
Debanjum Singh Solanky
1b55462fb0 Convert search_filter, conversation dir to proper modules
Add __init__.py files to their directories
2022-08-02 20:23:42 +03:00
Debanjum Singh Solanky
006a2aef57 Create setup.py to setup Khoj python application 2022-08-02 20:23:05 +03:00
Debanjum Singh Solanky
5108d45951 Wrap application startup steps into a method 2022-08-02 20:13:14 +03:00
Debanjum Singh Solanky
0ebfbb43ce Nest org, md results at level 2 on Emacs interface. Improve readability
- Makes it easier to fold/unfold, traverse and read results
- This 2 level nesting is already being used on the web interface

- Previously we were using the original nesting depth of the entry.
  This was aimed at providing more of the orginal context of the
  results. But currently this additional information does not provide
  as much, for the decreased legibility of the results
2022-08-01 04:01:18 +03:00
Debanjum
dbc7ca74c9
Minor cleanup of Readme. Move demo to docs/ directory
- Move Demo to docs/ directory to clean project root
- Minor updates to Readme
2022-08-01 02:46:44 +03:00
Debanjum Singh Solanky
bcaf86f25a Minor updates to Readme
- Help test workflows not triggered on PR with changes to docs
2022-08-01 02:42:48 +03:00
Debanjum Singh Solanky
f54f414893 Move demo video to docs/ directory to keep project root clean 2022-08-01 02:41:54 +03:00
Debanjum Singh Solanky
0797e9cef8 Run github workflows only when relevant paths are modified 2022-08-01 02:30:23 +03:00
Debanjum
06499da0c3
Align Config Web UI Code Layout, API with the rest of the application
- Put code to configure app via web interface under `src/interface/web` directory with the rest of the web interface code
- Make config web UI available under `/config` instead of at the generic `/ui` endpoint
- Rename `/config` API endpoint (that gets/sets config from/to yaml file) to `/config/data`
- Add Khoj App title, favicon to config web page
2022-08-01 01:53:21 +03:00
Debanjum Singh Solanky
1201bfddf3 Simplify name of config css from config-style.css to config.css 2022-08-01 01:34:00 +03:00
Debanjum Singh Solanky
075dba5d64 Use Khoj Title, Favicon in Config Page for Consistency 2022-08-01 01:27:14 +03:00
Debanjum Singh Solanky
56a4429f01 Move web interface to configure application into src/interface/web directory
- Improve code layout by ensuring all web interface specific code
  under the src/interface/web directory
- Rename config API to more specifi /config instead of /ui
- Rename config data GET, POST api to /config/data instead of /config
2022-08-01 00:53:42 +03:00