Debanjum e75e13d788 Create Tests to Measure Chat Quality, Capabilities ... Create Rubric to Test Chat Quality and Capabilities ### Issues - Previously the improvements in quality of Khoj Chat on changes was uncertain - Manual testing on my evolving set of notes was slow and didn't assess all expected, desired capabilities ### Fix 1. Create an Evaluation Dataset to assess Chat Capabilities - Create custom notes for a fictitious person (I'll publish a book with these soon 😅😋) - Add a few of Paul Graham's more personal essays. *[Easy to get as markdown](https://github.com/ofou/graham-essays)* 2. Write Unit Tests to Measure Chat Capabilities - Measure quality at 2 separate layers - **Chat Actor**: These are the narrow agents made of LLM + Prompt. E.g `summarize`, `converse` in `gpt.py` - **Chat Director**: This is the chat orchestration agent. It calls on required chat actors, search through user provided knowledge base (i.e notes, ledger, image) etc to respond appropriately to the users message. This is what the `/api/chat` API exposes. - Mark desired but not currently available capabilities as expected to fail <br /> This still allows measuring the chat capability score/percentage while only failing capability tests which were passing before on any changes to chat		2023-03-16 11:30:52 -06:00
..
interface	Bump khoj versions in obsidian, emacs files	2023-03-10 15:23:17 -06:00
khoj	Create Tests to Measure Chat Quality, Capabilities	2023-03-16 11:30:52 -06:00