5 commits

Author	SHA1	Message	Date
Debanjum	f967bdf702	Show correct example index being currently processed in frames eval ... Previously the batch start index wasn't being passed so all batches started in parallel were showing the same processing example index This change doesn't impact the evaluation itself, just the index shown of the example currently being evaluated	2024-11-10 14:49:51 -08:00
Debanjum	84a8088c2b	Only evaluate non-empty responses to reduce eval script latency, cost ... Empty responses by Khoj will always be an incorrect response, so no need to make call to an evaluator agent to check that	2024-11-10 14:49:51 -08:00
Debanjum	1ccbf72752	Use logger instead of print to track eval	2024-11-04 00:40:26 -08:00
Debanjum	791eb205f6	Run prompt batches in parallel for faster eval runs	2024-11-02 04:58:03 -07:00
Debanjum	96904e0769	Add script to evaluate khoj on Google's FRAMES benchmark ... Google's FRAMES benchmark evaluates multi-step retrieval and reasoning capabilities of an agent. The script uses Gemini as an LLM Judge to evaluate Khoj responses to the FRAMES benchmark prompts against the ground truth provided by it.	2024-11-02 04:57:42 -07:00