LLM Eval Harness
Reusable golden-set evaluation harness for LLM and RAG systems: faithfulness, answer relevance, and correctness metrics with LLM-as-judge and offline heuristic fallbacks.
Overview
A CLI + Python library that loads golden sets (YAML of questions, golden answers, and context), scores model outputs, and produces JSON + Markdown reports with per-metric mean/min/max — perfect for CI regression gates.
Metrics
- Faithfulness — how well the answer is grounded in the retrieved context
- Answer relevance — how on-topic the answer is to the question
- Correctness — lexical overlap against the golden answer
LLM-as-Judge + Offline Fallback
When an OpenAI-compatible API key is available, an LLM judges each pair on a 0–1 scale. Without a key, deterministic lexical scorers kick in — so evals run in CI for free and the pipeline never blocks on external services.
Usage
python -m eval_harness –golden-set golden_sets/example.yaml Used to gate the RAG Service pipeline in CI.
Tech Stack
PythonLLM-as-JudgeGolden SetsRAGASCLICI/CDPytest