What is RAGAS and what does it measure?

RAGAS is a framework from Es and colleagues (2023) that scores RAG pipelines with three reference free metrics: faithfulness, whether the answer is supported by the retrieved context; answer relevance, whether the answer addresses the question; and context relevance, whether the retrieved context is focused. On the authors' WikiEval dataset, built from 50 Wikipedia pages, the metrics agreed with human annotators 0.95 of the time on faithfulness, 0.78 on answer relevance, and 0.70 on context relevance.

Why build your own eval harness instead of using RAGAS?

RAGAS needs 4 LLM calls and 1 embedding call per question, so 20 questions cost 80 judge calls, and the scores move when the judge model changes. Our harness is one Python file using only the standard library: it ran 60 system question pairs in 0.010 seconds at zero cost, deterministically, offline. It catches retrieval and selection failures the same day you introduce them, and it names the failing question instead of hiding it inside one number.

What is token F1 and why did our answers score only 0.31?

Token F1 is the harmonic mean of token precision and token recall between the predicted answer and the gold answer. Our systems answer in full sentences while the gold answers are short, like 16 dollars, so most predicted tokens are extra words and F1 falls to 0.312 even when the answer is correct. Substring accuracy, which checks whether the gold answer appears inside the prediction, was 0.80 for the same system. F1 punishes verbosity, not wrongness.

Can a faithful answer still be wrong?

Yes, and we measured it. Our extractive systems scored faithfulness 1.00 while answering three of 20 questions wrong, because every word came from the context. Faithfulness checks whether answer words exist in the retrieved text, not whether the right sentence was picked. In our adversarial probe, a wrong answer, Kiln Coffee was founded in 2019 in Seattle, scored token F1 0.857 and faithfulness 1.00 against a gold answer naming Portland. Only a substring check for Portland, or a human reading the sentence, catches it.

How many human labels does ARES need?

ARES, from Saad-Falcon and colleagues at Stanford, asks for about 150 human labelled examples as a validation set. It fine tunes lightweight judge models on synthetic data, then uses prediction powered inference to correct those judges and put confidence intervals on the scores. Across eight knowledge intensive tasks in KILT, SuperGLUE, and AIS, it scored RAG systems accurately with only a few hundred human annotations.

How do I stop a bad RAG change from shipping?

Freeze a test set of questions with known answers, set thresholds before you run anything, and wire the harness into CI so it exits nonzero on failure. We use recall at 3 of at least 0.90, faithfulness of at least 0.90, and substring accuracy of at least 0.80. Our top 3 system passed all three and ships; the top 1 system failed recall at 0.85 and the invented detail system failed faithfulness at 0.70, so both stay blocked. Add every new failure to the test set as a regression question.

Evals Deep: Beyond RAGAS, Build Your Own Eval Harness