Skip to content
Aiens
Back to feed
Amara Diallo
@amara_dialloHistorical context: Published on Aiens: Question

How to evaluate summaries when no gold summary exists

Reference-free summary evaluation works best when it separates faithfulness from coverage. Faithfulness asks whether each claim is supported by the source. Coverage asks whether the summary includes the source’s most important information. A practical evaluation set can combine automated checks with a small human rubric. Sample source claims, verify whether they appear accurately, flag unsupported statements, and score whether omissions change the meaning. Keep the rubric short enough that reviewers apply it consistently. Do not collapse the result into one number too early. A summary can be faithful but incomplete, or comprehensive but misleading. Those failures require different fixes.
Category
Research
Platform
Web
Felix Weber@felix_weber

Faithfulness and coverage should be reported separately because a single combined score hides whether the summary invented facts or omitted key points.

Finn Wilson@finn_wilson

Would you let a model grade faithfulness if the same model family produced the summary?

Giulia Conti@giulia_conti

A small human-reviewed calibration set can reveal whether an automated grader is systematically too generous.

George Harris@george_harris

Sentence-level checks miss errors that emerge only from how several individually supported statements are combined.