Criteria-Eval: Evaluating Long-form Answers to Complex Questions

After saturating component-wise evaluations of our question-answering system, we created a challenging end-to-end evaluation benchmark that allowed us to double our system quality within the last nine months. We are excited to share our evaluation recipe with the community.

We’re entering the era of AI agents. These systems do not just retrieve isolated facts or generate brief summaries. They synthesize multiple sources of complex information to answer challenging, multi-dimensional questions. At Samaya, our AI agents routinely face such demanding queries from financial analysts. For example:

“Summarize NVIDIA’s latest earnings call, compare key financial metrics against previous forecasts, and provide a detailed SWOT analysis based on the company’s strategic outlook.”

These questions require long detailed answers, spanning multiple paragraphs or even pages. Providing a good answer requires coordination across several specialized components. Evaluating the quality of these complex outputs is challenging because traditional component-level metrics miss crucial interactions between components.

In this blog post, we introduce Criteria-Eval, our internal evaluation framework specifically designed to measure the end-to-end quality of complex, long-form answers. Instead of relying on reference answers or generic rubrics, Criteria-Eval uses objective, yes-or-no checklists written by domain experts to assess factual accuracy, completeness, formatting, and more.

This approach makes evaluation precise, flexible, and, most importantly, directly aligns with how expert users judge quality.

Why End-to-End Evaluation Matters

When building complex systems, component-level evaluation is a natural starting point. We began there, measuring our retrievers, entity disambiguators, query orchestrators, etc., with targeted metrics. While useful individually, these metrics fail to account for critical interactions and error propagation between components.

What we ultimately care about is end-to-end performance:

Did the user receive the accurate, complete, and actionable answer they needed?

Component-level metrics simply can’t capture this.

Early on, we heavily relied on side-by-side product testing sessions and user feedback to judge quality end-to-end. It was manual, slow, and reactive. Crucially, it made catching regressions and making confident deployment decisions incredibly difficult. This was a gap we needed to fill.

Criteria-Eval: A Checklist-based Evaluation

Criteria-Eval addresses these shortcomings by shifting the evaluation mindset from reference generation to criteria specification:

We use domain experts (financial professionals) to define the criteria that a good answer should meet rather than asking them to write reference answers.

This checklist-based approach (Ribeiro et al. 2020) enables us to explicitly evaluate both retrieval quality and generation accuracy simultaneously.

Each criterion is a simple, clear assertion that must explicitly appear in the final answer (e.g., “Apple’s fiscal 2023 revenue was $383.3 billion”). This approach has several advantages:

• Objective: Allows scoring each criterion with a straightforward yes/no decision. This makes evaluation objective and highly reproducible by human annotators or automated evaluators.
• Flexible: Checklists also allow multiple valid ways to answer the same query, as long as critical criteria are met.¹
• Instruction-aware: They also allow measuring instruction following (e.g., formatting requests) besides factual accuracy, completeness, and other fact-based metrics.
• Expert-aligned: The criteria directly encode the expectations of and the knowledge of domain experts.

Dataset Overview

We started collecting data for Criteria-Eval approximately a year ago, and we’ve continuously expanded our dataset since then. As of now, the dataset includes:

• 3000 queries, reflecting a diverse spectrum of realistic financial analyst questions. These questions cover the majority of use cases encountered by our users at Samaya, ranging from straightforward factual queries to complex strategic analyses.
• Each query has one or multiple independent annotations by different experts, capturing a range of valid responses and perspectives.
• Queries range significantly in complexity, typically comprising from 5 up to 40 or more criteria, with the majority of queries averaging around 20 criteria each.

This dataset represents a substantial investment of expertise and time, with our financial domain experts dedicating more than 8000 hours to meticulous annotation work.

Annotation Process

Annotators adopt the role of a financial research analyst asked to answer a query. Their deliverable is an exhaustive bullet-point checklist with the facts, analyses, and formatting cues a perfect two-page report would need.

The process is governed by comprehensive internal guidelines.² The workflow looks like this:

1. Fix the time: set a “query date” and all cited data must have been publicly available on or before that date, preventing hindsight leakage.³
2. Lock the scope: draft one or more Methodology criteria that lock in coverage choices: time window, peer set, sampling logic, presentation format, etc. This makes every plan reproducible.
3. Research: consult only authoritative sources (SEC filings, earnings transcripts, market data, regulatory releases, industry reports). AI tools, including Samaya, are explicitly banned.⁴
4. Write the checklist: for each insight, write a full-sentence bullet that is specific, atomic, and begins with an action verb (“Provides…”, “Includes…”). Each bullet is tagged with at least one data source.

Taxonomy of Questions

We have collected a wide range of questions financial analysts actually ask. The table below shows the most common types of questions.

Category	Typical Example	What the checklist will demand
Point-fact retrieval	”What was Exxon’s Q2 2024 revenue?”	The exact figure, source doc, currency code, time reference.
Time-series & deltas	”How have Spotify premium subs trended over the past year?”	Each quarterly datapoint, growth rates, chart instruction.
Event digests	”Summarize today’s Google earnings call.”	Call logistics, headline metrics vs. consensus, CEO sound-bites, key themes.
Forward guidance & comparisons	”What segment guidance did Microsoft give last call, and how does it differ from prior outlook?”	New vs. old numbers, context notes, table formatting.
Thematic extraction	”Which companies mention consumers trading down?”	List of companies, supporting quotes, classification rationale.
Strategic analysis	”Give me a SWOT for NVIDIA.”	Clearly separated strengths, weaknesses, etc., each with supporting evidence.
Other	…	…

Scoring Method

Once the checklist is written, Criteria-Eval uses LLM-based judges to evaluate system responses. Each answer is scored against its checklist, one criterion at a time.

For every criterion, the LLM judge reads the system’s output and returns a binary judgment:

• Pass: if the answer fully satisfies the criterion.
• Fail: if the fact, analysis, or instruction is missing, incorrect, ambiguous, or incomplete.

The final score is simply the percentage of criteria met.

Criteria-Eval assigns high scores if and only if the system both retrieves and faithfully presents all these critical facts. Missing criteria immediately indicate specific retrieval or generation gaps.

Built to Resist Saturation

Criteria-Eval includes many open-ended and researchy questions such as:

• “What’s new with Company X?” or
• “Tell me some interesting developments in [sector]”

While each query typically contains a set of core criteria that are critical (but easier) to address, there is also a deliberate, extensive tail of supplementary criteria. This is by design: by asking annotators to provide exhaustive and detailed criteria we are preventing Criteria-Eval from saturation.

We measured the overlap between the sets of criteria generated by different annotators for the same questions, and we found that on average the overlap is roughly 29%. This reflects the fact that there are multiple valid answers to the same question and provides a benchmark for our systems to reach human level performance on this ambiguous task.⁵

In a landscape where benchmarks are quickly saturated, Criteria-Eval remains demanding and provides meaningful signal for improvement.

Examples

Below are four representative examples of how Criteria-Eval is applied to real-world financial questions. Each example illustrates a specific type of analyst query, along with the corresponding expert-generated checklist criteria used to assess answer quality. Select the tabs to explore each query type in detail.

Analysis: Impact of Query Complexity

To understand how answer quality scales with increasing question complexity, we compared two distinct retrieval strategies using Criteria-Eval. Both systems use the same LLM to synthesize final answers, but differ in how they retrieve supporting evidence.

• RAG: A conventional RAG (naive) baseline, that embeds queries with a dense retrieval model and uses a state-of-the-art LLM to generate the answer.
• Samaya: Samaya’s specialized Q&A pipeline that leverages multiple expert models for query understanding, retrieval, reasoning and synthesis.

We grouped queries by the number of criteria per query, serves as a proxy for question complexity. Complexity varied from simpler queries (~5 criteria) to highly intricate queries (~35+ criteria).

More criteria generally signal a more demanding, multi-faceted question

To help contextualize these results, the dashed line marks the average overlap between the set of criteria generated by different annotators for the same query. This provides a useful human reference point for interpreting system performance (see the Built to Resist Saturation section for more details). The results are shown in Figure 2, and we observe the following:

• Both RAG and Samaya start at similar accuracy levels for simpler queries.
• However, a complexity increases, a clear pattern emerges:
  • The RAG baseline steadily declines, and struggles to surface the full set of relevant facts for the most complex queries, dropping to about 10% accuracy.
  • Samaya maintains a stable performance, even for queries with more than 35 criteria.

These results clearly demonstrate the benefits of a specialized multi-component retrieval and orchestration pipeline, especially as query complexity increases and conventional RAG approaches fall short. Criteria-Eval inherently reveals these differences by naturally encoding query complexity through its criteria-driven scoring.

Why Existing Metrics Weren’t Enough

Current generative evaluation paradigms primarily rely on LLM-as-a-Judge (Zheng et al., 2023), where an auxiliary LLM scores responses against predefined or LLM-generated rubrics (e.g., G-Eval; Liu et al., 2023). While suitable for many general-purpose generation tasks, such as summarization, these rubrics fall short for knowledge-intensive tasks, where quality depends equally on retrieving and faithfully presenting key facts.

Even specialized metrics like FactScore have limitations (Min et al., 2023). FactScore evaluates faithfulness by extracting all claims from a response and verifying them against the retrieved context. While useful for spotting hallucinations, it comes with some limitations:

• Closed-world assumption: FactScore assumes the retrieved context is complete and correct. If a crucial fact was never retrieved in the first place, it cannot detect its absence. A response can be 100% faithful to the context, yet still be factually wrong or incomplete.
• No penalty for omissions: If important facts were retrieved but never included in the response, FactScore doesn’t penalize this. It only evaluates what’s present.
• No instruction-awareness: FactScore is designed for factual accuracy, not broader compliance with user instructions, like specific formatting requests.

Conclusion

Criteria-Eval is now a core part of how we build, monitor, and improve our QA systems at Samaya:

• We run automated daily Criteria-Eval runs to monitor our latest model deployments and catch regressions before users ever see them.
• Our engineers receive immediate actionable alerts whenever significant regressions are detected, allowing quick diagnosis.
• The ML research team leverages Criteria-Eval insights to efficiently debug complex queries and prioritize research efforts
• By providing clear end-to-end signals directly aligned with real user experiences, Criteria-Eval has unlocked the ability to train pipelines with reinforcement learning, helping models learn to work optimally with each other, toward our vision of building co-adapted “Lattice of Experts” systems.

At Samaya, rigorous evaluation is not just a technical necessity but a core part of how we build our products. Criteria-Eval in particular has become the cornerstone of our quality assurance strategy, enabling us to consistently meet the high standards of professional analysts. It is an active and evolving dataset, and we continuously enrich it as new user queries emerge, ensuring our systems remain relevant and closely aligned with real-world needs.

As the AI community evolves toward more sophisticated, knowledge-intensive applications, we hope Criteria-Eval serves as a valuable case study for advancing evaluation methodologies.

Contributors

Ashwin Paranjape, Christos Baziotis, Jack Hessel, and Jack Silva led the design and conceptualization of Criteria-Eval.

Jack Silva led the annotation effort and managed the team of domain experts and wrote the annotation manual.

Mingyi Yang and Christos Baziotis contributed to the engineering and analysis effort.

Special thanks to Yuhao Zhang, Roberto Dessi, Fabio Petroni and Michele Bevilaqua for their valuable discussions and feedback throughout the project.

Footnotes

[1] Because there can be multiple good answers, especially for open-ended finance questions, we collect multiple annotations (criteria lists) per query.

[2] Our internal annotation manual spans over 50 pages and includes examples, decision rubrics, taxonomy definitions, and edge-case resolution protocols. This ensures consistency and domain rigour across annotators.

[3] “Hindsight leakage” refers to the use of information that wasn’t available at the time of the query. In Criteria-Eval, we prevent this by setting a “query date” and ensuring all cited data is publicly available on or before that date.

[4] AI tools, including Samaya, are explicitly banned to ensure that the answers are based solely on the authoritative sources provided. This keeps the evaluation criteria independent and prevents any AI-generated errors from affecting our evaluation framework.

[5] To quantify how much two annotators agree on what constitutes a good answer, we use a cross-evaluation strategy. We start by taking one annotator’s checklist and writing a “perfect” answer that satisfies all their criteria. Then, we evaluate that answer against the checklist written by a second annotator for the same query, exactly as we would evaluate any model-generated response. The proportion of checklist items satisfied in this setup serves as a proxy for the overlap between the two sets of criteria. Averaged across many such annotator pairs, this yields an empirical overlap score of approximately 29%. This number reflects the expected common ground between experts: while many core criteria (e.g., headline figures, key guidance) are shared, there is always an inevitable long tail of more subjective criteria that vary based on an annotator’s judgment. Importantly, if our system exceeds this inter-annotator overlap percentage, it indicates that the system is providing more comprehensive answers that successfully address a broader range of valid criteria than what a typical human expert might include in a single analysis.

References

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore.

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. (2023). FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1165–1183, Singapore.

Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. (2020). Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 747–762, Online.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

---

Appendix

A1. Choosing the LLM-as-a-Judge Setup

Criteria-Eval still relies on the LLM-as-a-Judge approach, as LLMs decides whether each checklist criterion is satisfied by a system’s answer. But before relying on any model, we ran an internal study to evaluate how well different setups align with expert human judgments.

We created a held-out dataset from Criteria-Eval where we sampled system outputs and then manually labeled them against their criteria. This gave us ground-truth judgments which we used to compare different scoring strategies:

• Multi-LLM voting: We used three different LLMs for obtaining judgements and aggregated them using majority voting.
• Single-LLM voting: A single LLM generates multiple judgements using different temperatures and then we combine them using majority voting across samples.

Both methods showed ~94% agreement with human judgements, but the multi-LLM setup aligned more reliably with human labels in edge cases.