# Evaluating Your Knowledge Base



<a id="section-goal"></a>

Answer quality and speed depend on the knowledge — its accuracy, completeness and links to evidence — but also on the model, the runtime, retrieval planning and how tools are executed. A good evaluation compares your knowledge base against an alternative, or against its own earlier version, on real questions, and then uses the run traces to find out *why* answers were slow or wrong.

```mermaid
flowchart LR
  set["Freeze a question set<br/>with criteria"] --> run["Run each setup"]
  run --> grade["Blind grading<br/>four levels"]
  run --> time["Time and tool calls"]
  grade --> compare["Compare on the<br/>common subset"]
  time --> compare
  compare --> trace["Read traces<br/>to locate causes"]
```


<a id="section-questions"></a>

## Build the question set

- Use **real developer questions**, covering the kinds of question people actually ask — locating an interface or page, interaction rules, parameters, permissions, version differences, debugging and environments.
- **Freeze the set before running** anything, so it cannot drift toward what a setup happens to answer well.
- **Write the grading criteria for each question in advance.**


<a id="section-measure"></a>

## Grade and measure

- Grade each answer against its criteria on **four levels**: fully resolved, partially resolved, materially wrong, not answered.
- Keep grading **blind** — the reviewer, a model or a person, should not know which setup produced the answer. A model reviewer that has not been calibrated against people is a limitation to state.
- Measure **end-to-end time** as the caller sees it, until the complete answer arrives. Report the mean, P50 and P90 (nearest-rank).
- Count the **tool calls** visible to the Agent.


<a id="section-compare"></a>

## Compare fairly

- To remove differences in question difficulty, also compare on the **common subset** — the questions every setup fully resolved.
- Change one thing at a time where you can. When a comparison changes both the knowledge setup and the model, add a **same-model control** to isolate each effect.


<a id="section-causes"></a>

## Find the causes

Numbers say *that* something is slow or wrong; traces say *why*. Read the run traces and classify what you see:

- **Missing knowledge** — the knowledge base does not cover the fact at all.
- **A key condition left out** — for example a component's branch condition, leading to an over-generalized answer. Turn such gaps into revision tasks.
- **Repeated searching** — the Agent keeps exploring instead of reading.
- **Waiting for startup** — environment preparation dominates the time.
- **Tool overhead** — many calls spent preparing repositories, running shells and reading files. Reading code remotely at a fixed version can replace much of this.


<a id="section-disclose"></a>

## Publish the limits with the numbers

State the conditions any reader needs to interpret the results: how many times each question ran (a single run says nothing about stability), differences between environments, system prompts, models, versions, dates and timeouts, whether the questions had been seen before, how many reviewers graded, what the timings include, and how any cost figures were estimated.

A worked example with its own limits is [Knowledge Retrieval Benchmark](</context/llms/pages/06602fd2c6c83d3113540066149a34b9.txt>).

---

## Sources

- note:20261012/usage-practices.md · usage-practices.md L73–L75
- note:20261012/benchmark-summary.md · benchmark-summary.md L19–L30
- note:20261012/benchmark-summary.md · benchmark-summary.md L32–L37
- note:20261012/benchmark-summary.md · benchmark-summary.md L38–L38
- note:20261012/benchmark-summary.md · benchmark-summary.md L59–L61
- note:20261012/benchmark-summary.md · benchmark-summary.md L15–L17
- note:20261012/usage-practices.md · usage-practices.md L75–L75
- note:20261012/benchmark-summary.md · benchmark-summary.md L69–L74
- note:20261012/benchmark-summary.md · benchmark-summary.md L76–L81