Evaluating Your Knowledge Base

Answer quality and speed depend on the knowledge — its accuracy, completeness and links to evidence — but also on the model, the runtime, retrieval planning and how tools are executed. A good evaluation compares your knowledge base against an alternative, or against its own earlier version, on real questions, and then uses the run traces to find out why answers were slow or wrong.

flowchart LR
  set["Freeze a question set<br/>with criteria"] --> run["Run each setup"]
  run --> grade["Blind grading<br/>four levels"]
  run --> time["Time and tool calls"]
  grade --> compare["Compare on the<br/>common subset"]
  time --> compare
  compare --> trace["Read traces<br/>to locate causes"]

Build the question set

• Use real developer questions, covering the kinds of question people actually ask — locating an interface or page, interaction rules, parameters, permissions, version differences, debugging and environments.
• Freeze the set before running anything, so it cannot drift toward what a setup happens to answer well.
• Write the grading criteria for each question in advance.

Grade and measure

• Grade each answer against its criteria on four levels: fully resolved, partially resolved, materially wrong, not answered.
• Keep grading blind — the reviewer, a model or a person, should not know which setup produced the answer. A model reviewer that has not been calibrated against people is a limitation to state.
• Measure end-to-end time as the caller sees it, until the complete answer arrives. Report the mean, P50 and P90 (nearest-rank).
• Count the tool calls visible to the Agent.

Compare fairly

• To remove differences in question difficulty, also compare on the common subset — the questions every setup fully resolved.
• Change one thing at a time where you can. When a comparison changes both the knowledge setup and the model, add a same-model control to isolate each effect.

Find the causes

Numbers say that something is slow or wrong; traces say why. Read the run traces and classify what you see:

• Missing knowledge — the knowledge base does not cover the fact at all.
• A key condition left out — for example a component's branch condition, leading to an over-generalized answer. Turn such gaps into revision tasks.
• Repeated searching — the Agent keeps exploring instead of reading.
• Waiting for startup — environment preparation dominates the time.
• Tool overhead — many calls spent preparing repositories, running shells and reading files. Reading code remotely at a fixed version can replace much of this.

Publish the limits with the numbers

State the conditions any reader needs to interpret the results: how many times each question ran (a single run says nothing about stability), differences between environments, system prompts, models, versions, dates and timeouts, whether the questions had been seen before, how many reviewers graded, what the timings include, and how any cost figures were estimated.

A worked example with its own limits is Knowledge Retrieval Benchmark.

评测知识库效果

回答的质量和速度取决于知识本身（准确性、完整性以及与证据的关联），也取决于模型、运行时、检索规划和工具的执行方式。好的评估会在真实问题上，把你的知识库与另一种方案或它自己的早期版本做比较，再借助运行轨迹找出回答慢或出错的原因。

flowchart LR
  set["冻结题集<br/>并附判据"] --> run["运行各配置"]
  run --> grade["盲评<br/>四档"]
  run --> time["耗时与工具调用"]
  grade --> compare["在共同子集<br/>上比较"]
  time --> compare
  compare --> trace["阅读轨迹<br/>定位原因"]

构建题集

• 使用真实的开发者问题，覆盖大家实际会问的各类问题：定位接口或页面、交互规则、参数、权限、版本差异、调试和环境。
• 在运行任何测试之前冻结题集，避免它逐渐偏向某个配置恰好擅长回答的问题。
• 提前为每道题写好判据。

评分与测量

• 按判据把每个回答评为四档之一：完全解决、部分解决、实质错误、未作答。
• 坚持盲评：评审者（模型或人）不应知道回答来自哪个配置。如果模型评审者未经过与人工评分的校准，应作为局限说明。
• 测量调用方视角的端到端耗时，直到完整回答返回为止。报告平均值、P50 和 P90（最近秩法）。
• 统计 Agent 可见的工具调用次数。

公平比较

• 为消除题目难度差异，还要在共同子集上比较，即所有配置都完全解决的那些题。
• 尽可能一次只改变一个因素。如果一次比较同时改变了知识配置和模型，就增加一组同模型对照，分别隔离各自的影响。

定位原因

数字告诉你哪里慢或出错，轨迹告诉你为什么。阅读运行轨迹，并对看到的情况分类：

• 知识缺失：知识库完全没有覆盖这个事实。
• 遗漏关键条件：例如遗漏了某个组件的分支条件，导致回答过度泛化。把这类缺口转成修订任务。
• 重复搜索：Agent 一直在探索，而不是去阅读。
• 等待启动：环境准备占据了大部分时间。
• 工具开销：大量调用花在准备仓库、执行 shell 和读取文件上。在固定版本上远程读取代码可以替代其中的大部分。

连同局限一起公布数字

说明读者解读结果所需的条件：每道题运行了几次（单次运行无法说明稳定性）、环境之间的差异、系统提示词、模型、版本、日期和超时设置、题目是否事先见过、由几位评审者评分、耗时包含哪些部分，以及成本数字是如何估算的。

一个附带自身局限说明的完整示例见知识检索测评。