# How to test whether agent distribution is working

> A practical evaluation design that separates discovery, understanding, completion, and real customer acquisition.

Canonical: https://byagents.dev/humans/evaluating-agent-distribution
Published: 2026-09-13
Reviewed: 2026-09-13
Publisher: ByAgents
Type: Playbook
Level: Mixed
Outcome: Run a reproducible small evaluation and report the result without inflating what it proves.

An evaluation is a controlled way to observe behavior. It becomes useful when the task, environment, introduction method, and expected result are recorded. Asking an assistant whether your site is good is feedback, not a distribution experiment.

## Separate two experiments

**Unbranded discovery:** give a realistic task and constraints without naming your business. Observe whether the environment searches, what it retrieves, which options it considers, and why it continues or stops. This tests an available route in that environment.

**Introduced-provider adoption:** supply the product or its URL, then test whether the agent can understand and use it. This diagnoses product information and handoff quality. It cannot establish that the business was independently found.

Both are useful. Mixing their results makes discovery look better simply because some runs were handed the answer.

## Build a task set before tuning the site

Include common customer tasks, difficult but relevant constraints, and tasks where your offering should be rejected. Reserve some tasks as a holdout that you do not use to write the new content. Specify the observable result for each task before seeing an answer.

For a fictional extraction API, completion could require correct invoice line items against a known fixture. For a service, it might require a correctly scoped inquiry with the requested service area. For a booking, completion means confirmed reservation state, not a draft message saying a reservation is possible. Use synthetic inputs and test environments where available.

## Record the environment

Capture date, application, model or mode when exposed, tool access, login state, location assumptions, language, time budget, and supplied context. Some consumer applications do not expose every setting. Record unknown values rather than inventing precision.

Keep fresh sessions where feasible and note persistent personalization. An agent with your docs preloaded has a different starting point from an ordinary unbranded user. Repeat selected tasks to reveal variability, but do not silently discard unsuccessful runs.

## Score stages with observable evidence

| Stage | Evidence | A common false positive |
| --- | --- | --- |
| Encountered | Actual source or tool retrieved | The model already knew the name |
| Understood | Correct facts with supporting sources | A persuasive but incorrect summary |
| Selected | An observed recommendation or chosen tool | Your URL was included in the prompt |
| Completed | Verified artifact or external test state | A 202 response or a promise to finish |
| Acquired | A qualified external customer event | Your own evaluation traffic |

A stage may be unknown when the environment does not expose its trace. Do not convert missing evidence into a failure or success automatically. Preserve the reason for the unknown state.

## Report denominators and failures

An **illustrative, invented** result might say: “Of 12 unbranded runs, 4 retrieved the guide; 3 described the offer correctly; 2 reached a valid next step.” These are not ByAgents results. Report both the full denominator and any conditional metric: 2 of 12 overall differs from 2 of 4 among retrieved cases.

Small task sets are primarily debugging tools. A few additional successes can change the percentage substantially. Include raw counts, environment, dates, and failure categories. Do not present repeated runs of one task as independent evidence about an entire market.

## Interpret before-and-after changes cautiously

Preserve the tasks, budget, and configuration; record everything you changed. Use an unchanged comparison group when possible. Search results, model versions, product availability, and platform behavior can change during the experiment. If you cannot isolate your intervention, describe an observed difference rather than a causal lift.

Tag testing so it does not inflate organic traffic, trial activation, or order reports. Keep evidence that the requested final state exists. Anthropic's [tool evaluation guidance](https://www.anthropic.com/engineering/writing-tools-for-agents) provides additional context on task-based evaluation.

## Turn the result into a repair

Cluster failures by the earliest broken step. Fix missing facts, ambiguous operations, invalid inputs, or inaccessible pages before trying to improve a composite score. Rerun the affected cases, then the held-out tasks. Publish the method and material limitations if sharing the result externally. Use our [evaluation record](https://byagents.dev/agents/evaluation-record-template) to preserve enough context for another person or agent to inspect the conclusion.

## Sources

- [Anthropic: Writing tools and evaluating them](https://www.anthropic.com/engineering/writing-tools-for-agents)

## Related reading

- [Measure agent discovery without fooling yourself](https://byagents.dev/humans/measuring-agent-traffic)
- [Template: a reproducible agent evaluation record](https://byagents.dev/agents/evaluation-record-template)
- [Pattern: measure agent activity without inventing attribution](https://byagents.dev/agents/attribution-event-contract)