How to test whether agent distribution is working

An evaluation is a controlled way to observe behavior. It becomes useful when the task, environment, introduction method, and expected result are recorded. Asking an assistant whether your site is good is feedback, not a distribution experiment.

Separate two experiments

Unbranded discovery: give a realistic task and constraints without naming your business. Observe whether the environment searches, what it retrieves, which options it considers, and why it continues or stops. This tests an available route in that environment.

Introduced-provider adoption: supply the product or its URL, then test whether the agent can understand and use it. This diagnoses product information and handoff quality. It cannot establish that the business was independently found.

Both are useful. Mixing their results makes discovery look better simply because some runs were handed the answer.

Build a task set before tuning the site

Include common customer tasks, difficult but relevant constraints, and tasks where your offering should be rejected. Reserve some tasks as a holdout that you do not use to write the new content. Specify the observable result for each task before seeing an answer.

For a fictional extraction API, completion could require correct invoice line items against a known fixture. For a service, it might require a correctly scoped inquiry with the requested service area. For a booking, completion means confirmed reservation state, not a draft message saying a reservation is possible. Use synthetic inputs and test environments where available.

Record the environment

Capture date, application, model or mode when exposed, tool access, login state, location assumptions, language, time budget, and supplied context. Some consumer applications do not expose every setting. Record unknown values rather than inventing precision.

Keep fresh sessions where feasible and note persistent personalization. An agent with your docs preloaded has a different starting point from an ordinary unbranded user. Repeat selected tasks to reveal variability, but do not silently discard unsuccessful runs.

Score stages with observable evidence

A stage may be unknown when the environment does not expose its trace. Do not convert missing evidence into a failure or success automatically. Preserve the reason for the unknown state.

Report denominators and failures

An illustrative, invented result might say: “Of 12 unbranded runs, 4 retrieved the guide; 3 described the offer correctly; 2 reached a valid next step.” These are not ByAgents results. Report both the full denominator and any conditional metric: 2 of 12 overall differs from 2 of 4 among retrieved cases.

Small task sets are primarily debugging tools. A few additional successes can change the percentage substantially. Include raw counts, environment, dates, and failure categories. Do not present repeated runs of one task as independent evidence about an entire market.

Interpret before-and-after changes cautiously

Preserve the tasks, budget, and configuration; record everything you changed. Use an unchanged comparison group when possible. Search results, model versions, product availability, and platform behavior can change during the experiment. If you cannot isolate your intervention, describe an observed difference rather than a causal lift.

Tag testing so it does not inflate organic traffic, trial activation, or order reports. Keep evidence that the requested final state exists. Anthropic's tool evaluation guidance provides additional context on task-based evaluation.

Turn the result into a repair

Cluster failures by the earliest broken step. Fix missing facts, ambiguous operations, invalid inputs, or inaccessible pages before trying to improve a composite score. Rerun the affected cases, then the held-out tasks. Publish the method and material limitations if sharing the result externally. Use our evaluation record to preserve enough context for another person or agent to inspect the conclusion.