Measure agent discovery without fooling yourself

Agent traffic is not one clean metric. A request from a crawler, a referral from an assistant, a tool invocation, and a paying customer are different events. A useful measurement plan preserves those distinctions rather than combining them into a single growth number.

Start with an evidence ladder

Record access first: could the page or tool be reached? Then record encounter: did the system actually retrieve it? Next record selection, useful completion, and returning usage. Each event supports a narrower claim than the next.

A crawler visit establishes a request. It does not prove that the page was indexed, cited, or recommended. A mention in a controlled test establishes behavior in that run. It is not an independent customer. A conversion may be assisted by an agent even when the original discovery source is unknown.

Keep four data streams separate

• Infrastructure requests: Server logs, response codes, and verified crawler signals.
• Referral visits: Requests carrying an observed referring source or campaign marker.
• Controlled evaluations: Tasks you initiated with recorded environments and expected outcomes.
• Customer events: Qualified inquiries, activated projects, purchases, and returning usage from first-party records.

Do not count an IP address as a unique agent, person, or company. User-agent strings can be copied. A missing referrer is unknown attribution, not proof that the visit came from an agent. Use provider verification guidance where available and retain uncertainty in the report.

Design a fair before-and-after test

Choose representative tasks before making changes. Record the environment, available tools, model or mode where known, introduction method, and task budget. Separate tasks that name the provider from tasks that require discovering one. Keep some tasks unchanged and outside the set used to tune the content.

Verify outputs directly: inspect the extracted fields, resulting booking status, or final artifact. Agent self-reports can help diagnose a failure but should not be the only completion check. Anthropic’s evaluation guidance emphasizes realistic tasks and observable outcomes. Writing tools and evaluations for agents.

Report counts and denominators

“Three of twelve unbranded discovery trials retrieved our guide in this environment” is more informative than “AI visibility up 300%.” Include failures, test dates, and meaningful changes to the runtime. Small samples can guide debugging; they are weak support for broad claims about all assistants.

Search results and models change. A before-and-after difference may reflect more than your edit. Describe an association when you cannot isolate causation. Mark evaluation traffic so it cannot inflate organic acquisition or conversion reports.

Connect to the business carefully

With appropriate permission, connect observed entry paths to first-party customer events. Decide how to handle existing accounts, repeat projects, assisted conversions, and attribution windows. Document those choices before comparing cohorts. Revenue belongs to the actual customer event, not to a model’s prediction that the prospect might buy.

ByAgents does not currently provide production attribution analytics or measured traffic guarantees. This framework describes the evidence needed to validate the distribution ambition, and the discipline we want the product to support.

Add platform reports without changing their meaning

Google's current Search Console help describes a Generative AI performance report for Search, including AI Overviews and AI Mode impressions. It is a platform exposure report, not a complete conversion ledger. Google report documentation.

Microsoft's AI Performance documentation describes citation activity and grounding-query insights. It explicitly cautions against interpreting the data as ranking or authority. Bing Webmaster Tools announcement.

Keep reporting windows, property scope, and metric definitions with each export. An impression, citation, visit, and confirmed customer event have different denominators and cannot simply be added. Inspect the features actually available in your account. The evaluation playbook covers controlled tests, while the event contract provides a separate first-party measurement pattern.