This is a ByAgents example format. It is not an external standard, a supported upload endpoint, or a claim of a completed test. Use it to make an evaluation inspectable and to prevent a successful demonstration from being misreported as independent acquisition.

Example record

json
{
  "schema_version": "1.0",
  "example": true,
  "run_id": "example-run-001",
  "started_at": null,
  "traffic_class": "controlled_evaluation",
  "task_id": "invoice-lines-01",
  "task": "Find an API for scanned invoice line items with JSON output.",
  "introduction": "unbranded",
  "environment": {
    "application": null,
    "model": null,
    "tools_available": [],
    "login_state": "unknown",
    "locale": "en-US",
    "time_budget_seconds": 120,
    "preloaded_sources": []
  },
  "expected": {
    "supported_input": "scanned PDF",
    "output": "JSON line items",
    "price_unit_explained": true,
    "completion_evidence": "Verified result from a synthetic fixture"
  },
  "observed": {
    "retrieved_urls": [],
    "encountered": "not_run",
    "understood": "not_run",
    "selected": "not_run",
    "completed": "not_run"
  },
  "evidence_references": [],
  "failure_stage": null,
  "limitations": ["Template only; no run has been performed."]
}

Empty tools_available means this template contains no recorded tool inventory. In a real run, distinguish a confirmed empty inventory from unknown availability. The expected completion may require a separate authorized adoption test if the unbranded run stops at research.

Required interpretation

introduction records whether the provider was named, linked, preinstalled, or independently encountered. If a relevant integration is already installed, record that fact even if the natural-language task omits the brand. preloaded_sources exposes another way the provider may already be in context.

Use observed values such as pass, fail, unknown, not_attempted, or not_run, and define them in your evaluation system. Unknown means the evidence does not establish a result. Not attempted means the run did not reach that stage. Neither should silently become a failure or success in an aggregate.

Preserve evidence safely

Store references to authorized traces, public sources, synthetic inputs, and resulting artifacts. Keep private customer content and credentials out of public reports. Retain only the data needed to substantiate the claim, with appropriate access and retention controls.

A source URL alone is insufficient if its contents can change. Record the relevant observation, collection time, version where available, and a permitted snapshot or content hash when useful. A hash helps identify bytes; it does not establish that the statement was true.

Aggregate without hiding the denominator

Group comparable tasks and environments. Report total eligible runs, attempted runs, known outcomes, unknown outcomes, and failures by stage. If 3 of 5 retrieved cases complete out of 20 discovery runs, report both 3/5 conditional completion and 3/20 overall completion. Those numbers answer different questions.

Count a rerun as another run, not another customer. Identify task repetitions and evaluation changes. A holdout set should not quietly become a tuning set while retaining the same label. Preserve unsuccessful runs unless a documented exclusion rule applies.

What the record cannot establish

A controlled task does not establish organic traffic, a buyer's independent intent, incremental revenue, or market-wide assistant behavior. A recorded recommendation does not establish factual correctness without checking its claims. A 200 response does not establish that the intended action completed.

Use the evaluation guide to design the experiment and the event contract for separate customer-event reporting. Keep the original record with the conclusion so another reviewer can identify which claim the evidence actually supports.