Worked example: from an invoice question to a usable API

This is a fictional design exercise. Example Invoice API is not a ByAgents customer, and the sequence below is not a measured agent run. The point is to make a proposed distribution system inspectable before claiming it works.

The customer task

“Find an API that extracts line items from scanned PDF invoices, returns JSON, and lets our application retrieve the output after processing. Explain the price unit and how to try it.” The provider is not named. The constraints include scanned PDFs, line items, asynchronous completion, pricing clarity, and a usable trial.

An evaluator should include an unsuitable task too: for example, a request for handwritten receipts if the fictional product does not support them. Correct rejection is evidence of accurate understanding, not a marketing failure.

Before: the source material is incomplete

Assume the homepage says “document intelligence for modern teams,” the pricing page lists credits without explaining the unit, and the API reference shows only POST /jobs. A tool listing promises extraction but does not say how the final output is obtained.

These are different defects. Broad positioning makes selection difficult. Ambiguous credits prevent cost comparison. Missing result retrieval breaks the task after selection. Publishing more listings would not repair the latter two.

After: one consistent product description

The fictional product page now states: “Extract line items from scanned PDF invoices into JSON. Processing creates a job; retrieve the job until it completes, then read its result. Billing is per processed page. Handwritten receipts are unsupported.” An actual business should replace these statements with verified product behavior and real pricing.

The page links to a sample fixture, a documented output, the price source, and the quickstart. The structured record and tool description repeat the same boundaries. No rating, compliance claim, or accuracy percentage is added without evidence.

The intended route

Customer task
  -> search or a relevant developer catalog
  -> task-specific explanation
  -> documented constraints and price unit
  -> trial / authorization
  -> create extraction job
  -> retrieve status and final output
  -> validate line items against the fixture

This is an intended route. The test must record which source was actually encountered. If the agent was handed the quickstart, the run starts at the quickstart; it does not prove the preceding search step happened.

The concrete repair backlog

Assign each repair to an owner. Do not combine all five into an opaque score. The integration owner can fix a missing retrieval operation; the content owner can reconcile a price explanation; neither should claim improved discovery before testing it.

What would count as evidence

Static review can show that documentation now contains the required information. An authorized execution can show that a specific input produces the expected output. A controlled unbranded run can show that a particular environment encountered the product. A real external activation can show customer usage, with acquisition attribution only when supported.

No one of those observations establishes all the others. Keep timestamps, versions, inputs, expected result, actual result, and limitations with each claim. The evaluation template and evidence record are reusable formats for this purpose.

Apply the pattern to another business

For a restaurant, replace file types with location, group size, and dietary constraints; replace job completion with reservation confirmation. For a freelancer, replace the sample fixture with a scoped brief and the API result with an accepted inquiry or agreed deliverable. The shared design question is whether the route preserves the customer's intent all the way to a real next step.