4.1
Measuring Commerce Agent Behavior:
The Machine Commerce Dataset
AI agents are increasingly deployed to complete commercial tasks across the web, yet no structured datasets exist for measuring how they actually perform end-to-end across real retail environments. We introduce the Machine Commerce Trace Dataset: a growing collection of structured agent sessions spanning product discovery through checkout across major e-commerce environments. This post describes our collection methodology, trace structure, and early findings on completion rates and failure modes. These traces form the foundation for CartBench, a forthcoming evaluation suite for commerce agents.
1. The Measurement Gap
1.1 Existing Benchmarks Fall Short
The most widely used LLM and agent benchmarks measure general-purpose capabilities: MMLU tests factual recall, HumanEval tests code generation, WebArena and WebVoyager test browser navigation. These are useful, but none were designed to measure whether an agent can successfully buy a product online.11. WebArena includes some shopping tasks on a sandboxed OneStopShopping environment, but the environment does not replicate the authentication flows, dynamic pricing, or bot detection that real retail sites deploy.
The gap matters because commercial deployments have already outpaced evaluation infrastructure. Teams building shopping agents today have no principled way to compare agent performance across vendors, measure regression after a model update, or know whether a reported 70% task completion rate on a sandbox corresponds to anything meaningful in the wild.
1.2 What Commerce Agents Actually Do
A commerce agent navigating a real retail site faces a fundamentally different problem than a benchmark task. Product pages are dynamic. Prices change between page load and checkout. Authentication walls appear mid-flow. CAPTCHA systems activate when request patterns look non-human. Cart state is session-bound and expires.22. We observed that 18% of agent sessions that successfully added a product to cart encountered a session expiry before checkout could be initiated, a failure mode not represented in any existing benchmark we are aware of.
The result is a class of failures that general-purpose benchmarks do not capture: product disambiguation errors, price-comparison abandonment, form-fill failures at checkout, shipping address validation failures, and payment method friction. Each of these failure modes has a distinct root cause and requires a distinct fix. Without structured traces, teams cannot even observe that these failures are happening, let alone prioritize which to address.
2. Collecting Commerce Agent Traces
2.1 Environment Setup
We run instrumented browser sessions against live retail environments. Sessions are initiated by an agent given a natural-language task description and access to a sandboxed browser. We do not use static replicas or cached pages. Every session hits real product inventory, real pricing, and real checkout flows.
Each session captures: a screen recording, DOM snapshots at each agent decision point, the full action sequence (clicks, keystrokes, navigation), task completion status, and any error states encountered. Sessions are isolated so that state from one run does not affect the next.
2.2 Task Taxonomy
We organize tasks into four categories, ordered by complexity:
- Discovery: find a specific product matching a description, including category navigation and search.
- Comparison: compare two or more products on a specified attribute (price, shipping time, specifications).
- Cart management: add a product to cart, verify cart state, apply a discount code.
- Checkout: complete a purchase end-to-end, including address input, shipping selection, and payment.
Each task has a binary completion criterion and an efficiency score based on the number of steps taken relative to the minimum possible path. We do not score style or phrasing of agent responses - only whether the commercial objective was achieved and how efficiently.
2.3 Trace Format
Raw session recordings are post-processed into structured trace records. Each record contains: task description, environment identifier, agent identifier, action sequence, completion status, per-step decision quality annotations, and a bottleneck classification for failed sessions.33. Bottleneck classification uses human annotators working from a 12-category taxonomy. We are building an automated classifier expected to reduce annotation cost by ~80%.
Trace records are stored in a structured JSON format with a normalized schema across all environments. This is the primary contribution of the dataset: a common format that allows direct comparison of agent behavior across heterogeneous retail environments without environment-specific parsing.
3. Early Findings
3.1 Completion Rates by Stage
Across an initial sample of sessions on major US retail environments, completion rates drop sharply with task complexity. Product discovery completes at a high rate for well-specified queries. Checkout completion is substantially lower, driven primarily by failures in the cart-to-checkout transition.
The most striking finding is not the overall checkout completion rate but its variance across environments. On some retail sites, checkout completes at more than twice the rate of others, for the same agent and the same task. This variance is almost entirely explained by form complexity: sites with guest checkout, autocomplete support, and minimal required fields see substantially higher completion rates.
3.2 Failure Modes
We classify failures into three primary categories based on the annotated trace records:
- Navigation failure: agent cannot locate the target product or cannot find the path to the next task stage. Most common on sites with non-standard navigation patterns or aggressive search ranking manipulation.
- Form-fill failure: agent fails to correctly populate a required form field, most often shipping address validation, phone number format, or credit card input. These failures are almost always recoverable with a single retry but current agents rarely retry correctly.
- Authentication friction: a login wall, CAPTCHA, or session expiry interrupts the agent mid-task. These failures are effectively unrecoverable without human intervention.
Authentication friction deserves particular attention because it is the failure mode most invisible to teams building agents in sandboxed environments. A sandboxed environment by design does not deploy bot detection. Teams that build and test in sandbox see no authentication failures at all, then encounter them at a high rate in production. The trace dataset captures these failures explicitly, which we believe is necessary for building agents robust to real commercial deployments.
4. CartBench
The Machine Commerce Trace Dataset is the raw material. CartBench is the evaluation suite built from it. We are designing CartBench to cover all four task categories across a representative sample of major retail environments, with test cases scored for completion, efficiency, and robustness to the failure modes described above.
CartBench is not a sandboxed benchmark. Test cases run against live environments with live state, which means scores will vary over time as retail sites evolve. We treat this as a feature: a score that degrades over time tells you your agent has become brittle relative to the environment. Static benchmarks cannot tell you this.
We expect to release a first version of CartBench alongside a subset of the trace dataset in early 2027. Labs and teams interested in early access can reach out at founders@generalmachines.ai.