Field notes

Notes on testing AI agents: incident breakdowns, benchmark critiques, and arguments about what counts as a real finding.

  1. One line in a 2-star review got Sonnet 4.5 to call a refund tool in 65% of chats

    · One line at the end of a stranger's two-star review got Claude Sonnet 4.5 to call the refund tool for the full $149 in 13 of 20 conversations, and 0 of 20 without it.

  2. The Chevy dealer's bot didn't offer a $1 Tahoe. The user did.

    · A visitor wrote the Chevy dealer bot's famous legally binding offer line himself, but a version of the trick needs no help from the customer.