Find out why your support agent breaks refund policy: replay real tickets as JSON decisions, then let code enforce the rule

An AI support agent granted a refund on day 47 under a 30-day policy, logged it as a success, and the owner only noticed through the payment notification. Rewording the prompt did not stop it recurring, and every wrong decision cost real money.

1. Force the agent to return a structured decision (decision, days since purchase, reason code, confidence) instead of a prose reply, so each answer can be checked. 2. Build about 30 test cases from real support transcripts, and write the expected decision for each from the actual policy document before running anything. (The author's first set, written from memory of the prompt, missed rules and passed while measuring nothing.) 3. Run each case about 20 times per prompt version through the Anthropic API with a short Python script and record how often the expected decision comes back. The author spent under $30 across five prompt versions. 4. Use the results to fix the prompt: move the policy rule where it wins (in the author's tests, the bottom of the prompt), scope tone lines so they cannot override decisions ('flexible in wording, never in the decision'), put the worked example right under the rule it shows, and cut length. 5. Do not delete rules that look redundant without re-testing: removing a separate 'chargeback threats always escalate' line let a chargeback case through to a refund. 6. Move the policy itself into ordinary code: a small function checks the agent's JSON against the policy table, and when code and model disagree the ticket goes to a person and is logged. Keep the prompt for tone, extraction and formatting.

The author reports that rule compliance depended heavily on placement (94% at the bottom, 23% at the top in their 30-case test), fell from 96% to 18% as the prompt grew from 190 to 1,410 words, and rose from 41% to 96% on late-refund cases after one scoping clause. A trimmed rule cost about $340 in wrong refunds over eleven days before it was caught. The final fix was a short code check plus a 240-word prompt. No figures after the change are given beyond these test results.

Self-reported by a single operator with no inspectable logs or full test set; only parts of the code are shown. Percentages come from the author's own 30 cases and one model, so they may not carry over. The post ends by selling the author's own prompt pack, so it is labelled vendor. The account belongs to a series of very similar dramatic posts published almost daily, so treat details with some caution. Business type and size are not stated beyond a store with refunds and Stripe.

An AI agent that handles support or refunds through an API, access to past support transcripts, and enough Python to run a test loop.

Write the expected answer for each test case from your policy document, not from your prompt.

• DEV Community (ULNIT) ↗ Vendor case study, published October 30, 2026

Source published: October 30, 2026. Last reviewed here: November 2, 2026. Spot a mistake? Tell us.