Description
advance. An AI feature does not give you one. Ask the same question twice and you get two different answers, and both may be right.
So you stop testing for the answer and start testing for properties of the answer. That shift is what this kit teaches, and it is what makes a test suite possible for a system whose output you cannot predict.
**Every defect is verified.** All 18 were confirmed by running them — each fires when armed
and passes when disarmed. That check is the expensive part, and it is the part a generated
case list does not have.
What is inside
The playbook, 27 pages. The oracle problem, retrieval versus generation defects, groundedness and citation integrity, hallucination, the over-refusal failure almost nobody tests, prompt injection including the indirect kind, and how to build an evaluation set that survives a model change.
The case bank, 95 cases. Plus an evaluation set template, a red team prompt library, an acceptance criteria sheet and a client sign-off sheet.
The practice sandbox. A deterministic simulator of an LLM support assistant with 18 switchable defects, and a working evaluation harness. It is deterministic on purpose: a real model would leave you unable to tell a caught defect from sampling noise.
The free 45-minute smoke test. For before every prompt change, model version change or knowledge-base update.
Who it is for. Teams shipping a chatbot, a support assistant, a summariser, or any feature with a model behind it.
How this differs from Juice Shop and DVWA
Those are excellent, free, and built for a different job — they teach you to attack an application. This teaches you to test one.
Juice Shop will not show you a subscription that keeps billing after cancellation, a proration that ignores an active discount, or an AI feature that refuses a legitimate question. And it has no switch: you cannot disarm a defect and confirm the same check now passes, which is where most of the learning is.
