AI for Product Management

A failure-mode table for one shipped AI feature you can use daily — a summariser, a search ranker, a support bot. Every row is a failure you produced yourself, not one you imagined.

1. Collect 30 real failures. Two hours with the feature, using adversarial and boringly typical inputs alike. Screenshot every output that is wrong, refused or useless.
2. Classify them. Confidently wrong, subtly wrong, refused when it should have answered, answered when it should have refused, correct but unusable, and wrong for one user group only.
3. Score three ways. Frequency out of 30, severity 1–5, and detectability — can the user tell it is wrong without checking elsewhere? The undetectable ones are the dangerous ones.
4. Design the interface response per class. Citations, a confidence display, an edit affordance, a fallback path, or a refusal that routes to a human. Specific copy, not a description of copy.
5. Set one escalation rule. The single condition under which the feature hands off to a person, plus what it costs to set that threshold too high and too low.
6. Accept one failure explicitly. Name the class you would not design around, what that decision costs, and who pays it. A map where every row gets fixed is a wish list.

Scope guard: one feature, 30 observed failures, one table. A map of imagined failures is a brainstorm.

The same 40 pieces of customer feedback synthesised twice — once by you, once by a model, independently — then compared. The comparison is the deliverable; the themes are not.

1. Freeze the corpus. 40 verbatim items from one product: support tickets, reviews or interview notes. Lock the set before you start so neither pass can cherry-pick. No feedback of your own? Take 40 reviews of a single product from the Amazon Fine Food Reviews set.
2. You go first. Tag every item, cluster into themes, record the count and two supporting quotes per theme. Timebox it and note how long it took.
3. Model goes second, blind. Same 40 items, one prompt asking for themes with counts and supporting quotes. Do not show it your themes, and do not iterate the prompt until it agrees with you — that measures your patience, not the method.
4. Line the two up. Themes both found, themes only you found, themes only it found, and quotes attributed to the wrong theme. Check every quote against the source text by hand.
5. Count the invented ones. Report fabricated or misattributed quotes as a fraction of all quotes produced. That single number decides whether the method is trustworthy on a real decision.
6. Write the operating rule. One paragraph: when you would use model synthesis unreviewed, when as a reviewed first pass, and when not at all.

Scope guard: one model, one prompt, one pass each. The value is in the divergence table, not in better themes.

A complete specification for one AI feature, written so an engineer could start on Monday and a lawyer could read the data section without asking you anything.

1. State the non-AI baseline. What the user does today, how long it takes, what it costs. If the feature cannot beat that, the document stops here and says so.
2. Acceptance criteria as thresholds. Not "accurate summaries" but "at least 90% of summaries contain no claim absent from the source, measured on the 50-case eval set, scored by two reviewers".
3. Specify every failure path. What the user sees on low confidence, on refusal, on timeout, and when the model is unavailable. Write the actual copy for each.
4. Name the data. One table: what is sent to the model, what is retained and for how long, where it is stored, what is excluded, and who approved that.
5. Budget cost and latency. Tokens per call times expected daily calls times price, plus the p95 latency you will accept. State what gets cut if the budget is breached.
6. Have it torn apart. Give it to an engineer, log their first five questions verbatim, and rewrite until the document answers them without you in the room.

Scope guard: one feature, thresholds instead of adjectives, no roadmap section. A PRD that survives five engineer questions unaided is finished; one that needs you is a conversation.

A 50-case evaluation set for the feature you specified, plus two scored runs that changed a decision. This is the artefact almost no product portfolio contains.

1. Build the distribution deliberately. 25 typical cases, 10 ambiguous, 10 out of scope where refusing is the correct answer, and 5 adversarial. Hand-written — generated cases test the generator.
2. Define pass per case. Each row carries the input, the expected behaviour, and a binary or 1–5 rubric a second person could apply without asking what you meant.
3. Validate the rubric before trusting it. Have someone else score ten cases independently. More than two disagreements means the rubric is ambiguous — fix it before scoring anything else.
4. Run one. Score the current implementation across all 50 and report per category, never as a single average. An 80% hiding 0% on out-of-scope refusals is a failing feature.
5. Change exactly one thing. One prompt edit, one retrieval change, or one model swap. Re-run and set the two columns side by side.
6. Report the regression. Something nearly always gets worse. Name it, and state whether you would ship anyway and on what grounds.

Scope guard: 50 hand-written cases beat 500 generated ones. No LLM-as-judge until a human has scored the same rows and you know how far the judge drifts from them.