False positive
The agent interrupted about something that did not deserve attention. Usually the source was weak, the consequence was low, or the rule was too broad.
How to review false positives in a personal AI agent
False positives are not just annoying alerts. They are evidence that the agent misunderstood timing, confidence, consequence, or user preference. A clean review loop turns each bad proactive message into a better system prompt.
Separate wrong alerts from wrong timing.
A personal agent can be factually accurate and still wrong to interrupt. The review should classify the failure before changing the prompt.
False negative
The agent stayed quiet when it should have warned, summarized, or escalated.
Too early
The alert mattered eventually, but the agent sent it before action was possible.
Too late
The agent noticed the issue after the useful action window had closed.
Duplicate
The agent repeated a signal the user had already seen or resolved.
Review the alert, name the failure, identify the missing source check, rewrite the system prompt, and verify the next similar event.
A weekly review loop is enough if the examples are concrete.
Do not tune the agent from vibes. Use the bad alert itself as the training artifact.
Save the interruption receipt.
Keep the original alert, source, timestamp, confidence, and recommended action. This is essential for a text message AI assistant because the channel is so expensive in attention.
Find the missing gate.
Ask what source, time window, human confirmation, or confidence threshold would have prevented the bad interruption.
Rewrite the instruction.
Tell the system prompt what went wrong and what should always happen instead. Replace vague preferences with precise rules.
Use the same review modes across channels.
The details differ by workflow, but the review taxonomy should stay stable.
Classify
Name the failure before editing the prompt or changing a threshold.
Verify
Require stronger source proof before another proactive interruption.
Rewrite
Translate the failure into exact replacement behavior.
Retest
Watch the next similar event and confirm the agent behaves differently.
False-positive review checklist
- Was the agent's fact correct but the interruption unnecessary?
- Did the agent have enough evidence to justify urgency?
- Was there a time window where the alert would have been more useful?
- Did the alert duplicate information the user already had?
- Should this have been batched, escalated, archived, or sent immediately?
- What exact prompt instruction would prevent this next time?
Prompt update template
When you see a similar event, do not interrupt unless the source is verified, the action window is open, and waiting creates real cost. If those conditions are not met, batch the item into the next digest and include the source for later review.
Where Supers fits
Supers workflows can span SMS, browser automation, and generated deliverables. Start by reviewing proactive texts, then apply the same false-positive taxonomy to computer-use cache workflows and progress updates from an AI agent that builds websites.
Sources
- Microsoft Research on interrupted work for why poor alerts carry cognitive cost.
- Nielsen Norman Group on notification design for alert clarity and user-control principles.
- Supers for personal AI agent workflows across SMS, browser work, and web tasks.
FAQ
How many examples are enough?
Five to ten reviewed alerts can reveal the first bad rules. High-cost misses should be reviewed immediately.
Should false negatives be in the same review?
Yes. Reviewing only noisy alerts can make the agent too quiet and miss important events.
What should change after review?
The system prompt, source gates, quiet-hour bypasses, or escalation thresholds should change in concrete language.
What is the goal?
Fewer unnecessary interruptions and fewer missed urgent events, with clear receipts for every proactive alert.
Turn every bad alert into a better personal agent.
Supers can help you test text, browser, and website-building agents with review loops that improve the next decision instead of merely hiding the last mistake.