The major product analytics platforms have moved past AI copilots that only answer typed questions. Amplitude’s AI Assistant, expanded this year, now surfaces context-aware suggestions proactively — flagging an anomaly, recommending a segment to investigate, or suggesting the next chart to build, without anyone asking first. That’s a different product from a query copilot, and it needs a different measurement approach. A copilot’s success is whether its answer to a specific question was accurate. A proactive assistant’s success is whether anyone actually did anything with what it surfaced — and that’s a much easier thing to ship without ever finding out.
The risk with proactive AI features specifically is that they’re easy to demo and easy to quietly ignore. A suggestion panel that appears on every dashboard load can look like adoption in a screenshot while actually training users to dismiss it on reflex, the same way banner-blindness developed for anything that looks like an ad. Without tracking what happens after a suggestion appears — not just that it appeared — a team can ship an AI assistant, watch impression counts climb, and never learn that nobody has acted on a single recommendation in months.
Data Points to Track
- Suggestion impression count, broken out by suggestion type (anomaly flag, segment recommendation, next-chart prompt) rather than one blended total
- Suggestion click-through or expand rate — whether a user engages with a surfaced suggestion at all before deciding to act or dismiss
- Follow-through rate — the share of suggestions that lead to a concrete action (a saved segment, a created chart, a shared insight) within the same session
- Dismissal rate over time per suggestion type, since a rising dismissal trend for one category is an early signal that its recommendations are missing the mark
- Time-to-action — how long after a suggestion appears the associated action happens, distinguishing an immediately useful nudge from one acted on much later, if at all
Setup Steps
- Instrument suggestion impressions as their own event type, tagged with suggestion category, so volume can be measured per type rather than as one aggregate feature.
- Track the specific action a user takes immediately after seeing a suggestion, linking the suggestion ID to any resulting saved view, chart, or segment.
- Log dismissals explicitly, distinguishing an active dismiss from a suggestion that simply expired unseen — the two mean very different things.
- Segment follow-through rate by user role or seniority, since proactive AI suggestions often land differently for an analyst who already knows what to check versus a PM who doesn’t.
- Review dismissal-heavy suggestion types on a regular cadence and treat sustained low follow-through as a signal to retune or retire that suggestion category, not just a metric to report.
Actionable Insights
A suggestion type with high impressions but a follow-through rate near zero is the clearest sign the feature needs retuning, not more visibility — pushing it harder in the UI without fixing relevance just accelerates the banner-blindness effect. A widening gap between impression count and dismissal rate over several weeks usually means the assistant’s suggestions have drifted out of step with what the team actually needs, often after a product change shifted which metrics matter. And strong follow-through concentrated in only one user segment — say, junior PMs but not analysts — tells you the assistant is filling a real gap for one audience rather than being universally useful, which should shape how it’s positioned rather than being read as a general success metric.
Related Resources
Need help tracking this in your app?
Our team sets up analytics pipelines for mobile and web teams every day. Talk to us and get your first events flowing in under an hour.
Talk to an expert