Docs

AI Session Summary Tracking

Mixpanel and Amplitude now auto-summarise session replays with AI. Bad event data makes a bad summary. What to instrument first.

Analytics

Session replay tools have started generating their own summaries: Mixpanel’s AI Summary reads a recorded session and describes what happened without anyone watching the replay, while Amplitude’s Session Replay Agent runs friction detection and navigation-flow analysis across many sessions at once and surfaces the patterns as structured findings. The pitch is obvious — nobody has time to watch hundreds of replays, and an AI summary turns hours of footage into a few readable sentences per session.

The quality of that summary is entirely downstream of the event data it’s built from. Both tools generate summaries from the governed event stream alongside the raw replay, not from pixels alone — which means a session with vague event names, missing properties, or gaps where nothing fired at all produces a summary that’s just as vague, or flatly wrong about what the user was trying to do. Teams that adopted session replay when it just meant “watch the video” often haven’t gone back to tighten up their event taxonomy for a world where an AI is now reading that taxonomy and writing conclusions from it — and a confidently wrong summary is more dangerous than an honest gap, because it gets trusted without anyone re-checking the replay.

Data Points to Track

  • Event name and property completeness per session: the share of a session’s actions that fire a named event with populated properties, versus generic clicks or taps that log no semantic meaning
  • Rage-tap and dead-click flags: explicit events for repeated taps on unresponsive elements, since these are exactly the friction signals summarisation tools weight heavily and will describe inaccurately if they’re inferred from timing alone rather than logged directly
  • Funnel-step tagging within the session: which stage of a known flow (onboarding, checkout, setup) each event belongs to, so a summary can correctly identify where in the journey friction occurred
  • Error and exception events tied to the session: client-side errors and failed requests logged with enough context to explain a summary’s “user encountered an error” line rather than leaving it as an unexplained guess
  • Summary-to-replay disagreement rate: a manual QA sample where a human checks a subset of AI summaries against the actual replay, tracked over time as its own metric

Setup Steps

  1. Audit the event taxonomy feeding session replay before turning on AI summarisation, specifically for generic or auto-captured events (unnamed clicks, autocaptured taps) that carry no semantic label an AI can reason about.
  2. Add explicit rage-tap, dead-click and error events where they’re currently absent, since these are the highest-value signals for friction-focused summaries and are rarely autocaptured with enough detail to summarise well.
  3. Tag events with the funnel or flow step they belong to as a property, not just an event name, so a summarisation model has the structural context to say where in a journey something went wrong.
  4. Turn on AI summaries for a limited cohort first and run a manual spot-check against the underlying replays before rolling out broadly, treating the first batch as a calibration pass on your own event data rather than a finished feature.
  5. Track the summary-to-replay disagreement rate on an ongoing basis, not just at launch, since taxonomy drift — new features shipping without matching event coverage — degrades summary quality quietly over time.

Actionable Insights

A high disagreement rate concentrated in one part of the product almost always traces back to thin event coverage there, not a limitation of the summarisation model itself — it’s a prompt to instrument that flow properly rather than to distrust AI summaries generally. Once summaries are reliable, the aggregate view across many sessions — which friction points recur, which drop-off points show up repeatedly — becomes more useful than any individual summary, since it’s the pattern across sessions that should drive a fix, not one dramatic-sounding replay.

Treat a consistently vague or generic summary as a data quality alert in its own right: if the AI can’t describe what happened in a session, that’s a legible signal that the event stream for that part of the product needs attention, surfaced automatically instead of waiting for someone to notice the summaries there are never useful.

Expert help

Need help tracking this in your app?

Our team sets up analytics pipelines for mobile and web teams every day. Talk to us and get your first events flowing in under an hour.

Talk to an expert