Docs

Mixpanel Agent Intelligence: Tracking AI Agent Usage

Mixpanel Agent Intelligence ingests OpenTelemetry spans to track AI agent conversations, cost and outcomes. See what to instrument and measure.

Analytics

Many products now ship an AI assistant, yet few teams can say whether it helps. Traditional analytics shows that a user opened the assistant. It does not show how many turns the conversation took, what it cost, where it failed, or whether the user then completed the task they came for.

On 1 October 2026, Mixpanel released Agent Intelligence in beta. According to its changelog, it automatically aggregates conversations, turns, cost, latency, errors and tool calls, offers a conversation view for inspecting turns, and links agent conversations to user events for funnel and retention analysis. It accepts OpenTelemetry spans: point an existing exporter at Mixpanel, include a user ID, and each span arrives as an event. Ingested spans count towards normal event volume.

Without this link, agent quality is judged on anecdotes and cost is discovered on the invoice.

Data Points to Track

  • Conversation ID and user ID, so agent activity joins to the rest of the user’s behaviour
  • Turns per conversation and conversation duration
  • Cost per conversation, split by model where you use more than one
  • Latency: time to first token and total response time per turn
  • Error type and count per conversation, including timeouts and failed tool calls
  • Tool call name, status and duration
  • Outcome event: the product event that signals success, such as a booking made or a report exported
  • Span volume per day, because spans count towards your event allowance

Setup Steps

  1. Join the beta and confirm Agent Intelligence is enabled for your Mixpanel project.
  2. Instrument the agent with OpenTelemetry, creating spans for each conversation, turn, model call and tool call.
  3. Attach the user ID to every span, using the same identifier as your existing Mixpanel events.
  4. Point your OpenTelemetry exporter at Mixpanel and verify spans appear as events.
  5. Define outcome events and link them to conversations in the outcome analysis view.
  6. Estimate volume. Multiply spans per conversation by daily conversations and check the result against your plan.

Actionable Insights

Compare retention and conversion for users who had an agent conversation against those who did not, ideally with a holdout. If agent users do not convert better, the assistant is a cost, not a feature.

Look for conversations with many turns and no outcome event. These are usually users the agent failed, and the conversation view shows where it went wrong. Slow or failing tool calls are the next most common cause, and they are usually quick to fix.

Watch cost per successful outcome rather than cost per conversation. A cheaper model that resolves fewer tasks can be more expensive overall. Finally, sample span volume early, since verbose tracing can use up your event allowance faster than expected.

Expert help

Need help tracking this in your app?

Our team sets up analytics pipelines for mobile and web teams every day. Talk to us and get your first events flowing in under an hour.

Talk to an expert