Lester Leong
Why Traditional Product Metrics Break for GenAI Features
The Measurement Problem Nobody Talks About
Product teams are shipping GenAI features faster than they are learning to measure them. The default playbook is to apply the same engagement and retention metrics that work for deterministic software: DAU, session length, feature usage counts, weekly retention curves. These metrics are well understood, every analytics platform supports them, and they have decades of validation behind them.
They also systematically mislead you when applied to AI features.
I have seen this firsthand. Working on a GenAI squad at a major finance technology company, our team shipped an AI-powered feature and initially measured it with standard engagement metrics. The numbers looked strong. Usage was climbing. Session counts were healthy. Then we dug into the qualitative data and realized a significant share of sessions ended with users rejecting or ignoring the AI output entirely. By every traditional metric, these were "engaged" users. In reality, they were frustrated users who kept trying because they wanted the feature to work, not because it was working.
This pattern is not unique to one product. Across 20+ consulting engagements with SMBs and startups through Gradient Growth, I have watched teams celebrate engagement numbers that masked a fundamental quality problem. The issue is structural, not situational. Traditional metrics were designed for products where the output is deterministic. AI features are probabilistic. That difference changes everything about how measurement should work.
Why Engagement Metrics Mislead for AI
When a user clicks "Generate Report" in a traditional product, they get a report. Every time. The output is predictable. Engagement with the feature is a reasonable proxy for value received.
When a user prompts an AI feature, the output quality is variable. The same user with the same intent might get an excellent result on Monday and a mediocre one on Tuesday. The quality depends on prompt specificity, input data, model capabilities, and context that the product team cannot fully control.
This variability breaks the fundamental assumption behind engagement metrics: that interaction equals value. In deterministic products, more usage means more value delivered. In probabilistic products, more usage might mean more value delivered, or it might mean the user is stuck in a retry loop trying to get an acceptable result.
High engagement on an AI feature can signal two completely opposite things. It can mean the user loves the feature and keeps finding new uses for it. Or it can mean the output keeps failing and the user keeps retrying. Traditional engagement metrics cannot distinguish between these scenarios. A user who generates five drafts because each one is useful for a different purpose looks identical to a user who generates five drafts because the first four were unusable.
The same problem infects session duration. Longer sessions on a traditional feature usually indicate deeper engagement. Longer sessions on an AI feature might indicate the user is spending time editing, correcting, or regenerating poor output. Time spent is not time valued.
Why Retention Curves Behave Differently
Standard retention analysis assumes a relatively stable experience across sessions. If a user has a good first session, their second session is likely to be comparably good. This assumption holds for deterministic products and drives the classic retention curve shape: sharp initial drop, then a flattening that represents the stable user base.
AI features violate this assumption. A user can have an excellent first experience (the model nailed their use case), return with high expectations, receive a poor result on the second attempt, and churn immediately. The reverse also happens: a user has a bad first experience, almost churns, but gives it one more try and gets a great result that hooks them.
The practical effect is that AI feature retention curves are noisier and take longer to stabilize. The early-week retention numbers are less predictive of long-term behavior than they would be for traditional features, because output quality variance introduces randomness into the experience that washes out over time but dominates in the short term.
I have also observed that AI feature retention curves often show a "second wave" pattern. There is an initial drop (normal), a further decay through weeks 2 to 4, and then a slight uptick in weeks 5 to 8 as users who churned early give the feature another chance after hearing about improvements or seeing colleagues succeed with it. Traditional retention analysis would miss this entirely because it focuses on monotonic decay.
The Four Metrics That Actually Work
After building measurement systems for AI features across multiple products, I have converged on four metrics that capture what traditional metrics miss. These are not replacements for engagement and retention. They are the additional layer required to interpret engagement and retention correctly for probabilistic products.
1. Task Completion Rate
The percentage of AI interactions where the user accomplished what they set out to do. This requires defining what "completion" means for your specific feature: did the user accept the generated output, export it, share it, or take the next logical action? Task completion rate is the single most important metric for AI features because it directly measures value delivery rather than activity.
At a consulting client running an AI content generation tool, overall feature engagement was high (averaging 12 sessions per active user per month). Task completion rate was 34%. Two out of three sessions produced output the user discarded. The engagement number was masking a severe quality problem.
2. Output Acceptance Rate
The percentage of AI-generated outputs that users accept without significant modification. This is more granular than task completion because it measures the quality of individual outputs rather than overall session outcomes. Track three tiers: accepted as-is, accepted with minor edits, and rejected or regenerated.
The ratio between these tiers tells you where your quality problem lives. A high "accepted with edits" rate suggests the model is directionally correct but needs refinement. A high rejection rate suggests the model is fundamentally misaligned with user expectations for certain query types.
3. Time-to-Value Variance
For traditional features, [time to value](/insights/time-to-value-onboarding-metric) is relatively consistent across users. For AI features, the variance in time to value is itself a critical metric. If some users reach value in 30 seconds and others take 10 minutes for the same task, you have a quality consistency problem that averages and medians will hide.
Measure the standard deviation and interquartile range of time-to-value, not just the median. A shrinking variance over time (through model improvements, better prompts, or UX guardrails) is a stronger signal of product improvement than a shrinking median.
4. Retry Rate
The percentage of interactions where the user regenerates, reprompts, or starts over. Retry rate is the inverse signal of output quality. In a deterministic product, retries indicate a bug. In an AI product, retries are a feature (users can iterate), but the rate at which they occur is a direct quality indicator.
Segment retry rate by query type, user segment, and time period. If retry rates are uniformly high, the model needs improvement across the board. If retry rates are high for specific query types, you can target those for fine-tuning or add UX guidance to set better expectations.
Building the Measurement System
Here is a Python implementation that computes all four metrics from a standard AI interaction event log. This assumes each interaction has a session ID, a user ID, timestamps, and outcome signals (accepted, edited, rejected, retried).
```python import pandas as pd import numpy as np
def compute_ai_quality_metrics( interactions: pd.DataFrame, session_col: str = "session_id", user_col: str = "user_id", outcome_col: str = "outcome", duration_col: str = "duration_seconds", retry_col: str = "is_retry", ) -> dict: """ Compute the four core quality metrics for an AI feature.
Parameters ---------- interactions : DataFrame where each row is one AI interaction. Required columns: session_id, user_id, outcome, duration_seconds, is_retry. outcome values: 'accepted', 'edited', 'rejected', 'abandoned'. is_retry: boolean indicating whether this interaction was a retry of a previous attempt within the same session.
Returns ------- dict with task_completion_rate, output_acceptance_breakdown, time_to_value_stats, and retry_rate. """ df = interactions.copy() total = len(df)
if total == 0: return {}
# 1. Task completion rate: sessions with at least one accepted # or edited outcome (user got usable output) session_outcomes = df.groupby(session_col)[outcome_col].apply(set) completed = session_outcomes.apply( lambda outcomes: bool(outcomes & {"accepted", "edited"}) ) task_completion_rate = completed.mean() * 100
# 2. Output acceptance breakdown outcome_counts = df[outcome_col].value_counts(normalize=True) * 100 acceptance_breakdown = { "accepted_as_is": round(outcome_counts.get("accepted", 0), 1), "accepted_with_edits": round(outcome_counts.get("edited", 0), 1), "rejected": round(outcome_counts.get("rejected", 0), 1), "abandoned": round(outcome_counts.get("abandoned", 0), 1), }
# 3. Time-to-value variance (for completed sessions only) completed_sessions = completed[completed].index completed_durations = ( df[df[session_col].isin(completed_sessions)] .groupby(session_col)[duration_col] .sum() ) ttv_stats = { "median_seconds": round(completed_durations.median(), 1), "p25_seconds": round(completed_durations.quantile(0.25), 1), "p75_seconds": round(completed_durations.quantile(0.75), 1), "std_seconds": round(completed_durations.std(), 1), "iqr_seconds": round( completed_durations.quantile(0.75) - completed_durations.quantile(0.25), 1, ), }
# 4. Retry rate retry_rate = df[retry_col].mean() * 100
return { "task_completion_rate": round(task_completion_rate, 1), "output_acceptance": acceptance_breakdown, "time_to_value": ttv_stats, "retry_rate": round(retry_rate, 1), "total_interactions": total, "unique_sessions": df[session_col].nunique(), "unique_users": df[user_col].nunique(), } ```
Run this weekly alongside your standard [engagement scoring](/insights/product-engagement-score). The combination of traditional engagement metrics and AI-specific quality metrics gives you the full picture: how much users interact with the feature (engagement) and how much value those interactions actually deliver (quality).
Interpreting the Numbers Together
The power of this framework is in the intersections. Here are the diagnostic patterns I have seen repeatedly:
High engagement, low task completion. Users want the feature to work but it is not delivering. This is the most dangerous pattern because traditional metrics look healthy. Prioritize model quality improvements.
Low engagement, high task completion. The feature works well but users do not know about it or do not understand when to use it. This is a [feature adoption](/insights/feature-adoption-rate-guide) problem, not a quality problem. Invest in discovery and education.
High retry rate, stable acceptance rate. Users are learning to iterate with the AI. This is not inherently bad, but monitor whether the retry rate declines over time as users develop better prompting habits. If it stays flat, the UX is not teaching users how to get good results.
Shrinking time-to-value variance. This is the strongest positive signal. It means the feature is becoming more consistently valuable across users and use cases. When the IQR tightens over successive weeks, your model and UX improvements are working.
What to Do Next
The teams that measure AI features well share one trait: they accepted early that traditional metrics are necessary but insufficient. They instrument both the standard engagement layer and the quality layer, and they make product decisions based on the intersection.
Start with task completion rate. It is the single metric that most directly answers whether your AI feature is delivering value. If you measure nothing else from this framework, measure that.
---
I help teams build measurement systems for AI features that capture what traditional metrics miss. [lester@gradientgrowth.com](mailto:lester@gradientgrowth.com)