Frustration Telemetry
- Entity ID:
ent-20260409-78224e9c51ca - Type:
service - Scope:
shared - Status:
active - Aliases: frustration regex, frustration detection
Description
Frustration Telemetry is a client-side analytics system that detects expressions of user frustration and profanity in conversation messages using regex pattern matching. When a match is detected, the system fires a telemetry event to Anthropic's analytics backend. The primary purpose is to serve as a leading indicator of model regression -- a spike in frustration events across the user population suggests the model's quality has degraded before formal benchmarks or user reports surface the issue.
Detection Mechanism
The system applies a set of regular expressions to each user message before it is sent to the API. These patterns target:
- Profanity -- Common English profanity and abbreviations thereof.
- Frustration expressions -- Phrases like "this is broken," "why can't you," "you keep," "stop doing," "that's wrong again," and similar patterns indicating user dissatisfaction.
- Repeated corrections -- Detection of the user re-stating the same instruction, which suggests the model failed to follow it.
The regex approach was chosen over LLM-based sentiment analysis for two reasons: it adds zero latency (no additional API call), and it works offline without consuming tokens. The tradeoff is lower accuracy -- sarcasm, code snippets containing profanity, and non-English frustration are not reliably detected.
Telemetry Event Structure
When frustration is detected, the system emits a structured telemetry event containing:
| Field | Content |
|---|---|
event_type |
user_frustration_detected |
pattern_matched |
Which regex category triggered (profanity, correction, expression) |
session_id |
The current session identifier |
turn_number |
How far into the conversation the frustration occurred |
model_id |
Which model version is active |
timestamp |
Event time |
Notably, the user's actual message content is not included in the telemetry event -- only the metadata about the match. This is a deliberate privacy choice.
Use as Regression Indicator
Anthropic's internal dashboards aggregate frustration events across all Claude Code sessions. A statistically significant increase in frustration rate, especially when correlated with a model version change or a growthbook flag flip, triggers investigation. This system has been described internally as one of the fastest signals for detecting model quality issues in production, since it captures real user reactions in real time rather than waiting for benchmark results or support tickets.
Privacy Considerations
The system operates entirely client-side within the Claude Code CLI. The regex matching happens before any data leaves the user's machine for telemetry purposes. The telemetry event itself contains only categorical metadata, not message content. Users who disable telemetry (--no-telemetry flag) also disable frustration detection events.
Integration
Frustration Telemetry runs within queryengine-ts as part of the message processing pipeline. It operates alongside the message-normalization system but fires before normalization transforms are applied. The detected events feed into the same analytics pipeline used by growthbook for feature flag decisions.
Key claims
- none yet
Relations
- none yet
Sources
src-20260409-4837ecedf1ce