Testing a churn theory before acting on it

The challenge

The company sells its software by subscription. A large customer cancelled and named an error message as the reason. Its leaders came to believe that errors were driving customers away and began planning a response: fixing the error, or giving free credits to customers who hit it. The link between errors and cancellations had not been tested.

The approach

The work moved from question to evidence in a matter of days. Nockpoint accelerated the analysis by bringing the company's product, subscription and customer data together, applying established recurring revenue logic, and providing a foundation for analyzing usage and churn without building the data environment from scratch.

Testing the error hypothesis

1. Tracked error messages. Verity implemented RudderStack to capture each error message shown to customers in the product.
2. Connected usage to paying accounts. Nockpoint joined those events with paying accounts from Stripe and customer segments from HubSpot.
3. Established a reliable churn measure. Stripe's subscription history and Nockpoint's recurring revenue configuration identified every paying account and every cancellation the same way.
4. Compared like with like. Accounts were separated into single-user and multi-user groups. Within each group, Verity compared churn among customers who saw an error with churn among customers who never did.

Looking beyond the original question

5. Expanded product telemetry. Verity extended RudderStack beyond errors to capture nearly every type of product event.
6. Analyzed usage and retention. With that activity flowing into the same Nockpoint data, Verity ranked customers into usage quartiles, four equal groups from least to most active, and compared their monthly churn.

The results

• The theory did not hold: among single-user accounts, customers who saw an error had a monthly churn rate of 17% across the four months, against 31% for customers who never did.
• Heavy users stayed: in one cohort of new customers, the most active usage quartile churned at 7% a month, against 48% for the least active quartile. Reaching an error takes use, which is the likeliest reading of the first finding.
• A decision on evidence: the company could weigh its response to errors on the behavior of every customer, beyond one cancellation.