ProdUX Lens's product auditing guide
- Catherine Tang
- Mar 7, 2024
- 6 min read
Updated: Mar 28, 2024
The get-go guide for product analytics and user research. No fuss, nor overcomplicated procedures.
In this guide you will learn about the following:
Now let's dive in.
What is Product Auditing and why should you care?
Product Auditing is a set of practices that help you to get to the bottom of what your customers DO with your product, so that
You truly understand and measure feature success through the lens of customer's (end-user) behaviors
You can uncover product experience gaps and find new product or business opportunities
It is done by analyzing feature usage data and paired with user feedback or qualitative research to fill in the context of why the usage data looks like what it is.
A quick example
If it still sounds too abstract, let’s see an example.
The first part of Product Auditing gives us a feature map showing how they are performing in terms of Retention and Popularity.
A graph like this can trigger a lot of questions and hypotheses on why certain features are performing this way.
In the second part of Product Auditing, we will go about validating or rejecting these hypotheses.
Together with qualitative data like customer feedback or user research results, the team will be able to uncover a lot of actionable insights such as
Feature 1 is powerful and should be promoted to a larger customer base…
Feature 11 is already "good enough" so we don't have to worry about them for now.
Interested in reading a real-life example? Read this case study we've previously shared.
How would Product Auditing benefit different teams in the company?
For Product team (PM, Designer, Engineer) - Insights for existing feature performance and new initiative prioritization
For GTM or Marketing team - Clarity for targeted value proposition and messaging
For Customer Success team - Direction for tailored onboarding and proactive support for churn reduction
For Revenue teams - Signals for appropriate Pricing and Packaging
Is Product Auditing the right fit for you?
Product Auditing is NOT for everyone
To have an effective Product Auditing, the following criteria have to be met:
The product has set up user behavioral data tracking for the features to be analyzed.
For a given feature, it has 100+ users using it every day and the usage is stable.
There is direct access to customers who use those features. Either to collect feedback or conduct interviews with them.
Signals that Product Auditing will bring immediate value to you
If you have launched the product for a while and you have been adding new features over time to acquire new customers, but…
You don't have a clear idea of which features bring the most value to existing customers, and which features are (almost) never used.
You noticed that your product stickiness (DAU/MAU) or retention rate (which are strong indicators for customer churn) is much lower than the industry standard and you don't know where to start to improve it.
You are wondering if your tiered pricing packages are aligned with your customers' natural usage patterns so that it's easier for them to make the decision and upgrade at the right point. Any of the above is a strong signal that Product Auditing could add value to your business instantly.
How to get started with Product Auditing?
Step 0: Review your data setup and confirm all events for feature usage are accurately tracked
Before doing any analysis, first, we have to make sure the user behavior data are accurately tracked.
Data are powerful, but inaccurate data are dangerous. This is such a simple point but yet too commonly missed.
We have seen data accuracy issues in almost every product that we have worked on. Sometimes it’s just simply caused by outdated event definitions or bugs occurred during iterations. It never harms to double-check before starting any analysis.
Step 1: Tag events to features
The first step is to define which customer behaviors (tracked as events) belong to a feature. So that if at least one of the events tagged in the feature is triggered, the feature is considered used.
If your team uses product analytics tools like Pendo, the feature tagging page looks like this.
Source: Pendo
Step 2: Calculate feature adoption rate and feature retention rate
Feature adoption rate
As mentioned above, we calculate the feature usage based on customer behavioral events that belong to the feature.
With this data in hand, the first analysis we can do is to understand how many % of the total active users have used this feature, indicating adoption or popularity.
Adoption Rate = # [Users Used Feature A] / # [Total Active Users]
An example could look like this.
Feature retention rate
However, using the feature at least once doesn't mean the feature is successfully delivering value to the customer. That's why it's also helpful to know whether the customers are repeatedly using this feature. Feature retention rate is a useful metric to indicate this.
📘 Note: There are many different ways of calculating retention rates. Here we show you a good go-to calculation - 7-day range retention.
Retention Rate (7-Day Range Retention) = # [Users Used Feature A At Least Once in Day 1-7] / # [Users Used Feature A in Day 0]
An example could look like this.
Step 3: Develop a clustered feature map with retention rates and usage rates
Feature adoption rate represents the breadth of customer value delivery, and feature retention rate represents the depth of customer value delivery. Combining those two metrics will bring an insightful overview of how the features are performing.
Step 4: Form hypotheses out of the data above
In the example above, some of the hypotheses could be:
Hypothesis 1: Feature 1 is such a powerful use case, and it is part of our core value proposition. There are only 20% of users using it because this feature is a bit hidden in the product, and not all the customers know about it.
Hypothesis 2: The retention rate seems a bit too low for a daily use case like Feature 5. The customers are not returning to it, maybe the usability of this feature has some issues.
Step 5: Validate or reject the hypotheses with user feedback or user research
For the two example hypotheses above, possible ways to validate or reject them are:
Hypothesis 1: we can look at user feedback data or simply pop a quick in-app survey to understand whether our customers know about this feature.
Hypothesis 2: we can run a usability test with 5 participants and see if usability is an issue.
With qualitative data in hand, the numbers are put into the right context for interpretation. As a result, the team will have more confidence on the next steps to take in improving the product.
Step 6: Translate the insights into actions and continuously monitor the feature status
Any adjustment to the acquisition, onboarding, or product development could change the usage and retention of the features. So it should be a standard action to take after every feature release or service change.
Why is Product Auditing hard to do right?
Data volume, stability, and accuracy are trickier than it seems.
Everyone wants to be data-driven nowadays. However, the law of statistics sets requirements for the data volume and stability, which is not easy to get for many low-volume business or early stage products. Also, just like no software can guarantee there’s no bug, data accuracy is a universal challenge for every product.
Behavioral data only tells us what, but it doesn't tell us why.
Making sense of the data requires a leap of faith to form hypotheses. Success is dependent on deep empathy with the customers and knowledge of the industry. Qualitative research helps to fill in the context and transfer the abstract numbers into relatable stories.
It is a continuous effort.
The customer and the market are ever-changing. We can always find out new learnings that prove us wrong. It is hard work, and it requires discipline to continuously do this practice, and constantly update our beliefs.
Summary
Product Auditing is extremely helpful in bringing actionable insights to help the business
Speak to customers in specific use cases
Sell to customers that fit their usage patterns naturally
Give customers the great product experience that delivers true value
There is a framework to do it step by step, but each step should be customized to fit the product and business type
It requires commitment and expertise to do it right, and it is a continuous effort to keep benefiting from it
Komentarze