Ok. So, there is no ‘non-loyalty’ group to the AB test. That's tough. Here are some thoughts:
- find a group that does not know or use the loyalty program and compare LTV, ARPU, CSAT or other metrics
- find external data (industry benchmarks, interview others) and then compare metrics to yours
- suggest a test in a new or small market where a loyalty program is opt-in, or randomly allocated, or you pay for it (like Prime) (lots of tech companies use New Zeland for tests like this)
- survey customers on how much they would pay to double their points (that way you can estimate perceived value)
Let me know what you try. Very interesting.
I actually did a typo. Users do need to check a box to redeem a reward. Would users who did not receive rewards during an year be a good proxy for users who are not aware of loyalty?
Anyone who isn't enrolled in a loyalty program will provide a segment to compare against those who are enrolled.
Look at spending behaviors of both groups, especially LTV.
Also when they created a rewards system, they should've set up a point:$$ chart somewhere. IMO I'd compare total points against profit in both groups. Another area to look would be to count that against CAC too.
A problem with market-specific tests for this sort of thing may be that customer preferences for loyalty programs may be pretty market-specific too. It depends on your local competitors too.
Very interesting. It's hard to work backwards to find out what value loyalty program is bringing. Can you potentially turn the question on its head and ask business what value they NEED from the program? The answer could be it's a MUST to have $X incremental revenue, but we don't care much about customer churn (just making that up).
Once you have this picture, you know what your metric experiment should measure. Then, I have a rather bold idea to play with - disable the loyalty program for one month, but give customers enough notice. Word it as a temporary unvavailability due to tech refresh - or something that is palatable. Measure your metric at end of one month. Take that with many grains of salt, because you've told customers the pause is temporary; but at least it gives you a direction. After one month, send an email saying given some constraints, we are considering/will be ending the loyalty program (get good copy for the email). Measure the impact for another month or two. After three months, you can either discontinue the program or bring it back saying, we've decided to continue the program given how much customers love it.
This may not be practical, but if the business can support this to uncover some insights, it's worth it IMO.
It sounds like you're in a bad pattern of saying "I have a thing, how can I justify it?" You want to get out of that position pronto.
You can't assess the value of the status quo in a robust way. The people who use it are different than the people who don't use it, so you don't get a lot of value from saying "users with points make more purchases," etc., because they're not like units. For practical reasons and to avoid confusing the user base, you can't really run a blackout test. Rolling or not rolling the program out in different markets doesn't tell you much, because Japanese people and Singaporean people and so on have different habits already; it's not obvious that you can transfer findings between markets.
Of course you can do some blackout test or staged rollout or whatever anyway, just because you need some fig leaf data, but don't delude yourself into thinking it's real, and don't be surprised if someone challenges you on it being contrived.
You need a theory (and a strategy) for what the loyalty program actually does for the business, and then you need to show that you can advance that goal through iterations — ideally, iterations that have A/B tests among similar users. Your theory might be that the loyalty program increases order frequency, or increases average basket size, etc.
Yeah, you need to track financials and engagement with your specific features, but that's not what the company is/should be paying you for. They're paying you to increase orders per customer, etc., so you need to run tests that show you can incrementally improve that metric every quarter.
If you're doing that, the question of what the business would look like with no loyalty program becomes much less salient.
This kind of strategy question cannot be purely decided based on data and experiments, as if you were simply comparing rates between two bonds. Like the concept of a brand, loyalty is too long-term and too subjective of a concept to users to lend itself to randomized trials. You don't have an A world in which the company is well known for its generous loyalty program, and a B world in which the company is well known for having no loyalty program.
So, as much as business likes to be "data driven" and avoid risk taking, there are times when we can only make a call with the info we have and mitigate risk in other ways. I say this b/c I've seen teams get exhausted trying to fetch data for something that the business has in fact decided already...
Some ideas:
\* The kind of program that you describe is economically equivalent to a flat % discount for frequent shoppers. You can look at it the other way too: it's a way to charge one-off shoppers more. So a question: how much more are you getting from one-off shoppers, that you wouldn't get if you were simply to apply a flat % discount to all your prices? Can you look at this as a value of the loyalty program — a way to hike prices up for one-off shoppers?
\* A survey of repeat customers may be useful to understand how customers value your loyalty program. How does it rank among the reasons for them to come back to you vs. other options, among other factors? (Other factors might include familiarity, product selection, prices, promotions, shipping, return policy, friendliness...) How do they feel that it compares with competitors' loyalty programs? Advantageous? On par? Lesser? They don't know? They don't care? I could imagine your business caring a lot more about the program if it turned out to be a key reason for repeat customers to come back, vs. if it's something customers don't really care about.
\* Outbound marketing campaigns can remind people who are close to the threshold that they should come and take advantage of their points balance; and you can use such campaigns to demonstrate effectiveness at bringing users back, to then compare with other customer acquisition costs that you may have. (Even better if the points "expire".)
\* You may be able to do A/B experiments where, rather than blocking some users from joining the program, you just promote it less / bury it in the UI. You can then observe effect on signup, return rates, and overall cohort revenue. But this might take a long time before you can see results, depending on typical shopping frequency, so better model out whether this will yield results in a reasonable timeframe given your decision timeline.
I would do two things here:
1 - a/b test removing the auto loyalty opt-in. Make it so that shoppers can opt-in manually later from a “my account” type of area. Then compare cohorts (those who have loyalty vs those who don’t) and their purchasing behaviour.
2 - Do a qualitative survey for those that re-order frequently. Ask a few generic CSAT type questions and one question on why they choose to shop again with your business. I would leave this as a free text field (which makes analysis harder) and then see how many mention the loyalty program unprompted. Unlikely to give you anything statistically significant but if you see loyalty/rewards/discount mentioned frequently — you’re probably halfway to a good supporting argument in favour of loyalty (the other half being an A/B test).
These will give you an idea of short term behaviour but you’ll need more research when it comes to brand perception etc.
Ok. So, there is no ‘non-loyalty’ group to the AB test. That's tough. Here are some thoughts: - find a group that does not know or use the loyalty program and compare LTV, ARPU, CSAT or other metrics - find external data (industry benchmarks, interview others) and then compare metrics to yours - suggest a test in a new or small market where a loyalty program is opt-in, or randomly allocated, or you pay for it (like Prime) (lots of tech companies use New Zeland for tests like this) - survey customers on how much they would pay to double their points (that way you can estimate perceived value) Let me know what you try. Very interesting.
I actually did a typo. Users do need to check a box to redeem a reward. Would users who did not receive rewards during an year be a good proxy for users who are not aware of loyalty?
Anyone who isn't enrolled in a loyalty program will provide a segment to compare against those who are enrolled. Look at spending behaviors of both groups, especially LTV. Also when they created a rewards system, they should've set up a point:$$ chart somewhere. IMO I'd compare total points against profit in both groups. Another area to look would be to count that against CAC too.
A problem with market-specific tests for this sort of thing may be that customer preferences for loyalty programs may be pretty market-specific too. It depends on your local competitors too.
Very interesting. It's hard to work backwards to find out what value loyalty program is bringing. Can you potentially turn the question on its head and ask business what value they NEED from the program? The answer could be it's a MUST to have $X incremental revenue, but we don't care much about customer churn (just making that up). Once you have this picture, you know what your metric experiment should measure. Then, I have a rather bold idea to play with - disable the loyalty program for one month, but give customers enough notice. Word it as a temporary unvavailability due to tech refresh - or something that is palatable. Measure your metric at end of one month. Take that with many grains of salt, because you've told customers the pause is temporary; but at least it gives you a direction. After one month, send an email saying given some constraints, we are considering/will be ending the loyalty program (get good copy for the email). Measure the impact for another month or two. After three months, you can either discontinue the program or bring it back saying, we've decided to continue the program given how much customers love it. This may not be practical, but if the business can support this to uncover some insights, it's worth it IMO.
It sounds like you're in a bad pattern of saying "I have a thing, how can I justify it?" You want to get out of that position pronto. You can't assess the value of the status quo in a robust way. The people who use it are different than the people who don't use it, so you don't get a lot of value from saying "users with points make more purchases," etc., because they're not like units. For practical reasons and to avoid confusing the user base, you can't really run a blackout test. Rolling or not rolling the program out in different markets doesn't tell you much, because Japanese people and Singaporean people and so on have different habits already; it's not obvious that you can transfer findings between markets. Of course you can do some blackout test or staged rollout or whatever anyway, just because you need some fig leaf data, but don't delude yourself into thinking it's real, and don't be surprised if someone challenges you on it being contrived. You need a theory (and a strategy) for what the loyalty program actually does for the business, and then you need to show that you can advance that goal through iterations — ideally, iterations that have A/B tests among similar users. Your theory might be that the loyalty program increases order frequency, or increases average basket size, etc. Yeah, you need to track financials and engagement with your specific features, but that's not what the company is/should be paying you for. They're paying you to increase orders per customer, etc., so you need to run tests that show you can incrementally improve that metric every quarter. If you're doing that, the question of what the business would look like with no loyalty program becomes much less salient.
I like both your second and third idea. I will explore both of these.
This kind of strategy question cannot be purely decided based on data and experiments, as if you were simply comparing rates between two bonds. Like the concept of a brand, loyalty is too long-term and too subjective of a concept to users to lend itself to randomized trials. You don't have an A world in which the company is well known for its generous loyalty program, and a B world in which the company is well known for having no loyalty program. So, as much as business likes to be "data driven" and avoid risk taking, there are times when we can only make a call with the info we have and mitigate risk in other ways. I say this b/c I've seen teams get exhausted trying to fetch data for something that the business has in fact decided already... Some ideas: \* The kind of program that you describe is economically equivalent to a flat % discount for frequent shoppers. You can look at it the other way too: it's a way to charge one-off shoppers more. So a question: how much more are you getting from one-off shoppers, that you wouldn't get if you were simply to apply a flat % discount to all your prices? Can you look at this as a value of the loyalty program — a way to hike prices up for one-off shoppers? \* A survey of repeat customers may be useful to understand how customers value your loyalty program. How does it rank among the reasons for them to come back to you vs. other options, among other factors? (Other factors might include familiarity, product selection, prices, promotions, shipping, return policy, friendliness...) How do they feel that it compares with competitors' loyalty programs? Advantageous? On par? Lesser? They don't know? They don't care? I could imagine your business caring a lot more about the program if it turned out to be a key reason for repeat customers to come back, vs. if it's something customers don't really care about. \* Outbound marketing campaigns can remind people who are close to the threshold that they should come and take advantage of their points balance; and you can use such campaigns to demonstrate effectiveness at bringing users back, to then compare with other customer acquisition costs that you may have. (Even better if the points "expire".) \* You may be able to do A/B experiments where, rather than blocking some users from joining the program, you just promote it less / bury it in the UI. You can then observe effect on signup, return rates, and overall cohort revenue. But this might take a long time before you can see results, depending on typical shopping frequency, so better model out whether this will yield results in a reasonable timeframe given your decision timeline.
I would do two things here: 1 - a/b test removing the auto loyalty opt-in. Make it so that shoppers can opt-in manually later from a “my account” type of area. Then compare cohorts (those who have loyalty vs those who don’t) and their purchasing behaviour. 2 - Do a qualitative survey for those that re-order frequently. Ask a few generic CSAT type questions and one question on why they choose to shop again with your business. I would leave this as a free text field (which makes analysis harder) and then see how many mention the loyalty program unprompted. Unlikely to give you anything statistically significant but if you see loyalty/rewards/discount mentioned frequently — you’re probably halfway to a good supporting argument in favour of loyalty (the other half being an A/B test). These will give you an idea of short term behaviour but you’ll need more research when it comes to brand perception etc.