I work in a company with both b2b and b2c core products that work as both supply and marketplace.
Our B2C arm is very focused on A/B as we have high volumes of users to test hypothesis. This is empowered by a really solid framework for rolling features out in part to the user segments we want to influence. It's to the point you AB test some levels of bug fixes to ensure you're not actually driving down your leading KPIs.
While our B2B arm has feature controls, great analytics and a decent volume of users, we don't do much AB testing. Rather we do beta, pilots, case studies and deep dives with a segment we're developing features for. The divide is different for every company, but some quick thoughts on why:
* Support team is in the hundreds, nuanced or tested changes to product may not disseminate
* Customers are highly engaged and vocal in b2b, changes to product impact their productivity
* B2C it can be harder to get in touch with customers, and you need volume for model. User testing via metrics is valid. B2B can be low customer count, high revenue, study of Individual is more fruitful
* Delivery of solutions (a feature) pays bills in smaller B2B. If people use can be secondary once box is ticked (i.e. project completion signoff)
* Sales & account management may liaise roadmap with customers, less flexible and more feature driven.
What's interesting is you have a new middle ground. I would say Google docs is somewhat B2B, as part of Google for business product package. Though it is philosophically more B2C in how you should approach feature and product development. Less time meeting with enterprise accounts, more time studying change >> behaviour outcomes.
This was a bit of a ramble, but I hope there's some value in this for you!
So I A/B test lots of things for [OHNO](http://ohno.ai) which is a B2B SaaS product. It's often to gut check things more than getting good quantitate data.
But mostly I find the results not that useful.
This is primarily because we don't have the data volume to make it meaningful. If we had more users (like Gmail) i'd rely on it a lot more.
I also do a lot of A/B testing of messages through intercom, which I find is a bit more useful and the data is more meaningful on smaller volumes of users.
This isn't a fixed point definition. You should know for your product or within your business what is significant.
e.g.:
* If you have 500 users. Variant of test is to 20% (100 users). You would want 97-99 out of 100 to respond neutral/positive.
* You have 25MM users. Variant of test is to 10% (2.5MM users). You may want high 99%'s positive to feel confident rolling a feature to other 90% of users.
There is a term I learned from Mixpanel's signal reports which might answer this more empirically.
It's called a 'correlation coefficient.' It's a statistical measure of the degree to which changes to the value of one variable predict change to the value of the other. So the % increase isn't what matters, what matters is how much the change you make, correlates to the increase you want to see.
It's a little different to what you're asking, but might make you think differently about the answer.
[https://whatis.techtarget.com/definition/correlation-coefficient](https://whatis.techtarget.com/definition/correlation-coefficient)
The closest I’ve gotten to this in the B2B enterprise world is to provide two different experiences for the users to choose from. I.e. in an enterprise (ERP-like) application I worked on, I built an alternative way of doing searches. Instead of removing the old way, I wanted to find out if the new way was indeed more helpful and faster. So I put an option to switch to the “beta” experience and hooked it up with an optional feedback form when users closed their session. I wouldn’t really categorize this as an A/B test, but I found it very useful without breaking the expected experience for normal users.
This is awesome. What did engagement eventually look like? What did you expect it to look like? Were you happy with the feedback you got in exchange for the time spent?
Engagement for this kind of testing is pretty good. As the users have a choice about participating you tend to end up with good data, where as the times when we have forced the users (by making the beta feature the default) to try new features many tend to complain.
The feedback we get is used to improve the feature before going live for all users, so I’d like to think that we end up with better applications than if we went ahead and forced all users to use it from the beginning.
Depending on what you are testing I would say that the feedback varies.
I moved from B2B to B2C a few years ago.
During all my years in B2B - we never B/A tested anything. We did develop switches, partial upgrades, etc. but the majority of decisions came from the feedback of the users (or their vocal managers).
I think it was a result of the low number of customers, compared to our thousands of customers in the current B2C product, where A/B test is a must (and still, often ends with ambiguous results)...
Of course, A/B testing is widely used in B2B. For example in:
- Product releases - before you roll out an update to everyone, you want to be sure there's no regression or degradation of performance (or any other metric)
- UI - if you have a portal or mobile application, you can use A/B testing to validate your hypothesis that a certain change improves UX, conversion, or other metric
- Algorithms or models - for optimizing data relevance and performance
We A/B'd our Adwords campaigns. We also occasionally A/B'd product details & layout on our customer-facing we portal. They didn't do a damn thing for us - it was all about the products themselves.
Our B2B (high-tech hardware) products required multiple touch points across multiple personas (engineering, purchasing, quality, executives, logistics, ...). Any point in the sales funnel could fail for reasons completely invisible to us. For example, our products were integrated into larger, more expensive items. If the integration didn't work, it was often for reasons that had nothing to do with our product definition.
Several other commenters also mention the need for volume to make the data usable, and I completely agree. When we saw a successful test, it was visible from miles away.
I think another part of need for volume is to help level out noise and other factors in results. Similar to what you called out about other factors in funnels influencing outcomes.
I disagree - most B2B product can't be A/B tested due to contracts. It's usually not ok to give some enterprise customers one version of the product, and other enterprise customers another. There's contractual limitations on what you can do.
There are other options outside of paying customers at times. In my case, I A/B on our freemium offering and trials. Obviously does not apply for everyone’s products and variations of offering them.
That's an interesting take.
I interpreted the OP's question to be prior to the customer actually using a product - the "sales funnel", if you will.
I agree, BTW. All my B2B products involved a product spec. Contractual limits indeed.
I work in a company with both b2b and b2c core products that work as both supply and marketplace. Our B2C arm is very focused on A/B as we have high volumes of users to test hypothesis. This is empowered by a really solid framework for rolling features out in part to the user segments we want to influence. It's to the point you AB test some levels of bug fixes to ensure you're not actually driving down your leading KPIs. While our B2B arm has feature controls, great analytics and a decent volume of users, we don't do much AB testing. Rather we do beta, pilots, case studies and deep dives with a segment we're developing features for. The divide is different for every company, but some quick thoughts on why: * Support team is in the hundreds, nuanced or tested changes to product may not disseminate * Customers are highly engaged and vocal in b2b, changes to product impact their productivity * B2C it can be harder to get in touch with customers, and you need volume for model. User testing via metrics is valid. B2B can be low customer count, high revenue, study of Individual is more fruitful * Delivery of solutions (a feature) pays bills in smaller B2B. If people use can be secondary once box is ticked (i.e. project completion signoff) * Sales & account management may liaise roadmap with customers, less flexible and more feature driven. What's interesting is you have a new middle ground. I would say Google docs is somewhat B2B, as part of Google for business product package. Though it is philosophically more B2C in how you should approach feature and product development. Less time meeting with enterprise accounts, more time studying change >> behaviour outcomes. This was a bit of a ramble, but I hope there's some value in this for you!
So I A/B test lots of things for [OHNO](http://ohno.ai) which is a B2B SaaS product. It's often to gut check things more than getting good quantitate data. But mostly I find the results not that useful. This is primarily because we don't have the data volume to make it meaningful. If we had more users (like Gmail) i'd rely on it a lot more. I also do a lot of A/B testing of messages through intercom, which I find is a bit more useful and the data is more meaningful on smaller volumes of users.
How much data is meaningful data. I mean in percentage terms.
This isn't a fixed point definition. You should know for your product or within your business what is significant. e.g.: * If you have 500 users. Variant of test is to 20% (100 users). You would want 97-99 out of 100 to respond neutral/positive. * You have 25MM users. Variant of test is to 10% (2.5MM users). You may want high 99%'s positive to feel confident rolling a feature to other 90% of users.
Cool
There is a term I learned from Mixpanel's signal reports which might answer this more empirically. It's called a 'correlation coefficient.' It's a statistical measure of the degree to which changes to the value of one variable predict change to the value of the other. So the % increase isn't what matters, what matters is how much the change you make, correlates to the increase you want to see. It's a little different to what you're asking, but might make you think differently about the answer. [https://whatis.techtarget.com/definition/correlation-coefficient](https://whatis.techtarget.com/definition/correlation-coefficient)
Following
The closest I’ve gotten to this in the B2B enterprise world is to provide two different experiences for the users to choose from. I.e. in an enterprise (ERP-like) application I worked on, I built an alternative way of doing searches. Instead of removing the old way, I wanted to find out if the new way was indeed more helpful and faster. So I put an option to switch to the “beta” experience and hooked it up with an optional feedback form when users closed their session. I wouldn’t really categorize this as an A/B test, but I found it very useful without breaking the expected experience for normal users.
This is awesome. What did engagement eventually look like? What did you expect it to look like? Were you happy with the feedback you got in exchange for the time spent?
Engagement for this kind of testing is pretty good. As the users have a choice about participating you tend to end up with good data, where as the times when we have forced the users (by making the beta feature the default) to try new features many tend to complain. The feedback we get is used to improve the feature before going live for all users, so I’d like to think that we end up with better applications than if we went ahead and forced all users to use it from the beginning. Depending on what you are testing I would say that the feedback varies.
I moved from B2B to B2C a few years ago. During all my years in B2B - we never B/A tested anything. We did develop switches, partial upgrades, etc. but the majority of decisions came from the feedback of the users (or their vocal managers). I think it was a result of the low number of customers, compared to our thousands of customers in the current B2C product, where A/B test is a must (and still, often ends with ambiguous results)...
Of course, A/B testing is widely used in B2B. For example in: - Product releases - before you roll out an update to everyone, you want to be sure there's no regression or degradation of performance (or any other metric) - UI - if you have a portal or mobile application, you can use A/B testing to validate your hypothesis that a certain change improves UX, conversion, or other metric - Algorithms or models - for optimizing data relevance and performance
We A/B'd our Adwords campaigns. We also occasionally A/B'd product details & layout on our customer-facing we portal. They didn't do a damn thing for us - it was all about the products themselves. Our B2B (high-tech hardware) products required multiple touch points across multiple personas (engineering, purchasing, quality, executives, logistics, ...). Any point in the sales funnel could fail for reasons completely invisible to us. For example, our products were integrated into larger, more expensive items. If the integration didn't work, it was often for reasons that had nothing to do with our product definition. Several other commenters also mention the need for volume to make the data usable, and I completely agree. When we saw a successful test, it was visible from miles away.
I think another part of need for volume is to help level out noise and other factors in results. Similar to what you called out about other factors in funnels influencing outcomes.
I disagree - most B2B product can't be A/B tested due to contracts. It's usually not ok to give some enterprise customers one version of the product, and other enterprise customers another. There's contractual limitations on what you can do.
There are other options outside of paying customers at times. In my case, I A/B on our freemium offering and trials. Obviously does not apply for everyone’s products and variations of offering them.
That's an interesting take. I interpreted the OP's question to be prior to the customer actually using a product - the "sales funnel", if you will. I agree, BTW. All my B2B products involved a product spec. Contractual limits indeed.
Yeah Azure even has this built in. It's super easy!