Develop a Hypothesis Before a Solution

Bart has been in ecommerce for over 20 years, and can't imagine a better job to have. He is interested in all things agile, or anything new to learn.

SHARE
Request to reuse this
Add to my favorites
Topics: Benefits Realization, Communications Management, Lessons Learned

My team recently launched a new feature for our product that we felt strongly about. Watching customers use the product, we witnessed first-hand the errors they were encountering and the frustrations they were experiencing. With that knowledge in hand, we had a great idea for what we should build to make the experience better, eliminate those errors, and make the flow smoother and easier for the customer. We had little doubt that the feature would be a success, and even if we didn’t entirely improve the user scenario, we certainly couldn’t make it any worse.

After the feature was designed, built and tested, we followed the same best practice for releasing as we always do. We created a way to A/B test the change and made sure the sample sizes for both the control and the treatment were large enough to see how customers reacted to the change. We released the feature to production to a percentage of users, let it run for a few weeks, and then collected the data to see if we had enough statistics to support giving the new feature to even more customers, and how much of an impact the feature was having. We had every reason to believe we would be adding more people to the experiment, up to the point where it would eventually become the control.

To our surprise, not only did the new feature not make things better, all the metrics that we were tracking got worse. Customers in the…

Please log in or sign up below to read the rest of the article.