Second-Order Benefits Of Full Stack Experimentation

Full Stack Experimentation has variety of potential second-order benefits that could impact the culture, communication patterns, and institutional memory of a company. While using experimentation can create a fast cycle of improvement for products and services, these second order benefits can actually change how a company does business.

These ideas, which had been percolating for a while, crystalized as I participated in a panel on May 15, 2018 organized by Split Software hosted at the Data Dog headquarters in the New York Times building. Trevor Stuart, co-founder and chief product officer of Split, moderated and Gabrielle Gianelli, Engineering Manager at Etsy, Brian Crofts, Chief Product Officer at Pendo and Ilan Rabinovitch, VP, Product at Datadog joined me on the panel.

In my past coverage of full stack experimentation I’ve focused mostly on the experimentation process itself (see “How Full-Stack Experimentation Enables Google-Speed Product Development”), although I had speculated that product market should be improved by experimentation (“Can A/B Testing and Full Stack Experimentation Inform Product Marketing?”).

Systematic Learning from Mistakes: All of the panelists agreed that one of the tough lessons learned when companies start a program of full stack experimentation was that most experiments fail to achieve a positive impact on the product in question. Many indeed have a negative impact. But, it occurred to me that the many failed experiments need not be wasted effort of the proper care was taken to analyze why they failed and what that said about both the product and the user segments involved. This sort of learning takes place informally as people running experiments analyze the results. The challenge is to take that to a higher level and start asking what do all the failures and successes say about the product and the users. When he speaks about the statistical approaches made popular in the book Moneyball, Paul DePodesta, the assistant general manager of the Oakland As, recommends that people keep a diary as they attempt to apply analytics. By doing just that about experiments, it is possible to use full stack experimentation to create a rich institutional memory that allows better experiments to be crafted, obvious mistakes to be avoided, and a more sophisticated understanding of users to spread throughout the organization.

Improved Cross-silo Communication: When full stack experimentation becomes wide spread, it acts as a forcing function for communication between groups that normally do not talk to each other enough. The product managers, product marketers, and others who come up with the experiments must work closely with the engineering team to implement the experiments. In this interaction, a better understanding of what sort of features are easy, and which are difficult is transferred from the engineering team to the product team. In addition, more of the subtle details of how the product is implemented is also transferred. This deeper understanding provides the product team with a better sense of the world of the possible so that impossible to implement features are avoided. In addition, the engineering team learns to think more like a product manager and understand the tradeoffs that must be made and how huge lists of potential features are grouped and prioritized. This mutual intimacy serves to improve collaboration and make for better experiments.

The Best Ideas Win: The Highest Paid Person’s Opinion (HiPPO) often holds sway in many companies in the absence of data. When fulls stack experimentation becomes widespread the successes are few and far between. This is humbling to just about everyone. It quickly becomes clear that if all ideas are likely to suck, it doesn’t really matter if they came from a senior executive. More people start to speak up. It pays to listen to everyone. The important thing is to run as many experiments as fast as possible and let the data tell you which ideas are truly great. In this way, full stack experimentation is a great leveler.

It was clear from the panel that every company goes through its own journey when adopting full stack experimentation. But by focusing on second order effects not just the results of experiments, the impact can be even larger.