Culture, Not Laws

Culture, Not Laws

Thoughts on creating an experimentation culture

This blog is written in the context of a centralized Experimentation Team, but it can apply more generally if you are working to expand Experimentation within your particular team, as well.

The first temptation when trying to create an Experimentation culture is not unlike the temptation faced by the would-be utopian reformer: we should write some laws. There should be rules for how to experiment, we think.

Unlike the utopian reformer, we (thankfully) don’t have the option to convert into a dictator when our Enlightened Laws prove difficult to enforce.

Rules for experimentation are not enforceable and cannot be enforced because an Experimentation Team lacks the two necessary requirements for any law enforcement agency:

1. Punishment.
2. Borders.

We do not have a mechanism to penalize people for not following our rules and we have no means of forcing them to run an experiment at all. Teams can simply secede from Experimentation Nation.

Think about it: How could an Experimentation Team possibly “block” direct code changes? How would that even work? It would clearly be a bad thing, in any case, because an Experimentation Team doesn’t have the information and skillset necessary to do this in a smart way.

Be humble.

The essential thing about a good experimentation culture is that people run experiments. Bad experiments are better than no experiments. Get people to run experiments and look at the data. If you show up to the gym everyday, eventually you’ll lift and get stronger — even if you don’t use the “best” workout routine.

If we try to use rules to create an experimentation culture, we will end up doing one of two things:

1. We will make the rules so weak that the rules become meaningless, or
2. We will be forced to engage with a significant number of teams about why their individual experiments should not launch despite their desire to do so. We will need to be comfortable being and empowered to be a blocker for them despite us having no direct stake in their product or roadmap and a limited, narrow understanding of the constraints they face. Experimentation will become an obstacle to overcome. A road block, not a road sign to know whether we’re going the right way.

So, we should not make rules or write laws. We should provide guidance that we make no effort to enforce but significant effort to disseminate.

Instead of laws, we should write guidelines.

We can make guidelines strong and prescriptive (and, therefore, meaningful and useful) because teams have the option of simply not following them, when they run into an exception — or they just don’t want to.

The key thing is that they are experimenting. They are looking at the data, making tradeoffs. There is a record of the tradeoffs they made. And they have some advice about what to do.

The Main Problem is not that teams don’t use the right standard errors or peek too much. The Main Problem is when things launch without looking at any metrics about what the impact of the product change is, or we only look at pre-post effects where most product impacts are masked by daily demand fluctuations. The Main Problem is when we have no record and no idea of how what we launched this quarter affected our revenue.

Solve the Main Problem with Culture and make the UI helpful enough that the Secondary Problems aren’t a big deal. Don’t let the Secondary Problems get you off-message.

In real life — which, arguably, work is a part of — Culture changes Law, not the other way around (see: Uber, LLMs). Carry this over into your thinking about how to make experimentation a larger part of your org’s decision-making.

A strong Culture does not need strong Laws.