Designer POV
A few years ago, a team showed me their test results. They’d spent 2 weeks testing button colours.
Red versus blue. Blue won by 5%.
I asked what problem they were solving.
Silence.
The real issue was that users didn’t understand what the button did. The label was confusing. The placement was buried. But they tested colours because colours are easy to test.
After 6 years leading teams, I’ve seen hundreds of A/B tests. Most measure surface-level changes because they’re quick to implement and easy to quantify.
Meanwhile, the hard questions go untested.
Should this feature exist? Is the information architecture broken? Are we solving the right problem?
Those questions require real research, not just analytics. They require killing features, not optimizing them.
A/B testing has become busywork disguised as rigour.
Teams run tests to feel data-driven without asking whether they’re testing what actually matters.
Before your next test, ask: if this wins, does it solve a real user problem or just make a mediocre solution slightly less mediocre?