Simpson's Paradox
A total can move one way while every group inside it moves the other way.
Why it matters
When the mix of groups changes, the total changes too, even if nothing got worse. Reading only the total can lead to the wrong decision.
How to use it
- Split results by device, channel and new or returning users.
- Check whether the mix changed.
- Ask why the mix moved.
Example
Phones rose from 2% to 2.2% and computers from 8% to 8.4%, yet the total fell from 5% to 3.44%, because most visits moved to phones.
Watch out
Do not split the data every way until you find a story you like. Decide on the main splits in advance.