Novelty Effect
People try something because it is new, so a lift can fade as the newness wears off.
Why it matters
A test that stops early can ship a change that adds nothing once people are used to it.
How to use it
- Run tests until the lift stops moving.
- Look at results week by week.
- Check new users on their own, since nothing is new to them.
Example
A redesign shows +20%, +12%, +6% and +1% over four weeks. The real effect is close to +1%.
Watch out
It also works the other way. People can dislike a change at first and come round to it, so an early drop may fade too.