Guidance on Limiting Audience per Experiment Path in Canvas

Hi Siva,

this is not confirmed by Braze or any documentation, but I'm pretty confident in why this is happening

When you have an A/B test or an experiment path, Braze uses probability to sort users into variants. It essentially flips a coin for every user 

The less coin flips (users) you have, the higher the margin for error is. 

For example, if you flip a coin 10 times, you are very likely to get outcomes like 7 heads / 3 tails, 6/4... which is far from the expected 50/50 split.

But if you flip a coin 10,000 times, you will get an outcome like 5,032/4,968 which is much closer to the 50/50 split.

This is a calculation for different audience sizes:

Users - Margin of error
50 - 13.9%
100 - 9.8%
500 - 4.4%
1000 - 3.1%
10,000 - 1.0%

• This is expected and happens in all tools that do any kind of A/B Testing
• The more users you have in your canvas, the more even the split will be
• The less variants you have, you'll get a more even split faster
• Using multiple canvases will not help, you'll get the same problem

In simple terms, a solution is to wait until you have at at least around 500 users per variant in your case.

Best,
Denis