Control Group Representivity

NatalieMc

We are looking at implementing Braze global control group for measurement.

We have typically done this at a canvas level using random buckets

Has anyone used these approaches and have any insight on the representivity of the global control vs canvas level controls. 

inespais

Hey @NatalieMc,

Before the out-of-the-box Braze Universal Control Group (UCG), I used a similar solution to your canvas one (I suspect) but through a Webhook campaign: new users would enter the webhook, we would assign them to the UCG randomly, based on the random buckets, and then we'd log a custom attribute in their profile to identify whether UCG was true/false.

With the current Braze Universal Control Group, the logic is similar but just more straightforward to setup/maintain. As far as I know, Braze also uses the random buckets to assign a random sample of your base to the UCG so it should be representative.

I think the latter is simpler, because not only you can handpick the % you want to allocate, but it's much easier to maintain and to ensure they are excluded from all campaigns, without relying on each team member to remember to add that filter to targeting every time. At the same time, you still have the flexibility to add "exclusions" and ensure that campaigns meeting certain conditions can still override the UCG (for example, transactional or key onboarding messages). You can easily pass the list of UCG members to other tools for reporting as well.

Overall, one good practice to keep in mind is that it might be worth re-setting your UCG every X periods of time. That's because if UCG users are never eligible to receive any messages, over time, they will naturally have a biased behavior and thus be less "comparable" to the treatment group. In the documentation, there're some useful tips on calculating size, things to watch out for, and experiment duration.

Hope that helps? good luck