nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Conversion Rate over 100%?

dazza

Hi,

I'm struggling to understand the performance of one of our Canvases.

I have both the conversion event and exit criteria as "any purchase", and users cannot re-enter the canvas.

When I analyze the variants, I see more conversions than entries, leading to a conversion rate over 100%, which shouldn't be possible.

Does anybody know what could be going wrong here?

thanks!

Find more posts tagged with

Canvas

Experimentation

Comments

TedScott

Hi,

Braze does not current track direct conversions (or even session based conversions).

In the simplest terms, the conversion is based on they are in the campaign or canvas and a conversion occurred in the conversion widow - There is a bit more to it than that.

How long of a conversion window are you using? The shorter the less noise. Also, are you using other means to help track such as google UTM's or promo codes?

There are improvements that are slated for conversion tracking but they have not surfaced yet.

dazza

We are using 3 days as the conversion window.

Perhaps I'm misunderstanding, but if Braze does not track direct conversions, how do we determine the performance of what we're setting up?

I have a canvas targeted at getting churned customers to make a purchase. I have a control group set up. How should I best evaluate the success of the churn campaign against the control group if I can't rely on the conversion rate?

DavidO

Hi @dazza

To add to what @TedScott noted, how Braze works is a user enters a canvas or campaign, then, once they have received all the messaging from that campaign or canvas, it watches the user for the time specified in your conversion window.

For example, if your canvas takes 5 days to get through, and your conversion window is 3 days, there are 8 days in total for Braze to measure a conversion. It works on the assumption that some users will get all the messaging but then not convert immediately. They may convert immediately, but quite often they will not.

Braze will only record the first conversion event, even if a user does the same event twice in the window, which still doesn't explain the over 100% conversion rate. I'm trying to recall if I've seen over 100% but not confident I have.

Conversion Events (braze.com)

I personally find Braze's conversion tracking pretty good, we do our analysis in Amplitude but Braze is a good temperature check. I wouldn't write it off.

What is the use case you are trying to measure converisons for?
🙂

dazza

This particular canvas is targeting the first purchase of a customer. So it starts with customers that have no purchases. The canvas target conversion action, and exit criteria, is making any purchase.

Here is a screenshot from the top level of the canvas, you can see 43 entered, 25 messages sent, and 45 conversions. Though this section seems to limit the conversion rate to a max of 100%.

However, I know that the conversion rate is incorrect because not all contacted customers converted. Some of the steps in the journey have too many conversions (one has 17 sends and 25 conversions, and another has 3 sends and 0 conversions).

But it becomes even more apparent when I click to "Analyze Variants", and it shows Variant 1 with a conversion rate of 120%

Further to that, when I create a Canvas Funnel Report, where the only step in the funnel is to make a purchase (ie the conversion event), the conversion rate looks completely different to the Analyze Variants report. They look more how I would expect.

Why is this report in the Canvas Funnel and the Analyze Variants so different?

DavidO

Yeah that 120% in the second image almost looks like it's recorded 6 extra conversions for the 30 people who entered. I wonder if some conversions are getting double-counted somehow? The 0.00% confidence, literally gives me no confidence this figure is correct either. Anything under 80-90% confidence I question pretty heavily. I would definitely flag this one with your CSM and see what they can unpack in the dashboard.