How Samsung Wallet, Panera, and BET+ Use Experimentation to Optimize Engagement

Marketing experimentation transforms uncertainty into clarity, allowing brands to ask questions, try new approaches, test hypotheses, and let data drive their decisions. At Forge 2024, we sat down with three industry leaders, Catherine Thacker from Panera, Angela Onwuka from BET+, and Peter Isaacson from Samsung Wallet, to discuss how experimentation fuels engagement and conversion.

During the session, the speakers emphasize that curiosity, combined with the right tools, can make experimentation more effective. They reveal that testing isn’t just a strategy; it’s a necessity for understanding what resonates with your audience. Join us as we explore their proven approaches to marketing experimentation and discover how you can implement these strategies to optimize your customer engagement efforts.

Our Hypothesis: Experimentation Drives Higher Engagement

OK, so it isn’t so much a hypothesis as a proven fact: Marketing experimentation drives higher engagement.

“We are constantly testing and evolving, enhancing our engagement from marketing, lifting MAUs, sessions, and stickiness significantly”, says Peter.

“If you aren’t testing then that’s a missed opportunity,” Angela adds. “Testing and optimization are important because they let you understand what resonates with your audience. This helps you get the most out of each touchpoint with your customers and prospects.”

Catherine Thacker agrees, commenting that “without testing, you don’t know that you’re landing the right message with the right audience in the right place at the right time.” And that knowledge drives results.

Getting Started: “Test Anything” Say Experts

One barrier to experimentation is overwhelm. Marketing experimentation can cover anything from copy and creative, to cadence and messaging. How do you choose what to test?

“Test anything,” encourages Peter. “Copy, button color, a new subject line… You will start to learn so much about your users in a way that surveys or static data can’t show you.”

Peter suggests using customer research as a launchpad. “Customer insights are a critical testing input. Conduct customer insight surveys. Is the messaging unclear? Should the offer be richer? The survey responses will provide insights to inform your tests.”

But how can you make sure you get your marketing experiments right the first time? Our experts give you permission not to.

“Don’t let perfection stand in the way of progress’ says Catherine. ‘Just because you can’t design the most ideal test for your scenario shouldn’t stop you from trying things in small increments. They’ll start adding up to bigger outcomes and impact. Start with what’s doable, what’s going to easily build that muscle, and demonstrate that it’s a worthwhile practice.”

Moving Forward: Be Strategic

As you grow in confidence and your culture of experimentation matures, how do you stay focused on the experiments that move the needle most for your business? And ensure your data is undeniable? Processes, prioritization, and measurement are key.

“There have to be processes by which you intake, prioritize, and execute the tests,” advises Catherine. “Aligning on measurement and methodology is important to ensure that when you report results, there’s credibility there.”

Angela recommends a “testing roadmap” to prioritize your tests and “understand the full scope of requirements to go live” so that you can align your resources such as your data team and your creatives to execute successfully.

Peter agrees saying “You need to be clear about what you’re optimizing with your test plan” before reiterating the importance of having the right people in place.

“You need the right team around you to support all that’s needed to plan and execute tests. You’ll need product and engineering teams; a designer or agency to develop creative iterations quickly; and someone to write the copy and plan the tests.”

Future Trends: AI Advances

With all the buzz around generative AI for copywriting, you may be surprised to hear someone advocating for old-fashioned human writers. While all of our experts are using Artificial Intelligence to enhance their productivity and results, they’re balancing that with human creativity.

“Gen AI content is something we’re still playing around with but seeing good results. When we’re running Gen AI subject lines at scale in a campaign, we consistently see 15% lifts in open rate and a lift in conversion rate,” says Catherine. However, they’re still experimenting to achieve their desired brand voice with AI.

Peter concurs that AI is advancing for copywriting but isn’t their go-to yet. “It’s always interesting to see what copy AI recommends but I still view it as a companion versus a necessary tool. I think, as AI develops, this will change and that could be very soon.”

However, when it comes to other uses of AI, past generative copywriting, such as analytics and intelligent timing, our experts are delighted.

“It’s extremely effective for running personalization at scale,” says Catherine. “You can set parameters like spend and profitability and starting audience, and it will continuously learn what offers and stimuli individuals respond to.”

Looking to the future, Angela believes AI will “begin to play a larger role in A/B testing predictions for marketers” and “deliver statistically significant test results more quickly”. This is exciting stuff and as we always say, there’s never been a better time to be a better marketer.

Learn More

This is just the tip of the advice iceberg. To learn more about experimentation, like the difference between tactical and strategic tests, when to use multivariate experiments, and how to integrate Braze and other tools into your tech stack, check out Forge on demand