Forum Discussion

DavidO's avatar
DavidO
Strategist II
2 years ago

How long do you run A/B or similar tests and why? Use cases?

Hi all, Curious about how long you run A/B tests, or any other tests for before making a call on the data. I imagine it varies with the use case and would be keen to hear other's thoughts and scen...
experimentation
  • ConorIrvine's avatar
    ConorIrvine
    2 years ago

    Until I get the result I want ğŸ˜‚

    Okay so the real answer would be until I get a statistically significant result. Sometimes that's a single send analysed the following week in 15 min and sometimes it's a longitudinal test that runs for 6 months and takes weeks to unpack. As you said it really does vary based on what you're testing, the size of your (engaged) base and what you're wanting to learn. 

ConorIrvine's avatar
ConorIrvine
Specialist
2 years ago

Until I get the result I want ğŸ˜‚

Okay so the real answer would be until I get a statistically significant result. Sometimes that's a single send analysed the following week in 15 min and sometimes it's a longitudinal test that runs for 6 months and takes weeks to unpack. As you said it really does vary based on what you're testing, the size of your (engaged) base and what you're wanting to learn. 

  • DavidO's avatar
    DavidO
    Strategist II
    2 years ago

    Statistically significant result. That actually makes a lot of sense to me. Great approach.

    As far as getting the results you want, just play with the X and Y axis until the graph goes up. 😆