Forum Discussion

DavidO's avatar
DavidO
Strategist II
2 years ago

How long do you run A/B or similar tests and why? Use cases?

Hi all,

Curious about how long you run A/B tests, or any other tests for before making a call on the data. I imagine it varies with the use case and would be keen to hear other's thoughts and scenarios.

Thanks 😊

experimentation
  • ConorIrvine's avatar
    ConorIrvine
    2 years ago

    Until I get the result I want ðŸ˜‚

    Okay so the real answer would be until I get a statistically significant result. Sometimes that's a single send analysed the following week in 15 min and sometimes it's a longitudinal test that runs for 6 months and takes weeks to unpack. As you said it really does vary based on what you're testing, the size of your (engaged) base and what you're wanting to learn. 

  • ConorIrvine's avatar
    ConorIrvine
    Specialist
    2 years ago

    Until I get the result I want ðŸ˜‚

    Okay so the real answer would be until I get a statistically significant result. Sometimes that's a single send analysed the following week in 15 min and sometimes it's a longitudinal test that runs for 6 months and takes weeks to unpack. As you said it really does vary based on what you're testing, the size of your (engaged) base and what you're wanting to learn. 

    • DavidO's avatar
      DavidO
      Strategist II
      2 years ago

      Statistically significant result. That actually makes a lot of sense to me. Great approach.

      As far as getting the results you want, just play with the X and Y axis until the graph goes up. 😆

  • 0203amit's avatar
    0203amit
    Active Member II
    2 years ago

    Hi DavidO ,

    Yes its true it varies from use case to use case.
    But minimum time should me more then 24 hour to get better result.

    Thanks