Skip to main content
Act-On Software

Using A/B Tests in Automated Programs

 

The A/B test condition within automated programs allows you to see which variations of messages perform best against a random and equally-distributed set of program contacts. After a specified test interval, the winning message (determined by the highest clickthrough rate) is then sent to future contacts entering that stage of the program.

The ability to select unlimited messages during the test allows you to test simple variables across two messages or design larger multivariate tests.

Create the A/B test

A/B tests are created through the Send Email program step, and can be applied at any stage of both new and existing programs. 

  1. Inside the program editor for a new or existing program, navigate to the Program Flow tab and add a Send Email step.
  2. Edit the Send Email step and click Add Conditional or A/B Test Message. 
  3. Choose A/B Test from the Mode drop-down menu that appears at the top of the window. 
  4. Click Add A/B Message to add messages to be tested. You may select as many messages as you would like. 
  5. Specify the number of clicks you would like the system to count before the winning message is determined. 
    • The test will run until the system receives the specified number of unique clicks across all test messages. The message with the highest clickthrough rate will then automatically be determined the winner and will be sent to any contacts who enter that program step in the future. The option to manually choose a winner will also be presented once the test begins.
    • The default Clicks value, and recommended minimum sample size, is 100.
  6. When the test messages and clickthrough threshold have been defined, save the step. After making any additional changes to your program, make sure to save the overall program as well. 

Once the A/B Test step is created and the program is saved, the test status and results can be viewed from the program dashboard.

Running the test and selecting the winner

  1. Once you're ready to start your program, click Start from the program’s dashboard to start the program and the test. 
  2. The A/B Test step will send each message in your test to program contacts in an equal and random distribution. For example, if there are two messages in the step, each message will be sent approximately 50% of the time. 
    Note: Note that with a very small sample set, the randomizer may result in a slightly different distribution between all messages. 
  3. On the program dashboard, click on the A/B Test step name to expand its description. The description will show the test in progress, how many clickthroughs are needed to complete the test, and the current test counts for each message. 

To select the winning message, you have two options:

  • System-defined winner – Once the specified number of unique clickthroughs has been reached, the test will end automatically, and the message with the highest clickthrough rate will be sent to each contact entering the program step in the future. The dashboard will refresh to show that the test has completed and the winning message is the one now being sent to contacts. A green checkmark will display to the right of the winning message. 
    • If the clickthrough results are the same, the first message will be chosen as the default winner. 
  • User-defined winner – You may also manually end the test at any time and choose any message to be the winner. To do this, click on the Mark as Winner text next to the message you want to be sent going forward. 

Pausing /editing a test

If you wish to pause your test, you can temporarily stop the overall program by clicking Stop at the top of the program dashboard. To resume the program and the test, click Start. Your test counts will stay intact.

You may edit the A/B Test step at any time by clicking Edit from the program dashboard. 

  • Running programs must first be stopped before making changes.
  • Messages can be added or removed from the test by editing the step from the Program Flow tab. Message content can be modified from the Program Messages tab. 
  • Note: If you replace any of the test messages in the step with new messages, your test results for those messages will be reset.

Any messages you have added in Conditional mode for a particular step will be preserved when switching to A/B Test mode, and vice versa.

Restarting a Completed Test

A completed test may be restarted at any time. Click Restart Test from the expanded step description on the program dashboard. 

  • This will clear the previous test counts and start a new test. Note: Although test counts will clear, the message’s total count history can still be viewed from the Program Messages tab on the dashboard.
  • If you wish to modify, add, or remove messages before restarting your test, the program will first need to be stopped before making changes.
  • Was this article helpful?
Support

Have a question about this topic?

Ask the community!