For example: you can test two different subject lines or test two different calls to action, or test the combinations (2 subject lines x 2 calls to action = 4 variations to test). Up to five email messages can be added to an A/B email test.
Under Outbound > A/B Test Messages > New A/B Test, you can do the following:
- Provide a name and description for your A/B email test (a unique name is advisable)
- Set the final run distribution list for the winning message
- Set the sample size for the test run. The maximum sample size is 33% of the distribution list total and must be comprised of at least 10 contacts.
- Select two or more emails to test
- Set the send time for the test run
- Set the send time for the final run
Three stages of A/B testing
- Setup and schedule
- Test run
- Final run of the winning message
Saving a new A/B email test creates a draft of the test, which you can resume editing later via Messages > A/B Testing > Drafts. Once saved, click Schedule Test to move the A/B email test from Drafts into the active listing (Messages > A/B Testing > Active). The system chooses the winning message, which is determined by the greatest number of unique clickthroughs at the time of the final run. A user-defined winner can also be chosen by hovering over your desired fully-run message and clicking the hover-over check box and then clicking Save.
Act-On does not take into account the message 'opens' for the final send, only 'clicks' when messages are being tested to determine the final send. If the clicks are the same (or there were none), Act-On will default and send the first message as the final message every time.
In the Drafts section, you can edit, delete, or schedule any of your draft messages.
In the Active section, you can see the status of test and final sends as well as the dashboard for any active A/B email tests. You can also cancel any active A/B email tests. Messages sent in a test run are also shown under Outbound > Triggered Messages > Other.
When canceling an active A/B email test in the Active section, one of two things will happen:
- If the test run has not run, the canceled A/B email test will return to the Drafts page (the test can be resumed by editing it).
- If the test run has run, you will be prompted to confirm the cancelation. If the cancelation is confirmed, the final run will be canceled and the record will move to the Completed tab. This cannot be undone.
If you do not wish to cancel the final run, but only unschedule it, edit the A/B email test via the dashboard. Change Final Run to Not Scheduled and then click Save.
Here you can see a list of the completed A/B email tests (or those canceled after the test run was sent). Clicking on a completed A/B email test in this list takes you to the test's report dashboard where you can see the testing and final results. Final-run sent email messages are also shown under the Messages > Sent menu. When viewing the message report for the final send, contacts who received one of the test versions will show as suppressed for "Suppression List".