What Is A/B Testing?

A/B testing is a controlled experiment in which two versions of a single campaign element - a subject line, opening line, call to action, or send time - are sent to randomly split segments of the same audience to see which performs better. Only one variable changes between version A and version B; everything else stays identical, so any difference in results can be attributed to that variable. In cold email, the metrics that matter are reply rate and positive reply rate, not opens, because open tracking pixels are unreliable and can hurt deliverability. A test needs enough volume to be meaningful: with typical cold email reply rates of 2-15%, small samples produce noise, not signal. Once a winner is confirmed, it becomes the new control and the next variable gets tested, compounding small gains over successive campaigns.

A/B Testing in Practice

In practice, agencies run A/B tests at the campaign level inside sending tools like Instantly or Smartlead, which rotate variants automatically across the lead list. A workable minimum is around 200-300 sends per variant before judging reply rate; below that, a 2% versus 3% difference is usually chance. A concrete example: an agency tests a question-based subject line against a two-word internal-style subject across 1,000 prospects, 500 each. Version A pulls a 4% reply rate, version B pulls 7%. B becomes the control, and the next test targets the call to action - asking for interest versus asking for a meeting time. The most common mistake is testing multiple variables at once: a new subject line, new offer, and new list in the same send tells you nothing about why results changed. The second most common mistake is calling a winner too early. Reply data trails sends by days because follow-up steps generate a large share of total replies, so a test judged 48 hours after launch will often reverse.