A/B Testing in AI-Driven Sales: How to Optimise Your AI Automations

A/B testing in AI sales sounds like something for corporate marketing departments, but at its core it is a simple idea: you run two variants of your automation against each other and see which one brings in more appointments or orders. This article shows when such a test is actually meaningful for a small business, how to set it up properly and what to do instead if you do not get enough enquiries.

What A/B testing in AI sales means

An AI-supported sales process is made up of many small decisions that someone took at some point. How does the chat greet a visitor? Does the AI employee ask about budget first, or about timing? Does the reminder about an open quote go out after three days or after seven? Usually these decisions are based on gut feeling or on the provider's default settings.

In an A/B test you change exactly one of these settings. Half of the enquiries get variant A, the other half variant B, assigned at random and in the same period. At the end you compare a metric you defined in advance, for example how many conversations lead to a booked appointment. The key is that only this one detail differs between the groups. If you change the greeting, the tone and the follow-up timing all at once, you will not know afterwards what made the difference.

An A/B test is not the same as a before-and-after comparison. If you use variant A in March and variant B in April, you are also measuring the difference between March and April: school holidays, weather, a discount campaign, roadworks outside the shop. That can be misleading.

When an A/B test in sales is worth doing at all

The honest answer for many small businesses is: less often than software vendors suggest. A test needs enough cases for a difference not to be mere chance. The following example calculation shows why.

Example calculation: online shop versus beauty salon

The figures are assumptions for illustration, not measured values.

Assumption	Online shop	Beauty salon
Chat and WhatsApp enquiries per month	800	60
Enquiries per variant per month	400	30
Conversions per variant with variant A	about 32	about 3
Time to a reliable result	a few weeks	hardly achievable

For the shop, 400 conversations per variant lead to a few dozen orders. If variant B is clearly ahead after a month, that is a useful signal. For the beauty salon, it may be three bookings against five. A single client who happened to land in group B can create that difference. Even after six months the result would be shaky.

As a rough rule of thumb: if you do not expect at least several dozen conversions per variant, a classic A/B test is wasted effort. Free online sample-size calculators help with a more precise estimate.

How to set up a clean A/B test

1. Write down a hypothesis. For example: if the chat asks for the order number first when handling return questions, fewer customers will drop out of the conversation. Without a hypothesis you are testing in the dark.
2. Define the target metric before the test starts. One main metric, such as booked consultations per conversation. You keep an eye on secondary metrics like response rate or conversation length, but you do not decide based on them.
3. Change only one thing. The wording of the first message, the order of the questions or the timing of the reminder. Everything else stays the same.
4. Assign at random. Most chat and automation platforms can assign conversations to a variant alternately or at random. Ask your provider how this works in your setup.
5. Fix the duration in advance. At least two full weeks, so that weekdays and weekends appear in both groups. Do not stop as soon as one variant briefly takes the lead; interim results like this often flip back.
6. Evaluate and document. Record what was tested, for how long and with what result. The winning variant becomes the new standard that the next test competes against.

Our overview of the key KPIs for AI sales automation describes which metrics work as a target metric.

What to test first

Not every setting matters equally. Test first where most prospects drop off. A look at the conversation logs usually shows quickly where customers bail out. How to map and measure your funnel stage by stage for this is shown in optimising your AI sales funnel.

What to test	Example for A and B	Sensible target metric
First message	Open question versus option buttons	Share of visitors who reply
Order of qualification	Needs first, then contact details, versus the other way round	Completed qualifications
Appointment offer	Two specific suggestions versus a calendar link	Appointments booked
Following up on quotes	Reminder after three days versus after seven	Quotes accepted
Tone	Matter-of-fact versus casual	Appointments booked, complaints

With wording in particular, it pays to prepare the variants carefully. You will find ideas in the article on AI sales scripts that convert.

Limits and common mistakes

The most common mistake is a test with too few cases whose result is then presented as an insight. The second most common: too many tests running in parallel. If you change the greeting and the follow-up rhythm at the same time, the effects get mixed up. For small teams, more than one running test per process step rarely makes sense.

There are also things you should not test. Telling customers that they are talking to an AI does not belong in an A/B test. We recommend always doing it; the EU AI Act requires this transparency. Prices or terms that differ by group are also tricky and can annoy customers if they find out. And if you use website tracking for the evaluation, you will usually need consent via your cookie banner.

The alternative for small enquiry volumes

If the volume is not enough, work with careful before-and-after comparisons. Change one thing, let it run for at least a month and compare it with the previous month and, if available, the same month last year. Add qualitative review: skim through twenty conversations every week. For a physiotherapy practice or a salon this often tells you more than any statistics, because you can see straight away which question customers get stuck on.

Frequently asked questions

What is the minimum number of enquiries I need for an A/B test?

There is no fixed number, because it depends on the conversion rate and the expected difference. What matters is the number of conversions per variant, not the number of conversations. If you only expect a handful of conversions per group, the result will tell you very little.

Can the AI decide for itself which variant wins?

Some systems automatically route enquiries to whichever variant is currently doing better. That is handy with large volumes. With small volumes, however, it amplifies random fluctuations. So leave the decision on the new standard with a person.

How often should I test?

A few well-planned tests are better than one a week. For most small and mid-sized businesses, one or two tests per quarter is realistic, embedded in a plan such as our 90-day plan for AI sales automation.

Do I have to tell customers that I am testing?

There is no specific duty to inform customers when you test different wording. Your general privacy notice must, however, describe which data you process in the conversation and for what purpose. If in doubt, have your data protection officer check it.

Conclusion

A/B testing helps you improve a running AI automation step by step, but only with enough enquiries and with a single change per test. If you get fewer enquiries, you will do better with clean before-and-after comparisons and regularly reading along. If you have not launched yet, first read how to implement AI sales automation in 7 days.

For retailers with lots of chat enquiries, the page AI sales assistant for online shops shows how an AI employee handles conversations and passes the results to your CRM.