Evaluating AI Providers: A Checklist for Managing Directors

If you have to evaluate AI providers, you quickly find yourself facing a pile of glossy demos that all sound alike. This checklist turns the selection into specific questions you can put to every provider, from conversation quality through data protection and contract terms to support after go-live. It is intended for managing directors who have no IT department of their own but want to make a sound decision.

Before the first meeting: what should the AI actually do?

The most important groundwork is yours. Write down on one page which task the AI should take on, through which channels and with what goal. For example: "Take calls and WhatsApp messages outside opening hours, book appointments in the calendar, forward urgent cases to the on-call service."

With this page you compare providers on the same basis. Without it, you are comparing presentations. Also prepare five to ten real customer enquiries, including difficult ones: the caller who mumbles, the question that does not fit the pattern, the angry message. You will run through these cases later in every demo. Also note a budget range for setup and ongoing costs. That way you can weed out offers that do not fit before you invest hours in meetings. Which tool categories are worth considering in the first place is set out in our comparison of AI sales tools for 2026.

The checklist: evaluating AI providers in six areas

1. Conversation quality and functionality

• Does the system understand your test cases, including dialect, background noise or typos?
• What happens when the AI cannot answer a question? Does it admit it does not know, or does it make something up?
• How and when does it hand over to a person, and does the context go with it?
• Does the AI identify itself as an AI? It should, not least with a view to the EU AI Act.

2. Data protection and security

• Where are the servers, and is data transferred to countries outside the EU, including by sub-processors?
• Is there a data processing agreement under Article 28 GDPR, and which sub-processors does it name?
• Is your conversation data used to train models?
• How long are conversations stored, and can you set deletion periods yourself?
• What security certifications are there, and what exactly do they cover? What an ISO 27001 certification tells you is explained in the article ISO 27001 when choosing a provider.

3. Integration

• Can your calendar, CRM or industry software be connected, or will data have to be maintained twice?
• Who sets up the connection, and who takes care of it when your software gets an update?
• Is there a documented interface for later extensions?

4. Introduction

• What does the process look like from signing to go-live, with dates and responsibilities?
• How much time does your team need to set aside?
• Is there a test phase before real customers talk to the system?

5. Support during operation

• Whom do you reach when there is a problem, through which channel and at what times?
• Who adjusts the answers when prices or opening hours change: you or the provider?
• Do you receive reports, for example on the number of conversations, handovers and open cases?

6. Costs and contract

• Which costs are one-off, which monthly, and what happens if there are more conversations than the plan includes?
• How long is the minimum term, and how long the notice period?
• Can you export all data in a common format when the contract ends?
• Is there a trial phase with an easy way out?

Warning signs in the sales conversation

Some signs count against a provider, however good the demo sounds:

• They quote specific increases in revenue or close rate without knowing your figures.
• They cannot say where the data is processed, or just refer to "the cloud".
• The demo runs exclusively on prepared examples; your own test cases are to come "later".
• The data processing agreement is only available after you sign.
• The price list is so convoluted that you cannot work out your monthly costs yourself.

Conversely, a provider who openly says what their system cannot do is usually a better partner than one who answers every question with "that's possible".

Your own test call

Do not rely on the provider's recordings when judging conversation quality. Ask for a test number or test access set up with your opening hours and a few of your answers. Then call yourself, several times:

1. Once with a standard request, such as an appointment request.
2. Once with a question the AI cannot know the answer to, for example about a special price.
3. Once with an urgent case that would require immediate forwarding.
4. Once, have a colleague call who speaks quickly or has a dialect.

Afterwards, check what has arrived in the calendar or CRM. A friendly conversation is of little use if the appointment ends up on the wrong day or the call-back number is missing.

When you do not need an AI provider

This is also part of an honest evaluation. If you only receive a few enquiries a day and are easy to reach during normal hours, an AI assistant does not solve a real problem. In that case a good voicemail greeting, an online calendar and fixed call-back times are often the better and cheaper solution.

Comparing providers: a simple scoring matrix

Score each provider in the six areas from 1 to 5 and weight the areas according to your situation. A medical practice will weight data protection more heavily than an online shop; a trades business with outdated software is more likely to weight integration.

Area	Weight (example)	Provider A	Provider B
Conversation quality	3
Data protection and security	3
Integration	2
Introduction	1
Support	2
Costs and contract	2

The matrix does not replace judgement; it makes it traceable, including for co-owners or the bank. The decision framework for SMEs offers a more detailed method. Whether a German or an international provider is a better fit is covered in the comparison German vs. international AI providers.

Frequently asked questions

How many providers should I look at?

Two to three on the shortlist are usually enough. More takes a lot of time without improving the decision. What matters more is that all of them are tested with the same cases.

Should I insist on a trial phase?

Yes, if possible. A short trial with clear criteria shows more than any demo. How to set one up is described in the article on the AI pilot project in sales.

Are customer references meaningful?

They help if you can talk to the reference customer yourself, ideally one from your industry. Do not just ask about the start; ask how things are going day to day after a few months.

Who in the company should be involved in the selection?

Besides management, the person who will later work with the system and, where personal data is involved, your data protection officer or data protection adviser.

Conclusion

A good choice of provider starts with a clear task description and your own test cases, not with the demo. Check conversation quality, data protection, integration, introduction, support and contract with the same questions for every provider, watch out for warning signs and document your assessment. Then the decision will still be understandable a year from now.

You can test how a digital employee takes enquiries by phone, chat and messenger in practice against your checklist using the example Digital assistant for service centres.