Assessing Third-Party AI: What to Demand From a Vendor

Assessing Third-Party AI: What to Demand From a Vendor

Most enterprise AI is not built. It is bought — as a feature inside a SaaS product you already use, as a startup's API, as a model behind someone else's wrapper. And most of it is assessed with a questionnaire that both sides understand to be a ritual.

The questionnaire asks whether the vendor tests for bias. The vendor says yes. The questionnaire asks whether there is human oversight. The vendor says yes. Nobody asks what the test was, what it found, or who the human is, and everybody signs. The document produced by this process has one function, which is to exist in a folder if something goes wrong.

The problem is not that procurement is lazy. It is that the questions are unanswerable in any discriminating way. "Do you test for bias?" has exactly one rational answer for every vendor, honest or otherwise. A question that every respondent answers identically carries no information.

Ask questions that cost something to answer

A good diligence question has a property the standard ones lack: a vendor who has not done the work cannot answer it without visibly failing. The answer requires an artefact, a number with a denominator, or an admission.

Compare:

The right-hand column is uncomfortable to ask and more uncomfortable to answer, which is precisely why it works. The vendor's reaction is data. A vendor who produces a report containing unfixed findings with severities and owners is a vendor with an evaluation function. A vendor who has never had an incident either has an immature monitoring capability or is not telling you about one.

The artefacts worth demanding

Ask for documents, not assurances. Five are worth the fight.

An evaluation report with negative results. If it contains no findings, either the evaluation was not adversarial or the findings were removed before you saw it. Ask which. The presence of open findings, with severity and remediation status, is a mark of maturity, not of weakness — and treating it as a red flag in procurement is how you train the entire market to hide them.

A system description, not a model card. You need to know what surrounds the model: the prompt layer, the retrieval corpus and its provenance, the tools the system can invoke and their side effects, the guardrails, the fallback behaviour when the model is unavailable. A model card from their upstream provider is not a description of their product.

The data flow. Where does your data go? Is it used for training, by them or by their upstream provider? Is it retained, and for how long? Which subprocessors see it, and in which jurisdictions? Is your tenant's data ever in the same retrieval index as another tenant's? That last question has produced more silence in vendor calls than any other on this list.

The change policy. How will they tell you when the underlying model changes, and how much notice will you get? This is the clause that decides whether your own evaluation evidence has a shelf life you control or one they control. A vendor who can swap the model silently has reserved the right to invalidate your assurance without telling you.

The dependency chain. Which model provider, which hosting, which vector store, which upstream API. Their concentration risk is your concentration risk, and their outage is your outage. This is also where you find out that three "independent" vendors in your portfolio are the same model behind three wrappers.

The split at the top is the one procurement usually misses. Claims about process — that a policy exists, that reviews happen, that staff are trained — can reasonably be handled through attestation and certification. Claims about behaviour — that the system is accurate, fair, safe, grounded, resistant to manipulation — cannot, because behaviour depends on the data it meets, and the data it will meet is yours.

Why their evaluation cannot be your evaluation

Even an excellent vendor evaluation, honestly conducted, does not transfer to you. It was run on their distribution, against their definition of correct, for their intended use, under their risk appetite.

Your users phrase things differently. Your documents have a different structure. Your regulator has an opinion the vendor's does not. Your workflow puts the output in front of a person with less time and less context than the vendor imagined. The vendor evaluated a capability; you are deploying a system, and the system includes your users, your data, and your consequences.

This is not a criticism of vendors. It is a structural fact about where accountability sits. When the system harms your customer, "our supplier tested it" is not a defence anybody has ever successfully run — not in financial services, not in medical devices, not in food safety, and not in AI. Regulatory regimes are converging on the position that the party deploying the system owns the risk it creates, and outsourcing the build does not outsource that.

So the vendor's evidence has exactly one legitimate use: it tells you where to aim your own.

Verifying independently, in practice

Independent verification does not mean rebuilding the vendor's system. It means running your own scenarios, on your own data, against your own definition of acceptable, and treating the result as your evidence rather than theirs.

That means negotiating an evaluation environment before signing, because you will never get one afterwards. It means constructing a scenario set from your actual traffic, including the awkward slices the vendor's demo never touches. It means scoring against a rubric that encodes your policy and your regulator's expectations, by evaluators qualified in your domain, with a measured level of agreement between them rather than a single opinion. And it means recording what you found in a form you can produce later — a versioned Evidence Report with findings, severities, and root causes — because the moment you need it is the moment nobody remembers what was tested.

Then build the re-evaluation trigger into the contract itself. The vendor tells you when the model changes; the change fires a re-evaluation; the re-evaluation updates the evidence. Without that clause, your assurance decays silently and you find out at the same time as your customers.

What to do tomorrow

Take your three most consequential AI vendors and, for each, ask one question: when did the underlying model last change, and how did you find out? If the answer is that you did not find out, you have discovered that your evidence describes a system that may no longer exist.

Then rewrite the questionnaire. Delete every question whose answer is the same for every vendor, and replace it with a request for an artefact. The vendors worth keeping will send you something. The ones who send you a longer answer have told you what you needed to know.