The AI Vendor Risk Questionnaire You Should Actually Be Sending

Title: The AI Vendor Risk Questionnaire You Should Actually Be Sending

Introduction

Early last year a mid-sized healthcare payer discovered their procurement process had a blind spot that could have been catastrophic. Their procurement team was rubber-stamping AI vendors because the security supplement on every RFP was the same 12-question form used since 2019. That form asked about encryption-at-rest, SOC 2 reports, and patch management - useful, but irrelevant to the real risk that arrived quietly: one vendor had trained a model on customer protected health information (PHI) in a way that violated the payer's Business Associate Agreement (BAA). The model surfaced PHI in outputs, and the payer only found out when a care manager noticed a patient identifier in an AI summary.

The vendor claimed ignorance; the contract language didn't clearly prohibit training on customer data. The cost: two vendors dropped, three renegotiated tighter contracts, a public incident response, and - crucially - an expensive two-month diversion from product roadmaps to fix governance and legal exposure. That payer rebuilt their AI vendor questionnaire around six themes that actually matter for AI economy readiness: training data lineage, model behavior controls, evaluation evidence, sub-processor disclosures, customer data isolation, and incident notification.

If your current questionnaire still looks like a generic security checklist, this article is for you. Below is a practical playbook - what a real AI vendor due-diligence questionnaire looks like, why each theme matters, sample questions you can copy, the red flags that should kill a deal, and how to fold this into your existing third-party risk management (TPRM) workflow so procurement becomes a source of advantage, not downstream risk.

Why most AI vendor questionnaires are useless

Traditional vendor questionnaires assume vendors are software vendors: they focus on perimeter security, physical controls, and compliance reports (SOC, ISO). AI changes the threat model:

- Risk lives in data and models, not just servers. A vendor can be secure and still train models on your data in ways that leak or repurpose it.
- Model behavior is emergent. You cannot "test into safety" by checklist alone - you need evidence about evaluation, drift monitoring, and mitigation controls.
- Sub-processors matter more. Pretrained models, foundation model providers, and cloud AI services are nested supply chains.
- Contracts matter in new ways. You must specify training prohibitions, IP ownership, and allowable reuse.

A modern AI vendor questionnaire must be framed by how AI systems are built and operated - not by pre-AI assumptions. It should align to risk frameworks such as NIST AI RMF, the EU AI Act (where applicable), and emerging standards like ISO/IEC 42001 - but be operational and litigation-aware, not legalistic.

The six themes your questionnaire must cover

Below are the six themes the payer used - each explained, with sample questions and the red flags that should kill the deal.

1) Training data lineage: know what your vendor trained on
Why it matters: Knowing the origin, labeling, and consent status of training data is the single most important control for protecting regulated or proprietary inputs (PHI, financial data, IP).
Sample questions:
- Describe the datasets used to train the models delivered to our organization. For each dataset, state origin (customer, public, licensed, synthetic), date(s) of collection, and whether data contained regulated information (e.g., PHI, PCI).
- For any customer data used in training, provide explicit consent and contractual basis that permits such use; specify the timeframe and controls applied.
- Do you use external pre-trained models? List them, their providers, and version hashes if available.
Red flags that should kill a deal:
- Vendor cannot provide dataset provenance or claims "we don't track that."
- Vendor admits to using customer data for training without explicit contractual authorization.
- Training data includes scraped sources with unknown copyright/consent status.

2) Model behavior controls: how they prevent harmful outputs
Why it matters: Models can hallucinate, expose sensitive data, or produce discriminatory outputs. Behavior controls are the operational guardrails.
Sample questions:
- Describe in detail the guardrails implemented to prevent leaking of PII/PHI (prompt filters, response redaction, model abstention).
- What methods do you use for toxicity, bias, and fairness mitigation? Include specific metrics and thresholds.
- Do you provide a configurable safety policy for customers to set allowed/disallowed content? How is it enforced in real time?
Red flags:
- No runtime control or only post-hoc logging without real-time enforcement.
- "We rely on model training to prevent bad outputs" with no evidence of mitigation testing.
- No mechanism to disable learning from a customer's data stream.

3) Evaluation evidence: can they prove their claims?
Why it matters: Vendors make performance and safety claims that executives rely on. You need objective evidence - tests, datasets, and validation methodology.
Sample questions:
- Provide results from internal and third-party evaluations relevant to our use case, including test datasets and evaluation epochs.
- Share information on adversarial testing and out-of-distribution behavior analysis. How often are these evaluations run in production?
- Do you allow independent model audits or provide model cards and data sheets?
Red flags:
- No reproducible evaluation artifacts or refusal to share test datasets under NDA.
- Evaluations limited to proprietary benchmarks with untestable claims.
- No continuous evaluation plan for model drift or data distribution changes.

4) Sub-processor disclosures: who else touches the model/data
Why it matters: AI systems are supply chains. A downstream provider can introduce risk that invalidates your assurances.
Sample questions:
- Provide a list of all sub-processors, model suppliers, and cloud providers used in the development and inference pipeline; include locations and contract termination notice periods.
- Do any sub-processors perform model training, fine-tuning, or access raw customer data? If so, describe controls and contractual terms.
- How do you assess and monitor sub-processor security and compliance?
Red flags:
- Vendor refuses to name critical sub-processors or claims "we can't disclose."
- Sub-processors located in jurisdictions with weak data protections for regulated data without compensating controls.
- Use of open weights or models from unknown third parties with no provenance.

5) Customer data isolation: technical and contractual protections
Why it matters: You must guarantee your data is not commingled and cannot be used to train models without consent.
Sample questions:
- Describe how customer data is isolated at rest and in processing pipelines. Do you support dedicated tenants or single-tenant deployments?
- Do you use customer data for ongoing model updates? If yes, specify opt-in/opt-out mechanisms and the contractual basis.
- Provide details on data retention, deletion procedures, and verification proofs (e.g., deletion certificates).
Red flags:
- Multitenant systems with no tenant isolation and an "opt-out later" training policy.
- No verifiable deletion or retention guarantees.
- Vendor uses default settings to train on customer inputs without a clear opt-in.

6) Incident notification and governance: response must be fast and specific
Why it matters: AI incidents - data exfiltration via outputs, model poisoning, drift causing regulatory breaches - require fast, expert response.
Sample questions:
- Commit to notification timelines for incidents involving data leakage or model misuse (e.g., initial notification within 24 hours, detailed root cause within 7 days).
- Describe your incident response playbook for AI model incidents, including roles, forensics capabilities, and communication protocols.
- Are security events, model misbehavior, and sub-processor breaches included in your SLA and contractual remedies?
Red flags:
- Generic "we will notify" without timelines, or a conflict where vendor can delay notification for business reasons.
- No AI-specific incident handling; incidents are treated like traditional IT outages.
- No willingness to provide forensic artifacts under NDA.

Which red flags should absolutely kill a deal?

- Unverifiable training claims: If a vendor cannot demonstrate where models were trained or refuses to provide lineage under NDA, stop.
- Uncontracted use of customer data: Any admission that customer data may be used for model training without explicit, documented consent is non-starter for regulated sectors.
- No isolation options for regulated data: Multitenant training with no contractual prohibition on re-use is a deal killer for PHI/PCI.
- No real-time behavior controls: If a vendor offers no mechanism to prevent leakage or harmful outputs in production, walk away.
- Opaque sub-processor chain: If critical upstream providers are undisclosed or based in jurisdictions that create regulatory exposure, do not proceed.

How to integrate this into existing TPRM workflows

You don't need an entirely new program. Embed these checks into your existing TPRM gates and procurement lifecycle.

- Reclassify AI vendors by impact: Create AI-specific vendor tiers (e.g., model providers, fine-tuning vendors, inference platforms) and map them to your risk taxonomy (data sensitivity, regulatory exposure, business criticality).
- Add an AI pre-RFP checklist: Before sending an RFP, require prospective vendors to complete the six-theme questionnaire. Throw out bids that fail the prelim screen.
- Update contract templates and SLAs: Work with legal to add clauses on training prohibitions, data reuse, sub-processor approval rights, deletion verification, and AI incident notification timelines.
- Make evidence non-optional: Require model cards, evaluation artifacts, and sub-processor lists under NDA. Put a timeline for remediation or audit access if evidence is incomplete.
- Add technical gates: Require penetration testing for inference APIs, red-team outputs testing, and a runbook for on-demand model snapshotting for forensic purposes.
- Continuous monitoring: Treat AI vendor risk as active - not a one-time checkbox. Add periodic attestations (quarterly), drift reports, and just-in-time audits when model changes are pushed.
- Board and executive reporting: Include AI vendor risk metrics in your TPRM dashboard: number of AI vendors in each risk tier, remediation status, and incident response exercises performed.

Real outcomes: why this approach pays off

In the payer's case the new questionnaire surfaced problems fast. Two vendors were immediately disqualified: one could not prove provenance for its training data, the other refused to sign contract terms disallowing training on customer PHI. Three vendors agreed to contract amendments - including explicit prohibitions on training with customer data, independent audit rights, and 24-hour incident notification. The payer avoided regulatory fines and rebuilt trust with customers while keeping strategic partners that were willing to adopt responsible practices.

Treat governance as an enabler, not a drag

The intuition many executives feel - that tighter controls slow innovation - can be true if controls are ad hoc or fear-driven. But the right questionnaire makes procurement a source of speed. Vendors that survive this scrutiny are more reliable, require fewer product interruptions, and are less likely to drag your org into public incidents. Governance that focuses on data lineage, model behavior, evaluation evidence, sub-processors, isolation, and incident response reduces uncertainty and makes it easier to scale AI where it creates the most economic value.

Conclusion - one concrete readiness move

If you only do one thing this quarter: replace your AI vendor security supplement with a six-theme questionnaire and run it against your top 10 AI vendors within 30 days. Use the results to classify vendors into "continue," "remediate," or "terminate" buckets and require contract changes for anyone in remediate. Add a quarterly attestation and a one-page AI risk playbook to your TPRM portal. Invite legal, compliance, and product risk into a single review - and treat a clear "no" as equal to a failed SOC2.

Being AI-economy-ready means governing AI so it accelerates your business rather than becoming a recurring liability. Start with questions that reveal how models are built and behave - not more questions about firewalls.