AI Governance in Procurement

The third-party AI risk landscape

Third-party AI risk is materially different from traditional software procurement risk. When an organisation procures a conventional software tool, the tool does what it is configured to do, predictably. When an organisation procures an AI-powered tool, the tool's behaviour may change over time as models are updated, retrained, or replaced by the vendor. An AI feature that performed acceptably at the point of procurement may drift in ways the procuring organisation cannot observe without active monitoring.

The risk is compounded by opacity. Many AI vendors treat their models as proprietary and provide limited visibility into how decisions are made, what data was used for training, or how model updates are tested before deployment. The procuring organisation is accountable for the AI's outputs but may have limited ability to understand or challenge how those outputs were produced.

Under the EU AI Act, the deployer of a high-risk AI system (typically the procuring organisation, not the vendor) bears specific obligations around human oversight, transparency to affected individuals, and monitoring. These obligations cannot be delegated to the vendor by contract alone; the deployer must be able to demonstrate that they are met in practice.

What to include in AI vendor contracts

Effective AI governance in procurement requires contractual provisions that go beyond standard software licensing terms. These provisions should cover several categories.

Transparency requirements: the vendor should disclose where AI is used within their product, what data the AI processes, and what decisions it makes or informs. The procuring organisation needs this information to meet its own transparency obligations to affected individuals.

Model change notification: the vendor should notify the procuring organisation before making material changes to AI models that affect the organisation's use case. This includes model retraining, replacement with a different model architecture, or changes to the training data composition. The procuring organisation needs time to assess whether the change affects its risk profile.

Data governance: the contract should specify what data the vendor's AI may process, whether that data is used for model training, how data is retained and deleted, and what happens to data on contract termination. For organisations processing personal data, these provisions interact directly with GDPR data processing agreements.

Performance and fairness: the contract should include service levels for AI accuracy, provisions for bias testing and reporting, and the organisation's right to audit or require independent testing of the AI system's fairness characteristics.

Incident reporting: the vendor should be obligated to report AI-specific incidents (model failures, bias detections, data quality issues) within defined timeframes, distinct from general service incident reporting.

Assessing vendor governance maturity

Before procurement, organisations should assess the AI governance maturity of prospective vendors. This assessment goes beyond standard security questionnaires. Key areas to evaluate include: whether the vendor has a documented AI governance framework; whether they conduct bias testing and can share results; whether they maintain a model change log; whether they have an AI incident response process; and whether they can provide the technical documentation needed for the procuring organisation's compliance obligations.

This due diligence is particularly important for high-risk AI deployments. If the AI system will make or significantly inform decisions affecting individuals (recruitment, credit, insurance, healthcare), the procuring organisation's accountability is heightened, and the vendor's governance maturity becomes a material procurement criterion.

Building procurement governance into your framework

Third-party AI governance should be integrated into the organisation's existing procurement workflow rather than treated as a separate process. This means adding AI governance criteria to the vendor evaluation process, including AI-specific clauses in standard contract templates, and establishing an AI review checkpoint in the procurement approval workflow.

Several Veridio governance documents support this integration. The Third-Party AI Dependency Register tracks vendor AI systems and their governance status. The AI Contract Clause Library provides ready-to-use contractual provisions covering the categories described above. The AI Risk Classification Framework helps determine which procured AI systems warrant enhanced governance scrutiny.