Procuring AI is not like procuring standard software. Standard software does what it is programmed to do. AI systems produce outputs that may vary, may be wrong in systematic ways, may reflect biases in their training data and may degrade over time. Responsible procurement requires asking questions that most traditional vendor evaluation processes were not designed to address.
What Makes AI Procurement Different
Three characteristics of AI systems make procurement more complex. First, AI outputs are probabilistic, not deterministic — the same input may produce different outputs, and the model may be confident even when wrong. Second, AI systems reflect their training data — if the data is biased, incomplete or unrepresentative, the model's outputs will be too. Third, AI systems require ongoing oversight — they do not stay calibrated on their own.
The Checklist
Before procuring any AI system, seek clear answers to the following questions from vendors:
On data and training: What data was used to train this model? How was that data collected and validated? Is the training data representative of the population that will be affected by this system's outputs? How is the model updated when new data becomes available? What data from our organization will be used to improve the model?
On performance and validation: How is model performance measured? What is the error rate in realistic conditions (not just benchmark conditions)? Has the model been tested on populations similar to ours? What happens when the model is uncertain — does it indicate low confidence or provide a high-confidence wrong answer?
On transparency and explainability: Can the system explain why it produced a specific output? Is there an audit log of decisions made with AI assistance? Can we inspect the model's behavior on our own data?
On risk and accountability: What is your process when the model produces harmful or incorrect outputs? Who is responsible within your organization for model performance? What are the contractual remedies if systematic errors cause harm?
On data privacy: How is our organizational data isolated from other customers' data? Does our data contribute to model training shared with other customers? What data access rights does your organization retain?
Our full AI governance checklist provides additional structured questions for organizational use, and our AI governance framework guide covers the broader governance architecture.