TwinLadder Weekly

Issue #29 | March 2026

Editor's Note

Last month I sat in a conference room in Rotterdam, watching a European logistics company's procurement team evaluate AI vendor responses for a new freight optimisation platform. Four vendors, four glossy proposals, four promises of "full EU AI Act compliance." I was there as an advisor. Within twenty minutes, I knew we had a problem.

The procurement team had a solid IT evaluation framework -- uptime guarantees, data residency, API documentation, SLA terms. They knew how to buy software. What they did not know was how to buy AI. Not one of the four vendor proposals specified the training data used in their models. Not one disclosed a measured error rate for the company's specific use case. Not one explained, in operational terms, how a warehouse manager should verify the system's routing recommendations before acting on them. And the procurement team had not asked for any of this -- because they did not know they should.

When I raised these gaps, the head of procurement said something that has stayed with me: "We assumed the vendor handles the AI part. We just need it to work."

If that sounds reasonable to you, keep reading. Because that sentence captures the single most dangerous assumption in European AI procurement today. Under the EU AI Act, the vendor does not handle the AI part. You do. And what you buy either makes that obligation manageable or makes it impossible.

The Procurement Blind Spot: When Buying AI Means Buying Compliance Risk

Why Article 4 Changes Everything About How You Evaluate AI Vendors

Alex Blumentals, with legal analysis by Liga Paulina and technical analysis by Edgars Rozentals

Here is a number that should concern you if anyone on your team buys AI tools: only 14% of organisations have formal AI vendor evaluation frameworks that go beyond standard IT procurement criteria. [cite:gartner-ai-vendor] Meanwhile, 65% of organisations now regularly use generative AI -- nearly double from ten months prior. [cite:mckinsey-ai-procurement]

14% | Organisations with AI-specific vendor evaluation frameworks | highlight
65% | Organisations regularly using generative AI | ↑2x in 10 months
18% | Organisations that have adapted procurement for AI risks | highlight

The gap between AI adoption speed and procurement sophistication is not a nuisance. It is a compliance liability that sits in your organisation right now.

You are almost certainly still treating AI purchasing like any other software procurement. Feature comparison, price negotiation, security review, contract signature. The vendor says "AI-powered." Your team says "sounds good." Nobody asks what that means in practice -- what the model can and cannot do, where it fails, how your staff should verify its outputs, and whether the documentation supports your legal obligations as a deployer.

Article 4 changes this equation entirely. [cite:eu-ai-act-article4] The regulation places AI literacy obligations on the deployer -- your company, the one that buys and uses the AI system, not the company that built it. This means every AI tool you purchase either helps you meet your Article 4 obligations or creates a gap you must fill yourself. And your procurement team almost certainly has no way to tell which.

What your vendors are not telling you

Let me be specific about what I have observed in vendor proposals and sales processes across Europe over the past six months. I have reviewed or advised on AI procurement for eleven organisations in seven countries. The patterns are consistent enough that you will recognise them.

"AI-powered" is not a feature specification. It is a marketing term. When a vendor tells you their contract review platform is "AI-powered," that tells you approximately as much as a car manufacturer saying their vehicle is "engine-powered." Every vendor proposal I reviewed used the phrase. Not one defined it with technical precision. Does "AI-powered" mean a large language model? A fine-tuned classifier? A rule-based system with a machine learning layer? The answer matters enormously for understanding reliability, limitations, and compliance requirements -- and it is almost never provided.

I shared the vendor claims from the Rotterdam meeting with Edgars over coffee the following week. He barely glanced at the proposals before responding. "If a vendor cannot explain their AI architecture in two sentences," he said, pushing the papers back across the table, "they either do not understand their product or do not want you to understand it. Both are disqualifying."

Training data provenance is rarely disclosed. Of the eleven procurement processes I observed, only two vendors provided any information about what data their models were trained on. Neither provided sufficient detail to assess whether the training data was representative of the deployer's specific domain. The Stanford Foundation Model Transparency Index found an average transparency score of just 37 out of 100 across major providers, with training data documentation among the weakest areas. [cite:stanford-hai-transparency]

Error rates are almost never published. I asked every vendor I encountered a simple question: "What is the measured error rate of your AI system for this client's specific use case?" Not one could answer. Some offered generic accuracy benchmarks from controlled test environments. None had domain-specific error data for the industries they were selling into. One sales representative told me, memorably, "Our system is highly accurate." When I asked for the number, he said, "We don't publish that."

Try this: go back to the last AI vendor proposal your team evaluated. Search for the words "error rate," "failure mode," or "limitation." If you find nothing specific, you bought a system whose weaknesses nobody disclosed -- and nobody on your side asked about.

"Compliance" is marketing, not a legal guarantee. Seven of the eleven vendor proposals I reviewed claimed some form of "EU AI Act compliance." I asked Liga to review three of these compliance claims in detail. She spent an evening with them and called me the next morning. "Every compliance claim I examined was either a statement of intent -- 'we are committed to compliance' -- or a reference to features that partially address one article of the regulation while ignoring others," she said. "No vendor I reviewed offered a contractual guarantee that their system, as deployed, would satisfy the deployer's specific obligations under Articles 4, 13, 14, or 26."

This last point is critical. The vendor's compliance is not your compliance. Even if a vendor's AI system is perfectly compliant from the provider's perspective, that does not discharge your separate obligations as a deployer. And your obligations -- particularly the Article 4 literacy requirement and the Article 26 human oversight requirement -- depend entirely on how the system is used, by whom, and with what understanding. [cite:eu-ai-act-article26]

The deployer's dilemma

Liga and I spent an afternoon mapping the legal architecture for deployers. She pulled out her marked-up copy of the regulation and laid it out plainly. "The EU AI Act creates a shared responsibility model," she said. "The provider must build a transparent, documented system. But the deployer must ensure that the people using that system understand it well enough to use it appropriately. [cite:eu-ai-act-articles13-14] These are separate obligations, and they cannot be satisfied by the same actions. A vendor's product documentation does not automatically become your AI literacy programme."

She then walked me through three specific deployer obligations that your procurement team is almost certainly overlooking. Read the middle column against the right-hand column. The gap between what the regulation demands and what your team currently asks is where your compliance risk lives.

Obligation	AI Act Article	What It Requires of You as Deployer	What Your Procurement Team Probably Asks
AI literacy	Article 4	Your staff must have sufficient understanding of AI system capabilities and limitations	"Does the vendor provide training?" (checkbox)
Human oversight	Article 26	Your staff assigned to oversight must have necessary competence, training, and authority	"Does the system have a human-in-the-loop?" (feature check)
Transparency to affected persons	Articles 13-14	You must be able to explain AI-assisted decisions to affected individuals	"Is the system explainable?" (yes/no)

Be honest: does the right-hand column describe your current procurement process? If it does, you are evaluating AI tools with questions designed for traditional software. That is like using a vehicle safety checklist to evaluate an aircraft.

The five questions every AI vendor RFP should ask

Based on what I have observed -- the procurement processes that went well, the ones that produced regret, and the regulatory guidance emerging across Europe -- here are five questions that should appear in every AI vendor evaluation you conduct. None of them are standard in current procurement frameworks. All of them are necessary.

1. What are the documented limitations of your AI system in our specific use case?

Not generic limitations. Not "AI can sometimes make errors." Specific, documented, use-case-relevant limitations. If you are buying a contract review tool, what types of clauses does it consistently miss? If you are buying an HR screening system, what candidate profiles generate unreliable results? If the vendor cannot answer this question with specifics, they have not tested their system in your domain -- or they have tested it and do not want you to see the results.

I shared this framework with Edgars, and he suggested a practical test you can run immediately. "Ask the vendor to provide three examples of cases where their system produced incorrect or misleading results in a comparable deployment," he said. "If they say there are none, they are either not measuring or not being honest. Both are problems."

2. What training does your platform provide to help our staff understand AI output reliability?

Article 4 requires you, as the deployer, to ensure sufficient AI literacy. [cite:eu-ai-act-article4] If the vendor provides training, that training needs to go beyond product tutorials. It needs to teach your staff how to assess whether the AI output is reliable in their specific context. Ask for the training curriculum. Ask how it addresses limitations and failure modes. Ask whether it differentiates by user role and expertise level.

If the vendor's training is a 30-minute onboarding video followed by a quiz about interface features, that is product training, not AI literacy training. You will need to budget for the gap.

3. What is your measured error rate for our domain, and how was it established?

This question is uncomfortable for vendors. Ask it anyway. A vendor that cannot provide domain-specific performance data is asking you to deploy their system based on faith. The measurement methodology matters as much as the number -- was it tested on a representative sample of your data? By whom? Under what conditions?

4. How does your system support human oversight as required by the EU AI Act?

Not "does your system have a human-in-the-loop option?" That is a feature question. The compliance question is: how does the system's design enable the specific oversight capabilities Article 26 requires? [cite:eu-ai-act-article26] Can your human overseer understand the AI's reasoning? Can they override the system? Can they identify when the system is operating outside its reliable parameters? How does the interface design support -- not just permit -- meaningful human oversight?

5. What documentation do you provide that we can use as Article 4 compliance evidence?

Ask this explicitly. You need to demonstrate compliance. That means documentation -- about the system's capabilities and limitations, about the training provided to your users, about human oversight mechanisms, about how your organisation ensures ongoing AI literacy. If the vendor's documentation is designed for their own compliance rather than yours, you have a gap that your legal and compliance teams will need to fill.

The vendor competence framework

These five questions are a starting point. But individual questions are not a framework. What you need -- and what almost no organisation currently has -- is a structured methodology for evaluating whether an AI vendor supports or undermines your Article 4 obligations.

Over the past three months, working with procurement teams, legal advisors, and technical evaluators across seven organisations, I have developed a scoring framework that applies TwinLadder's AI competence methodology to the vendor evaluation context. It assesses vendors across five dimensions. Use it as a self-assessment tool: score your most recent AI vendor against each row.

Dimension	What It Evaluates	Compliance-Supporting	Compliance-Neutral	Compliance-Undermining
Transparency	How clearly the vendor documents system capabilities, limitations, and architecture	Detailed technical docs, domain-specific limitation disclosures, architecture explanation	Generic product documentation, high-level accuracy claims	"Proprietary" response to technical questions, no limitation disclosure
Training support	Whether vendor training builds genuine AI literacy or just product proficiency	Role-specific training, failure case studies, verification exercises	Product tutorials, feature walkthroughs, generic "AI awareness"	No training, or training that positions AI output as reliable by default
Documentation quality	Whether vendor docs serve your compliance needs	Deployer-facing compliance documentation, Article 4 mapping, oversight guides	Provider-focused documentation only, no deployer compliance guidance	No documentation, or documentation with disclaimers that shift all responsibility
Error transparency	Whether the vendor measures and discloses performance data	Published domain-specific error rates, ongoing monitoring data shared with you	Generic accuracy benchmarks, controlled-environment metrics only	No error data, or claims of near-perfect accuracy without supporting evidence
Human oversight design	Whether the system's UX supports meaningful human control	Interface designed for verification, uncertainty indicators, override mechanisms	Basic approval/rejection workflow, no uncertainty signals	Automated defaults, friction to override, confidence displays without calibration

Each dimension can be scored on a 1-5 scale. A vendor that scores 3 or above across all five dimensions actively supports your Article 4 posture. A vendor that scores below 3 on any dimension creates a compliance gap you must fill with your own resources -- training, documentation, oversight protocols -- at your own cost. A vendor that scores 1 on transparency or error transparency is, in my assessment, not suitable for deployment in an Article 4 environment without substantial internal mitigation.

This is not abstract. I have watched procurement teams use a version of this framework in live evaluations. It changes the conversation. When you tell a vendor, "We scored your training support at 2 out of 5 because your onboarding programme does not address system limitations," you get a very different response than when you ask, "Do you provide training?" The specificity creates accountability.

Try this: take your most recently purchased AI tool and score the vendor against these five dimensions right now. Where they score below 3, that is a compliance gap your organisation currently owns -- whether you have budgeted for it or not.

What Europe is already doing

The smartest procurement frameworks I have seen are emerging from public sector organisations -- not because governments are more sophisticated buyers, but because public procurement rules force transparency that the private sector can avoid. The question is whether you will wait for these standards to become mandatory, or adopt them before your competitors do.

The Dutch government maintains a public algorithm register that requires government agencies to document AI systems including purpose, data sources, human oversight mechanisms, and known limitations. [cite:dutch-algo-register] More importantly, Dutch public procurement frameworks now require vendors to complete transparency documentation before contract award. The Netherlands was early on this. Other jurisdictions are following.

Germany updated its federal procurement guidelines in 2025 to include AI-specific evaluation criteria. [cite:german-procurement-ai] Vendors bidding on public contracts must provide technical documentation covering training data provenance, known error rates, and human oversight design. The Bitkom industry association published complementary guidance helping procurement teams evaluate these submissions -- recognising that asking for the documentation is only half the battle. You also need to know how to read it.

The European Commission published procurement guidance recommending that public buyers evaluate AI systems on transparency, explainability, and human oversight capabilities -- not solely on functionality and price. [cite:ec-ai-procurement] The guidance is advisory, not binding. But it signals where the regulatory expectation is heading.

In the Nordics, I have observed private-sector organisations building their own vendor assessment frameworks. A Swedish financial services company developed a 40-point AI vendor evaluation checklist that includes questions about model retraining schedules, data retention policies, and deployer notification requirements when the model changes. A Finnish healthcare organisation now requires vendors to conduct a "failure mode workshop" as part of the procurement process -- a structured session where the vendor walks the buyer through scenarios where their AI system is known to perform poorly.

The Baltic picture is less mature. In Latvia, Lithuania, and Estonia, most organisations I work with are still using standard IT procurement frameworks for AI purchases. The regulatory infrastructure is developing -- the Latvian CDPC has signalled interest in AI procurement guidance -- but the practical tools are not yet in place. This is a gap that will close, but it has not closed yet. If you operate in the Baltics, you have a window to get ahead of the curve -- but it is narrowing.

The market that does not yet exist

Here is where this analysis becomes uncomfortable, because it points to a capability gap that no existing service adequately fills -- and it may describe your own organisation.

Your procurement team does not have AI literacy. Not in the Article 4 sense. They understand purchasing. They understand vendor management. They do not understand, in most cases, how to evaluate whether a vendor's AI system will support or undermine your compliance posture. They cannot assess training data quality. They cannot evaluate whether an error rate is acceptable. They cannot determine whether a vendor's human oversight design is genuine or cosmetic.

This is not a criticism of your team. It is a structural problem. Article 4 created an obligation that cuts across every function that uses AI -- including the function that buys it. Your procurement team is not exempt from the literacy requirement. And yet procurement-specific AI competence is almost entirely absent from the training market, from consulting offerings, and from regulatory guidance.

I was on a call with Edgars last week when this came up again. "I regularly get calls from procurement teams asking me to evaluate an AI vendor's technical claims," he said. "They know enough to sense that the vendor's pitch does not add up. They do not know enough to articulate why. That gap -- between intuition that something is wrong and the technical knowledge to identify what -- is where organisations get hurt."

Ask yourself: if your procurement team evaluated an AI vendor last month, could any of them explain what a hallucination rate is, why training data provenance matters, or how to tell genuine human oversight from a cosmetic checkbox? If the answer is no, you have identified your next training priority.

The emerging role is something like an "AI procurement assessor" -- someone who can sit between your procurement team, your legal team, and the technical evaluation to ask the questions that none of those functions, individually, knows to ask. Some organisations are building this capability internally. Most are not. The consultancies that serve procurement functions have been slow to develop AI-specific evaluation competence. The AI vendors themselves have an obvious conflict of interest.

This is, candidly, where we see TwinLadder's assessment methodology extending naturally. Our competence framework -- the same seven pillars and four maturity levels we use for internal AI literacy assessment -- applies directly to vendor evaluation. If you can assess your own organisation's AI competence, you can assess whether a vendor supports or undermines it. The same diagnostic questions, the same evidence requirements, the same scoring methodology. Applied outward instead of inward.

We are building this. I mention it not as a pitch but as a disclosure -- because the editorial observation and the product development come from the same analysis, and you should know that.

The Competence Question

Your insurance company buys an "AI-compliant" claims assessment tool in early 2026. The vendor's proposal emphasises speed -- 70% faster claims processing -- and includes a section titled "EU AI Act Compliance" that describes the provider's internal governance framework. Your procurement team evaluates the tool on processing speed, integration requirements, and cost. They check the compliance section, confirm it exists, and move forward.

Eight months later, a policyholder disputes a claim denial. The claim was flagged by the AI system as potentially fraudulent and routed to a human reviewer on your team, who affirmed the denial. The policyholder's lawyer asks a straightforward question: "On what basis did the AI system flag this claim?" Your reviewer cannot answer -- the system provided a fraud probability score but no explanation of the underlying factors. The training the vendor provided covered how to read the dashboard and process claims efficiently. It did not cover how to interrogate the system's reasoning or when to override it.

The policyholder files a complaint with the national data protection authority. The authority asks your company to demonstrate that the staff involved had "sufficient AI literacy" to understand the system's capabilities and limitations -- the Article 4 standard. [cite:eu-ai-act-article4] You produce the vendor's training certificates. The authority asks a follow-up question: "Did the training address the specific limitations of the AI system in claims fraud detection? Can the reviewing staff explain how the model generates its risk scores?" You cannot demonstrate this. The vendor's training was product training, not competence training. And your procurement team never asked for the difference.

The Article 4 obligation falls on you -- the deployer -- not the vendor. The compliance gap was built into the purchase.

When was the last time your procurement team asked an AI vendor: "What will our staff still not understand after your training programme?"

What To Do

Add Article 4 requirements to your standard AI vendor RFP template. Include the five questions above -- limitations disclosure, training adequacy, error rates, human oversight design, and compliance documentation. Do not accept "we are committed to compliance" as an answer. Require specifics. If your RFP template does not have an AI-specific section, you are evaluating AI tools with a framework designed for traditional software. That is like using a vehicle safety checklist to evaluate an aircraft.
Require vendor transparency documentation before purchase, not after. The time to discover that a vendor cannot explain their model's limitations is during evaluation, not after deployment. Request the vendor's technical documentation -- architecture description, training data summary, known limitations, performance benchmarks -- as a prerequisite for advancing to commercial negotiations. If the vendor declines, that is your answer.
Test vendor claims with domain-specific scenarios. Do not accept generic demonstrations. Provide the vendor with test cases from your actual operations -- representative documents, realistic edge cases, known-difficult scenarios. Ask them to run their system against your data and share the results, including errors. A vendor that refuses domain-specific testing is asking you to buy a product they have not validated for your use case.
Budget for the competence gap your vendor does not close. If the vendor's training covers product features but not AI literacy in the Article 4 sense, budget for supplementary training. If the vendor's documentation does not map to your deployer obligations, budget for creating that mapping internally. The cost of filling these gaps should be part of your total cost of ownership calculation -- not a surprise discovered six months after deployment. Try this: take your most recent AI vendor contract and calculate the true cost -- purchase price plus the training, documentation, and oversight infrastructure you had to build yourself. That is the real price of buying AI without procurement competence.

Quick Reads

EU Guide for Public Procurement of AI -- official procurement playbook, written for public sector but broadly applicable.
Stanford HAI, Foundation Model Transparency Index -- average transparency score: 37 out of 100.
Dutch Government Algorithm Register -- Europe's most mature public AI transparency initiative, with documentation requirements that mirror good procurement practice.
McKinsey, The State of AI in 2025 -- 65% adoption, 18% procurement adaptation.
EU AI Act, Article 26 — Deployer Obligations -- competence, training, and limitation-awareness obligations apply regardless of vendor promises.

One Question

Your AI vendor says their system is "EU AI Act compliant." Can anyone on your procurement team explain what that claim actually means -- which specific articles it covers, which deployer obligations it leaves unaddressed, and what competence gaps your organisation still owns? If the answer is no, you are buying compliance risk and calling it compliance.

TwinLadder Weekly | Issue #29 | March 2026

Helping professionals build AI capability through honest education.