When the Best Vendor Isn’t the One That Delivers
- Darren Emery
- Oct 13
- 6 min read
What CTOs Get Wrong About Supplier Selection
CTOs obsess over feature checklists and price, then wonder why the roadmap slips. The real mistake isn’t who you chose, it’s how you judged their ability to deliver. Data, a cautionary tale, and a simple framework to fix it.
If your roadmap is late, your board is nervous, and your teams keep promising “we’ll fix it next sprint,” pause before you blame scope or “bad sprint discipline.” As a CTO you weren’t hired to shop; you were hired to deliver outcomes.
Unfortunately, procurement decks make vendors look like products. They list features, SLAs and glossy references. But they rarely answer the question that matters most once the real work begins: Can this supplier deliver, week-in, week-out, against your cadence and quality standards?
This mistake - treating vendor selection like feature selection - is why so many transformations stall. A large body of research shows that the majority of digital transformations fail to meet their objectives, and a non-trivial percentage of IT projects blow up badly enough to threaten the actual existence of the organisation itself. Those aren’t abstract numbers; they’re expensive, reputation-breaking realities for companies that underestimated delivery risk.
Below: an engaging real-world cautionary story, the data-backed logic of what CTOs miss, a practical evaluation template, and a simple vendor selection playbook you can use in your next procurement cycle.
A short cautionary story: Target’s Canadian debut - not a strategy failure, an execution issue
UK readers may not know the story of US discount retailer, Target.
Between 2013–2015, the firm decided to expand into Canada. And, that expansion failed spectacularly.
The story is often told as a geography + pricing + brand mismatch. That’s true, but the crash-landing was the result of execution and delivery failures in the supply chain and IT systems: inventory systems that misreported stock, distribution centres that couldn’t replenish shelves reliably, and a catalogue of integration and timing problems. The result: empty shelves in newly opened stores, angry customers, and a withdrawal costing billions.
Fast forward to your world. Imagine you’re a fintech CTO and you select a “best-in-class” vendor to build your payments reconciliation and customer dispute microservice. They have great references, low cost, and promise a 6-month delivery. You strap the module onto your backlog and plan the Q3 merchant onboarding.
Two things happen:
Their sprint throughput is half what they promised; milestone commitments slip.
They deliver code that passes unit tests but doesn’t integrate cleanly with your reconciliation engine or your fraud monitoring pipeline, and so you end up with duplicate transactions and customer disputes.
The cost? Rework, extra staff to manage exceptions, delayed merchant rollouts and a near-miss with the regulators. If that vendor had been evaluated on delivery capability - throughput history, integration tests with third-party services, CI/CD maturity, staff stability - you might have chosen differently or staged the work as a time-boxed, end-to-end slice of functionality.
Target’s newspaper headlines traded brand for execution. Your headline will be different, but the cause is the same: the vendor that looked good on slides could not deliver at scale and in context. Use these insights now, not as bullet-points in the lessons learned.
The core mistake: vendor features ≠ vendor delivery capability
Procurement bias shows up in three predictable ways:
The team chooses vendors that shine in demos or check-boxes (features).
Contract negotiation secures the financial risk and liabilities, but rarely mandates the demonstrable proof of weekly delivery.
Proof of concept becomes prescribed: a “demo” sprint rather than a representative, integrated slice of work.
That’s a problem because software and systems aren’t static components you buy and bolt on. Instead, they operate inside a socio-technical system - your teams, pipelines, governance, data, and customers.
A vendor's technical brilliance counts for little if they cannot integrate seamlessly with your operational cadence: your reporting cycles, incident response protocols, data governance, and regulatory timelines.
If you find yourself whispering “why is my digital transformation failing,” check whether vendor delivery capability was under-weighted in the original selection. Too often it is.
A few data points you can’t ignore
• About 70% of digital transformation programs fail to meet their objectives.
That pattern repeats across multiple industry studies and large consultancies. The primary cause is execution and capability gaps, not vision alone.
• A non-trivial minority of large IT projects fail so badly they threaten the company.
McKinsey/Oxford work highlighted that roughly 17% of large IT projects can go catastrophically wrong. That’s not theoretical risk; it’s existential.
• Vendor/outsourcing engagement failure rates are high.
Studies and industry analyses indicate a large portion (commonly cited ranges of 25–50%) of outsourced software efforts under-perform due to misalignment, communication and execution problems. Treat those ranges as a wake-up call, not a comfort.
Whilst those figures should provoke serious reflection within procurement, it is entirely possible to build processes that materially reduce the risk.
Two dimensions every CTO must score - and how to measure them
When you evaluate a vendor, score them on Supplier Potential and Delivery Capability. Give delivery capability at least 40–50% of the total weight.
Delivery Capability - scoring checklist (hard metrics you can request)
Throughput history - show three months of completed stories or delivered features, with scope/context.
Schedule adherence - % of committed milestones delivered on time over the past year.
Escaped defects - defects per release / per 1,000 story points.
Time-to-fix critical issues - mean time to resolution for P1 / P2 incidents.
CI/CD & test automation coverage - frequency of deployments, percentage of automated tests in pipeline.
Staff stability & bench - turnover rate of the delivery team, coverage plan for sickness/attrition.
Integration maturity - examples of previous integrations with third-party APIs similar to yours (and their test harnesses).
Transparency & governance - do they provide live dashboards, a RACI for escalation, and clear SLAs for defects and handover?
Reference evidence - ask for references that closely match your scale and domain, and validate delivery metrics with them, not just the sales team.
Demand the raw data - not marketing summaries. If they can’t or won’t provide it, move on.
Supplier Potential - what still matters (but shouldn’t dominate)
Domain knowledge, architecture fit, security certifications, commercial terms and cultural fit. These are important. But they’re not the decider.
A short procurement playbook (a reasonable next procurement cycle)
Proof sprint (4–8 weeks): Require a paid, representative vertical slice - integrated with your dev pipeline, CI/CD, and monitoring. Measure throughput, defects, and collaboration behaviours. This kills illusion.
Delivery dossier: Require the vendor to submit a delivery dossier: historical metrics, a sample sprint backlog, a demo environment, and a contingency plan.
Weighted scoring: Make delivery capability ≥ 40–50% of the scorecard. Walk away if they fail the dossier minimums.
Contract levers for delivery: Include clear acceptance criteria for increments, transparency obligations (dashboards), and constructive SLAs tied to milestones - not punitive only.
Co-governance model: From day one, put vendor leads into regular joint planning, retros, and risk calls. Make them part of your rhythm.
Escape routes: Architect so the vendor’s work is modular and portable. You want the ability to switch partners without a system rewrite.
Post-award proof: First 3 months are probationary - if delivery metrics underperform consistently, you have a modular exit or re-plan clause.
If this reads heavy - it’s supposed to. It’s far lighter and cheaper than the alternative: a mid-programme rescue or a public failure.
Realistic Case Example
Context: mid-sized fintech wants to roll out an embedded payments capability for merchants (onboarding, reconciliation, chargeback management, settlement).
Core requirements: PCI compliance, 99.9% transaction integrity, phased merchant onboarding (5, 50, 500 merchants across 12 months), and integration with existing fraud and reconciliation engines.
What many CTOs do: pick a vendor with a competitive price and impressive references. They write a contract and plan a 6-month implementation.
What the delivery-capability approach looks like:
Pilot Phase (6 weeks): This must include the merchant onboarding screen, process a test payment through a staging environment, run a reconciliation task, and integrate seamlessly with your existing fraud checks and system monitoring. The entire slice must be deployed using your established engineering pipeline.
Metrics required from pilot: average cycle time per story, number of escaped defects in pilot, time to resolve P1 incidents, and successful live transactions in staging environment.
Pass/fail thresholds: ≥ 80% of committed stories delivered; < 2 escaped defects; P1 resolved within 24 hours; A successful, zero-data-loss migration from the staging environment to the live production platform.
Governance: weekly steering calls with product owner, vendor delivery lead, and engineering lead; shared dashboard; joint retros.
Contract: phased delivery timeline, clear acceptance criteria, and a rollback/portability clause covering both the source code and data schemas.
Outcome if the vendor passes: scale with confidence, onboard merchants in stages with reduced operational risk.
Outcome if vendor fails the pilot: swap vendors early or transition the subsequent work back internally - you’ve avoided months of disruption and regulatory risk.
This is not theoretical - it’s replicable and budget-light. You invest 4–8 weeks for evidence rather than 6–12 months chasing unknown risk.
