The room already knew
We opened a conference session last October with a question for the audience: why did your last AI pilot stall? Nobody blamed the model. The answers came quickly and they clustered. Nobody owned the system after the demo. The data was not production grade. Governance slowed every change. The people meant to use it did not. And the cost climbed as soon as the volume did.
That is the pattern we see in our own work. A pilot succeeds, a steering group agrees it should go further, and six months later it is still a pilot, or it has been quietly switched off. The technology did what it was asked to do. The organization around it was never asked to do anything.
What a pilot is allowed to borrow
A pilot exists to prove that something can work, as quickly and as cheaply as possible. To do that, it borrows five things that production will have to own:
- An owner. The sponsor of the experiment, who is rarely the person accountable for the process once it runs.
- Data. A clean extract prepared by hand, not the live feed with its gaps, delays and changes of format.
- Permission. A pass on the review a production system would face, on the understanding that it is only a test.
- Users. Volunteers who are curious about it, not the team whose work it changes.
- A budget. A few weeks of effort, with no running cost to defend.
None of this is a mistake. It is what makes a pilot fast. The failure comes later, when the pilot is declared a success and all five loans fall due at once.
The pilot did not fail. It was never set up to become anything else.
Five practices, and a test for each
The session set out five practices. Each one repays a loan before it falls due, and each comes with a test you can apply today to the pilot you already have.
An owner with a budget, a measure and the right to decide
- Where it breaks
- The sponsor moves on, and nobody inherits the decision to fund the system, change it or stop it.
- What to do instead
- Name the executive who owns the outcome, with a budget line, the two or three measures the system must move, and the authority to change the process around it.
- The test
- Can you name the person who can stop it, and the number they will be judged on?
The decision behind it: Value & Economics: what is it worth, and what will it cost?
Data that holds up in production, not just in the demo
- Where it breaks
- The pilot ran on an extract someone cleaned by hand. The live feed arrives late, with gaps, and in a different shape.
- What to do instead
- Run the proof on the feeds production will use, on the schedule it will use them, and fix the integration before a launch date depends on it.
- The test
- Is the pilot reading the same data, as fresh, as the production system will?
The decision behind it: Data Readiness: is it the model, or the data underneath it?
Governance that is lean, and set by risk
- Where it breaks
- Every change waits for the same review, so either the review is skipped or the system stops moving.
- What to do instead
- Sort use cases by what they decide and whom they touch. A low-risk tool clears a short review; anything that affects a customer, an employee or a regulated decision gets the scrutiny it needs, and the evidence is kept.
- The test
- Does a low-risk change clear review in days, while a high-risk one is still examined properly?
The decision behind it: Governance & Accountability: if it is wrong, who owns the decision?
Adoption planned from the start
- Where it breaks
- The tool launches, a training session runs, and the work carries on the old way.
- What to do instead
- Put the people whose work will change into the pilot, redesign the workflow with them, and decide in advance how you will know they are using it.
- The test
- Have the people who will use it changed how they work during the pilot, or only tried it?
The decision behind it: Adoption & Workforce: you launched it; are people working differently?
Run cost, reliability and value, tracked for each workflow
- Where it breaks
- The pilot cost a few weeks of effort. In production the cost grows with every request, and nobody priced that.
- What to do instead
- Measure what one answer costs at production volume, how often it is right, and what it is worth to the workflow it serves. Review all three on a schedule.
- The test
- Do you know what the system costs per answer at full volume, and what that answer is worth?
The decision behind it: Pilot to Production: why did the pilot work, but production did not?
None of the five is a technology decision. Each is a decision someone in the business has to make, about money, data, risk, people or cost, and each is cheaper to make before the demo than after it. A pilot that has answered all five before it starts is not slower. It simply has somewhere to go when it works.
The pilot-to-program ladder
The session’s second idea was a ladder: five rungs from the first conversation to a program that scales, each with a gate you can measure. A system moves up a rung only when it clears the gate, and the gates are agreed before the work starts, not after the demo has made everyone optimistic.
1
Discovery
One workflow, one owner, and the measure it should move.
GateWhat it is worth is written down before anything is built.
2
Proof
A working system on real data, judged against criteria agreed in advance.
GateThe exit criteria are met: quality, data and cost per answer.
3
Hardening
About two sprints on everything the demo skipped.
GateProduction feeds, evaluation, access controls, monitoring and a fallback are in place.
4
Production
Live with two or three early teams.
GateEach team has a named service owner and an operations playbook.
5
Scale
More teams, then more workflows.
GateCost, reliability and value measured long enough to decide: expand, change or stop.
Gates do two jobs. They stop a system from moving up before it is ready, and they give its sponsor a clean way to stop one that will not get there. A failed gate is a decision, not an embarrassment, and it is far cheaper at the second rung than at the fifth.
Most stalled pilots skipped the third rung. Hardening is unglamorous, because nobody demos an access control. But it is where the five loans are repaid, and it stays short when the first two rungs were done honestly.
Three questions from the room
When does a proof of concept end?
When it meets exit criteria agreed before it started: the quality it has to reach, the data it has to run on, and the most an answer may cost. Without them a proof never ends. It just gets extended.
Why two sprints of hardening?
Two sprints are long enough to connect the production data, build the evaluation and the controls, and rehearse what happens when the system is wrong, and short enough that the momentum survives. If hardening needs much longer than that, the proof was not run on real conditions.
Who should go first?
Two or three teams, not the whole organization. Choose teams with a clear owner for the service and a playbook for running it, so the first problems are found, fixed and written down before anyone else depends on the system.
If your pilot has already stalled
Do not start another pilot. Start with the five loans: which of them fell due, and which one is blocking the launch. It is usually one or two, not all five, and it is a question that can be answered with evidence from your own systems in a couple of weeks.
Where to start
- A readiness review of the pilot you haveRun as the AI & Data Assessment Sprint: the pilot scored as it runs, the gaps ranked by what blocks production first, and what it will cost to run.
- AI & Agent PilotFor a use case that is not proven yet: a controlled evaluation, scoped to one function and one use case.
- AI Governance QuickstartWhen the review path is what is holding it: one use case taken through to a decision.
