How to Measure AI ROI: Why Pilots Fail and What to Track Next Time
Why your AI pilot failed to show a return
- ✓
No baseline. Nobody timed the process before the tool arrived, so “it feels faster” is the only evidence.
- ✓
Wrong use case. The team picked something impressive to demo rather than something expensive to run by hand.
- ✓
Messy inputs. The tool read six versions of the same template, or scanned PDFs with no structure, and its output needed so much checking that the saving disappeared.
- ✓
No owner. IT ran the pilot, the operations team was “consulted”, and when the trial ended the project simply stopped.
- ✓
Adoption never measured. Licences went to 40 people. Eight used them, and only on Fridays.
How to measure AI ROI, step 1: pick the use case by value and feasibility
Choose the use case that scores well on both value and feasibility, not the one with the best demo. Value is the cost of doing the work by hand today. Feasibility is how likely AI is to do it reliably with the data you actually have.
Score each candidate from 1 to 5 on both axes, then multiply. A process that costs 20 hours a week and runs on clean, consistent documents might score 5 × 4 = 20. By contrast, a clever chatbot for a question asked twice a month scores 1 × 3 = 3, however good it looks.
This is exactly the AI use case assessment we run in our AI readiness and governance work. Sometimes the honest outcome is that the best “AI” use case needs a simple rule or an integration instead, and we say so. If you’re unsure whether to buy a tool or build something, our guide to build vs buy software sets out the trade-offs.
Step 2: take a baseline before anything changes
- 01
Hours per item. How long one contract, invoice, report or enquiry takes, from opening it to closing it.
- 02
Turnaround time. Elapsed days from arrival to completion, including time spent waiting in someone’s inbox.
- 03
Error or rework rate. How often the output gets sent back, corrected or causes a problem later.
- 04
Cost per item. Hours multiplied by the loaded hourly cost of the people doing it.
Step 3: define success thresholds before the pilot starts
Success thresholds are the numbers the pilot must hit for you to roll it out, agreed in writing before the pilot begins. Setting them afterwards invites people to move the goalposts to fit whatever happened.
A useful threshold has three parts: the metric, the target and the minimum acceptable result. For example: “Review time per contract falls from 3 hours to under 1.5 hours, with no increase in missed clauses. Below a 30% saving, we stop.”
Also set a guardrail metric that must not get worse. Quality, accuracy or client complaints are the usual ones. So a tool that halves the time but doubles the errors fails, even if the hours look good.
Step 4: run the pilot on real work
Run the pilot on live work with the people who will use the tool every day, not on a hand-picked sample in a sandbox. Real work shows you the messy inputs, the edge cases and the adoption problems that a demo hides.
A good pilot lasts six to twelve weeks and covers one team or one contract type. Track usage weekly. If adoption drops after the first fortnight, find out why while it’s cheap to fix.
Where you can, test against history too. In our AI contract review case study for an MEP subcontractor, we piloted the tool on past contracts, including the three that had led to disputes. It caught the missed clauses in all three before it went near a live tender.
Step 5: count the full cost of the pilot
- ✓
Licences: per-seat or per-document fees, plus any usage charges.
- ✓
Integration and set-up: connecting the tool to your document store, practice management system or ERP.
- ✓
Human review time: someone still checks the output, and those hours come off your saving.
- ✓
Governance: a data protection impact assessment, an AI use policy and security checks. The ICO’s guidance on AI and data protection explains what it expects, and the NCSC’s guidelines for secure AI system development cover the security side.
- ✓
Change management: training, updated procedures, and the dip in output while people learn.
Which metrics to track for each AI use case
The right metric depends on what the AI is doing. Document review is judged on time and accuracy, while customer-facing tools are judged on response speed and resolution. Use this table to pick your primary metric and your guardrail.
| Use case type | Primary metric | Guardrail metric | Where to get the baseline |
|---|---|---|---|
| Document or contract review | Hours per document | Missed clauses or issues found later | Timesheets, review log |
| Document drafting | Minutes per standard document | Partner or manager corrections per document | Document management system |
| Data extraction (invoices, forms) | Cost per item processed | Error rate at month-end reconciliation | Finance system exceptions |
| Enquiry triage and email | Time to first response | Wrongly routed or escalated enquiries | Shared inbox or CRM |
| Reporting and summaries | Hours per weekly report | Figures queried or corrected | The Thursday report itself |
| Knowledge search | Time to find an answer | Answers later found wrong | A short sample survey |
Pick one primary metric per pilot, two at most. Otherwise a poor result hides behind a good one.
Worked example: an AI document-review pilot
- 01
Net monthly benefit: £5,040 − £1,680 = £3,360.
- 02
Payback: £25,000 ÷ £3,360 = about 7.4 months.
- 03
Year-one ROI: (£60,480 − £20,160 − £25,000) ÷ (£20,160 + £25,000) = £15,320 ÷ £45,160 = about 34%.
What real AI ROI looks like
Real AI ROI is measured time saved against a known baseline, plus a guardrail that held. In the MEP project, review time fell by 65% per contract, and early warning notice compliance held at 100% for twelve months. Our guide to AI contract review in construction covers the wider approach.
In our legal document automation case study, a City firm cut standard document production time by 70%. But the key step came first: six competing NDA templates became one. Clean inputs made the saving possible.
Common mistakes when measuring AI ROI
- ✓
Measuring the demo instead of the rollout. Five keen volunteers tell you little about 50 reluctant users.
- ✓
Ignoring review time. If a person checks every output line by line, count those hours.
- ✓
Changing the process and the tool at once, then crediting the AI. Sometimes the gain comes from fixing the process. That’s fine, but know which is which.
- ✓
No stop rule. Without a threshold for “this didn’t work”, pilots drift into permanent licences.
- ✓
Skipping governance until after go-live. A late data protection review can pause the rollout and wipe out months of payback.
How Severus approaches AI ROI
We start with the business problem and the process, then decide whether AI is the right answer at all. Sometimes an integration or a simple automation gives the same return for a fraction of the cost.
As a business automation and operations partner, we’re vendor independent. That means we can recommend building, buying, integrating or automating without a product to push. Many clients start with a business operations audit to find where the hours really go. If the real goal is fewer repetitive tasks, read our guide on how to reduce manual work, or see our work with construction companies.
AI often sits inside a wider modernisation programme (US vendors call it modernization). Either way, knowing how to measure AI ROI starts with the baseline.
Frequently asked questions
How do you calculate AI ROI?
Take the annual benefit, which is usually hours saved multiplied by loaded hourly cost and adjusted for actual adoption. Subtract the annual running costs and the one-off costs, then divide the result by total cost. For payback, divide one-off costs by the net monthly benefit. Always compare against a baseline you recorded before the pilot, not an estimate made afterwards.
Why do so many AI pilots fail to show ROI?
Most pilots have no baseline, so there is nothing to compare against. Others pick a use case chosen for the demo rather than the cost, feed the tool messy data or never track who actually uses it. When the pilot ends, nobody owns the decision to scale it. So the technology is rarely the main problem; the process around it is.
What should an AI business case include?
An AI business case should state the process, its current cost and the baseline figures. It should also set out the success thresholds, a guardrail metric and the full cost, including integration, review time, governance and training. Add a payback period and a stop rule. Keep it to two pages so the people approving it actually read it.
How long should an AI pilot run?
Six to twelve weeks on live work is usually enough to see a stable result. Shorter pilots mostly measure the novelty effect, when people try the tool because it's new. Longer ones drift. Set a review date up front and agree what result means scale up, adjust or stop.
Is AI ROI only about time saved?
No. Time saved is the easiest benefit to measure, but fewer errors, faster turnaround and lower risk often matter more. For example, one missed contract deadline can cost more than a year of licences. Still, only count those benefits when you have a baseline for them. Otherwise, keep them as upside and build the case on hours.
Business problem first. Technology second.
Planning your next AI pilot and want the numbers right this time? Book a first consultation with Severus