Ten ways to measure AI returns

In the last edition, three of KIT’s five messages were good enough to send with basic changes. The other two exposed weaknesses we could fix.

That gave us a starting point. Now we need to work out whether using KIT is worth the research, review and follow-up involved. The same question applies to the use case you chose for your company.

So let’s revisit the return you expected and look at ten places that value could come from.

Go back to your original ranking

When you discovered and ranked your use cases, you estimated how much each one could improve the business. Those estimates helped you choose where to start.

Now you’ve run a prototype, you have real experience to work from: the quality of the output and the time saved or added. Use both to update your expected return. Count the checking, corrections and failed attempts, as well as any extra work a good result creates.

For KIT, leave out the manual spreadsheet compilation when estimating the return from the automated version we plan to build. Keep that saving labelled as a forecast. The review work still counts, and we’ll need to include the cost of building and running the automation.

Run it a few times on work that reflects a normal week. One good batch is encouraging. Consistent results give us a much stronger reason to invest.

Ten places your returns on AI implementation can come from

Choose the two or three that fit your use case. For each, record what happens today, what changed with AI and how you measured it.

1. Time saved

Measure total staff time per completed task, including preparation and review. Then decide where the saved time will go. For KIT, faster research could lead to more useful conversations and more follow-up work. Count both.

2. Better-quality work

Compare outputs against the same quality criteria, ideally without the reviewer knowing which used AI. Is the work more complete, relevant or useful? For KIT, score the relevance of the article and the reason for sending it.

3. Fewer errors and less rework

Track mistakes, missing information and corrections per completed job. Record correction hours and any refund or remediation costs. If those hours are already included in your time-saving calculation, count them once.

4. Faster turnaround

Measure elapsed time from a request arriving to the work being completed. Include time spent waiting for information or approval. A quote delivered sooner might help win the job; check whether faster responses actually improve conversion.

5. More work completed

Count additional jobs completed to the required standard with the same team. Check whether there’s enough demand to use that capacity. It creates a financial return when you can take on more profitable work or avoid spending you had planned.

6. More sales

Track qualified opportunities, proposals, wins and contribution per sale. Compare similar prospects and allow enough time for sales to come through. For KIT, a reply is an early signal. Follow the conversation through to an opportunity and, eventually, a sale.

7. Better customer retention

Look for improved renewals, repeat purchases or fewer customer losses. Ask customers what influenced their decision. More regular contact could help maintain the relationship, although price, service and other changes may also explain the result.

8. Lower operating costs

Compare actual spending on overtime, contractors, outsourced services or tools for equivalent work. Subtract the extra cost of the AI system. Before calling something a saving, check that the expense has actually gone away.

9. Fewer missed commitments and risks

Track overdue actions, missed deadlines and confirmed issues caught early. KIT might surface a promise buried in old meeting notes. Report estimated avoided losses separately from actual savings, because spotting a warning doesn’t prove a loss would have happened.

10. New capabilities or services

AI might make a service affordable to deliver for the first time. Test demand, then track paying customers and contribution after delivery costs. For an internal capability, measure its intended outcome, such as previously neglected contacts receiving useful follow-up.

Watch for double counting. Time saved, extra capacity and additional sales can be stages of the same benefit. Adding all three together would make the return look bigger than it is.

Saving an employee time does not automatically reduce their salary cost. The company still pays the full salary.

Say the sales team spends less time researching and drafting messages. Agree that the saved time will go into finding suitable prospects and following up. Make room for it in the working week, otherwise the next round of meetings will happily fill the gap.

Then estimate its value: additional qualified prospects the team can handle, multiplied by its historical conversion rate, multiplied by contribution per sale. Contribution is the revenue left after the variable costs of delivering that sale.

Treat that number as a forecast to test. Track the extra activity and eventual sales, then subtract the setup and running costs of the AI system over the same period.

Be deliberate about where the saved time goes, then check what changed. Compare similar teams or periods where you can. Changes in demand, staffing or pricing could also explain the result, so record those alongside the AI work.

Better quality, better service and a more manageable workload can be worthwhile returns too. Measure and report them directly when you can’t put a credible financial value on them.

Keep the information in one shared sheet so you can compare the results.

1. Choose your measures. Pick two or three returns from the list above. Define what you’ll count, what acceptable quality looks like and where the information will come from.

2. Assign an owner and review date. Name who will collect the information and review it. Allow enough time for the outcome to appear; use 90 days as a starting point if it suits the workflow.

3. Record the baseline. Use existing records or observe a representative batch of work without AI. Log time per task, quality, volume and the business outcomes you chose.

4. Run a small pilot and keep a log. Use the same measures for comparable work with AI. Record the date, task, time including review and corrections, quality and outcome. Agree who checks outputs before use.

5. Track costs and other changes. Record setup, tools, support and additional work. Note where saved time goes, whether people use the workflow, and changes in demand, staffing or case difficulty.

6. Compare and decide. At the review, compare results per task or equivalent period. Separate observed improvements from forecasts and consider other explanations. Update the expected return, then decide whether to stop, improve or expand.

So go back to your original ranking and update the expected return. Choose your measures, give someone responsibility for collecting them and set a review date. You’ll have a way to decide whether this use case deserves more investment.

You now have a method to take your next AI idea from a business problem to a measured result. Revisit any step here:

Edition 1: Bringing AI into your business? What you need to know

Understand the AI Champion role and what it takes to turn a business problem into an AI solution.

Edition 2: How to know which AI ideas are worth building

Map the parts of an AI system, then discover, assess and rank opportunities for your business.

Edition 3: Design comes first

Define what the solution should achieve, understand, decide and do before building it.

Edition 4: Prototyping - the key to "worked the first time"

Run KIT with a spreadsheet and prompt, inspect the results and improve the workflow before automating it.

Edition 5: Is your AI use case worth the investment? (this edition)

Revisit your assumptions, measure the value and decide whether the return justifies further investment.

You’ve now got a way to take an AI idea through to a business decision. In the coming editions, we’ll build on that so you can spot useful opportunities and make better calls about where to spend your time and money.

We’ll unpack the ideas and industry changes that could affect your business, so you can judge what needs your attention and what you can leave for later.

And we’ll explore use cases you can try yourself or bring to your team: ways to get tedious work off your plate, improve what you deliver and offer something you couldn’t before.

Whats been happening this week

Decision models give us another way to handle routine choices. TypeSafe’s Jev and Liquid AI’s d1 are built for questions with a defined set of answers: is this a lead, which team should get this ticket, how urgent is this request? They return structured answers and probabilities that software can act on. TypeSafe reports lower costs and faster responses than general-purpose models on its decision tasks.

So look at the small, repeated decisions in your workflows. If you’re paying a powerful general-purpose model to make them, test a decision model on the same work. Compare accuracy, speed and total cost before switching. A cheaper call is useful when the decision is good enough for the job.

Agents are making purchases through channels customers already use. DoorDash has opened a US beta waitlist for ordering through Apple Messages. Stripe’s Link is adding ways for agents to handle checkout obstacles and request approval when the final price changes. OpenAI’s ‘Sign in with ChatGPT’ also makes account access easier across supported services, though that is a separate development from agents completing purchases.

For a business, the opportunity is to make buying easier wherever the customer starts. Check whether your product details, prices and availability are accurate and accessible to software, and whether your booking or checkout process can support an agent acting with