How to Train AI to Triage Customer Emails by Real Urgency

How to Train AI to Triage Customer Emails by Real Urgency - feature image for the Rapid Ready Ai blog.

What you will end up with

You will end up with an AI email tool that flags only the customer emails that genuinely need you today: payment problems, service failures and time-sensitive requests. Routine thank-yous, confirmations and general questions get sorted into lower-priority groups, so they stop competing for your attention.

The AI is not clever by default. It sorts well only when you give it clear tiers and real examples. This guide shows you how to do that in five steps.

Before you start

You need three things:

  • A list of emails you must see immediately. Think complaints, payment problems, cancellation requests, and anything with a deadline.
  • A list of emails that can wait. Thank-yous, order confirmations, delivery acknowledgements and newsletter replies are common ones.
  • An AI email tool ready to configure. If you want one, Rapid Ready Ai offers AI Inbox & Email. A role that needs a mailbox connection is quoted before you order. Any AI tool that lets you write your own instructions will work with the steps below.

Also have your last 20 customer emails handy. You will use them in Step 3.

Set it up in five steps

Step 1: Define your urgency tiers

Create three tiers and write a plain-English description of each.

  • Urgent: needs you today. A customer says they were charged twice, your service is down for them, or they need something done before a deadline.
  • Important: needs you this week. A quote request, a question about an upcoming job, a supplier query.
  • Routine: can wait. Thanks, confirmations, "got it" replies and general questions that are already answered on your website.

Keep each description to a sentence or two. If you can't tell which tier an email belongs in, the AI won't be able to either.

What you should see: a short written definition of each tier that a new staff member could follow on their first day.

Step 2: Write your triage instructions

Now give the AI your tiers and specific examples of each. Be concrete. Write them the way your customers actually write.

Here is an illustrative example for a plumbing business in Hamilton:

  • Urgent: "Your invoice was charged to my card twice." "The leak is back and it's getting worse." "I need this done before Friday or I lose my tenant."
  • Important: "Can you quote for a new hot water cylinder?" "Can we move next week's visit?"
  • Routine: "Thanks so much, all sorted!" "Please find the signed form attached." "Received your confirmation."

Include the thank-you emails on purpose. A message like "Thanks, that fixed it!" contains words such as "fixed" and "issue" that can trick a vague setup into flagging it. Tell the AI plainly that thanks and praise are Routine, even when they mention a past problem.

What you should see: one block of instructions with your three tiers and at least a few examples under each.

Step 3: Test with real emails

Run the AI over your last 20 customer emails. Don't tell it the answers. Then check each result against your own judgement.

Look for two kinds of mistakes:

  • False positives: routine emails marked Urgent.
  • False negatives: real problems placed in Important or Routine.

False negatives matter more. Write down every miss, and note what the email said that fooled the AI.

What you should see: most emails sorted sensibly, and a short list of mistakes you can learn from. Expect some mistakes the first time. That is normal.

Step 4: Adjust and lock in your rules

Use your mistake list to improve the instructions.

  • Over-flagging? Add more examples of routine emails, especially the ones it wrongly marked Urgent.
  • Missing problems? Add examples of the problems you care about most, using the exact wording from the emails it missed.

Change a little at a time, then re-run the same 20 emails. If you change five things at once, you won't know which one helped. When the results match your judgement on nearly all of them, save that version and stop fiddling.

What you should see: the same 20 emails sorted the way you would sort them, with no real problems missed.

Step 5: Set up your review workflow

Sorting is only useful if a person acts on it. Decide this now:

  • Urgent: one named person checks it every working day, at set times. Pick a backup for sick days.
  • Important: one person checks it weekly, say every Monday morning.
  • Routine: one person skims it weekly, or the AI handles the groupings and you only look when needed.

Keep a person in the loop for anything the AI flags as urgent. The AI works only from what you tell it, and it can get things wrong. Once a fortnight, glance through Routine for anything that should have been higher.

What you should see: a simple list of who checks which group and when, written down where your whole team can see it.

If it does not work

The AI flags nearly everything as urgent

Your instructions are probably too vague. Words like "important" or "time-sensitive" mean different things to different people. Fix it by giving the AI 5 to 10 real examples of each tier, and make sure the Routine examples include the polite, friendly messages your customers send most often.

The AI misses a type of problem

You likely didn't mention it. A Tauranga cafe that sells gift vouchers, for instance, might forget to list "my voucher code isn't working". Add that problem type, with two or three examples in your customers' own words, and re-test.

The AI is right, but nobody acts on the flags

This is a people problem, not an AI problem. Give the Urgent group one owner, with a set time each day to check it. If it belongs to "everyone", it belongs to no one.

Next steps

Once triage is working reliably for a few weeks, consider using the AI to draft replies to routine emails, such as confirmations, thank-yous and common questions. A person should read each draft before it goes out, at least until you trust the wording.

Revisit your examples every few months too. New products, new prices and new kinds of customer questions all change what counts as urgent.

If you have questions about setting up AI Inbox & Email, you're welcome to get in touch through our contact page.