Contents6 sections
  1. Removal is the only clean measurement
  2. What comes back on the first evening
  3. What does not come back
  4. The thing nobody predicts
  5. How to run it without hurting anyone
  6. What the week is actually for

Field notesARTICLE

Turn It Off for a Week

After the first month there is no clean comparison left, and every number becomes an argument. Removal is the one measurement nobody can dispute, and it is worth doing on purpose.

Costa4 min read

Short answer

Once an assistant has been running for a month, no report can separate what it does from what the business does, because the baseline moved. Switching it off deliberately for a fixed period restores the comparison, and the honest version is a few nights rather than a week. What returns first is the overnight backlog and the repeated questions, both immediately and both measurable.

Key facts

  • The assistant can be switched off per channel and per hour without touching anything else.
  • Conversations that arrive while it is off are still recorded, so the two periods are comparable.
  • Customers see the honest out-of-hours message rather than silence.
  • The test is scheduled against a comparable week rather than against whatever week is convenient.

By the second month there is no honest comparison left. The before-state is stale, the numbers move for reasons that have nothing to do with the assistant, and any claim about what it contributes is an argument rather than a measurement.

There is exactly one clean experiment available, and it is the obvious one nobody runs: switch it off.

Removal is the only clean measurement

Attribution after the fact is hopeless. Response time improved, and so did three other things, and the season changed, and somebody rewrote a page in September.

Removal sidesteps all of it. Whatever comes back is what the assistant was holding. It requires no model of the business and no agreement about what to count, which is precisely why it settles arguments that a report cannot.

What comes back on the first evening

The overnight pile. It reappears immediately and completely, and it is the least surprising item on the list.

The second thing is the repeated question, and this one lands harder than people expect. Somebody in the business answers it three times before lunch on the first morning and remembers, physically, what that used to be like. Businesses that had been putting off the page fix for two months tend to do it in the week after the test.

What does not come back

Usually more than expected, and this is the half worth being honest about.

The channel nobody was really using. The category of question that turns out to arrive twice a week rather than twice a day. The elaborate rule somebody spent an afternoon writing for a scenario that has occurred once since launch. Roughly half the businesses that run this find at least one thing they were proud of automating and that nobody misses.

That is not a failure of the assistant. It is a correction to a story the business had been telling itself, and it makes the next round of rules better.

The thing nobody predicts

The staff response, which splits.

Some people are relieved, because answering customers was the part of the job they liked and it had quietly gone. Others discover within two days that they had reorganised their week around not having a queue, and had not noticed doing it. Both reactions are information about how the work actually got redistributed, and neither of them appears in any report.

How to run it without hurting anyone

The out-of-hours message has to be in place and honest before anything is switched off: office hours, when the answer will come, and no pretending a person is about to appear.

Pick a comparable period rather than a convenient one, and say the dates in advance so nobody quietly re-enables it on the second morning. Keep recording everything so the two periods can be laid side by side. And nothing on the limits list changes: complaints and money still go to people, exactly as before.

A few nights is usually enough. A full week only makes sense where the volume is low enough that a shorter run proves nothing, and the discipline of naming the end date in advance matters more than the length.

What the week is actually for

Not for proving that the assistant works. Run it to produce a number for a slide and you have given your customers a worse week to settle an argument they were not part of.

It is for finding out which parts of it matter, because that determines where the next month of editing goes. The list that comes out of a removal test is short, specific and impossible to argue with, and it is the only artefact in this entire subject that a business gets for free by doing nothing for a few days.

We suggest it to customers roughly six months in, and the ones who take it up usually turn something off permanently afterwards. That is the correct outcome and it is worth saying out loud, since it is not the one we are commercially supposed to be hoping for.

Questions this raises

Isn't this just an expensive way to prove something obvious?
It is a cheap way to find out that something is less obvious than assumed. Every business that runs it is surprised by at least one item on the list, and about half discover that a category they were proud of automating was barely being used.
Won't customers notice?
They will notice slower answers at night, which is the thing being measured. What they must not see is silence with no explanation, so the out-of-hours message has to be honest and in place before the test starts.
How long should it be?
Long enough to include a weekend and short enough that nobody is tempted to end it early. For most businesses that is a few nights, and a full week only where volume is low enough that a shorter period proves nothing.

Share

Read next

Next step

Agent-fit audit

Free, in the browser, no call required.

Run Agent-Fit Audit