The complaint that arrives first after an agent goes live is rarely "it was wrong". It is "it was slow".

The useful thing we learned is that most of that waiting is not the model thinking. It is the shape of the sequence around it — which means it is something you can design, and therefore something you can ask for.

The measurement

In our own routing layer we moved the decision about which specialist handles a question earlier in the sequence, and let that specialist answer directly when its confidence was high enough, instead of passing the result back through a coordinator.

Before After
Same question, end to end 12.7–17.3 s 7.9 s
Model unchanged unchanged
Routing accuracy in the same test 37 of 37 37 of 37

That is a reduction of roughly 55% on this path, with the denominator being the full round trip for that class of question. Nothing about the model changed.

Where the seconds actually live

A single answer from a knowledge-backed agent is several steps: work out what is being asked, choose which sources to search, retrieve, judge whether that was enough, possibly search again, then write. A model call sits inside some of those steps, but the arrangement decides the total.

Two things dominate:

  • How many round trips the design requires. Every extra deliberation step costs a full model call, and the steps that add the least are the easiest to add.
  • How early the system commits to a route. An agent that works out what kind of question it faces before doing the expensive work can skip the work that does not apply.

What we publish as a range

  • Ordinary questions: about ten seconds.
  • Answers assembled from several sources: thirty to sixty seconds.

A range rather than an average, because an average hides the tail and the tail is what people remember.

Speed can be part of the specification

If the workload is latency-sensitive — a customer waiting on the line, a technician asking with gloves on, a line that cannot stop — say so at the start and the system gets designed for it, the same way we designed for it ourselves.

That is the practical consequence of the number at the top: a 2× difference was available in the orchestration. It is a requirement you can state, not a property you have to accept.

Why it is worth the engineering

Slow does not read as slow. It reads as broken.

Someone who waits twenty seconds with no feedback assumes it has hung, tries again, and tells a colleague it does not work. The same answer at eight seconds — or at twenty with a visible sign that it is still searching — earns a completely different reputation inside the company.