Two weeks spent making a voice assistant faster changed how I think about latency.
For most of my career, the slow part of a system was some kind of waiting: a disk, a lock, a network hop, a query against a table that had outgrown its indexes. The code made its decisions in microseconds and then stood around for the world to deliver. The craft of performance engineering, as I learnt it, was to shorten that waiting without touching the decision. An optimisation that changed the output was not an optimisation. It was a bug with good timing.
Then we shipped voice at YORO Health, and that rule stopped applying.
Families now speak to our care assistant and hear it reply, in Marathi, Hindi or English. A few seconds is a long time in a conversation, so I did what I have always done. I measured where the seconds go.
The profile is upside down
The lookups were quick. Availability, prices, the family’s own record: all of it came back in a small fraction of the turn. Most of the time went into the model deciding what to do. Reading the message, choosing what to check, reading the result, choosing again, and only then composing a reply. The time was going into deliberation, not I/O.
That is an unfamiliar profile, but the old reflexes still applied. Give the assistant the details it always fetches, up front. Stop the loop when the reply is already complete. Cache the part that never changes. The turns got faster, and then something more interesting happened.
Faster is a different conversation
Once the assistant had a family’s details in hand from the first word, it stopped confirming the address before a booking. Nothing in its instructions had changed. We had removed a step it used to perform for itself, and a step we cared about went with it. In my old world that would be impossible. A prefetch cannot alter business logic. Here there is no such separation. The thing you speed up is the thing that decides what to say, and the material you give it to be faster is the material it reasons from.
The same insight showed up from the other side. In one review, the screen showed one visit type and one price. The voice read out three. The speech layer, optimised for speed, had read what the system looked up during the turn rather than what it actually said. In every system I had worked on before, what a service knew and what it told you were the same fact. Here they were two different facts, and the shortcut had picked the wrong one.
We caught both, because we treat latency changes as behaviour changes. Every speed improvement in a conversational feature goes through the same review as a product change. Does it still ask what it used to ask? Does it say only what it meant to say?
The seconds are doing work
That framing cuts both ways. Some of those seconds are not waste at all. They are the assistant checking before it speaks. A turn that gets too quick can simply be a turn that has stopped checking. In a cache we call that serving stale data. In a care conversation, we call it not doing the job.
So the target was never “as fast as possible”. It is fast enough that the conversation feels natural, and slow enough that the assistant has actually thought. That balance is now part of how we build every conversational feature at YORO Health: a latency budget that respects judgment, and a conversation review alongside every performance change.
The old rule survived in a new shape. An optimisation must not change what the system does, except now what the system does includes how it thinks.