Financial technology / Regulated software provider
Making production LLM performance observable and governable
How a regulated product team established quality, latency, and cost controls for an LLM workflow before expanding it across customer operations.
The operating constraint
An LLM-assisted workflow had moved from prototype to production, but the team could not reliably answer three questions: whether responses remained useful, why latency changed, and what each successful customer task cost.
Application monitoring captured HTTP health while model-provider dashboards showed aggregate token usage. Neither described the full chain across retrieval, prompts, model calls, tool execution, and the final outcome. Expansion would have multiplied an unmeasured operational risk.
The intervention
Detakai designed an observability model around the task rather than the individual API call. A single trace connected retrieval quality, prompt version, model selection, token consumption, tool calls, latency, safety checks, and user-visible outcome.
We established a controlled evaluation loop:
- Define quality criteria with product and compliance owners.
- Build a representative evaluation set from reviewed production scenarios.
- Trace every component contributing to latency and cost.
- Route regressions by prompt, model, data source, and failure class.
- Gate material prompt and model changes against quality, cost, and safety thresholds.
No single score was allowed to hide a trade-off. A cheaper response only counted as an improvement when it still met the agreed task-quality threshold.
What changed
Trace coverage reached 98%, exposing repeated context, unnecessary model calls, and retrieval paths that added latency without improving answers. Targeted changes reduced cost per successful task by 42% and P95 latency by 37%.
Evaluated answer quality increased from 71% to 93%. Product owners gained a release-level view of model behaviour, while compliance teams could inspect the evidence behind a flagged response.
The durable system
Prompt, model, and retrieval changes now pass through the same evidence-based release process. Dashboards connect technical signals to successful tasks; alerts trigger on outcome degradation rather than token counts alone.
The team can adopt new models without losing its operating controls. Performance comparisons use the same evaluation set, trace structure, and business thresholds—turning model choice into a governed engineering decision.