Gateway latency diagnostics: find slow AI agent dispatch before changing models
Gateway latency diagnostics help you locate a slow AI agent request before you change models, trim prompts, or add another cache. The useful question is not “How long did the request take?” It is “Where did it wait?” OpenClaw v2026.9.2 breaks Gateway RPC latency into phases in OpenTelemetry and Prometheus diagnostics, separating admission, queue wait, handler work, and first response.
That distinction changes the repair. A model that starts generating late needs a different response from a message that spent most of its time waiting in a queue. Recent community discussion lands in the same place: one operator wrote that agent latency often comes from asking an origin for information that was already available, while another described a 30 to 60 second agent workflow before investigating its execution path. The model is often part of the wait. It is rarely the whole story.
Contents
- What gateway latency diagnostics measure
- Read the phases before changing a model
- A practical diagnosis workflow
- What to watch in OpenClaw
- FAQ
What gateway latency diagnostics measure
A Gateway is the layer that receives an action, decides whether it can run, schedules work, invokes the appropriate handler, and returns a response. In an agent system, a single user-visible delay may include channel delivery, policy checks, queueing, context assembly, model work, tool calls, and response rendering.
A single end-to-end number hides that chain. Phase-level diagnostics expose enough structure to choose the next investigation.
| Phase | What it usually means | If it is slow, investigate |
|---|---|---|
| Admission | The Gateway is deciding whether the request can enter work | policy checks, overload handling, auth, routing, or contention |
| Queue wait | The request is accepted but waiting for execution | lane depth, parallel work, blocked workers, or a slow earlier run |
| Handler | The Gateway is actively processing the request | context assembly, tool work, model calls, serialization, or application logic |
| First response | The time before the caller gets the first usable response | streaming, channel bridge behavior, early events, and handler startup |
OpenClaw’s Prometheus integration exposes matching Gateway RPC histograms, including openclaw_gateway_rpc_admission_seconds, openclaw_gateway_rpc_queue_wait_seconds, openclaw_gateway_rpc_handler_seconds, and openclaw_gateway_rpc_first_response_seconds. The OpenTelemetry export can provide traces alongside metrics and logs, which matters when an aggregate spike needs to be traced back to a particular request path.
Read the phases before changing a model
Use the phase that dominates the slow request as the starting point.
High admission time
High admission time points to work before the handler starts. Review whether the request is hitting an access decision, channel routing path, rate limit, or overload guard. Treat admission time as a control-plane signal: speeding up inference will not reduce a request that is still waiting to be accepted.
A small controlled test helps. Send the same low-risk request through the same channel with normal background work paused. If admission improves, contention or policy evaluation is a more likely lead than the model.
High queue wait time
A high queue-wait phase means the Gateway has accepted the work but cannot run it immediately. Check queue depth and look at what occupied the lane ahead of it. OpenClaw exports queue lane size and wait metrics as well as session state and recovery signals, so a slow request can be correlated with the workload already in flight.
Do not solve this by simply raising concurrency. More parallel turns can increase provider pressure, tool contention, duplicate work, and memory load. First identify whether the queue holds a few legitimate long runs, stuck sessions, or an unexpected burst from a channel or automation. The agent scheduling guide is relevant here: unattended work needs explicit cadence and recovery behavior, or it eventually competes with interactive requests.
High handler time
High handler time is where model selection may matter, but it is still a broad bucket. A handler can spend time building context, calling a provider, waiting for a tool, formatting output, or recovering a state transition.
OpenTelemetry traces are useful because they preserve the request path as spans. OpenClaw documents spans for model usage, model calls, tool execution, webhooks, message processing, context assembly, and tool loops. Find the longest child span before changing a global setting. If a search API or browser action dominates, changing the LLM will not move the number much.
LangChain makes the same operational point: diagnose the bottleneck before selecting a fix. Depending on the trace, the remedy may be fewer model calls, a smaller context, parallel independent work, or a different user experience. It is a targeted decision, not a default upgrade ritual.
High first-response time
First response is the user-facing boundary. A run can finish in an acceptable time and still feel broken if it stays silent for too long. Inspect whether the Gateway emits a response or progress event promptly, whether the channel bridge buffers it, and whether the UI renders it as soon as it arrives.
A clean model trace does not prove the request felt fast. Pair this phase with first-reply latency checks to inspect the user-visible boundary.
A practical diagnosis workflow
Start with a single slow request and work outward. A broad dashboard is helpful for detecting a trend; a trace is what usually identifies the owner.
- Capture the request identity and time window. Include the channel, RPC method, and whether the run was interactive, scheduled, or a recovery attempt. Avoid putting sensitive prompt content into labels or dashboards.
- Compare the four Gateway phases. Identify whether admission, queue wait, handler, or first response accounts for most of the delay. Use a percentile view for the fleet and one trace for the outlier.
- Inspect the immediate parent. For queue wait, inspect the lane and prior work. For handler time, inspect child spans for model, tool, context, or delivery work. For first response, inspect streaming and channel behavior.
- Change one constrained variable. Pause a noisy automation, reduce an unnecessary tool call, test a smaller context, or adjust a known policy. Do not make a model swap, queue change, and prompt rewrite in the same experiment.
- Run the same test again. Compare the same phase, not only end-to-end time. A faster total can conceal a worse queue or first-response experience for other users.
Keep telemetry inside a security boundary. OpenClaw’s diagnostics docs describe redacted health snapshots and omit endpoint values, headers, certificates, payloads, and raw errors from status projections.
What to watch in OpenClaw
OpenClaw has two official diagnostic paths that suit different operations setups.
| Need | OpenClaw path | Useful output |
|---|---|---|
| Existing Prometheus and Grafana stack | diagnostics-prometheus plugin | Protected metrics endpoint with Gateway RPC, queue, run, model, tool, session, and liveness metrics |
| Trace-level request investigation or OTLP backend | diagnostics-otel plugin | OTLP/HTTP traces, metrics, and optional logs from Gateway diagnostic events |
Prometheus is a good fit when you want dashboards and alerts around phase percentiles or queue growth. The plugin exposes its endpoint at /api/diagnostics/prometheus and requires the same Gateway authentication used by operator clients. It is deliberately not a public status route.
OpenTelemetry is better when a dashboard says “handler latency is high” and you need the chain behind it. The diagnostics-otel plugin exports metrics, traces, and optional logs through OTLP/HTTP. Its docs note that exporters attach only when diagnostics and the plugin are enabled, which keeps default in-process cost near zero. It can also hot-reload its exporter service without taking down unrelated Gateway connections.
If you are starting from scratch, make the first dashboard boring:
- p50, p95, and p99 for each RPC phase by method
- queue lane size and queue wait time
- run duration by channel, model, and outcome
- tool execution duration and error category
- first-response latency for the interactive methods that users actually feel
Then create alerts for changes in shape, not just a single high number. A queue-wait spike with stable handler time suggests a different incident from handler time rising while queues stay empty.
For broader Gateway health and exporter state, pair this with OpenClaw Gateway host status. A healthy process does not prove that interactive work is moving through the right phase quickly enough.
FAQ
What are gateway latency diagnostics?
Gateway latency diagnostics divide a request’s delay into operational phases such as admission, queue wait, handler execution, and first response. They help an operator tell whether a slow AI agent request is waiting to start, doing slow work, or failing to become visible to the caller.
Does high handler latency mean I need a faster model?
Not necessarily. Handler time can include context assembly, provider calls, tools, output formatting, and delivery work. Inspect a trace or the child operation timings before switching models.
What should I check when queue wait is high?
Check queue lane depth, active long-running work, stuck-session signals, incoming channel volume, and scheduled automations. Increasing concurrency before finding the blocked or noisy workload can make the system less predictable.
Can OpenClaw send latency diagnostics to Prometheus or OpenTelemetry?
Yes. The diagnostics-prometheus plugin exposes protected Prometheus metrics, and the diagnostics-otel plugin exports OpenTelemetry metrics, traces, and optional logs over OTLP/HTTP when diagnostics and the plugin are enabled.