Metrics and observability

Envoy records task, step, and connector measurements during execution, uses them for run summaries, and can send them to configured backends.

What is measured

Task measurements include:

• total duration;
• output item count;
• completion status.

Step measurements include:

• total wall-clock duration;
• per-item execution duration;
• items in and out;
• execution errors;
• current in-flight calls;
• items skipped by conditions.

Connector measurements include call count, duration, and errors.

Server metrics configuration

The control plane maintains metrics-backend configuration under Settings → Metrics Backends. Enabled server configuration is injected into a task run when the task YAML does not define its own metrics section.

A new server creates a default durable backend appropriate to its state configuration. SQLite deployments normally use metrics.db; MSSQL deployments can use the configured MSSQL service. In a remote topology, choose storage reachable from the executor and control plane.

After changing a backend, execute a small task and confirm that the run detail shows a metrics summary.

Per-task configuration

A CLI task can declare one or more backends:

metrics:
  - backend: sqlite
    config:
      path: ./metrics.db
 
  - backend: statsd
    config:
      host: ${STATSD_HOST}
      port: 8125
      prefix: envoy

When no metrics backend is present, the runtime uses a no-op collector.

Use a task-level section when the task needs to override deployment defaults or when running entirely outside the control plane.

Run summaries

The run detail page reads metrics associated with that run and displays aggregate task and step values. Use it to identify:

• a step with disproportionate wall time;
• item loss or amplification between steps;
• high connector call count;
• errors hidden by per-item continue behavior;
• concurrency that is not producing throughput.

Metrics describe what was measured; logs explain individual failures. Review both.

StatsD

The StatsD backend sends UDP packets and does not wait for delivery acknowledgement. It can target a compatible agent such as Datadog Agent, VictoriaMetrics, Telegraf, or Graphite.

Confirm routing and packet-size limits from the executor. Missing StatsD metrics do not fail the task merely because UDP delivery was lost.

Capacity signals

Start with:

• task duration by task name;
• step duration and items per second;
• connector error count and latency;
• executor active tasks versus capacity;
• pending and failed run counts;
• scheduler dispatch failures;
• state and log-store disk usage.

Do not infer record correctness from a success counter. Add explicit validation steps and destination checks to migration workflows.

Data management

Metrics storage has its own retention behavior. SQLite metric backends support age-based pruning in their backend config. Deleting a run through Envoy also removes associated run metrics from the server metrics store on a best-effort basis.

Back up a metrics database only when historical performance data is part of the recovery objective. It is separate from the main SQLite state database.