Debugging tasks and runs

Debug in layers: validate structure, reproduce with a bounded input, inspect the first incorrect step, then verify external state. Avoid enabling broad payload logging on a production dataset.

Validation

Validate task YAML before execution:

envoy validate tasks/example.yaml

Validation checks structure and expression compilation. It does not call every connector or evaluate expressions against live records.

Debug output and structured logs

Envoy separates:

• human-readable debug output on stderr;
• task data output on stdout;
• structured application logs in the configured application log;
• per-run logs in the configured run-log store.

Use the CLI debug flag for a diagnostic run or debug: true on a narrow step. Do not parse debug stderr as pipeline data.

Inspect one step at a time

1. Bound the source to one known record.
2. Add or enable inspection after the source.
3. Verify runtime types, not only displayed values.
4. Inspect the normalized transform.
5. Disable the destination write or use a test connector.
6. Verify the write request and response.
7. Remove or disable verbose inspection before scale testing.

Earlier step output is available as steps.<name>.output, which helps distinguish source, transformation, and connector problems.

Read metrics with logs

Useful counters include processed, emitted, skipped, and errors per step. A growing input count with no output may indicate a slow connector, filter, blocked write, or batching boundary.

Metrics indicate where to look; logs provide the causal detail.

Common failure categories

Configuration

Unknown connector or step type, missing slot mapping, invalid argument, or malformed YAML.

Expression

Missing fields, wrong runtime type, ObjectId/string mismatch, invalid date, or an unbounded $callMethod().

Connector

Authentication, network, permission, provider validation, rate limit, or pagination failure.

Orchestration

Failed dependency, stopped process, expired lease, executor unavailability, or run concurrency limit.

Data quality

Duplicate identity, missing required relationship, unsupported attachment, or destination constraint violation.

Sensitive data

• Never log authorization headers or connection strings.
• Mask only the minimum diagnostic field.
• Avoid full-record debug on personal or regulated data.
• Share request IDs, run IDs, and narrow timestamps with support.
• Remove temporary debug steps and exported files after the incident.

Escalation package

Provide:

• Envoy version and deployment topology;
• task, job, and run IDs;
• failure timestamp and time zone;
• first causal error;
• relevant step name and connector type;
• bounded sanitized input shape;
• provider request ID;
• expected versus observed behavior;
• whether a retry or partial write occurred.

See Live run monitoring and Monitor and control job runs.