Troubleshooting

Troubleshoot from the outside in: availability, authentication, dispatch, connector access, task logic, then throughput. Preserve the first useful error before retrying.

Gather a minimum incident record

Record:

• deployment and image or binary version;
• time range and time zone;
• affected task, job, run, connector, or trigger IDs;
• whether the failure is repeatable;
• the first error and its source;
• recent configuration or network changes;
• executor selected for the run.

Redact passwords, tokens, API keys, TOTP secrets, authorization headers, and record payloads.

Control plane will not start

Check:

1. the process or container exit output;
2. envoy-server.yaml syntax;
3. required environment substitutions;
4. encryption-key presence and validity;
5. state database connectivity and permissions;
6. bind-mounted directory ownership;
7. port conflicts.

The authenticated application cannot load until state migrations complete. Do not repeatedly restart against a failing managed-database migration without preserving the original error.

GET /api/health is the control-plane health endpoint. A healthy HTTP response does not prove that the state, executor, and every connector are operational for a real run.

Login or TOTP fails

• Confirm the account exists and the invitation is current.
• Correct clock drift on the server, browser device, and authenticator device.
• Start a new login after a pending TOTP session expires or reaches its attempt limit.
• For SSO, confirm the account was created and linked before OAuth login.
• Check the exact redirect URI and identity-provider tenant policy.

Do not disable TOTP as a first diagnostic step in production.

Connector test fails

Separate four failure classes:

• name resolution or routing — the host cannot be reached from the testing process;
• TLS — certificate name, trust chain, protocol, or interception failure;
• authentication — invalid, expired, or mis-scoped credentials;
• authorization or schema — login works but the requested resource or method is denied.

A test from the control plane can differ from execution on a remote executor. Reproduce with one bounded task on the executor network.

For file connectors, verify absolute container paths and mounts on both services. For connected-account APIs, inspect account health and reconnect when the provider authorization is inactive.

Run stays pending

Check Executors:

• at least one executor is healthy;
• capacity is available;
• labels satisfy the requested selector;
• control-plane callback and advertised executor URLs are mutually reachable;
• the executor key matches.

Then inspect the control-plane log for No healthy executors available with sufficient capacity.

Do not create repeated runs while capacity is unavailable; that obscures the original dispatch failure.

Run appears stuck

Compare:

• run status and last heartbeat;
• executor active runs;
• current step in live logs;
• upstream API or database latency;
• rate-limit and retry messages;
• process and memory limits;
• output item counts.

A long connector call may be healthy. A stale lease can be reaped by the control plane after executor loss. Avoid manually changing run rows.

Use Stop and wait for cancellation when the task must end. Force-killing the executor is a last resort because it interrupts other runs on that executor.

Task validation or expression failure

Run static validation first. Common causes include:

• undeclared connector or step references;
• connector type mismatch;
• invalid JSONata syntax;
• a task argument not declared in the schema;
• a job template placeholder not declared under template arguments;
• a non-acyclic job graph;
• transformed data that does not match a destination method.

Enable debug: true on the narrowest relevant step and rerun against a small input. Debug output can expose records.

See Live run monitoring and JSONata.

Job is paused or failed

Open the job run and find the first failed or validation-stopping node. Correct the external state, connector, mapping, or job configuration before resuming.

continue failure policy can allow unrelated nodes to finish, so the last completed node is not necessarily the root cause.

Trigger did not run

• Confirm it is enabled.
• Verify the cron expression and IANA time zone.
• Check the displayed next-run time.
• Confirm the target job still exists and is usable.
• Look for another in-flight run of the same job.
• Confirm scheduler and executor health.

Disabling and re-enabling without finding the cause can shift scheduling state and remove useful evidence.

Logs or metrics are missing

For logs, confirm the configured run log store is writable by the executor and readable by the control plane. For file storage, confirm shared mounts.

For metrics:

• confirm an enabled backend;
• confirm the task did not override it;
• verify remote reachability;
• run a small task;
• inspect run summary and backend storage.

StatsD uses UDP, so network loss may not produce a task error.

Retention did not clean up

Confirm:

• max_age_days is positive;
• the cleanup cron is valid;
• runs are terminal and old enough;
• the control plane remained running at cleanup time;
• log-store and database delete permissions;
• retention errors in server output.

Retention is best effort across stores. Investigate partial cleanup before deleting rows manually.

Escalation package

Provide:

• sanitized server and run logs for the time range;
• relevant YAML with secrets replaced;
• connector type and tested method;
• run and executor IDs;
• deployment topology;
• exact expected and observed result;
• one minimal reproduction where possible.

Do not include the state database, connector database, environment file, or full configuration unless an approved secure transfer process is in place.