Curated recipes

These recipes combine product workflows rather than replacing the technical references. Validate every connector method against Connectors reference and every task against Task structure.

Copy CSV records into SQLite

Use the bundled csv-to-sqlite task for a first local pipeline. It:

1. creates a destination table if absent;
2. streams users.csv;
3. derives full_name;
4. upserts by id.

Prepare connectors.yaml:

connectors:
  csv_source:
    type: csv
    config:
      directory: ./data/input
 
  db:
    type: sqlite
    config:
      path: ./data/output.db

Run:

envoy validate tasks/csv-to-sqlite.yaml
envoy csv-to-sqlite --config connectors.yaml --output none

Inspect the destination, run it a second time, and confirm the primary-key upsert does not duplicate rows.

In Docker, replace relative paths with shared absolute paths under /app/data and /app/dbs.

Add a safe dry-run switch

For a task that deletes, overwrites, or bulk-updates data, make preview the default:

arguments:
  dry_run:
    type: boolean
    default: true
 
steps:
  - name: preview
    type: console
    run_if: arguments.dry_run = true
    config:
      method: log
      prefix: "Would update: "
 
  - name: write
    type: call_method
    run_if: arguments.dry_run = false
    config:
      connector: destination
      method: upsert
      args:
        table: records
        data: input
        conflict_columns: [id]

Keep the preview bounded. A dry run should exercise source queries and transforms without calling the mutating method.

Make a load idempotent

Choose a stable source identifier and make it a unique destination key. Then:

• prefer upsert over insert;
• map the same source item to the same destination key;
• avoid using execution time as identity;
• store external IDs separately from display labels;
• verify the second run changes no unintended rows.

When an upstream API lacks stable IDs, derive a documented deterministic key from immutable fields. Hashing mutable labels causes duplicates after edits.

See Author a production task for batching, fan-out, validation, and reconciliation guidance.

Separate extraction, validation, and load

Use a job template when a migration has independently observable stages:

tasks:
  extract:
    task: extract-records
    connector_mapping:
      source: source
 
  validate:
    task: validate-records
    depends_on: [extract]
    connector_mapping:
      staging: staging
 
  load:
    task: load-records
    depends_on: [validate]
    connector_mapping:
      staging: staging
      destination: destination
 
on_failure: pause

Use task results to report validation messages and stop downstream work when acceptance thresholds fail. Resume only after fixing the data or configuration.

See Job templates for the exact result contract and dependency behavior.

Keep runtime choices editable

Use job_config for operator choices such as:

• batch size;
• dry-run mode;
• notification toggle;
• stage;
• field mapping.

Use template arguments for identifiers that should be substituted when a job is created.

This distinction lets operators edit behavior without cloning the template, while each run retains a snapshot of the values it used.

Save a reusable Data Explorer query

Build and run a narrow read query, then save it as a team preset. Parameterize only the values that should vary:

{
  "_id": {
    "$oid": "{{record_id}}"
  }
}

Declare record_id as an ObjectId parameter. For a friendlier form, use a record parameter backed by a bounded, searchable Mongo collection.

Do not place credentials or private user-specific values in team presets. See Use Data Explorer presets in practice.

Build an operational dashboard

Start with a job-bound internal dashboard containing:

1. a status stat;
2. task progress;
3. a scoped exceptions table;
4. a field-mapping or options form if operators need it;
5. a confirmed run action.

Only after internal testing should you expose a public page. Mark diagnostic pages public: false, keep queries scoped, and enable public input or actions only for explicitly reviewed widgets.

The bundled kitchen-sink demo shows every dashboard widget against a self-contained SQLite dataset. Treat it as a schema demonstration, not a production topology.

See Dashboards.

Promote a tested task into a schedule

Before adding a trigger:

1. validate the task or job;
2. run with a bounded dataset;
3. rerun to prove idempotency;
4. verify logs and metrics;
5. choose a time zone;
6. avoid overlap with another trigger for the same job;
7. define who reviews failures.

Scheduling is the last step of productionization, not the first.

Reconcile after a migration

Do not rely on source and destination counts alone. Compare:

• total expected records;
• records accepted, skipped, and failed;
• unique source IDs present at destination;
• representative transformed fields;
• relationship counts;
• attachment or child-resource counts;
• a deterministic sample plus high-risk edge cases.

Store a machine-readable reconciliation result with the run and surface discrepancies on the job dashboard.