# Curated recipes

> Source: https://docs.clonepartner.com/guides/recipes/

These recipes combine product workflows rather than replacing the technical references. Validate every connector method against [Connectors reference](/reference/connectors) and every task against [Task structure](/reference/tasks/task-structure).

## Copy CSV records into SQLite

Use the bundled `csv-to-sqlite` task for a first local pipeline. It:

1. creates a destination table if absent;
2. streams `users.csv`;
3. derives `full_name`;
4. upserts by `id`.

Prepare `connectors.yaml`:

```yaml
connectors:
  csv_source:
    type: csv
    config:
      directory: ./data/input

  db:
    type: sqlite
    config:
      path: ./data/output.db
```

Run:

```bash
envoy validate tasks/csv-to-sqlite.yaml
envoy csv-to-sqlite --config connectors.yaml --output none
```

Inspect the destination, run it a second time, and confirm the primary-key upsert does not duplicate rows.

In Docker, replace relative paths with shared absolute paths under `/app/data` and `/app/dbs`.

## Add a safe dry-run switch

For a task that deletes, overwrites, or bulk-updates data, make preview the default:

```yaml
arguments:
  dry_run:
    type: boolean
    default: true

steps:
  - name: preview
    type: console
    run_if: arguments.dry_run = true
    config:
      method: log
      prefix: "Would update: "

  - name: write
    type: call_method
    run_if: arguments.dry_run = false
    config:
      connector: destination
      method: upsert
      args:
        table: records
        data: input
        conflict_columns: [id]
```

Keep the preview bounded. A dry run should exercise source queries and transforms without calling the mutating method.

## Make a load idempotent

Choose a stable source identifier and make it a unique destination key. Then:

- prefer upsert over insert;
- map the same source item to the same destination key;
- avoid using execution time as identity;
- store external IDs separately from display labels;
- verify the second run changes no unintended rows.

When an upstream API lacks stable IDs, derive a documented deterministic key from immutable fields. Hashing mutable labels causes duplicates after edits.

See [Author a production task](/guides/task-authoring) for batching, fan-out, validation, and reconciliation guidance.

## Separate extraction, validation, and load

Use a job template when a migration has independently observable stages:

```yaml
tasks:
  extract:
    task: extract-records
    connector_mapping:
      source: source

  validate:
    task: validate-records
    depends_on: [extract]
    connector_mapping:
      staging: staging

  load:
    task: load-records
    depends_on: [validate]
    connector_mapping:
      staging: staging
      destination: destination

on_failure: pause
```

Use task results to report validation messages and stop downstream work when acceptance thresholds fail. Resume only after fixing the data or configuration.

See [Job templates](/reference/job-templates) for the exact result contract and dependency behavior.

## Keep runtime choices editable

Use `job_config` for operator choices such as:

- batch size;
- dry-run mode;
- notification toggle;
- stage;
- field mapping.

Use template arguments for identifiers that should be substituted when a job is created.

This distinction lets operators edit behavior without cloning the template, while each run retains a snapshot of the values it used.

## Save a reusable Data Explorer query

Build and run a narrow read query, then save it as a team preset. Parameterize only the values that should vary:

```json
{
  "_id": {
    "$oid": "{{record_id}}"
  }
}
```

Declare `record_id` as an ObjectId parameter. For a friendlier form, use a record parameter backed by a bounded, searchable Mongo collection.

Do not place credentials or private user-specific values in team presets. See [Use Data Explorer presets in practice](/guides/data-explorer-presets-in-practice).

## Build an operational dashboard

Start with a job-bound internal dashboard containing:

1. a status stat;
2. task progress;
3. a scoped exceptions table;
4. a field-mapping or options form if operators need it;
5. a confirmed run action.

Only after internal testing should you expose a public page. Mark diagnostic pages `public: false`, keep queries scoped, and enable public input or actions only for explicitly reviewed widgets.

The bundled kitchen-sink demo shows every dashboard widget against a self-contained SQLite dataset. Treat it as a schema demonstration, not a production topology.

See [Dashboards](/reference/dashboards).

## Promote a tested task into a schedule

Before adding a trigger:

1. validate the task or job;
2. run with a bounded dataset;
3. rerun to prove idempotency;
4. verify logs and metrics;
5. choose a time zone;
6. avoid overlap with another trigger for the same job;
7. define who reviews failures.

Scheduling is the last step of productionization, not the first.

## Reconcile after a migration

Do not rely on source and destination counts alone. Compare:

- total expected records;
- records accepted, skipped, and failed;
- unique source IDs present at destination;
- representative transformed fields;
- relationship counts;
- attachment or child-resource counts;
- a deterministic sample plus high-risk edge cases.

Store a machine-readable reconciliation result with the run and surface discrepancies on the job dashboard.