Module Packaging Guide

Data Generators

How to materialize API data before rescile builds a graph.

Edit this page on GitHub

Data Generators

Data generators materialize remote API data as JSON inputs or CSV assets during Phase 0. Request workflows live in external/*.toml; module.toml controls when a workflow runs and where rescile stores its result.

Define the API workflow

Create a named HTTP or GraphQL fetcher:

# external/catalog.toml
[external]
name = "catalog"
result_step = "load"
ttl = "6h"

[[external.steps]]
name = "load"
method = "GET"
url = "https://api.example.com/catalog/{{ args.region }}"

The external fetcher format is HTTP-only. It has no runtime field. Fetchers support GET, POST, GraphQL, sequential response-dependent steps, pagination, headers, JMESPath filtering, Vault secrets, and exact-key response caching.

Bind the workflow to Phase 0

Reference the fetcher from module.toml:

[generators.catalog]
external = "catalog"
target_input = "catalog.json"
args = { region = "{{ params.region }}" }
condition = "on_missing"
abort_on_failure = true

Each binding supports:

  • external: fetcher name, resolved as external/<name>.toml in the module.
  • target_input or target_asset: persisted destination. Specify exactly one.
  • args: literals or templates using params.<name> and env.<name>.
  • jmespath: optional generator-specific result shaping before materialization.
  • condition = "on_missing": skip resolution when a valid target exists.
  • abort_on_failure: abort graph construction when fetching or validation fails.

Phase 0 arguments cannot use model-only values such as origin_resource, value, or responses. Vault secrets belong in the external fetcher and are not generator arguments.

A generator must not combine external with inline request, command, environment, secret, pagination, runner, response-filter, or TTL fields. Cache TTL belongs to the external fetcher.

Validation and output

rescile validates generated data before replacing the target:

  • target_asset: converts an array of JSON objects to CSV and validates declared asset columns and types.
  • target_input: writes JSON and validates the declared input format and fields.

Target replacement is atomic. Invalid output does not replace an existing file.

Caching and offline mode

Fetcher definition and rendered arguments identify the cache entry. Generator bindings and model external! bindings share that entry when both use the same fetcher and arguments.

Resolution order:

  1. condition = "on_missing" keeps an existing target unless --refresh-generators is set.
  2. A fresh exact-key cache is materialized without a request.
  3. Online stale or missing cache sends the request and updates the cache.
  4. --offline accepts a fresh or stale exact-key cache.
  5. Offline cache miss fails when generation is required. No HTTP request is sent.

--refresh-generators bypasses fresh fetcher cache while online. --offline takes precedence. --ignore-generators skips Phase 0 generators.

Deprecated inline and command generators

Inline HTTP fields in module.toml remain executable for the current major release and emit one deprecation warning per generator and importer run. They will be removed in the next major release.

Command/script execution is unsupported. Obsolete command fields remain parseable during the current major release so rescile can emit a focused migration warning before returning the unsupported-runner error. These fields will also be removed in the next major release.

Move inline request fields into external/<name>.toml, then set external = "<name>" on the generator binding.

Run legacy scripts before rescile from CI, systemd ExecStartPre, or another operator-controlled process. Write their results to data/assets/ or data/input/, then start rescile. No replacement command-hook runtime is planned.