Data Migrations

Between module releases the expected data format can change: an asset CSV gains columns, a new CSV appears, an input JSON changes shape. The workspace data directory is user-mutable and is deliberately never overwritten by rescile mod sync — so a module upgrade alone leaves old-format data in place. Data migrations close that gap:

• Modules own the migration logic for the release that changes the data schema.
• The controller owns detection, ordering, execution, and the startup gate.
• The packager only ships migration files — it never interprets or runs them.

Data format is tracked separately from both the module version and rescile_version, because both are unreliable witnesses: two module releases can carry the same version while changing data shape.

For concrete recipes — “a new column is added”, “a column must be renamed”, “one column must be split into two” — see Migration Examples.

Declaring the data schema

Each module declares the data format it expects with an integer:

# module.toml
[module]
name = "app-runtime"
version = "0.0.2"
data_schema = 2    # integer; absent means 1 (all pre-migration modules are format 1)

Bump data_schema in the release that changes the format, and ship the matching migration steps with that release. The workspace root module participates in the same mechanism: a workspace can declare its own data_schema and ship migrations/ next to its module.toml (authored by the workspace owner, not a package), which is how user-owned data that no dependency bootstrapped gets migrated.

Migration files

Migrations live in the module package under migrations/ — one TOML file per step, with the step’s Rhai script (when it has one) co-located under the same stem:

migrations/
├── 001-instance-fields.toml    # ops step: declarative [[op]] column operations
├── 002-server-csv.toml         # script step: metadata + script reference
├── 002-server-csv.rhai         # the script itself
└── 003-cleanup.toml

A module ships one file per step — each release appends new step files and never edits old ones (the same discipline as Rails’ db/migrate/ and Django’s numbered migration modules, so the per-release diff shows only additions). The step id is the file stem, and each file declares the schema range it covers plus — critically — the targets: the only files the step is allowed to touch.

# migrations/001-instance-fields.toml
from = 1
to = 2
targets = ["instance.csv"]
description = "Add retention_days and backup_policy columns, fill defaults"

# ...the step's body: [[op]] column operations, or a script reference.

A step is either a list of declarative column operations or a Rhai script — never both:

# migrations/002-server-csv.toml
from = 2
to = 3
targets = ["server.csv", "config.json"]
description = "Introduce server.csv, move zone keys in config.json"
script = "custom-name.rhai"     # omitted: defaults to the co-located 002-server-csv.rhai

Validation rules enforced by the controller:

• Steps must chain contiguously per module (1 → 2, 2 → 3, …) and have unique IDs (unique file stems).
• targets are root-relative, exactly like script paths: instance.csv (and subdir/instance.csv) resolve under assets/, config.json under input/. Writing the root explicitly ("assets/instance.csv") is rejected — it would aim at assets/assets/instance.csv. Only .csv and .json files can be targets.
• Forbidden as targets (validation error): the vault secrets file, action_queue.json and state-dir files, module.toml, rescile-lock.toml, bootstrap-state.json, migration-state.json, anything outside data/, and files the owning module’s generators write (target_asset / target_input outputs — generator output is derived state; fix the generator, don’t migrate its files).
• No two modules may declare the same target — shared files must move into the owning module.

Multi-step releases add one step file per schema hop; the chain is validated across files (each step’s from must equal the previous step’s to):

migrations/001-instance-fields.toml   # from = 1, to = 2
migrations/002-server-csv.toml        # from = 2, to = 3
migrations/003-cleanup.toml           # from = 3, to = 4

What belongs in a migration — and what does not

Migrations are structural recipes, not content scripts:

A step that hardcodes per-row business values to make a diff look byte-exact is a smell: derive what follows from the data itself, add the rest of the columns empty, and let the workspace owner (or the importer’s column defaults) fill the content.

The full step vocabulary — column operations and the sandboxed script API — is documented with one example per use case in Migration Examples.

Migration workflow

# 1. Stop the controller and any other writers (runners can keep idling).
systemctl stop rescile-ee

# 2. Back up the workspace data directory (migration also keeps per-step backups,
#    but a full copy is the safe baseline).

# 3. Plan: show the steps that would run, never touching data.
rescile-ee migrate --data-dir ./data --check

# 4. Apply, verify, and record.
rescile-ee migrate --data-dir ./data

# 5. Start; the controller re-validates schema compatibility on startup.
systemctl start rescile-ee

Note: The migrate subcommand ships in every component that reads data assets — rescile-ce, rescile-ee, and rescile-controller — all sharing one implementation. Substitute the binary you deploy in the examples above. The service binary rescile-controller exposes the same flow as mode flags instead of a subcommand:

rescile-controller --artifact-url <url> --migrate-check --migrate-data-dir ./data
rescile-controller --artifact-url <url> --migrate-apply --migrate-data-dir ./data

--check prints the plan — module, from → to, step IDs, the actual column operations or script paths — and exits non-zero when migrations are required, so it can gate deploy pipelines.

The controller never migrates silently on startup — an incompatible data format is instead a hard error, and serve, save, and validate refuse to run until migrate has been applied. An existing deployment with no migration-state.json and a module requiring schema > 1 is also refused — the starting schema is unknown, so the controller does not guess. Record it explicitly:

rescile-ee migrate --data-dir ./data --baseline 1

--baseline records the starting schema and immediately applies the steps from there, so a stopped workspace migrates in one invocation. The same flags exist on rescile-ce (migrate subcommand) and rescile-controller (--migrate-baseline).

What apply does per step

1. Acquires an exclusive lock on migration-state.json (concurrent serve or migrate is refused).
2. Copies every target to data/.migrations/backup/<module>/<step-id>-<timestamp>/, preserving permissions. Backups older than 30 days are pruned at the start of each run (--keep to retain everything).
3. Runs the step — the compiled column operations, or the Rhai script, with file access restricted to the declared targets.
4. Verifies outputs parse (CSV shape, JSON validity).
5. Records the step in migration-state.json atomically.

After the last step, the controller runs a full graph build. On failure it reports the step and the backup path — there is no automatic rollback; restoring is a plain copy from the backup directory.

The execution record

Applied steps are tracked in data/migration-state.json, separate from bootstrap-state.json (which tracks packaging, not data format):

{
  "version": 1,
  "modules": {
    "app-runtime": { "data_schema": 2, "applied": ["001-instance-fields"] }
  }
}

Crash recovery is re-planning: recorded steps never re-run; the interrupted step runs again. Declarative steps are idempotent by construction; script steps must be written idempotently (see the examples).

Startup gate

Every component that reads data assets — rescile-ce, rescile-ee, and rescile-controller — compares the schema each module requires against the recorded state during serve, save, and validate, and fails closed when a migration is outstanding (the message names the component printing it):

ERROR: workspace data for module 'app-runtime' is at schema 1, module requires 2.
       Run: rescile-ee migrate --data-dir ./data --check

rescile mod add and mod update warn when a newly resolved dependency requires a data migration.

Ordering, downgrades, and derived data

• Ordering: modules migrate in lockfile dependency order (dependencies first); steps within a module run strictly in sequence.
• Downgrades: migrating down is not supported. Rolling a module back to an older release means restoring data from the step backups or re-pinning an older lockfile — migrate --check reports the conflict instead of running destructive steps.
• Derived data: saved graph artifacts are invalidated, not migrated — the graph is rebuilt from migrated data on the next serve.

Safeguards summary

Note: rescile mod sync is unchanged — it still bootstraps new installs with starter files and never overwrites existing data. Upgrading the data of an existing workspace is exclusively the job of migrations. rescile mod update warns when a dependency update requires a data migration.