Progressive Delivery, Feature Flags & Zero-Downtime Schema Changes

Ship code dark, turn it on gradually

Progressive delivery is the practice of shipping code dark and then turning it on gradually, independent of the deploy. The two tools are feature flags for behavior and the expand/contract pattern for schema. Both exist because, during any rollout, old and new code run at the same time, so every change in flight must be both backward compatible (old code tolerates the new state) and forward compatible (new code tolerates the old state).

Feature flags

Feature flags are runtime conditionals (if flag('new_pricing_engine', user)) evaluated against a flag service (LaunchDarkly, Unleash, Statsig, or a homegrown config-plus-Redis setup). They decouple deploy from release: you deploy the new code disabled, then flip it on for 1% of users, then 100%, and if it misbehaves you flip it off in seconds without a redeploy. That kill switch is the point. Flags also target: by percentage, by user segment, by geo, by tenant, or by an allowlist, which is how you dogfood internally, then beta a cohort, then GA. The same flag doubles as a feature circuit breaker, cut a struggling feature to shed load during an incident. The tax is flag debt: every flag is a live branch, so you must remove flags after full rollout or they rot into untested dead paths.

Expand/contract (parallel change)

Expand/contract migrates schema in ordered, individually-safe steps so that at no point does the deployed code disagree with the schema:

1. EXPAND    add the new column/table (nullable, additive) -- old code unaffected
2. DUAL-WRITE deploy code that writes BOTH old and new; reads still from old
3. BACKFILL  copy historical rows old -> new, throttled + idempotent + restartable
4. MIGRATE READS switch reads to the new column (behind a flag), verify parity
5. CONTRACT  once nothing reads the old column, stop dual-writing, then drop old

Each arrow is a separately deployable, separately reversible step. You never combine "add new" with "drop old" in one deploy, because that is exactly the destructive change that makes rollback impossible.

For the physical DDL on a large hot table, a naive ALTER TABLE can lock the table and stall writes. Online schema-change tools (gh-ost, pt-online-schema-change for MySQL) build a shadow table, backfill it while capturing live changes via triggers or the binlog, and swap it in with a brief atomic rename, so the table stays writable throughout. Backfills must be throttled (chunked, watching replica lag), idempotent (safe to re-run), and restartable (checkpoint progress) because a multi-hour backfill will get interrupted.

Interview nuance: the classic disaster is renaming a column. ALTER TABLE users RENAME COLUMN email TO email_address looks trivial and is a trap: the instant it runs, every still-deployed old instance that selects email breaks, and you cannot roll the code back because the column named email no longer exists. The correct answer is expand/contract: add email_address, dual-write, backfill, migrate reads, then drop email in a later deploy. A rename is never one step in a live system.

Recap: flags decouple release from deploy and give you a per-feature kill switch and targeting, expand/contract migrates schema in individually-safe reversible steps, everything in flight must be backward and forward compatible, and never pair "add new" with "drop old" in a single deploy.