Anatomy of a Change Event

Two ways to notice that a row changed

A warehouse needs to know when the operational database changed. There are exactly two families of answer, and interviewers ask you to compare them by name.

Query-based capture polls the source: SELECT * FROM customers WHERE updated_at > :last_seen. It is simple, it needs no special permissions, and it works on any database with a decent timestamp column. It also has three problems that get worse as the table grows. It loads the source with a scan on every poll, it only sees the LATEST state of a row (two edits between polls collapse into one), and, the fatal one, it misses deletes entirely. A deleted row does not have an updated_at any more. It simply is not there, and a polling query cannot see an absence.

Log-based capture reads the database's own write-ahead log: the Postgres WAL, the MySQL binlog, the SQL Server transaction log. Every committed change is already in there in commit order, because that is how the database itself guarantees durability. A connector (Debezium is the standard one) tails that log and emits one event per change. Cost to the source is near zero, deletes are captured like anything else, and you get every intermediate state instead of a sampled one. The price is operational: you need replication privileges, a slot that will fill your disk if the consumer stalls, and a plan for schema changes.

The envelope

Debezium emits a structured record per change, an envelope of {op, before, after, ts_ms, source}. Flattened into a sink table that keeps both images, which is the shape this module grades, it is this:

• op is the operation: 'c' create, 'u' update, 'd' delete. (Snapshots also emit 'r' for read, which you will meet the first time a connector backfills a table.)
• The before image holds the row as it was. It is NULL on a create, because there was no row.
• The after image holds the row as it now is. It is NULL on a delete, so the before image is the only address you have left for a row that is gone. Capturing a delete at all is the entire reason log-based capture beats polling.
• ts_ms is when the source committed the change.
• lsn is the log sequence number, the position in the write-ahead log. It is the only field that defines true order, and the next lesson is built on that fact.

A delete event is not a tombstone, and a CDC round will check that you know it. The delete event is the record above: op = 'd', before image present, after image null. Debezium then emits a tombstone right after it, a separate record carrying the same key and a null VALUE, whose only job is to let Kafka log compaction drop that key from the topic entirely. It is controlled by tombstones.on.delete, which defaults to true. The warehouse-side DELETE that the next lesson writes is still worth calling the tombstone branch, and that shorthand is fine, as long as you can say what the tombstone record itself is.

Table

One row's whole life as change events. Which image is NULL tells you the operation even before you read op.

lsn	op	before_email	after_email	what the source did
4101	c	NULL	grace@northwind.io	INSERT: no before image exists
4104	u	grace@northwind.io	ghopper@northwind.io	UPDATE: both images present
4113	d	ghopper@northwind.io	NULL	DELETE: after image is NULL

Rebuilding state from events

The changelog is the truth, but nobody queries a changelog to answer "what tier is customer 103 on right now". You reduce it: for each key, keep the event with the highest lsn, then read its after image. That is ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY lsn DESC) and keep rn = 1, and it is the first half of every changelog consumer ever written. If that winning event has op = 'd', the key does not exist any more, and the second half of the consumer deletes it rather than writing a row full of NULLs.

Common mistake: ordering the reduce by ts_ms instead of lsn. Source timestamps have millisecond resolution and a busy table commits several changes inside one millisecond, so ts_ms ties and the tie is broken arbitrarily. Worse, a timestamp comes from a clock, and a clock can be stepped backwards by a correction, so ts_ms can disagree with commit order outright. Customer 106 in this changelog is that case: its update carries an earlier ts_ms than its create, so a reduce ordered by ts_ms hands you the email and tier the customer no longer has. The log position never ties and never goes backwards, because it is a position in a file that only ever grows.

Interview nuance: the question is usually phrased "CDC versus dual writes". Dual writes means the application writes to the database and then also publishes an event, and it is wrong for a reason worth being able to say out loud: those two writes are not one transaction, so a crash between them leaves the database and the stream permanently disagreeing. Log-based CDC has no such window, because the log IS the transaction. Follow that with "and the hardest part of a CDC pipeline is schema evolution", which appears close to verbatim in 2026 question banks, and you have covered both halves of the answer.

On a real platform this differs. Here the envelope is nine plain columns in SQLite. Debezium emits it as Avro or JSON onto a Kafka topic with the before and after images as nested structs and a source block carrying the database, table, and log coordinates. The stock unwrap Single Message Transform, ExtractNewRecordState, flattens that envelope to the AFTER image plus optional metadata columns such as __op and __source_ts_ms, and under it a delete carries no before image at all. Keeping both images, as this sink table does, is a deliberate choice you make when the consumer needs the departed row. AWS DMS produces the same idea with its own column names, Microsoft Fabric mirroring hides it behind a managed replica, and Apache Hudi holds the streaming-CDC niche among the table formats. The reasoning is identical in all of them: op tells you the branch, the after image tells you the value, and the log position tells you the order.