Feature Stores & Training/Serving Skew

A feature store exists to kill training/serving skew

Training/serving skew is the single most common cause of a model that looks great offline and quietly underperforms in production, and it is subtle enough that teams ship for months before noticing. If you learn one thing in this lesson: the same feature value the model saw at train time must be the value it sees at inference time, and that is harder than it sounds.

The two sources of skew

First, code divergence: the training pipeline computes "average order value over the last 7 days" in a Spark job, and the serving path recomputes it in Java service code, and the two implementations disagree on time zones, null handling, or rounding. Second, time divergence: at training you accidentally use the feature's current value instead of its value as of the moment the labeled event happened, which leaks future information into the past.

The dual-store architecture

                 +------------------ feature definition (one) ------------------+
                 |                                                              |
   raw events -> feature pipeline ---> OFFLINE store (warehouse / Parquet) ---> point-in-time join -> training data
                          |                                                                          (train time)
                          +---------> ONLINE store (Redis / DynamoDB)  ---> low-latency get -> inference
                                                                                            (serve time)

The offline store holds the full history of every feature value with timestamps, in a warehouse or Parquet on S3, optimized for large point-in-time joins. The online store holds only the latest value per entity, in Redis or DynamoDB, optimized for single-digit-ms point lookups by entity key. Both are populated by one pipeline from one definition, which is what guarantees the serving path and the training path compute the feature identically.

Point-in-time correctness

When you build a training row for "user U at event time T," every feature must be joined as-of T, using the last value known strictly before T, never a value computed after T. If a user's "total lifetime purchases" feature is joined at its current value while the label is a purchase from six months ago, the model learns from the future and posts fantastic offline numbers that collapse in production. Feature stores implement this with an as-of join keyed on entity and event timestamp.

Interview nuance: if the interviewer asks "how do you know your feature store works," the strong answer is not "we tested it," it is "we log served feature vectors and compare them to the offline-computed vectors for the same entity and time; skew shows up as a mismatch rate."

Freshness tiers

Batch features (7-day average spend) recompute hourly or daily. Streaming features (clicks in the last 5 minutes) update within seconds via Kafka plus Flink. On-demand features (distance between user and merchant) are computed at request time from request inputs because they cannot be precomputed. A registry tracks each feature's definition, owner, freshness, and lineage so features are reused rather than reinvented, and so you can reason about high-cardinality features whose online storage cost (one row per user times millions of users) can dwarf everything else.

Recap: a feature store uses a dual offline/online store fed by one definition to kill code-divergence skew, enforces point-in-time as-of joins to prevent label leakage, tiers features by freshness SLA, and proves correctness by comparing served vectors to offline vectors.