Shard-Key Selection, Hotspots & the Celebrity Problem

The decision you cannot cheaply undo

Consistent hashing spreads keys evenly given good keys. The shard key itself is the higher-leverage decision, and it is the one you cannot cheaply undo. A bad shard key silently rebuilds a single-node bottleneck inside your distributed system, and changing it later means migrating the whole dataset.

A good shard key has three properties. High cardinality: many distinct values, so load can actually spread. Sharding by country or status (a handful of values) means a handful of partitions, and one value dominates. Even access distribution: not just many values, but roughly uniform traffic per value. A high-cardinality key where 1% of values get 99% of reads is still a hotspot. Alignment to the dominant query: the key the most frequent/critical query filters on, so that query hits one partition instead of scatter-gathering across all of them.

These pull against each other, which is the whole difficulty. user_id gives high cardinality and even spread but forces "all posts in this group" to scatter-gather. group_id co-locates a group's posts for cheap reads but hotspots a giant group. Naming the tension and committing to a side (usually: align to the dominant query, then mitigate the resulting hotspot) is the senior move.

The celebrity / hot-key problem

Shard a social graph by user_id and one celebrity with 100M followers and 1000x normal traffic maps to one partition, which then owns 1000x the load of its peers. No amount of consistent hashing helps: it is one key, so it is one node. Mitigations, roughly in order of reach:

• Salting / key-splitting: append a bucket suffix to spread one logical key across K physical partitions (celebrity_id:0..K-1). Writes pick a random bucket; reads fan out to all K and merge. Use it for the few known whales, not everyone.
• Sub-partitioning: split a hot partition's range into finer ranges dynamically when it exceeds a load threshold (DynamoDB adaptive capacity and split-for-heat do this for you).
• Dedicated shards for whales: route known celebrities/mega-tenants to their own isolated nodes so their traffic cannot starve normal users. Common in multi-tenant SaaS.
• Caching + fan-out-on-read: for a celebrity's timeline, cache aggressively and read the celebrity's posts at read time rather than fanning out writes to 100M follower inboxes (the classic Twitter hybrid).

Entity groups / co-location: deliberately put data that is transacted together on the same partition so common operations stay single-shard (Google Megastore's entity groups, and the reason you shard an e-commerce order and its line items together by order_id).

Compound keys serve multi-tenancy: (tenant_id, entity_id) isolates tenants (a tenant's data is co-located) while entity_id preserves cardinality within a tenant. Combine with dedicated shards for the biggest tenants.

Interview nuance: always say resharding is expensive and must be planned before you need it. Pre-split into more logical partitions than nodes (e.g. 1024 logical shards on 16 physical nodes) so growth is a cheap remap of logical-to-physical, not a re-key. And design online migration: double-write to old and new, backfill, verify, then cut over reads, so you never take downtime to reshard.

Recap: a good shard key is high-cardinality, evenly accessed, and aligned to the dominant query; the celebrity problem defeats plain hashing because one key is one node, so mitigate with salting, sub-partitioning, or dedicated shards; use entity groups and compound keys to keep transactions single-shard; and plan resharding and online migration early.