ECMP (Equal-Cost Multi-Path) (ECMP)

ECMP is a network routing strategy where a router has multiple next-hop paths to the same destination, all with identical routing metrics. Instead of picking one path and leaving the others idle, the router spreads traffic across all of them. This gives you more aggregate bandwidth and built-in redundancy without any application-level load balancing.

The router decides which path a packet takes by hashing a combination of fields from the packet header, usually source IP, destination IP, source port, destination port, and protocol. All packets with the same hash go to the same path, which keeps individual TCP connections on a single path (preventing out-of-order delivery). Different connections naturally spread across paths.

CDNs use ECMP extensively in their backbone networks. A CDN edge PoP might have multiple uplinks to transit providers or to the CDN's own backbone routers. ECMP across these links means the PoP's aggregate capacity scales with the number of links. If one link fails, traffic shifts to the remaining paths automatically, within the convergence time of the routing protocol (typically under a second with BFD).

ECMP interacts with Anycast in interesting ways. When a CDN advertises the same IP prefix from multiple PoPs via BGP, routers in the network see equal-cost paths to that prefix. ECMP at intermediate routers can split traffic between the PoPs. This is usually fine for UDP (like DNS), but for TCP it can cause problems because packets from the same connection might end up at different servers. CDNs handle this by ensuring consistent hashing at each hop or by using Anycast only for the initial connection and then pinning to a specific server.

At the data center level, ECMP enables fat-tree (Clos) network topologies. In a leaf-spine architecture, every leaf switch has equal-cost paths through every spine switch. ECMP across spine switches means the network can scale by adding more spines, and any single spine failure only reduces capacity proportionally rather than breaking connectivity.

The hash algorithm matters for load distribution. A poor hash can lead to polarization, where traffic is unevenly distributed across paths. Modern routers use high-entropy hash functions that consider all five tuple fields. Some support configurable hash seeds to ensure different routers in the path don't use the same hash, which could cause the same subset of flows to always take the same path at every hop.