Peering

Peering, also called Internet peering, is the voluntary interconnection of two Internet networks, the autonomous systems the Internet is built from. The two networks exchange traffic directly, instead of relying on an intermediary transit provider to carry it (LINX). Most peering is settlement-free: no money changes hands. It takes two physical forms. Public peering crosses the shared switch fabric of an Internet exchange point (IXP). Private peering uses a direct cross-connect between exactly two networks (Netnod).

Peering is not transit. It is also not a technology. Transit is the paid service in which one network carries a customer's traffic to the whole Internet (Netnod). A peering session only reaches the peer and the destinations connected downstream of it (Cloudflare). So a peer is never a route to everywhere else. The mechanism underneath is ordinary BGP. What makes peering peering is the commercial and routing arrangement. For a CDN the payoff is a shorter, better-controlled path from its edge servers to the networks that serve users. That path costs less than buying the same traffic as transit.

How it works

Peering begins as an agreement. It "can be as informal as a handshake agreement, or defined by contractual terms" (Cloudflare). The two networks then interconnect physically. They exchange routes over BGP. Each network announces the blocks of IP addresses it controls. These blocks are called prefixes. Each network also announces the addresses connected downstream of it. For an ISP, that means its customers (Cloudflare).

• Public peering happens at an IXP. An IXP is an Ethernet switch, or a set of switches, in a colocation facility. Every peering network in the building connects to it. One connection lets a network peer with many others, though each peering arrangement still has to be negotiated separately. No new cabling is needed (Netnod).
• Private peering is a direct connection between exactly two networks. Cloudflare's term for it is a private network interconnection (PNI). Cloudflare says "some PNI connections are simply a large fiber optic cable that plugs into a physical port on each network". Other PNI connections are virtual, running through a third-party network (Cloudflare). Netnod describes the same thing inside a colocation facility. Two networks put routers in the same building, but they run a direct cable between them instead of going through the exchange switch. This is most useful when the volume of traffic "won't fit on a shared connection to an exchange point" (Netnod).
• Bilateral or multilateral: sessions can be negotiated one peer at a time. Or a member can connect to the exchange's route server and "replace some or all of your separate BGP sessions to your peers with one single session". This also makes it easier for new peers to start exchanging traffic at the exchange "from day one" (Netnod; LINX).
• Settlement: settlement-free is the norm. Cloudflare states that peering is free in over 99% of cases and calls the free form "settlement-free peering" (Cloudflare). Netflix runs settlement-free interconnection as a named program component (Netflix Open Connect). The exception is paid peering, "also known as 'partial transit'". It looks like a normal peering session from outside, but one network pays the other. This happens "when one network values the arrangement more than the other does" (Netnod).

Who to peer with, and where, is looked up in PeeringDB. Cloudflare points peering candidates at its PeeringDB record for the list of mutual locations. It uses that record for automated provisioning and network notifications (Cloudflare peering policy; PeeringDB).

Why it matters for a CDN

A CDN has not delivered anything until its bytes reach the network that runs the user's last mile. Peering is the handoff at that boundary. Sending traffic directly there, rather than "through additional intermediary backbone networks", is more efficient routing. Peering "can shorten network paths and reduce latency" (Cloudflare). Peering also shapes the middle mile between a CDN's own sites. But the delivery win is the edge-to-ISP handoff.

Control matters as much as distance. Traffic sent over transit crosses the Internet by whatever path the transit provider chooses. When there is a problem, "slow connections or packet loss, for instance — the network is at the mercy of its transit provider". By contrast, an operator that peers "has more control over external paths, and can easily adjust routing to avoid problem network segments" (Netnod).

Peering is also a cost lever, because transit is billed per unit of capacity. Netnod's arithmetic: at EUR 10 per Mb/s per month, 100 Mb/s handed to a peer willing to take it for free saves EUR 1,000 per month. The caveat is that "the infrastructure required to peer also costs money" (Netnod). The Internet Society puts the same point structurally: networks that peer "avoid paying transit fees to third-party providers, which can reduce service fees for users" (Internet Society).

The strongest form of the idea is putting the cache inside the ISP network, rather than meeting it at an exchange. Akamai's Accelerated Network Partner program minimizes "transit and other costs through deployment of Akamai servers in operator facilities" (Akamai). Cloudflare offers an embedded cache program to networks that exchange at least 10 Gbps of traffic with it. Cloudflare adds that it "can sometimes accommodate smaller networks" (Cloudflare peering policy). Netflix's Open Connect Appliances "can be embedded in your ISP network" and "are provided to qualifying ISP partners at no charge" (Netflix Open Connect).

What CDNs do

• Cloudflare (AS13335) has an open peering policy. It "will peer with networks that have a presence in mutual locations". There is "no need to be a Cloudflare customer to peer with Cloudflare" (Cloudflare peering policy; Cloudflare). Public peering is automated through its Peering Portal. Networks exchanging more than 10 Gbps of peak traffic with it in a single location may request a PNI (Cloudflare peering policy). It reports being interconnected with over 13,000 service provider, cloud and enterprise networks (Cloudflare network page). Its PeeringDB record lists an open policy, contracts not required, and presence at over 350 exchanges (PeeringDB).
• Fastly (AS54113) has "a selective but generally open peering policy — peers are selected based upon performance, capability, and where traffic needs to be delivered". It expects peers to exchange traffic "at all available IXPs that are shared in common". Fastly warns that it has no backbone, so it does not announce a consistent set of prefixes across IXPs. This means "peers will not receive a consistent routing table from Fastly across multiple points of interconnection". Its traffic profile is "dominantly outbound" (Fastly).
• Akamai (AS20940) "will openly peer with any network at IXP locations where mutually present, and will also consider private interconnection relationships on a case-by-case basis". Its network partnership program goes further, deploying Akamai servers inside operator facilities (Akamai).

Watch out for

• A peer is not a route to the rest of the Internet. A peering session reaches the peer and the destinations downstream of it, not everywhere (Cloudflare). Transit still supplies global reach, while peering supplies "cost-effective, high-performance local connectivity" (LINX). Fastly makes the rule explicit: peers must "only send traffic destined for prefixes announced by Fastly" and must not point a default route at it (Fastly).
• Settlement-free is the norm, not a guarantee. Paid peering exists (Netnod). Depeering can end an arrangement: "the two networks may unplug from each other altogether, or one network may simply decide to start charging the other network for transit". This is likelier when one network is larger and "has an advantage in the market" (Cloudflare).
• Peering infrastructure costs money. A port from the exchange operator, colocation space, metro Ethernet, possibly a new router, and the people to find and negotiate with peers all have to be paid for. So peering is not automatically cheaper than the transit it displaces (Netnod).
• Route-server peering with a CDN may not get you everything. Cloudflare recommends bilateral BGP sessions for optimal traffic delivery, "as we advertise a limited number of prefixes over route servers" (Cloudflare peering policy). So a multilateral session alone can leave a peer without the routes it wanted.
• Prefixes differ per location. A CDN with no backbone scopes announcements to the point of interconnect. It offers local POP-specific prefixes plus selected regional or global anycast prefixes. So the same peer sees different routes at different exchanges (Fastly).
• Congestion is a shared obligation, and sessions can be suspended. Fastly requires peers to "maintain congestion free interconnection" and augment capacity as needed. Fastly reserves the right to suspend without notice on "a severe quality of service issue such as high latency, packet loss, or jitter" (Fastly). Cloudflare reserves the right "to suspend, without notice, any peering connection experiencing severe quality of service issues". Cloudflare expects both parties to upgrade port capacity in a timely manner as part of its PNI requirements (Cloudflare peering policy).
• There is an operational bar. Cloudflare requires a complete and updated PeeringDB entry, a 24x7 NOC responsive to peering issues, and adherence to BCP-38 with robust route filtering. RPKI validation and ROAs are highly recommended, rather than required (Cloudflare peering policy). Fastly adds technical rules: a publicly routed ASN, no IPv4 prefixes with a mask longer than /24, and a default maximum of 250 prefixes per session (Fastly).

Best practice

• Peer at every location you share with the networks that serve your users. Cloudflare recommends "establishing sessions in all mutual locations" (Cloudflare peering policy). Fastly expects traffic exchange at all shared IXPs for traffic distribution and redundancy (Fastly).
• Use the route server for reach and for day-one turn-up. But establish bilateral sessions with content networks that advertise a reduced prefix set over route servers (Netnod; Cloudflare peering policy).
• Keep PeeringDB, IRR route and route6 objects, and RPKI ROAs complete and current. Fastly builds its BGP prefix lists from exactly that data (Fastly; Cloudflare peering policy).
• Decide and publish your policy: peer with anybody who asks, or evaluate case by case (Netnod). Back it with a peering contact and a 24x7 NOC (Fastly; Cloudflare peering policy).
• Discipline each session: prefix limits, robust route filtering, announcements consistent at every point of interconnection unless agreed otherwise, and port upgrades before interfaces saturate (Fastly; Cloudflare peering policy).
• Run the break-even per candidate peer: how much traffic could shift, what transit it displaces, and what the connectivity and the people to manage it will cost (Netnod).