Content Negotiation

Content negotiation is the HTTP mechanism by which one URL serves different representations of the same resource to different clients. The client states what it can handle in request header fields. The server selects which representation to return. It is defined in RFC 9110, section 12. It is not a redirect, not a separate URL per variant, and not client-side feature detection. The URL stays the same, and the choice is made server side. HTTP itself is not aware of the resource semantics. The consistency of what comes back is entirely down to whatever selects or generates the responses.

RFC 9110 names three patterns that are visible in the protocol. In proactive (server-driven) negotiation, the server selects based on the client's stated preferences. In reactive (agent-driven) negotiation, the server returns a list of alternatives, and the client selects. In request content negotiation, a server states in a response which media types or codings it prefers for later requests. CDN work is almost entirely the proactive form, because it settles the answer inside the first response instead of costing a second request. The bill arrives at the cache. A proactively negotiated response is only conditionally reusable. As a result, the origin has to advertise the request headers that selected it in the Vary response header. Every shared cache on the path has to key on them.

How it works

The client sends its preferences as request header fields. Accept carries preferences for response media types, Accept-Encoding for content codings, and Accept-Language for natural languages. Accept-Charset still exists, but it is deprecated because UTF-8 has become nearly ubiquitous, and a detailed charset list mostly wastes bandwidth and eases passive fingerprinting. Each entry may carry a quality value: a q weight normalized to the range 0 through 1. A weight of 0.001 is the least preferred, 1 is the most preferred, and 0 means "not acceptable". An entry with no q parameter defaults to a weight of 1. Absence of a negotiation field means the sender has no preference on that dimension. It does not mean the sender prefers the default.

The server compares the representations it can produce against those preferences and returns the one it judges best. That is proactive negotiation. HTTP deliberately leaves the selection algorithm to the server. Suppose a negotiation field is present and none of the available representations is acceptable under it. The origin may honor the field with a 406 (Not Acceptable) response. Or it may disregard the field and answer as though the request were not negotiated on that dimension. A client therefore cannot rely on its preferences being honored.

Because the response now depends on request headers, the origin declares which ones mattered in Vary. Vary describes what parts of the request, aside from the method and target URI, might have influenced selection. It also expands the cache key. A cache must not use a stored response carrying Vary, without revalidation, unless every request header field named there matches the field in the request that produced the stored response. Matching permits only semantics-preserving tidying: whitespace, combining repeated field lines, and normalization known to be identical in meaning, such as case or ordering where order is not significant. An absent header matches only an absent header. The practical effect is one stored entry per distinct value of each named header.

In the reactive form, the server sends a list of alternatives instead. The 300 (Multiple Choices) and 406 (Not Acceptable) responses carry information about the available representations. The user or user agent selects among them, paying for a second request. HTTP defines no mechanism for automatic selection among them. That is a large part of why the reactive form is rare in practice.

Why it matters for a CDN

Among the serious disadvantages RFC 9110 lists for proactive negotiation is that it "limits the reusability of responses for shared caching". A CDN is that shared cache, so the entire cost lands there. Fail in one direction, and the edge stores the first variant it happens to serve, say AVIF, then hands it to a client that can only decode JPEG. Fail in the other direction, and the edge keeps a separate object for every distinct header value it sees.

The second failure is the expensive one. Vary widens the cache key. Real Accept and Accept-Language values differ across browsers, versions and locales in ordering and q weights. As a result, identical bytes get stored many times over. Each copy has to be filled from origin separately. Each copy can also be evicted before it is ever reused. So the hit ratio drops, and origin traffic rises. Compression is where this bites hardest. As Fastly puts it, compression support is one of the most common reasons to vary output on a request header. Near enough every text response on a CDN is negotiated over Accept-Encoding.

What CDNs do

The shared move is normalization: rewrite the negotiating request header to a canonical value before it reaches the cache, so near-identical clients collapse onto one entry. Note that this is something the edge does to the request. A cache on its own may only apply semantics-preserving matching. So it cannot decide that "gzip, deflate" and "deflate, gzip" deserve the same entry as a coarser bucket would.

• Fastly: normalizes Accept-Encoding on every inbound request, before vcl_recv, because it "has a huge impact on cache performance when responses include Vary: Accept-Encoding". The value is cut down to a single token. It becomes gzip if the original mentioned gzip. It becomes deflate if the original mentioned deflate instead. Otherwise the header is removed. On services with Brotli enabled, the FASTLY recv macro reduces it to br when the original mentioned br. The original is preserved in Fastly-Orig-Accept-Encoding. For the other dimensions, Fastly ships VCL functions: accept.encoding_lookup, accept.language_lookup, accept.media_lookup, accept.charset_lookup. These pick the best match from a list of representations you actually have.
• CloudFront: its cache policy carries explicit Gzip and Brotli compression settings (EnableAcceptEncodingGzip, EnableAcceptEncodingBrotli). They are opt-in, not on by default. With both enabled, a viewer advertising both has the header normalized to Accept-Encoding: br,gzip. That normalized value, not the viewer's original, goes into the cache key. A viewer advertising only gzip gets Accept-Encoding: gzip. A viewer advertising neither has Accept-Encoding left out of the cache key altogether. CloudFront then sends Accept-Encoding: identity on the corresponding origin request. Identity is an origin-request value, not a cache-key value. AWS warns to leave these settings off when the origin does not actually return compressed objects, because enabling them can then decrease the cache hit ratio.
• Cloudflare: with Polish's WebP option, the WebP version is served only when the browser's Accept header includes WebP and the WebP is significantly smaller than a lossy or lossless recompression of the original format. Polish converts only standard formats to WebP, and it leaves origin-served WebP alone. Polish may not be applied at all to an origin response that contains a Vary header. The only accepted one is Vary: Accept-Encoding. Its documented conversions stop at WebP. AVIF comes from image transformations with format=auto, which automatically serves the most efficient format the requesting browser supports. It is invoked either through the /cdn-cgi/image/ URL or from an edge function. In a hand-written Worker, you parse Accept yourself.

Watch out for

• No Vary at all. The classic bug: a cacheable negotiated response with no Vary lets a shared cache hand one client's variant to everybody.
• Vary missing from the default response only. RFC 9111 calls this out by name. Some resources send Vary on negotiated responses but omit it from the plain one. This causes caches to pin that default for later requests, even when better variants are stored. When a cache holds several responses for a URI and one lacks Vary, it should prefer the most recent stored response that has a valid Vary value.
• Over-Varying. Name only the headers that actually change the bytes. Each extra name multiplies entries. Vary on User-Agent or Cookie, and the cache becomes effectively per-client. Authorization never needs naming, because reuse of such a response for a different user is already prohibited by that field's definition.
• Vary: *: a stored response whose Vary value contains the member "*" always fails to match. So it can never be reused without going back to the origin. A proxy must not generate it.
• q=0 and the wildcard. A client sending Accept: */*;q=0 is explicitly asking for a 406 if a more preferred format is not available. But it still has to handle a different response, because the server is allowed to ignore the preference.
• Client Hints are negotiation too. Client Hints (RFC 8942) are an opt-in proactive-negotiation framework advertised with the Accept-CH response header. When a server adapts a cacheable response to them, it must also generate Vary naming those hints. The same fragmentation arithmetic applies, on headers that vary far more finely than Accept does.
• Fingerprinting. Sending complete linguistic preferences on every request may be contrary to the user's privacy expectations. Making passive fingerprinting far too easy is one of the stated reasons Accept-Charset was deprecated.

Best practice

• Send Vary on every cacheable response whose content you selected from request headers. RFC 9110 makes that a SHOULD for origin servers. Send it on the default and uncompressed responses too. Fastly's guidance is to include Vary: Accept-Encoding in all responses where compression was considered, whether or not the response was actually compressed.
• Name the smallest set of headers that genuinely selects the bytes. RFC 9110 explicitly allows eliding Vary when the variance matters less than Vary's performance impact on caching.
• Normalize negotiating headers into a handful of canonical values at the edge. Use the CDN's built-in Accept-Encoding normalization where it exists. Bucket the rest yourself, for example one image-format token derived from Accept, so near-identical requests share a cache entry.
• Prefer the CDN's own image negotiation, such as format=auto, over generating a variant per client at the origin. One canonical object is stored, and the format is chosen at the edge.
• Ship a safe default representation. Test the ugly paths too: no Accept header at all, Accept with q=0, a response with no Vary, a client that supports nothing modern. Don't just test a current browser's happy path.