ESI

Edge Side Includes (ESI) is a declarative, XML-based markup language. It lets a CDN edge server, the HTTP "surrogate", assemble one web page out of separately fetched fragments. Each fragment carries its own cacheability and TTL. The origin returns a template studded with esi: tags. The edge fetches what those tags point at, splices the results in, strips the tags, and hands the browser ordinary HTML.

What ESI is not. It is not edge compute. None of your code runs at the edge, only fixed assembly directives. Still, the spec does call ESI an "in-markup scripting language". It carries conditionals, expressions and request variables, so calling it logic-free overstates the case. It is not Server Side Includes. The spec calls ESI only "conceptually similar" to SSI. SSI runs on the origin web server, whereas ESI is "primarily intended for processing on surrogates". And it is not a standard. ESI 1.0 was published as a W3C Note on 4 August 2001 and never advanced. The URL for the language's latest version still serves that same 2001 document.

The one-line version: ESI is the oldest and simplest way to make a page that is 95% static actually cacheable. It works by refusing to let the other 5% decide the fate of the whole document.

How it works

ESI splits a page into a template and fragments. The template is the resource the user's URL actually names. It holds the ESI elements that tell an ESI processor which fragment URIs to fetch and include. Each fragment is a separate resource, so it gets its own cache metadata. The spec's own example gives a TTL "of several days" for the template. It gives "a much lower TTL" for a fragment holding a frequently changing story or ad. Some fragments may be "marked uncacheable" outright.

1. On the way in, a surrogate advertises what it can do. The spec's grammar names the request header Surrogate-Capability. Its prose and examples write Surrogate-Capabilities instead. Either way, the value is a device token and a capability list, as in Surrogate-Capabilities: def="Surrogate/1.0 ESI/1.0". ESI's capability token is ESI/1.0.
2. The origin asks for processing with the Surrogate-Control response header and its content directive. For example: Surrogate-Control: no-store, content="ESI/1.0". A capability token is consumed by whichever surrogate acts on it. It is not passed further downstream.
3. By default, processing happens after caching: "cached entities have any applicable processing applied before being served". So the template can sit in cache and be re-assembled on each request. The spec allows processors and extension directives to change this ordering.
4. For every esi:include, the processor issues a separate request for the fragment URI. The URI is resolved relative to the template and checked against cache first. Implementations "may use the original request's headers (e.g., Cookie, User-Agent, etc.)" when doing so.
5. Each fetched object replaces its element in the markup. Every esi: element is stripped from the output. The assembled page is served. ESI tags never reach the browser.

Templates normally declare the ESI namespace http://www.edge-delivery.org/esi/1.0 on their top-level element. The ESI 1.0 vocabulary is small:

• esi:include: fetch a fragment and splice it in. Optional alt names a fallback URI. Optional onerror="continue" changes failure handling.
• esi:inline: demarcate a fragment delivered inside another response. The processor extracts it and stores it under the given name. A fragment can be marked fetchable="no", meaning it can only be obtained by re-requesting its carrier. Support is optional and advertised separately as ESI-Inline/1.0.
• esi:choose / esi:when / esi:otherwise: conditional content. The first when whose test evaluates true wins. An optional otherwise catches the rest.
• esi:try / esi:attempt / esi:except: run the except branch if the attempt fails. Only include and inline failures trigger it.
• esi:vars plus read-only request variables: HTTP_COOKIE, HTTP_USER_AGENT, HTTP_ACCEPT_LANGUAGE, HTTP_HOST, HTTP_REFERER, QUERY_STRING. They support dictionary access such as $(HTTP_COOKIE{id}) and defaults such as $(HTTP_COOKIE{id}|default).
• esi:remove and the <!--esi ... --> construct: graceful degradation when nothing processes the page.
• esi:comment: author notes, deleted before output.

One asymmetry to internalise: response headers on a fragment, such as Set-Cookie, Server, Cache-Control and Last-Modified, "may be ignored, and should not influence the assembled page". The spec puts that as a may and a should rather than a prohibition. So treat it as unreliable in both directions, and never let a fragment try to steer the client's response. The same 2001 submission also carried a companion ESI Invalidation Protocol. This is a way to purge fragments cached in surrogates when the origin content changes.

Why it matters for a CDN

ESI addresses the page shape a CDN handles worst: almost entirely static, with one small piece that varies per user or per minute. Consider a cart count, a greeting carrying the customer's first name, or a personalised recommendation strip. Any single one of them makes the whole document uncacheable. So every request travels to the origin to rebuild a page that is nearly identical every time. ESI dissolves that all-or-nothing choice. In the spec's words, it "allows surrogates to treat parts of pages as cacheable resources, which gives them the ability to serve resources from cache in more situations."

The payoff is ordinary CDN arithmetic. The shell serves from the edge. Only the small dynamic fragments reach the origin. Origin load falls, and cache hit ratio rises. Surrogates may be deployed "close to the origin server, or throughout the network". The Edge Architecture Specification observes that this is the configuration often called a Content Delivery Network. Because of that deployment, assembly happens near the user instead of a round trip away.

What CDNs do

Implementations deliberately diverge. Nothing enforces conformance. Check your provider's subset before you design a template around a tag.

• Varnish has built-in ESI. It is switched on per response in VCL with set beresp.do_esi = true; inside vcl_backend_response. Its ESI guide is blunt: "we've only implemented a small subset of ESI". That subset is just esi:include, esi:remove and <!--esi ... -->. And "Content substitution based on variables and cookies is not implemented." The rest, its docs argue, is "easier and better done with VCL". Nested includes work, but depth is capped by the max_esi_depth parameter.
• Fastly enables ESI by setting beresp.do_esi to true, which has the same effect as its esi statement. The documented default is false. Fastly's own glossary states that "the version supported by Fastly is only a subset of this and supported only in our VCL platform". So ESI is a VCL-platform feature, not a Compute one.
• Akamai exposes ESI as a property behaviour. Its engineers authored and edited the original specification. "Enable the ESI Processor" turns processing on. Alternatively, "Enable through Response Headers" narrows it to content your origin marks with Edge-control: dca=esi. Sub-options govern passing origin-set cookies and the client IP to the processor, character sets for transcoding, and "Detect and Deny ESI Injection".

Watch out for

• Every include is another request. "When an ESI template is processed, a separate request will need to be made for each include encountered." Implementations "may limit the number of includes used in a single ESI resource" and the number or depth of recursion. This is precisely so ESI does not "monopolize resources or impact end-user perceived performance". Fine-grained fragments buy freshness with edge work and assembly latency.
• Know which failure default you are running under. The spec's default is loud, not quiet. If a processor can fetch neither src nor alt, it "returns a HTTP status code greater than 400 with an error message". Silence is opt-in, through onerror="continue", which makes processors "delete the include element silently". Varnish inverts that. Its docs say the attribute "is ignored by default" and that Varnish "will treat failures to deliver ESI fragments as if there was the attribute" set to continue. It honours onerror only once you set param.set feature +esi_include_onerror. So a missing fragment can vanish without a trace on one platform, and return a 4xx on another.
• Personalised fragments are cached by URI. A fragment is just a resource. Cache it, and the next requester of that URI receives it. Yet a processor "may use the original request's headers (e.g., Cookie, User-Agent, etc.)" when fetching. So a fragment can be personalised on the way in and then shared on the way out. Put the identity in the cache key, or mark the fragment uncacheable.
• The origin emits invalid markup on purpose. The spec is explicit: "the markup that is emitted by the origin server is not valid; it contains interposed elements from the ESI namespace." When nothing processes it, browsers "ignore markup it doesn't understand". So an unprocessed include renders as nothing at all. Any path that bypasses the processor, such as direct origin access, an unmatched content type, or a route where ESI was never enabled, ships a page with silent holes.
• Turning it on everywhere is not free. Akamai warns you "should NOT enable this behavior for entire sites since parsing all files for ESI increases the time required to deliver content." Varnish adds a sniff in the other direction. It peeks at an object's first byte. If that byte is not <, it assumes you did not really mean to ESI-process it, and quietly skips JSON unless you set feature +esi_disable_xml_check.
• It costs compression ratio. Varnish compresses the parts of an ESI response separately and stitches them together during delivery, "which has a negative impact on compression ratio". Set do_esi on a gzipped response and it "will be uncompressed and recompressed part-wise during the fetch". That is because "back-references in the gzip data stream cannot point outside its own part".
• ESI injection. esi:include makes the edge fetch whatever src names. So unsanitised user input reflected into a page that the edge then parses turns into an edge-side request under your surrogate's authority. Akamai ships a "Detect and Deny ESI Injection" option for exactly this class of attack. Sanitise before the edge parses, not after.
• A 2001 Note, not a standard. The W3C's own boilerplate on the document says publication "indicates no endorsement by W3C or the W3C Team, or any W3C Members". It also says "No W3C resources were or are allocated to the issues addressed by the NOTE", and "W3C has had no editorial control over the preparation of this NOTE." Nine authors from eight companies wrote it. Ten W3C members submitted it, and Akamai supplied its editor. Nothing since has replaced or ratified it. That is why vendor subsets differ, and a template that works on one platform may not on another.

Best practice

• Scope ESI processing to responses that genuinely contain ESI: by content type, by path, or by a marker header such as Akamai's Edge-control: dca=esi. Never scope it site-wide. Varnish's docs add that do_esi "is not required, and should be avoided, for the included fragments, unless they also contains" includes of their own.
• Give the template a long TTL and each dynamic fragment its own short one, or mark it uncacheable. Per-fragment freshness is the entire point. One TTL across the whole set means you have gained nothing over a plain uncached page.
• Keep fragments coarse and few. Stay inside the implementation's include-count and recursion-depth limits. Remember: every extra include is another request in the assembly path.
• Never serve a personalised fragment from a shared URI. Key it per user, or mark it uncacheable. Do not depend on fragment response headers such as Set-Cookie surviving assembly.
• Always ship a fallback. Use esi:remove and <!--esi ... --> so an unprocessed page still renders sensibly. Use alt or esi:try / esi:except so a dead fragment degrades on purpose rather than by accident. Remember: esi:try catches only include and inline failures.
• Set the failure behaviour explicitly instead of inheriting whatever your platform defaults to. Alert on fragment fetch failures. A silently dropped include looks exactly like a page that simply has no cart.
• Test both through the CDN and directly against the origin. Assert that no raw esi: tag ever reaches a browser.