A recent article by Cloudflare has shed light on a fascinating issue: the impact of CNAME ordering in RFC specifications. This seemingly minor detail caused a major outage for Cloudflare's 1.1.1.1 service, highlighting the importance of even the smallest technical nuances.
The Unclear RFC and its Consequences
On January 8, a routine update to Cloudflare's DNS service led to a significant change in the order of CNAME records in responses. This change caused some DNS clients to fail when resolving names, as they expected the alias records to be listed first. While modern software often treats record order as irrelevant, Cloudflare's team discovered that certain implementations rely on CNAME records appearing before other record types.
The Root Cause
Sebastiaan Neuteboom, a systems engineer at Cloudflare, explains that the issue stemmed from improvements made to reduce memory usage in their cache implementation. A subtle change in CNAME record ordering was introduced on December 2, 2025, and deployed on January 7, 2026. This change caused the CNAME records to appear at the bottom of responses, after the final resolved answer.
The Impact
When a DNS resolver encounters a CNAME record, it typically caches each step with its own expiry time. If part of this chain expires in the cache, the resolver only re-fetches the expired portion and combines it with the valid parts to form the complete response. However, with the altered order, this process failed, leading to a significant outage of the popular 1.1.1.1 DNS service.
The Discussion
Many DNS client implementations are not affected by the order, such as systemd-resolved. However, others, like the getaddrinfo function in glibc, handle the resolution chain by expecting CNAME records to appear before any answers. This has sparked a debate among users, with some questioning the clarity of the RFC and others suggesting that Cloudflare developers may have misinterpreted it.
Patrick May offers an interesting perspective, citing Hyrum's Law and Postel's Law: "With a sufficient number of users, all observable behaviors will be depended on by somebody." and "Be conservative in what you send, be liberal in what you accept."
Cloudflare's Proposal
In an Internet-Draft to be discussed at the IETF, Cloudflare proposes an RFC that explicitly defines how to handle CNAME records in DNS responses correctly. This proposal aims to clarify any ambiguity and prevent similar issues in the future.
The incident timeline shows that Cloudflare began the global rollout on January 7, reaching 90% of servers by January 8 at 17:40 UTC. The company promptly declared the incident, started reverting the change at 18:27 UTC, and completed the rollback by 19:55 UTC.
And this is the part most people miss...
While the technical details are crucial, it's also essential to recognize the impact of such incidents on a global scale. As one user commented, Cloudflare's post-mortem analysis showcases a high standard in engineering, but it also raises questions about their testing practices and the potential for more comprehensive global impact assessments.
Thoughts?
What are your thoughts on this incident? Do you think the RFC is unclear, or was it a misunderstanding by the developers? Join the discussion and share your insights in the comments!
