Whatever happens, wherever it happens, it eventually all comes back to DNS.
May 5th, 2021
DNS is fascinating and affected by more seemingly innocent actions on the network than many realize.
To illustrate the truth of this statement, here’s the story of how Chromium users instigated an accidental DDoS attack against the DNS root servers.
One of the Chromium browser’s most used features is the omnibox: a UI element that can act as an address bar or a search box (and many, many other things that work with text inputs) based on what’s typed into it.
For the average user, the omnibox is a convenient way of interacting with “the internet.” But what goes on behind the scenes to enable this convenience has caused a lot of trouble for the engineers and guardians of the internet and DNS.
A DDoS attack is a fairly simple affair: it overwhelms a DNS server (using bogus queries) to the point that it no longer answers even the legitimate ones.
Resilience against DDoS is constantly improving, but so are DDoS attacks. There are some outstanding solutions available (such as our very own xDNS Redundancy, built into Micetro for no extra cost, and providing cross-platform DNS redundancy), but it’s always an uphill battle. DDoS attacks are almost always automated, carried out by — through malware-infected computers — proxies that don’t even know about it.
While this particular story in question wasn’t malicious or carried out through malware-created botnets, it still wreaked the same havoc on the unsuspecting root servers.
As pointed out by Matthew Thomas on the APNIC blog, Chromium’s omnibox lookup code resulted in the root servers being hammered by billions upon billions of requests. And, technically, these queries made by a legitimate feature are almost indistinguishable from a DDoS attack.
In many cases, the omnibox has an easy job figuring out what is a search term and what is a legitimate url: anything more than one word (i.e., there’s a space) is a string to search for. But what about single-word searches? And what about private networks, such as those a business might run, that have private domains such as ‘marketing’?
To further complicate the situation, some ISPs routinely hijack the results for mistyped/non-existent DNS queries (called the NXDOMAIN response) and redirect the user to custom pages that often display ads. It’s a way to generate revenue in case the user mistypes a domain. (We’ll let you decide the degree to which this behavior is ethical or not.)
Chromium’s omnibox needs to figure out what to do with single-word search terms. Should they display the search, with a “Did you mean…” on the top? Or try and get a real answer, going through the DNS hierarchy until there’s an answer or no more servers to ask?
To figure out how the omnibox should behave, Chromium performs a test by looking up three randomly generated (and statistically unlikely to be valid) domains. The result for these three lookups will inform the browser how to handle future inputs.
Because of how DNS hierarchy works and how the randomized queries are performed, Chromium’s three random “domains” skip past the normal chain of DNS requests and reach up to the root servers on the top.
The root server system is designed to handle large volumes of traffic. Being the ultimate authority for DNS, they need to. However, Chromium is the most popular modern browser, with 70% combined market share between Google’s Chrome, Microsoft’s Edge, and Vivaldi’s Opera, to name just the three major ones. And in the 21st century, that means a lot of traffic.
How much? 45,80% of the traffic to the DNS root servers was attributed to Chromium’s intranet redirection tests. Almost half of over 140 billion queries per day.
Shortly after APNIC revealed the findings in August 2020, Chromium 87 was released in November 2020 with a redesign to the redirection test code and a new feature for a multi-state DNS interception policy. The results were quick and impressive: Verisign reports that the number of queries per day has rapidly fallen to 84 billion (from 143).
A happy ending? Yes. Kind of.
Although Chromium has been updated and the unintended consequences averted (for the most part), this story once again illuminates how vital (and vulnerable) DNS is.
Everything that happens on the network, whether it’s your small office intranet or the global internet itself, is tied to the health of DNS. And as much resilience as we can build into the hierarchical, distributed system of DNS, sometimes accidents happen.
For the critical role DNS plays in everything we do online, often it is hidden away, particularly from the end-user who just wants to replace this:
As the internet and other networks permeate our lives more and more, it’s essential that those who are tasked to facilitate the smooth operations for users are mindful of the pitfalls. And whatever happens, wherever it happens, it’ll eventually all come back to DNS.
Want to do DNS better? Try Micetro by Men&Mice!