How one rent-a-botnet army of cameras, DVRs caused Internet chaos

Enlarge / We’re also mad you’re connected to the Internet, toaster et al.

Disney

Welcome to the Internet of Evil Things. The attack that disrupted much of the Internet on October 21 is still being teased apart by investigators, but evidence thus far points to multiple “botnets” of Internet-connected gadgets being responsible for blocking access to the Domain Name Service (DNS) infrastructure at DNS provider Dyn. Most of these botnets—coordinated armies of compromised devices that sent malicious network traffic to their targets—were controlled by Mirai, a self-spreading malware for Internet of Things (IoT) devices.

in a blog post on the attack, Dyn reported “tens of millions” of devices were involved in the attack

But other systems not matching the signature of Mirai were also involved in the coordinated attack on Dyn. “We believe that there might be one or more additional botnets involved in these attacks,” Dale Drew, CSO of Level 3 Communications, told Ars. “This could mean that they are ‘renting’ several different botnets to launch an attack against a specific victim, in which multiple other sites have been impacted.”

The motive may have been blackmail, since the attacker sought a payout by Dyn to stop. But Drew warned that the huge disruption caused by the attack “could result in large copycat attacks, and [a] higher [number of] victim payouts [so] as to not be impacted in the same way. It could also be a signal that the bad guy is using multiple botnets in order to better avoid detection since they are not orchestrating the attack from a single botnet source.”

Mirai has played smaller roles in previous attacks. It factored into last month’s extended distributed denial of service (DDoS) attack on the website of information security reporter Brian Krebs and an even larger DDoS against the French cloud provider OVH. Mirai clearly was the star of the attack on Dyn, apparently controlling multiple groups of bots.

But even in the midst of the Dyn attack, some of the Mirai-infected devices were being used to attack another target—the infrastructure of a gaming company, according to Allison Nixon, the director of security research at security company Flashpoint. That idea matches up with what others who had some insight into the attack have told Ars confidentially—that it was also pointed at Sony’s PlayStation Network, which uses Dyn as a name service provider.

For now, it’s not clear that the attacks on Dyn and the PlayStation Network were connected. And with a criminal investigation underway, a Dyn spokesperson declined to confirm or deny that Sony was also a target. “We are continuing to work closely with the law enforcement community to determine the root cause of the events that occurred during the DDoS attacks last Friday,” Adam Coughlin, Dyn’s director of corporate communications, told Ars. “Since this is an ongoing investigation, we cannot speculate on these events.”

Regardless of the reasons behind it, the attack on Dyn further demonstrates the potential disruptive power of the millions of poorly protected IoT devices. These items can be easily turned into a platform for attacking anything from individual websites to core parts of the Internet’s infrastructure. And Mirai has demonstrated that it doesn’t take “zero-day” bugs to make it happen; attackers only need poorly implemented security on devices that can’t be easily fixed.

From tiny cameras, mighty botnets grow

Mirai is hardly the first IoT botnet to make headlines. In December 2014, LizardSquad’s “stresser” service—built on compromised home Wi-Fi routers—announced that it was ready for business with Christmas attacks on the PlayStation Network and Microsoft’s Xbox Live service. (The service was eventually hacked itself.) And while Mirai played a supporting role in the 620-gigabit per second attack on Krebs on Security and the 1.5 terabit-per-second attack on OVH, those attacks also leveraged Bashlight, another (larger at the time) IoT botnet. By the time it was over, more than 30,000 Internet-connected surveillance cameras and DVRs were involved in the OVH attack. It lasted for over a week.

There are a few things that make Mirai stand out from previous IoT botnets. First and foremost, its code has been published openly on the Internet. On September 30, in the wake of the attacks on OVH and Krebs, someone claiming to be the malware’s author published the botnet and command and control (C&C) server code on Hacker Forums. Suddenly, anyone could access step-by-step instructions for its configuration and use.

The post to Hacker Forums that started it all.

On the plus side, the published source code gives researchers a great deal of insight into how Mirai operates. On the downside, however, it makes it possible for anyone who can compile the code and has access to Internet-connected servers to build their own botnet. This opportunity provides more ambitious botnet builders a proven platform to improve upon.

The simplicity of Mirai’s C&C structure makes scaling it up relatively simple. “One of the things we noticed during the Dyn attack was that the C&C domain would change its address,” Nixon explained. “That way, the C&C network could segment its botnet.” By simply changing a DNS entry, the attacker could use the same domain to create and operate multiple separate botnets simultaneously.

When a Mirai bot is created, it sends a request to the Domain Name Service for the “A” address of a domain configured by its creator. Once it has the Internet address associated with that “A” address, it locks onto that IP address. “When one C&C server fills up, [the botnet operator] can just change the IP address associated with that ‘A’ name,” Nixon explained. New bots will connect to the new address while older bots continue to communicate with the previously labeled server.

While this scheme can cause problems with resiliency of the botnet—if a C&C server gets identified and its traffic is shut down, the bots fail—it’s not a big problem for the botnet long-term. The botnet can easily be re-established from another server by simply re-discovering vulnerable devices.

Checking for open doors

Still, the worst thing about Mirai is that it leverages the horrible security decisions made by a handful of manufacturers of Internet-connected devices. And despite growing public alarm, these IoT devices and their shortcomings will likely persist on the Internet for years. A majority of the devices compromised by Mirai connect to the Internet via firmware from one company: the Chinese electronics supplier XiongMai Technologies. The attack led XiongMai to issue a recall for some of its products sold in the US, Fortune reports.

The reason XiongMai’s firmware is such an easy target for Mirai is that it includes a setup interface that is essentially a hard-coded “backdoor”—an unchangeable administrative username and password, common across entire lines of devices. While the user can set their own credentials, the default credentials are hard-coded into the firmware.

Mirai simply uses a hard-coded library of default usernames and passwords to log in to the devices it discovers. This is the equivalent of walking through a parking lot, checking for unlocked car doors, and finding the keys sitting on the driver’s seat. These devices included Panasonic printers, SNC and ZTE routers, and dozens of network-connected cameras and digital video recorders. Some of these passwords were simply factory-set defaults, but others were permanent—meaning they could not be changed by their owners.

Some of the passwords hard-coded into Mirai.
Types of attacks programmed into Mirai’s bots.

To be compromised by Mirai, a device also had to be on a network with very weak firewall configurations (or no firewall at all). An analysis of the botnet’s code by Ars revealed that Mirai uses Telnet and SSH—the channels used to connect to a system remotely and log in to a text command prompt—to compromise and control devices. While home routers generally can be configured to block connections from outside the local network from Telnet and SSH, these connections are often left open by default.

In a statement issued on October 24, a XiongMai spokesperson wrote, “XiongMai closed the Telnet port on related products before April 2015. Therefore, for the product in April 2015 after, the hacker is simply no way to use the port to attack, and for products produced before April 2015, XiongMai has provided a firmware upgrade, [and] if [users are] really worried about the risk, it can be solved by upgrading.” The spokesperson claimed that even if the patch was not applied, there would be no harm to the device by hacking attempts.

Using Telnet or HTTP traffic—which is unencrypted—makes it relatively easy to catch Mirai botnet traffic with deep packet inspection. Flashpoint had visibility into one of the botnets attacking Dyn, Nixon said, and while others had described the attack on Dyn as coming in two or three waves, “it was more like every once in a while, I would see another line of attack instructions coming in. I had seen something like 20 or 30 lines of commands.”

Mirai is loaded with a variety of configurable attacks, executed in response to those command lines:

Two types of UDP (User Datagram Protocol) flood attacks intended to overwhelm a target with raw network traffic (one “generic” attack with various payload options and another “plain” attack “optimized for speed”)
A UDP attack tailored for taking down Valve game servers by overwhelming them with queries for game connections
TCP (Transmission Control Protocol) attacks based on successive SYN (synchronize) or ACK (acknowledge) floods—attacks that use TCP’s “three way handshake” against the target by fooling its network interface into dedicating resources to spoofed connections
GRE (Generic Routing Encapsulation) attacks that use the Cisco-designed tunneling protocol to get Internet Protocol (IP) and Ethernet packet floods past hardware used to block DDoS attacks
“Proxy knockback connection,” apparently for going after Minecraft servers
An HTTP Layer 7 flood attack focused specifically on taking down Web servers

Also in the mix is a “DNS Water Torture” attack—a UDP-based attack designed specifically to target domain name servers. It creates DNS requests targeting a specific domain, adding random strings of text to it formatted at subdomains. It also randomly selects the path for those requests to take, selecting from four different public DNS providers (including Google’s public DNS). The random string—which is used as the name of a subdomain or host in a lookup request sent to the DNS server—forced the DNS service pinged to send a request to Dyn, and it forced Dyn’s servers to do a fresh look-up for each. The requests, laundered through legitimate DNS services, looked like legitimate pass-along requests and were less easily screened out.

The tale of the traffic

Level 3’s Drew provided Ars with a record of observed traffic as part of the DDoS against Dyn. The first plot shows the attack traffic last Friday “compared to a typical day for this same IP space,” Drew explained. The vast majority of the attacks were largely SYN flood attacks against DYN’s DNS and the “DNS Water Torture” prefix label attack, according to Level 3’s data.

Level 3 traffic plot showing traffic to Dyn’s segment of Internet address space on October 21 (in blue) vs. a normal day (in red).
A chart showing periods when the attacks against Dyn were coming largely from non-Mirai botnets.

“There are two distinct attack waves,” Drew said. “The first begins at 1110 to ~1310 UTC and the second (even bigger attack) begins around 1550 and goes hard for about an hour, then dropping significantly in volume. As can be seen, there were a few smaller attacks in between the two major waves, but each was short-lived. This is important because it shows that the bad guy is using multiple botnet networks to launch his attack.”

Eventually, the server that Flashpoint was monitoring began to have connectivity issues. Then, it “may have died a serious death,” Nixon noted.

“It was having intermittent issues late in the day,” said Zach Wikholm, a security developer at Flashpoint. “And at some point, we couldn’t get to it any more. It died at different times for different places.” That may have been indicative of network owners blocking the communications of the C&C servers once they were identified.

The new normal

Mirai’s creator may have simply released the code because he or she had already moved on to another, better alternative. Using the screen name “Anna-senpai,” the alleged author complained that because Internet service providers began shutting down traffic on protocols used to spread the malware after the attack on the Krebs site, it was getting harder to build massive armies of bots. “With Mirai, I usually pull max 380k bots from Telnet alone,” the author wrote in the post to Hacker Forums. “However, after the Kreb DDoS, ISPs been slowly shutting down and cleaning up their act. Today, [the] max pull is about 300k bots and dropping.”

After the Dyn attack, more network providers are bound to take measures to block the sort of traffic associated with Mirai. But it will likely be years before devices vulnerable to Mirai are either properly protected from attack or removed from service. And while consumer device manufacturers have become generally more serious about IoT security, there are still a vast number of devices on the Internet that are configured with default or permanent passwords—passwords that another botnet developer could easily add to a targeted library.

Now that the potential of Mirai has been demonstrated, plenty of people will be ready to try. And just as many are eager to take credit. Wikileaks urged its “supporters to stop taking down the US internet,” saying they had “proved point”. And someone calling themselves the “New World Hackers” claimed responsibility on Twitter for the attack, posting: