How the U.S. Hobbled Its Hacking Case Against Russia and Enabled Truthers

Sometimes, in his covert influence campaign against America, Vladimir Putin need do nothing but sit back and chuckle mirthlessly while U.S. officials shoot themselves in the foot. Such was the case last week when the Department of Homeland Security and the FBI released a technical exposé of Russia’s hacking that industry experts are slamming as worse than useless—so jumbled that it potentially harms cybersecurity, so aimless that it muddies the clear public evidence that Russia hacked the Democratic Party to affect the election, and so wrong it enables the Trump-friendly conspiracy theorists trying to explain away that evidence.

“At every level this report is a failure,” says security researcher Robert M. Lee. “It didn’t do what it set out to do, and it didn’t provide useful data. They’re handing out bad information to the industry when good information exists.” At issue is the “Joint Analyses Report” released by DHS last Thursday as part of the Obama administration’s long-awaited response to Russia’s election hacking. The 13-page document was widely expected to lay out the government’s evidence that Russia was behind the intrusions into the Democratic National Committee’s private network, and a separate attack that exposed years of the private email belonging to Hillary Clinton campaign chair John Podesta.

Instead, the report is a gumbo of earnest security advice mixed with random information from a broad range of hacking activity. One piece of well-known malware used by criminal hackers, the PAS webshell, is singled out for special attention, while the sophisticated Russian “SeaDuke” code used in the DNC hack barely rates a mention. A full page of the report is dedicated to listing names that computer security companies have assigned to Russian malware and hacking groups over the years, information that nobody is asking for.

Rather than focusing on the Russian intelligence services, the U.S. seemingly opted to gather all Russia-sourced hacking under a single rubric, code named “Grizzly Steppe,” putting everything from online bank heists to identity theft in the same bucket as the Kremlin-linked intrusions into the White House, State Department, and the DNC.

Though the written report is confusing, it’s the raw data released along with it that truly exasperates security professionals. The department released 876 internet IP addresses it says is linked to Grizzly Steppe hacking, and urged network administrators everywhere to add the list to their networking monitoring.

Lists of IP addresses used by hackers can be useful “indicators of compromise” in network security—admins can check the list against access logs, or program an intrusion detection system to sound the alarm when it sees traffic from a suspect address. But that assumes that the list is good: carefully culled, and surrounded with enough context that administrators know what to do when they get a hit.

The DHS list is none of these things, as Lee, founder of the cyber security firm Dragos, discovered when he ran the list against a stored cache of known clean traffic his company keeps around for testing. The results stunned him. “We had thousands of hits,” he says. “We had an extraordinary high amount of false positives on this dataset… Six of them were Yahoo e-mail servers.”

It turns out that some, perhaps most, of the watchlisted addresses have a decidedly weak connection to the Kremlin, if any. In addition to the Yahoo servers, about 44 percent of the addresses are exit nodes in the Tor anonymity network, The Intercept’s Micah Lee reported Wednesday. Tor is free software used primarily for anonymous web browsing. Russian hackers use Tor, but so do plenty of other people.

“If you just create a list of all the IP addresses that could deliver you a virus or an attack, Tor exit nodes belong there—that’s true,” says security expert and blogger Robert Graham. “But it’s not useful. If it’s Yahoo, it’s not useful. It’s not something that you can blacklist or watchlist.” Yahoo servers, the Tor network, and other targets of the DHS list generate reams of legitimate traffic, and an alarm system that’s always ringing is no alarm system at all.

The consequences of the over inclusive list became apparent last week, when a Vermont utility company, Burlington Electric Department, followed DHS’s advice and added the addresses to its network monitoring setup. It got an alert within a day. The utility called the feds, and The Washington Post soon broke the distressing news that “Russian hackers penetrated [the] U.S. electricity grid through a utility in Vermont.”

The story was wrong. Not only was the laptop in question isolated from the utility’s control systems, the IP address that triggered the alert wasn’t dangerous after all: It was one of the Yahoo servers on the DHS list, and the alert had been generated by a Burlington Electric employee checking email. The Post article was later corrected, but not before Vermont Senator Patrick Leahy issued a statement condemning the putative Russian attack.

The incident illustrates why the DHS watchlist—with a high false-positive rate, and no explanation of why a particular address made the list—is useless to network administrators already fighting “alert fatigue,” says Lee. “When they alert you, you have no context, you don’t know what to do,” he says. “Your only course of action then is to call the government.”

The Grizzly Steppe report also gives succor to those who argue that the identity of the DNC and John Podesta hackers is unknown, and perhaps unknowable—a position reiterated by President-elect Donald Trump this week. “[A] 14 year old could have hacked Podesta,” Trump tweeted Wednesday, quoting WikiLeaks founder Julian Assange.

Much of the skepticism about the Russian hacking, including Donald Trump’s, follows a simple and intuitive narrative: The administration is publicly accusing Russia, while jealously hoarding whatever evidence it has to support that accusation. The press is guilty of regurgitating the government’s claims on pure faith, just as it did with Iraq and weapons of mass destruction.

By kind-of-but-not-really publishing forensic data on the DNC and Podesta hacks, and mixing it with other material, the administration fed right into that story line and fattened it up.

WikiLeaks made hay—and 6,500 retweets—off the report’s random inclusion of the PAS webshell—common malware that nobody has connected to the DNC hacks. “‘Russian hacking’ sample provided by U.S. government is common malware,” the group wrote. The Kremlin-controlled news outlet Sputnik News latched on to the report’s many problems to write the headline “Experts Destroy White House ‘Proof’ of Russian Hacking.”

The administration, though, never claimed that the Grizzly Steppe report would prove anything, and thanks to a recent BuzzFeed scoop, we now know that the FBI didn’t even examine the DNC’s harddrives, a development that was perplexing Trump late Thursday. (“So how and why are they so sure about hacking if they never even requested an examination of the computer servers?” Trump tweeted. “What is going on?”) Instead, it was a respected computer security company called Crowdstrike that examined the servers, and publicly revealed Russian’s involvement in the DNC hacks last year. It backed up the claim with specific technical information far more useful than anything in the DHS report. Crowdstrike competitors, including Symantec and FireEye, have examined the forensic data from the DNC hack themeselves, and endorsed Crowdstrike’s conclusion that two particular hacking groups were the culprits: “Fancy Bear” and “The Dukes.”

To skeptics, those hacking groups are shadowy apparitions, as likely to be Julian Assange’s “14-year-old” hacker or Donald Trump’s “400 pound guy” as any national government. But to analysts in the computer security industry, the hackers are old, familiar adversaries that they’ve been watching under a microscope for the better part of a decade.

The first group, called “Fancy Bear” or APT28 has been active since at least mid-2007. The group typically begins its attacks with targeted spearphishing emails crafted to trick the recipient into clicking on a link or downloading a malicious file. Then the group installs backdoors controlled through a cloud of command-and-control servers deployed around the world. Its targets have included NATO, several U.S. defense contractors, the German parliament and, after Russia’s doping scandal began, the World Anti-Doping Agency. One of the command-and-control servers used in the DNC hack was reportedly also used in the Bundestagand intrusion.

The other group, commonly called “the Dukes” or APT29, was first spotted operating in Chechnya in 2008. Stealthier and more cautious than Fancy Bear, the Dukes have nonetheless been detected infiltrating the White House, the State Department, and the Joint Chiefs of Staff. Known for innovation—one attack campaign used Twitter as a command-and-control channel—they have their own fleet of customizable malware, including a program called Seaduke that they only bring out for the really important targets, and which was found again on the DNC’s network.

Security companies can tell you much more about these groups, their code, their infrastructures, and their methods. (The Finnish security firm F-Secure has an excellent 34-page write-up of the Dukes, and FireEye has a deep dive into Fancy Bear, among many other reports by different companies.) (PDF) From analysis of the dozens of malware packages used exclusively by these hackers, researchers can tell you that they’re usually compiled on machines with the language set to Russian. Both groups operate during working hours in Russia, and take Russian holidays off. Their targets are radically different from those of for-profit criminals hackers in Eastern Europe or anywhere else—no banks, no retailers with credit card numbers to steal—always governments, companies, journalists, NGOs, and other targets that the Russian government would be interested in.

In other words, these hackers don’t operate like 14-year-olds. They sometimes use off-the-shelf hacking tools, but more often they deploy industrial scale malware no teenagers have access to. They hit targets of interest to spies, not kids. And virtually all the public analysis of these two groups concluded—well before it became a political issue with the DNC hack—that they are likely controlled by the Russian government.

The evidence, then, that Russia interfered with the election is already solid, and is supported by years of work by the security industry. “If you’ve been following along, all the evidence that matters is already public,” Lee notes. “This is one case out of hundreds that they’ve investigate involving the same hackers. It’s all very, very consistent, it all makes sense, it’s all very, very solid,” he says. “It’s just that the government is now confusing everyone.”

DHS defends the broad list, and urges network administrators to follow up on any hits. “We know the Russians are a highly capable adversary who conduct technical operations in a manner intended to blend into legitimate traffic,” the department said in a statement provided to The Daily Beast. “Because the IPs are in the logs does not mean there has been malicious activity. It is, however, cause for a further look to determine if malware, for example, may be resident.”

Just don’t tell that to Graham, the cybersecurity specialist who found a watchlisted Yahoo address in his own logs. He says the discovery didn’t move him to call DHS, nor to examine his hard drive for evidence of attack.

“The Russians didn’t hack my browser,” he says. “I just used Yahoo.”