To understand the hunt for the villains behind the WannaCry ransomware attack, imagine the cybersleuthing in this whodunit unfolding like a Hollywood flick.
In one story arc, you have cops and detectives from an alphabet soup of U.S. and international investigative agencies, learning to work together while fighting over jurisdiction and pushing the boundaries of what the law allows to catch a crook.
In the other plotline, you have a hodgepodge of private researchers, hackers and cybersecurity companies — a melange of meddling youths, amateur Sherlock Holmes and technology-savvy private investigators who are less bound by the constraints of rules and regulations.
Faced with an explosion of hacks and viruses from increasingly sophisticated pirates and nation-states, it often takes a combination of these detectives — sometimes working in harmony, sometimes at odds — sifting through the intricacies of a cyberattack to unmask those responsible and, occasionally, bring them to justice.
"It's something we started to see more and often in the past few years," said Matt Suiche, founder of the Dubai-based Comae Technologies, who played a key role in unearthing a vital clue in the WannaCry mystery. "What we see is that security needs to be a joint effort. Open source collaboration and cooperation are very important to help us work together on these issues."
The WannaCry hack is far from resolved. Still, actions by Suiche and others slowed the virus’ march and show how cybersleuths can team up to combat an elaborate blackmail scheme that slowed factories, froze computers and forced hospitals to cancel surgeries. Victims discovered their hard drives had been encrypted and were told to pay $300 in the electronic currency
First reports of WannaCry in Britain, one of the harder-hit countries, surfaced Friday about 9 a.m. The virus moved quickly, but so too did the efforts to contain it.
Later Friday, the U.S. Computer Emergency Readiness Team issued a nationwide alert in cooperation with the Department of Homeland Security, the National Cybersecurity and Communications Integration Center and the FBI.
By Saturday, the virus had hit 126,000 computers in 104 countries. A day later, there were 200,000 victims in 150 countries. By Monday, the infections exceeded 300,000.
Europol, the continental police force, called the attack "unprecedented" and began working closely with cybercrime units in countries worldwide. "I think what is important to mention is that everyone — public and private sector — joined forces and are doing their best to get to the bottom of this," said Europol spokesman Alex Niculae.
The trick, according to security experts, is to preserve as much evidence as possible while trying to cripple the virus.
Phil Lieberman, president of the security firm Lieberman Software, said that includes protecting and examining infected servers, trying to trace any paths a virus may have taken to enter a system and tracking any ransom payments.
The problem from here becomes working across borders. Even with allies, it very quickly becomes complex in terms of what agencies can share when dealing with information about corporations or individuals. And when it comes to nations such as Russia and China, cooperation can be nonexistent.
"Agencies like the FBI and [Department of Justice] have jurisdiction for crimes against citizens and companies in the U.S., but they don't have the ability to project power outside the U.S., so if the criminals are outside the U.S., then law enforcement has little power or options," Lieberman said.
When it comes to borders, amateur cybersleuths and private companies have more freedom and often are able to work much faster.
The first hero of this story did not come from the ranks of the cops, but rather was an unsung security expert toiling away alone in the bedroom of his parents' house in a quiet English seaside town.
About 5½ hours after WannaCry began spreading through Britain, 22-year-old Marcus Hutchins realized the virus was sending messages to an Internet domain that had not been registered. Hutchins, who works for the Los Angeles-based firm Kryptos Logic, registered the domain to fool the virus and then walked away.
"My job is to look for ways we can track and potentially stop botnets (and other kinds of malware), so I'm always on the lookout to pick up unregistered malware control server domains," he wrote in a blog post recounting his work. "In fact I registered several thousand of such domains in the past year."
He thought that he had perhaps disrupted some small portion of the attack involving versions of the virus linked to that domain. He'd actually done much more. What he realized only later was that because WannaCry's creators weren't very clever, all the infected computers were trying to reach the same domain — and that action halted the virus.
"Humorously at this point we had unknowingly killed the malware," he wrote.
It takes time for a new domain registration to be recognized across the Internet, so several hours would pass before his domain registration triggered the "kill switch" for all newly infected machines.
But even as Hutchins found himself becoming an unwitting international media celebrity, a new variation of the virus had been launched. From his office in Dubai, Suiche spotted it and likewise registered a domain that blunted its effects.
"It's worked pretty well," Suiche said. "It's stopped more than 50,000 infections."
But as Suiche was focused on that work, he also spotted a cryptic tweet from Neel Mehta, a well-known Google security researcher.
Mehta gained a measure of fame for discovering "Heartbleed," a vulnerability in a computer program that provided encryption for about two-thirds of all servers on the public Internet. On Monday at 10:02 a.m. Pacific, Mehta tweeted to his 5,893 followers:
"9c7c7149387a1c79679a87dd1ba755bc @ 0x402560, 0x40F598
ac21c8ad899727137c4b94458d7aa8d8 @ 0x10004ba0, 0x10012AA4
"Attribution" is the term the security industry uses to refer to finding the person responsible for something.
One Twitter user dismissed the code Mehta shared as "fairly routine," but Suiche quickly went to work. He realized the jumble of numbers and letters were two remarkably similar lines of computer code, with one coming from Mehta's analysis of the WannaCry virus. But what was the other one?
He soon unlocked the mystery: The second line of code came from a virus used by the Lazarus Group, an organization that has been linked to the 2014 hack of Sony Pictures that was attributed to North Korea.
About an hour after Mehta's tip, Suiche referred to the Democratic People's Republic of Korea in a tweet: "Similitude between #WannaCry and Contopee from Lazarus Group ! thx @neelmehta - Is DPRK behind #WannaCry ?"
"If validated, this means the latest iteration of WannaCry would in fact be the first nation-state-powered ransomware," Suiche wrote in a blog post detailing his work. "This would also mean that a foreign hostile nation would have leveraged lost offensive capabilities from Equation Group to create global chaos."
A few minutes later, researchers for the Moscow-based global cybersecurity firm Kaspersky Lab tweeted a similar conclusion. In a statement, Kaspersky was quick to note there could be an alternative explanation, for instance, the possibility that someone planted the code to throw detectives off the trail.
"This can be an attempt to cover traces conducted by orchestrators of the WannaCry campaign," said the Kaspersky statement.
Suiche declined to say whether he had teamed up with any law enforcement agencies, but Hutchins told British media that he was now working with the FBI and Britain's National Cyber Security Center to help prevent the ransomware attack from causing further damage.
Travis Farral, director of security strategy at Anomali, a cybersecurity firm based in Redwood City, Calif., said that even when government and private experts team up, identifying the culprit generally remains a long shot.
"Due to the inherently difficult task of attributing computer-based crimes," he said, "finding an individual author or authors will likely boil down to mistakes made by the actor or actors behind the malware."
Becky Pinkard, vice president of service delivery and intelligence at the British cybersecurity firm Digital Shadows, said a slip-up might occur when crooks try to access the ransom money. They will either have to transfer it to other accounts, try to exchange or sell it for real-world currency, or use it to buy something from a vendor that accepts bitcoins.
In any of the cases, the thieves will start to leave digital footprints, or create interactions with other humans that could reveal more personal information.
"To get that money out, they're going to have to engage middlemen," Pinkard said. "Essentially, they are going to have to find a way to launder this money and it will have to pass through several different hands before they are going to be able to use it."
And that could mean an end to the whodunit.
O'Brien is a special correspondent.
Special correspondent Christina Boyle in London contributed to this report.