San Francisco firm defends Hong Kong vote from online attack

Two weeks ago, Matthew Prince, the chief executive of San Francisco tech company CloudFlare, had no clue that people in Hong Kong were preparing to hold a controversial online referendum on democratic reforms.

By Thursday night, half a world away from the southern Chinese city, he found himself on the front lines of a battle to defend the nonbinding, unofficial vote from sabotage.

Amazon Web Services and Hong Kong’s UDomain had initially been onboard to support and protect the voting website. But at the last minute, both bowed out, saying the expected size of the cyberassault could affect their other customers.

That was a somewhat worrying sign for Prince and team, whose small, 5-year-old company specializes in making websites run more quickly and smoothly and preventing disruptions and recently launched a pro bono service for situations just like this.

Going it alone, Prince and about 20 employees gathered in their office near 3rd and Townsend streets and girded for a test of their wits and ingenuity. As Friday morning broke in Hong Kong and the 10-day vote began, the fight was on.

The website, popvote.hk, was being flooded with requests that look like legitimate users trying to call up or contact the site. The queries were being sent by armies of commandeered computers, known as botnets, around the world. At the height of the assault, more than 200 million requests were coming in every second, Prince said – a clear bid to overwhelm the site and knock it offline.

“It was easily one of the largest the Internet has ever seen, on any metric,” Prince said in a phone interview, adding that the attacker or attackers kept changing strategies. “It felt like there was an intelligent adversary on the other side that was not just trying one technique but adopting new techniques to try to find ways around our mitigation efforts.”

Measured by another metric, the attack generated 300 gigabytes per second of traffic, according to CloudFlare. That’s like downloading nine HD movies per second, every second, according to an expert at GreatFire.org, which monitors Internet usage, censorship and hacking in China. On top of that, attackers chose methods that chewed up a lot of CPU server resources at CloudFlare. “The volume of the attack is impressive; so is its sophistication,” said the GreatFire.org expert who requested anonymity for fear of retaliation.

Halfway into the balloting, the attacks are persisting, but the efforts of Prince’s team seem to be working – no major outages have been reported and more than 700,000 people had voted by Tuesday evening, organizers said. That’s nearly 10% of the city’s population, and a far greater number than organizers anticipated.

Neither CloudFlare nor officials at the University of Hong Kong’s Public Opinion Program -- which designed and administered the voting website on behalf of a local civic group – could say where the attacks are being directed from.

“We don’t have any strong evidence … but there’s lots of speculation that it’s coming from China,” said Karie Pang, assistant director of the Public Opinion Program. “We have reported it to police. … Maybe one day we can get to the monster behind the scene.”

The balloting has been organized not by the city government but by a local civic group called Occupy Central with Peace and Love. The results are nonbinding, but those behind the effort hope that the process will help push forward democratic reforms in Hong Kong, a former British colony that reverted to Chinese rule in 1997 but retains a high degree of political autonomy from the Communist-led mainland under a legal framework known as “one country, two systems.”

Participants in this week’s referendum are being asked to express a preference for one of three methods of selecting the city’s chief executive in 2017, the first time the balloting is supposed to be open for a citywide, one-person, one-vote election.

But authorities in Beijing have denounced all the options on the referendum as illegal and the referendum itself as “invalid.” Ahead of the referendum, China issued a white paper essentially admonishing Hong Kongers not to push things too far – and warning that Beijing has final say over all matters in the city.

Willy Lam, a Hong Kong-based political analyst, called the referendum turnout “a big surprise” and said the white paper was a decisive motivating factor.

“People are totally mad about this,” said Lam. “This is seen by many people as Beijing taking away their autonomy. It’s basically pronouncing ‘one country, two systems’ dead.”

Lam said it was possible that the final vote tally could surpass 1 million and increase the likelihood that residents will eventually take to the streets to press their demands. Organizers of Occupy Central have said if the ultimate vote rules for the 2017 election don’t meet “international” standards, they will call on people to hold sit-ins and other forms of nonviolent, civil disobedience in Hong Kong’s main financial district, known as Central.

No date for an actual Occupy Central sit-in has been declared, but the next test of the movement’s strength is likely to come July 1, a traditional day of democratic protest in Hong Kong. Lam said he expected 150,000 to 200,000 people to march the designated route.

But Joshua Wong, a student leader in the Occupy Central movement, said he thinks the numbers could match those seen in 2003, when about half a million people turned out, angered by proposed anti-subversion legislation drafted at the behest of Beijing. In wake of that demonstration, the legislation was shelved indefinitely.

Robert Chow, a leader of the group Silent Majority, which opposes this week’s referendum process, said he believes the turnout numbers for the 10-day vote have been exaggerated. “I believe the figure is around 150,000 or 200,000,” he said, based on what he saw during in-person voting Sunday and in previous turnout for elections in Hong Kong.

Still, he conceded, the “number is sizable and the government should be wary.”

Back in San Francisco, Prince said he still doesn’t really understand the details of what Hong Kongers are voting on -- but his company’s involvement with the balloting has brought positive publicity. CloudFlare has been mulling the idea of opening an engineering center in Asia, he said, looking at cities including Singapore and Sydney, Australia. After this week, Hong Kong may be moving up the list as the company has been flooded with resumes.

“We had been leaning toward Singapore but ... CloudFlare is a much better known brand in Hong Kong today than we were at the beginning of this week,” Prince said. “I’ve been very touched by the number of talented people who have reached out to us from Hong Kong and said if you ever have an office here we’d like to come work for you.”

For news out of East Asia follow @juliemakLAT on Twitter