The game industry has been under attack for a long time. Security professionals have often had to deal with distributed-denial-of-service (DDoS) attacks going back years.
It seemed like the problem was solved not so long ago, but then, the vector for attacks changed. With the rise of the Internet of Things (IoT), hackers were able to get their hands on many more compromised machines, and in turn, they were able to marshal those machines in much larger DDoS attacks. And so, the game companies are finding that they are getting flooded with attacks once again.
Nokia Deepfield helps companies defend themselves against such attacks. I spoke with Craig Labovitz, general manager of Nokia Deepfield, about the game industry’s ongoing vulnerability to DDoS attacks. That may not sound like the specialty you’d expect Nokia to have, but Nokia acquired Deepfield back in 2016 to ensure real-time network security and performance.
Here’s an edited transcript of our interview. GamesBeat and Akamai will hold a breakfast at the Electronic Entertainment Expo (E3) on June 14 to talk about games and security. Contact us through deantak on Twitter if you’d like to attend.
GamesBeat: Tell us about your interest in security and game companies.
Craig Labovitz: I’ve been doing DDoS for about 20 years now. I was a founder and chief architect at Arbor Networks, one of the first commercially successful DDoS companies. I was with Arbor for 12 years. After we left Arbor, we started Deepfield about five years ago, but our history goes back 25 years doing security, doing DDoS, particularly focused on unusual traffic blocking, traffic floods, things like that.
Deepfield had its start trying to do the next generation of security for both the large cloud guys, the large game guys, and the large carriers. Deepfield was an independent company for about five years. We grew pretty quickly, to cover about 90 percent of North America. We’d just started to enter Europe and Latin America and other parts of the world when we joined Nokia, about a year ago. Since then, we’ve been able to — Nokia provided additional investment. We’ve grown our technology, grown the base. Now, we’re deployed all over the world, doing both engineering and DDoS security.
GamesBeat: Why has this problem persisted for so many years? It sounds like an almost unsolvable issue in some ways, the fact that people can still do DDoS attacks.
Labovitz: Well, I’d actually say the opposite. When we left our last company, one of the reasons I left is I thought we were done. If you go back to 2011, all the carriers deployed appliances. It’s always an arms race between attackers and defenders, whether it’s war or security. In 2011, the defenders had the upper hand. Everyone had deployed the tech they bought from Arbor Networks. Generally, while DDoS was a nuisance, it wasn’t on the front page.
Back in 2000, when we started Arbor, DDoS was on the evening news. All the major brand names were under attack. 2011, there were still attacks, but most of them were easily mitigated. Technology had advanced to a point where we thought it was basically over. We saw the market declining. There wasn’t a lot of growth. It wasn’t in the news. Everyone who was going to buy had already bought: 80 or 90 percent of the large cloud and game companies. Then, things started to change, and you get to where we are today, which of course is a very different market.
GamesBeat: 2011 was a big deal in gaming security, because it was the year of the PlayStation Network hack.
Labovitz: Right. That was when things began to change, in that time frame. I left Arbor in 2011, and in the last five or six years, we’ve seen the resurgence. As far as why things changed, a couple of things have really changed the marketplace to where you’re seeing DDoS be such a pain point for our customers and for games, as well as other verticals.
What changed is, number one, the platforms changed, in the sense of we went from compromising PCs in consumer homes to millions of mobile devices. On a regular basis, we’re seeing cloud DVRs and other home devices participating in attacks. The number of compromised devices participating in botnets has tilted the balance of DDoS back to favor the attackers.
The second thing is just the bandwidth available. In 2010, I had a megabit, a couple of megabits at home? Now, I have hundreds. Other people have gigabits. You see significant last-mile advances in bandwidth and not just to consumers. We’ve seen the explosion of cloud servers and VMs, all of which we see being used as part of DDoS today. The firepower in terms of bandwidth has grown dramatically.
Now, we’ve gone from one device in a home you can compromise to as many as 30 or 40. We’re seeing some of these IOT devices participate in DDoS — like webcams. It’s gotten much easier for criminals to hijack devices all around the world. These devices aren’t connected to just a megabit anymore. Some of them have gigabit bandwidth to the rest of the Internet.
GamesBeat: And that sends a much higher volume of junk requests?
Labovitz: Correct. The number of devices to compromise has grown by a factor of 10 or, in some cases, 100, and the bandwidth to those devices has grown in the same way. All this has really happened since 2010, 2011, where we’ve seen the balance of DDoS tilt back to the attackers.
GamesBeat: What’s been the reaction on the defensive side?
Labovitz: Well, concern. It puts you in a tough position when your attackers grow by 10 or 100 times. It’s hard to counter that. That’s why DDoS, particularly in the last few years, is making headlines again and becoming more of a challenge.
It’s a pretty fundamental shift in the way people are thinking about security. When attacks are occasional, when attacks are small, whether you’re a game company or a provider you respond by adding stuff to the network, by adding servers or different security devices. When you get to this scale of attacks, when the attackers are 10 times bigger than any capacity you have, it’s no longer a matter of just adding more devices to the network. You have to fundamentally shift how you think about security, particularly with an eye toward things like DDoS.
GamesBeat: What has that shift been like?
Labovitz: Back in the day, I used to have a Palm Pilot. I had an MP3 player. I had five different devices that I carried with me that were all sort of adjunct. Similarly, in networking, you used to have a separate device for every possible function. You had a firewall, a DDoS box, an analysis box, a router, a management box. You tried to scale by scaling up all five or six of these things, and that worked for a good 15 [to] 20 years.
The problem, of course, is your attackers are now so much bigger than you are. It’s hard to scale each of those things separately by 10 or 100 times. What you’re seeing now across the market is a shift to move away from that Palm Pilot view of the universe and look to have this embedded in the network, embedded in the infrastructure. You can’t just add it on as an afterthought.
For years, security was an afterthought. You build your network, your game, or your data center, and then, you added security to it. The real shift today is it needs to be part of how you build it from day one. It needs to be everywhere, ubiquitous, embedded. It needs to scale at the same rate you scale your game servers and your network. That’s what we’re seeing in the market today.
GamesBeat: If you had to tick off, say, five things game companies have to worry about, where would you put DDoS in that spectrum of security problems?
Labovitz: It’s kind of like asking a homeowner how they consider security. If they’ve never been burglarized, that’s the last thing on their list. Someone who’s just been broken into or someone who’s made the front page of the Wall Street Journal because they just lost five percent off their stock value, they might have a different opinion. Having done DDoS for 20 years, our best sales were the day after. We used to call them the day-after sales. The day after someone made the front page of the Financial Times, those were the easiest sales we ever had. You hear similar stories about home alarm systems.
When we started doing DDoS 20 years ago, we had to convince people they needed DDoS protection. I think the market has largely matured, and people believe they need it. The question is how much. Clearly losing all your game infrastructure for a period of hours or days is catastrophic to the business. In terms of things you worry about, that would probably be near the top of the list. Things that pose an existential threat to a company are good things to worry about.
GamesBeat: As far as where the online game operators are at, are they effectively all outsourcing this function to the likes of Akamai or Amazon? Do they say to the providers, “Hey, if I get attacked, just give me some more compute resources and get me through it?” Or, is there a different mix of infrastructure.
Labovitz: If you look at the game companies, what’s been interesting over the last three or four years is they’ve come to look a lot like network providers. They’re starting to not only do DDoS themselves, but they’re building their own data centers, laying their own dark fiber, handling more and more as performance becomes a competitive element in games. We see the top five game companies take over more and more of their own infrastructure, down to dark fiber. They’re building out their own global networks.
We did see a period of outsourcing, but now, the opposite is happening, as performance and latency and jitter become more important. As scale has grown, the major game operators — certainly in the U.S. and also in other parts of the world — have made big investments in infrastructure.
GamesBeat: We haven’t talked much about platforms yet, but are we talking about consoles or PC or even mobile? I know that on mobile now, the fast interaction has been very important for games like Clash Royale or Arena of Glory. These are multiplayer team games. They seem to be very sensitive to latency problems. If they’re getting attacked, is that another layer to the problem?
Labovitz: There are definitely attacks there. I think most of the issues we see and hear about from our game customers and carrier customers are more the first-person shooters. We see a ton of — it’s just constant. At any given time for some networks, as much as five or 10 percent of traffic is just people with Xboxes or other console games trying to block someone else.
When we talk about DDoS with respect to gaming, there are two types of attacks. One is you’re specifically targeting another consumer, trying to knock them off, knock their IP address off. The other is you might have monetary incentives. You might go after one of the main game companies and attack their servers. We see both of those. Less frequent, though they happen on a regular basis, are the attacks against servers. But we do see a constant, never stopping wave of gamers attacking each other for whatever motives.
GamesBeat: In that case, they’re going to the trouble of finding a farm to use to attack someone?
Labovitz: I don’t know if it’s a farm exactly. There are just sites that you can go to, pay $10 or whatever, and get a link. I don’t think it’s that much trouble. If you have a credit card or Bitcoin, you too can launch a DDoS.
GamesBeat: Now, we’re getting to another part of the problem, then, that something like this isn’t getting shut down.
Labovitz: No, they’re not. It used to be a big deal, to find a machine that [had] a gigabit of bandwidth. Today, you can rent one. We’ve seen an explosion of bandwidth, an explosion of devices out there, servers and others. Stuff on the edge has grown by 10 or 100 times. You’re left with the guys in the middle of the Internet facing — I remember I had a pool growing up, and sometimes, the algae in it would just explode overnight. I think that’s how a lot of game companies and carriers feel, facing 10 times the devices with 10 times the bandwidth. You can buy any of it for a few bucks.
GamesBeat: How do you mitigate this?
Labovitz: As I say, we have pretty broad coverage. Our customer base includes a large cross section of the major game companies, as well as providers, in North America. The game companies do two things. We work with them on traffic engineering and visibility. We can detect unusual spikes, unusual shifts in traffic. We also work with devices on the network, particularly — a lot of our focus is not on third-party devices, that Palm Pilot world, but we’re working with a lot of the router vendors. Nokia, of course, is a big focus there, but we’re also working with other providers that do the plumbing of the Ethernet.
Deepfield’s big idea, instead of what we used to do when security was something you added to the network, we’ve been working with all of these providers to make sure it’s built in. Every bit of networking device has the capability to block and to filter. We’re working with them to build these blocking capabilities, to build this intelligence in the network, so we can accommodate this huge explosion of devices and bandwidth over the last few years.
GamesBeat: I’ve written a lot about semiconductor companies like ARM that are trying to build trust networks and physical hardware security for IoT chips. Is any of that helping yet? Or do we have too many unprotected devices already out there?
Labovitz: I’ve never won my battles against the moles in the backyard, and that’s never going to happen on the Internet. We’re never going back. Pandora’s box is open. Just as an example, do you know what’s the most popular domain name on the Internet as far as DNS queries?
GamesBeat: Maybe some kind of movie pirating site?
Labovitz: Nope. It’s not Google or Facebook either. The single most popular thing queried on the Internet is time.netgear.com because eight years ago, a bug was introduced into the firmware of routers, where devices would make regular queries well beyond anything they really needed to do to set their time. That bug was fixed long ago, but what’s fascinating is that it’s still by far the most popular thing queried on the Internet. That speaks to how hard it is, once firmware gets out there — the changes of that getting permanently fixed, it’s like a radioactive half-life. We’re stuck with it forever.
GamesBeat: As far as game companies go, are they collectively addressing this in some way? Do they have their own security conferences or other signs they’re approaching this as a group?
Labovitz: Certainly, there’s a very tight security community. It’s not very big. All of us know each other and travel in the same circles. There’s a lot of collaboration. It’s not just the game companies, of course. Whether you’re a game company, a financial company, or one of the ISPs, security crosses all of those. There’s a lot of interaction as we push on initiatives and share information about the threats we’re seeing, as well as working with vendors like Nokia as we work on solutions and try to implement them.
We spend a lot of time talking to different groups and working with different parties. I’m not aware of a specific security organization just for gaming, but certainly, there are a lot of discussions, a lot of engineering meetings. It’s a fairly small community, and it works together.
GamesBeat: As far as other problems besides DDoS, what do you see in security that relates to games?
Labovitz: I can only tell you about what we deal with. I read articles about other things, like loot box fraud, but the problems we deal with in the market, what I personally interact with — it’s just keeping everything running as this stuff continues to scale. Keeping it running, keeping the latency and performance up. Part of that is blocking DDoS, but it’s also just managing the complexity of traffic.
It used to be that whenever you went to Netscape.com, you went to a single server. Today, if you play a game or watch a video, a lot of infrastructure needs to work together from different game servers, different telemetry servers, and content distribution. Power has come at the cost of complexity. Traffic comes from a lot of places. Lots of things go into playing a game. Managing that traffic as it makes its way across the Internet, having the real-time visibility into quality so that as things shift, you can adjust, and, of course, having real-time visibility into DDoS and security. We really help with all of that: just managing stuff, keeping it up and running, and maintaining basic levels of quality in the experience.
GamesBeat: When you do that, are you interacting directly with game companies, or do you work through intermediaries like Amazon or other games-as-a-service vendors?
Labovitz: We do a little bit of both. We do have direct companies we interact with that are game companies.
GamesBeat: Do you have some predictions on this front? It seems like it can only get to be a bigger and bigger problem.
Labovitz: I’m lousy at predicting the future. Like I said, in 2010, I predicted that DDoS was over. I left my previous DDoS company thinking we were done. But I can give you some predictions with that in mind.
I think we’re in the early days of IoT. I’m one of those guys who vowed to never have an IoT device at home, and now — well, I don’t want to talk about what I have in my home. But if you take my mother, she has a Nest doorbell. She has connected speakers. We’re still in the early days of things in the home that have IP addresses. We’re also in the early days of bandwidth. The bandwidth predictions we’re seeing these days are wild. If you look at 5G, suddenly, we’re talking about every phone having huge amounts of bandwidth available in addition to IoT devices.
I don’t think we’ve made the advances we need to in terms of figuring out how to secure servers, how to secure IoT. I don’t think we’ll win that. There’s no magic bullet. We’ve been trying to win as far as protecting PCs and protecting servers for 30 or 35 years. It hasn’t happened yet. It’s not likely to happen any time soon. We’re seeing new threats even at low levels. The threat will continue to grow.
My main prediction is we need to be able to build this stuff into the network itself. You mentioned ARM and others. We’re seeing significant advances in the basic chipsets. Nokia makes our own hardware, so we like to think we’re ahead of the curve, but we’re seeing even some of what’s called merchant silicon, the commodity chips market. They’re a little bit behind, but we’re seeing a lot of advances in merchant silicon as well.
I have high hopes that if we can build this into the network, if we can make sure the hardware advances continue, and if security isn’t an afterthought but really starts to become a part of how we build everything, we can have a chance of improving or at least maintaining the status quo. I don’t know if we’ll ever win.
GamesBeat: I had a couple of questions about streamers. A few years ago, there was a streamer who became very popular broadcasting on Twitch, and he was followed by a bunch of DDoS attack groups. They had a sort of sparring conversation. He would go play a game, and then, the attackers would take down that game while he was trying to stream and repeat the process every time he started a new game. People would watch this, and the audience got bigger and bigger as the day went on. Every game he tried to play, the attackers took down. Some of these streamers have enormous audiences now, with hundreds of thousands of concurrent viewers. I wonder if there’s a way they have of protecting themselves now.
Labovitz: That’s another big thing. Like I say, there are two types of attacks we see. You have attacks against servers and then attacks against players or even streamers. Previously, I think most of the focus was on the servers, higher up on the network. But we’re seeing the volume of malicious traffic — and a lot of that is DDoS — becoming so large that it’s a performance win if your provider can automatically block this traffic when it first enters the network. We’re starting to see carriers — including probably your provider because we’re working with a lot of the U.S. providers — who are trying to add these capabilities for blocking traffic before it ever enters the network.
Going back 5 [to] 10 years, DDoS protection was so expensive that it was just the big banks and a handful of other companies that were purchasing it. Of course, those numbers have come down. You can protect web pages. But the cost of protecting your business traffic or your traffic at home is still prohibitive. Sometimes, that’s not even technically available.
What we are seeing, though, is DDoS protection going from something you add to the network to something that is available, that’s already in place for every customer. It’s just part of the network. We’re starting to see the buildout of infrastructure and capability to block DDoS everywhere in the network, and that capability could be available, whether automatically or for a fee, to every home user and every business. We’re seeing DDoS go from something available to dozens or hundreds of companies to something that’s available to everyone as the problem has become more significant [and] more ubiquitous.
As I say, this has taken a while, but we’re finally seeing a convergence of technology and incentives. This stuff is cyclical. Back in 2010, I thought we had won. Then, the world changed on us. In hindsight, the ways it changed are obvious, but hindsight is always obvious. We’re starting to see more capabilities built into the network, and that’s quite encouraging.