How we just defended a huge attack that cold-booted almost 1,000 servers

At 22:12 UTC on Monday, one bot knocked on more than 1,400 websites behind the Edge in a single minute.

The serverless app behind those sites normally runs about 16 copies of itself at once. Sixty seconds later it was running 926. Every new site the bot touched needed a server, and almost none were awake, so they cold-booted, hundreds at a time.

Over the next five minutes the same bot sent 84,161 requests to 1,398 websites. Then it stopped.

Nothing went down. Here's what happened, what it was after, and what I changed so its network gets a 403 the next time it knocks.

What a cold boot is

Most of the 3,597 websites behind the Mr. Nice Try Edge are served by one serverless app on AWS Lambda. Serverless means there's no fixed fleet of servers. When a request arrives and no copy of the app is free, AWS starts a fresh one: it boots, loads the app, and only then answers. That first answer is slow. That's a cold boot.

On a normal evening that happens a few times a minute and nobody notices. The app keeps a handful of copies warm and reuses them.

This bot didn't hit one site hard. It hit every site at once: more than 1,400 different websites in the first minute, so hundreds of requests landed at the same moment, each wanting a server, and the free copies ran out instantly.

Copies of the server running at once

The serverless app behind the sites · same 20 minutes

The slowest 5% of responses that minute took 3.2 seconds instead of the usual 0.1 or so. That's the boot time. Once the copies were up, the app was fine: by the busiest minute, at 41,345 requests, the slowest 5% were back to 41 milliseconds.

AWS throttled nothing and the app never crashed. Out of 84,161 requests, 4 came back with a server error. Serverless did exactly what it's sold to do. It just did it all at once.

One bot, one IP, 41,345 requests a minute

All of it came from a single IP address in 45.138.12.0/24, on a hosting network called TC DATACENTER LIMITED (AS218785).

Requests per minute, all 3,597 sites

October 19, 22:05–22:24 UTC

The usual background noise across all 3,597 sites is a few hundred requests a minute. At its peak this one bot sent 130 times that, alone.

What it wanted: your keys

It didn't want the websites. It wanted the files people forget to hide. For each site it asked for the homepage, then ran down a list:

What it asked for	Times
/ (the homepage, to see if anyone's home)	4,897
/.env	761
/%2F.env and //.env (the same file, sneaking past simple filters)	841
/.aws/credentials	403
/.env.production, .env.local, .env.staging, .env.backup, .env.bak…	about 365 each, 20+ versions

A .env file holds an app's secrets: database passwords, API keys, cloud credentials. One that's been left reachable on the web gives all of it away in a single request. That's why bots ask for it more than almost anything else.

None of these sites have one to give. 75,634 requests (90%) got a plain 404.

The bot also wore a costume. 79,248 of its requests claimed to be Firefox 47 on Windows 7, a browser from 2016. But every request opened its connection the same way, with one TLS fingerprint, and it isn't Firefox's. That's a script with a borrowed name tag, and the borrowed name was ten years old.

The first wave got through

I want to be straight about this part: the first five minutes reached the servers. The bot was in and out before I'd looked at a chart.

What saved the evening wasn't a clever block. It was that there was nothing to find: no .env, no .aws folder, no backups, nothing in the web root that shouldn't be there. Every one of those requests cost a few milliseconds and found nothing.

If you take one thing from this post, take that. A blocklist helps. Not having the file is what actually protects you.

My own chart couldn't see it

When I opened my traffic chart, I couldn't see my normal traffic at all. The spike was 41,000 tall, so the usual few hundred a minute were drawn about one pixel high. Every minute before and after the attack looked empty.

So I fixed the chart. Now, when one bar is far taller than the usual, the scale follows the usual bars instead, and the giant gets cut short with a little break across its top. Hover it and you still get the real number.

Banning the whole network

Here's the thing about that IP: banning it alone would have been almost pointless. Over the past 12 days, TC DATACENTER used 40 different IPs from two small blocks to ask 2,318 of the sites for 171,880 things, and 88% of those were probes like the ones above. It rotates. Ban one address and the next request comes from the one next door.

So I looked at who else on the internet does nothing but this, and found three more networks just like it. Then I banned all four networks, not their IPs, with one rule on the Edge:

BAN ip FOR 30 DAYS
WHERE asn IN (218785, 48090, 202412, 211590)

The Edge checks it on every request, so any IP on those networks is caught the first time it knocks, even one it has never seen before. Banned visitors go to a waiting room that ends in a plain 403. The scripts don't run JavaScript, so they never get past the waiting room. Either way, nothing reaches the servers.

They came back. Since the ban, those four networks have tried 12,958 more times. All but 170 were stopped at the door.

The full list, and how to ban them from your own app in a couple of lines, is in the next post: Ban these networks from your application right now.

And then I banned myself

To check the 403, I opened one of the sites on my phone and asked it for a probe path. The Edge did its job. It tagged me, sent me to the waiting room, and turned me away.

And I had no quick way to let myself back in. The tag sat on whatever IP my phone carrier had given me that hour, and the list of tags only showed one tag at a time.

So that got fixed too. There's now one search box for every tag and ban, by IP or by a whole range like 172.58.0.0/16, and a Lift mine button that spots the IP you're browsing from and lets you go, wherever your phone has put you tonight.

What I'd take from it

• Serverless scales sideways, and so can attacks. This bot wasn't heavy on any one site. It was wide. Wide is what makes a serverless app cold-boot hundreds of copies at once. If you run many sites on one app, watch for breadth, not just volume.
• The best defense is having nothing to find. Keep .env, .git, backups and credentials out of your web root. Then a scan like this is just a few thousand 404s.
• Ban networks, not IPs. Scanners rotate inside their block. Ban the block.
• Make sure your charts can show a spike and the normal day. If one can't, you'll miss everything around the thing you're looking at.
• Test your ban from outside, and build the unban first. You will lock yourself out. Make getting back in one click.

84,161 requests, 926 servers woken up, nothing found and nothing broken. The next time they knock, they get a 403.

If you want to see who's knocking, I'm @MrNiceTry on X.