Mr. Nice Try Notes from the edge

· 4 min read

Mistakes were made

The plan was to move 1,900 websites onto the Edge 5% of the traffic at a time. Testing particular sites through the load balancer turned out to be the hard part, so I edited my hosts file and went from 5% to 100%.

On Sunday, about 1,900 websites moved onto the Mr. Nice Try Edge. It now watches the front door of 3,565 sites.

Nothing crashed. But it didn't go the way the plan said it would, and how I got there is worth writing down.

Mr. Nice Try shrugs over his coffee, one hand on a big dial cranked all the way round, as every dot in the pipe pours into the new branch.

The plan: 5% of the traffic

The plan was a dial. The load balancer in front of these sites sends 5% of the traffic to the new Edge, then 25%, then 50%, then 100%, with a day or more at each stop.

5% of the traffic means 5% of the visitors, from every site, picked at random. That's great for finding problems under real load: a small slice of everyone, so anything that breaks only breaks for a few people.

The hard part: testing one site on purpose

Moving most of these sites was a no-brainer.

A few needed quite a bit more attention. For those I wanted solid end-to-end tests: open the site in a browser, click around, and see every page come back right through the new Edge.

That's where the dial got in the way. With a weighted load balancer, you don't get to choose. Whether you land on the old server or the new Edge is a dice roll, and with stickiness on, wherever you land is where you stay. I couldn't say "send me to the new Edge for this site, please."

A robot receptionist in a bow tie rolls dice to decide which door each visitor goes through, while Mr. Nice Try holds up a clipboard of the sites he wanted to pick himself.

So I edited my hosts file

Every computer keeps a tiny phone book of its own, and it checks it before asking real DNS: /etc/hosts on a Mac or Linux machine (Windows has one too). Add a line, and your computer believes whatever you tell it about where a site lives.

So on my laptop, and only on my laptop, I pointed the tricky sites straight at the new Edge:

# /etc/hosts (with a made-up address)
203.0.113.10  example.com www.example.com

My browser skipped the dice roll and went exactly where I sent it. Everyone else kept going through the load balancer as usual.

That let me test each tricky site end to end, in a real browser, as many times as I liked. When I was done, I deleted the lines and I was back to normal.

Two gotchas if you try this. Your browser and operating system cache DNS answers, so a change might not show up straight away: restart the browser, or on a Mac run sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder. And for HTTPS, the server you point at needs a valid certificate for the site, or you'll get a browser warning instead of a test.

Mr. Nice Try strolls past the baffled, dice-rolling robot receptionist along a coral dashed path he drew himself, straight to the new door, laptop and coffee in hand.

Everything looked good

The tricky sites worked through the new Edge. And the logs for the 5% already going there, real visitors clicking real pages on every site, looked great too.

Mr. Nice Try sits on a stool highlighting rows on a log printout that unrolls across the floor to a little robot reading it upside down.

The undo button came first

Before touching the dial, I wrote a script to put everything back in one go: the old server to 100%, the new Edge to 0%.

Then I tested it both ways: scale up, scale down. Moving traffic over and moving it back was the part I'd actually rehearsed, so it was the part I trusted most.

Mr. Nice Try leans on a pedestal with a big coral undo button, sipping coffee, while a nervous little robot peeks out from behind it.

So I went all the way

With the tricky sites tested, clean logs and a tested undo, I picked a slow time on Sunday and moved everyone else across. No 25%. No 50%. Straight from 5% to 100%.

Was I nervous? Honestly, no. The old server kept running the whole time, undo was one command and a few seconds, and the worst case was a short blip for a few visitors at the quietest time of the week.

What I'd take from it

  • A traffic split can't aim. It's great for real-world load and no help when you need to test one site on demand. Plan for both.
  • Your hosts file is the oldest trick in the book. Point a domain at a new server on your own machine, test it for real, then delete the line. Nobody else is affected.
  • Write the undo before the do. Then run it. A rollback you've never tried is a hope, not a plan.
  • Watch real traffic. A small slice of real visitors, watched closely, finds things your own tests won't.
  • Pick the quiet hour. If you're going to skip steps, skip them when the fewest people are around.

Mistakes were made. Nothing broke. 3,565 websites are on the Edge now, and the bots are finding out.

If you want to see what they get up to, I'm @MrNiceTry on X.