Systems that hold when things break.
Things will break. The question is what your system does next, and whether anyone had to be awake for it. I lean toward the simplest design that will still hold up at scale.
Below is a small production system, running live in your browser. Try to take it down.
Three regions. About a thousand requests a second. Each light is one request, slowed right down so you can watch it.
Every number and every failure comes from a model running here, with real queues, timeouts and retries. Nothing is scripted.
Go on. Break it.
It's a simulation running in your browser, not my production system or a client's.
What I've actually run
Multi-region AWS
A live-streaming platform, as co-founder and CTO for six years. Containerised services on ECS, with WebRTC and Cloudflare for delivery and the infrastructure defined as code in the AWS CDK. One region in the US and one in Asia, so a global audience wasn't served from a single continent.
Tencent Cloud and AWS China
I stood up and ran servers for a creative studio in the Chinese market, for regional compliance and performance.
The usual kit
AWS, Docker, Kubernetes, Terraform, CI/CD.
Have a real one?
Tell me what's worrying you about your system, or the decision you're trying to make.