A complicated system design with multiple loosely related systems dynamically handing off work to each other with a race condition, REALLY doesn't fit the single point of failure concept.
this post was submitted on 27 Oct 2025
414 points (98.8% liked)
Technology
76418 readers
2614 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
It's funny, because I've heard a variety of reasons why the outage happened, why it wasn't caught in time, why it signaled a problem with hardware versus software or human error versus automation.
I think its safe to say the company is increasingly over-managed and under-staffed, no matter how you slice it. Maybe its time to just break the mega-corp up already and let some good old fashioned free market competition fix this mess.
Amazon just did more layoffs today.
some good old fashioned free market competition
This kills the billionare
We need to democratize the internet again, every generation there's a ma bell pretending they own the internet. Current Gen is Google, AWS, Azure and the like, with ISPs just making sure they get their cut.
I don't have an issue with these services existing, but in such a way that everything depends on a couple companies? Dangerous for everyone.
load more comments
(16 replies)
this headline has mad ‘it was was in the last place i looked’ vibe.
We also need more individuals paying for “business” Internet connections at home. We need self-hosters to be able to feel comfortable running public services from their homes. And so we need a set of practices and recipes to follow, so a self-hoster can feel confident that, if one thing gets broken into, the other few dozen things they’re hosting will stay safe.
The “family nerd” hosting things for the family needs to be a thing again. Sorry, friends, I know family tech support sucks. It’ll suck so much more when it’s a web site down and nobody can reach their kid’s softball team page, and there’s a game next weekend, etc. But we’ve seen what happens when we abdicate our responsibilities and let for-profit companies handle it for us.
(I wish so hard that I had a solution ready, a corporate LAN in a box, that someone can just install and use. I’m working on something, but I’m pretty sure I over-complicated it. It doesn’t need to be Fort Knox, it just needs to be pretty good. And I suck at ops stuff.)
We need more places offering the same upload as download so people can do these things from home. Here I know spectrum only offers like 10mb upload even if you’ve got like 3gb download.
It's not just speed, CGNAT is a near complete "fuck you" to self-hosting. You can work around it with a VPN endpoint "in the cloud", but that still means you are reliant on someone else's computer.
It's not only upload (I'm so lucky living in France) but CORS, DNS and other stuff that de-decentralise things IMO.
load more comments
(10 replies)
The last time I looked at bus internet for my location, they wanted 3x the price for 2/3 the speed. At my actual job 20m away, I have the same provider and a business acct. Same downtime, same equipment, only unlock port 80 and 443.
You’re right to be frustrated. Mine is the same way. It’s ok to be passionate about that, and to value punishing greedy ISPs by not paying extra for a business account. (In many cases you could even need both, if you might worry about occasional denial of service attacks and need to be sure attackers can’t also knock out your ability to work from home, for example.)
I think there’s a compelling argument in favor of protecting diversity of hosting and preventing a monoculture or a monopoly. It’s not super compelling, but it’s out there.
load more comments
(5 replies)
We need to ditch cloud entirety and go in house again.
We need to ditch cloud entirety and go in house again.
For many many companies that would be returning to the bad-old-days.
I don't miss getting an emergency page during the Thanksgiving meal because there's excessive temperature being reported in the in-house datacenter. Going into the office and finding the CRAC failed and its now 105 degree F. And you knew the CRAC preventive maintenance was overdue and management wouldn't approve the cost to get it serviced even though you've been asking for it for more than 6 months. You also know with this high temp event, you're going to have an increased rate of hard drive failures over the next year.
No thank you.
There's a huge gulf between pub clowd and shitty on-prem. My daytime contract is with an organization almost completely on-prem for privacy, although on-prem to them means priv-cloud. Space has been rented. Redundant everything piped in. Redundant everything set up. We run VMs by terraform. Wheeeeee
Point is, posing shitty on-prem as the alternative to the clowd is moving the goalposts a bit.
There’s a huge gulf between pub clowd and shitty on-prem.
We agree on this.
Redundant everything piped in. Redundant everything set up. We run VMs by terraform. Wheeeeee
For that customer of yours, is that a single datacenter or does is represent multiple datacenters separated by a large distance across a nation, or perhaps even across national borders?
Point is, posing shitty on-prem as the alternative to the clowd is moving the goalposts a bit.
I think ignoring that shitty on-prem represented a large part of IT infrastructure prior cloud providers is ignoring a critical point. Was it possible to have well-run enterprise IT data centers prior to cloud? Sure. Was everyone doing that? Absolutely not, I'd argue the majority had at least a certain level of jank in their infra and that that floor is raised with cloud providers. Just the basic facilities is enterprise grade irrespective of the server or app config.
We don't have to. It is entirely possible to engineer applications and services in a way that they're not dependent on any one cloud service, while also using cloud services for IaaS. Netflix famously does this, and sure enough Netflix experience no service interruptions during this latest outage despite having a large AWS presence.
If we want a truly robust system, yeah, we kinda do. This sort of event is only one of the issues with allowing a single entity to control pretty much everything.
There are plenty of potential issues from a corrupt rogue corporation hijacking everything to attacks to internal fuck-ups like we just experienced. Sure, they can design a better cloud, but at the end of the day, it's still their cloud. The Internet needs to be less centralized, not more (and I don't just mean that purely in terms of infrastructure, though that is included of course).
Bit of an over-reaction to one incident. I'd be willing to bet the uptime, reliability and scalability of AWS is significantly better than what the vast majority of in-house solutions could do. It's absolutely not worth going back.
Millions of customers using AWS also weren't affected - the company I work for certainly wasn't, although some of our tools like Jira were.
The problem is far more pervasive than any single incident, allowing a single megacorporation to control most of the Internet is a bad idea for many reasons. The Internet is supposed to be decentralized, it was even designed to withstand a nuclear war with that principal in mind. Even with a robust, distributed network with redundant backups, if it's still all controlled by one company, that is still a very precarious situation.
Agreed, but other cloud providers exist and it would be good if there was stronger competition in this space. But going back to self hosting is a huge step back and I think if a CTO said they were going to move from the cloud back to a self hosted solution, pretty much everyone would hate it.
There are still self hosted places today, not everything is cloud based.
Also, there isn't more competition largely because of Amazon so, while I agree with the sentiment that it could improve things, in practice it's a moot point.
I certainly don't miss dealing with air conditioning, dry fire protection, and redundant internet connections.
I also don't miss trying to deal with aging servers out and bringing new hardware in.
That work is still being done by someone in a data centre. But all these jobs went from in-house positions to the centres.
That work is still being done by someone in a data centre. But all these jobs went from in-house positions to the centres.
The difference is scale. When in-house, the person responsible for managing the glycol loop is also responsible for the other CRACs, possibly the power rails, and likely the fire suppression. In a giant provider, each one of those is its own team with dozens or hundreds of people that specialize in only their area. They can spend 100% on their one area of responsibilty instead of having to wear multiple hats. The small the company, the more hats people have to wear, and the worse to overall result is because of being spread to thin.
The inverse of the old axiom "The cloud is just someone else's computer" is "Yes, duh, that's how you get economies of scale".
In-housing would mean an enormous increase in demand for physical hardware and IT technical services with a large variance in quality and accessibility. Like, it doesn't fix the underlying problem. It just takes one big problem and shatters it into a thousand little problems.
That's good though. It means half the internet wouldn't fail.
I think some of you younger folks really don't know what the Internet was like 20 years ago.Shit was up and down all the time.
I worked on a project back in 2008 where I had to physically haul hardware from Houston to Dallas ahead of Hurricane Ike just to keep a second rate version of a website running until we got power back at the original office. Latency at the new location was so bad that we were scrambling to reinvent the website in real time to try and improve performance. We ended up losing the client. They ended up going bankrupt. An absolute nightmare.
Getting screamed at by clients. Working 14 hour days in a cramped server room on something way outside my scope.
Would have absolutely killed for something as clean and reliable as AWS. Not like it didn't even exist back then. But we self-hosted because it was cheaper.
Reminder to everyone, if you aren't necessarily worried about uptime too much, and have a spare device at home, you can host personal websites and various services that might be useful for yourself or friends and family. To keep it simple, all you would really need is
- an up-to-date router that isn't end-of-life
- a firewall that geo blocks traffic from outside your country and blocks all ports except 80 and 443
- port forwarding 80 and 443 to your device
- setup dynamic dns service (some routers can handle this)
- a domain name
Keep your device and router updated and reboot it every once in a while to load the updated kernel. Then just install some web server software or whatever on your device and point your domain to it.
Together, we can decentralize the web a little bit 🙂
As a novice DON'T put your devices on the open Internet. Use something like tailscale or Splittunnel. You can give your friends access through that.
load more comments
(1 replies)
load more comments
(5 replies)
We need to put Amazon in the cloud.
Cuz, you know, the cloud never goes down 👎. /s
view more: next ›