this post was submitted on 20 Oct 2025
420 points (98.6% liked)

Selfhosted

52365 readers
1508 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] db0@lemmy.dbzer0.com 234 points 17 hours ago (9 children)

It's wild that these cloud providers were seen as a one-way stop to ensure reliability, only to make them a universal single point of failure.

[–] joel_feila@lemmy.world 3 points 6 hours ago (1 children)

Well companies use not for relibibut to outsource responsibility. Even a medium sized company treated Windows like a subscription for many many years. People have been emailing files to themself since the start of email.

For companies moving everything to msa or aws just was the next step and didn't change day to operations

[–] NotMyOldRedditName@lemmy.world 3 points 4 hours ago

People also tend to forget all the compliance issues that can come around hosting content, and using someone with expertise in that can reduce a very large burden. It's not something that would hit every industry, but it does hit many.

[–] Nighed@feddit.uk 114 points 16 hours ago (2 children)

But if everyone else is down too, you don't look so bad 🧠

[–] clif@lemmy.world 10 points 10 hours ago* (last edited 10 hours ago)

One of our client support people told an angry client to open a Jira with urgent priority and we'd get right on it.

... the client support person knew full well that Jira was down too : D

At least, I think they knew. Either way, not shit we could do about it for that particular region until AWS fixed things.

[–] queerlilhayseed@piefed.blahaj.zone 60 points 16 hours ago (2 children)

No one ever got fired for buying IBM.

[–] cdzero@lemmy.ml 15 points 14 hours ago (2 children)

I wouldn't be so sure about that. The state government of Queensland, Australia just lifted a 12 year ban on IBM getting government contracts after a colossal fuck up.

[–] queerlilhayseed@piefed.blahaj.zone 44 points 13 hours ago* (last edited 13 hours ago) (2 children)

It's an old joke from back when IBM was the dominant player in IT infrastructure. The idea was that IBM was such a known quantity that even non-technical executives knew what it was and knew that other companies also used IBM equipment. If you decide to buy from a lesser known vendor and something breaks, you might be blamed for going off the beaten track and fired (regardless of where the fault actually lay), whereas if you bought IBM gear and it broke, it was simply considered the cost of doing business, so buying IBM became a CYA tactic for sysadmins even if it went against their better technical judgement. AWS is the modern IBM.

[–] NotMyOldRedditName@lemmy.world 2 points 4 hours ago* (last edited 4 hours ago)

if you bought IBM gear and it broke, it was simply considered the cost of doing business,

IBM produced Canadian Phoenix Pay system has entered the chat with a record 0 firings.

[–] sugar_in_your_tea@sh.itjust.works 20 points 13 hours ago

AWS is the modern IBM.

That's basically why we use it at work. I hate it, but that's how things are.

[–] ByteJunk@lemmy.world 4 points 13 hours ago

Such a monstrous clusterfuck, and you'll be hard pressed to find anyone having been sacked, let alone facing actual charges over the whole debacle.

If anything, I'd say that's the single best case for buying IBM - if you're incompetent and/or corrupt, just go with them and even if shit hits the fan, you'll be OK.

[–] Auli@lemmy.ca 5 points 12 hours ago

Yes but now it is nobody ever got fired for buying Cisco.

[–] corsicanguppy@lemmy.ca 10 points 11 hours ago (1 children)

universal single point of failure.

If it's not a region failure, it's someone pushing untested slop into the devops pipeline and vaping a network config. So very fired.

[–] 4am@lemmy.zip 7 points 10 hours ago

Apparently it was DNS. It’s always DNS…

[–] GissaMittJobb@lemmy.ml 56 points 15 hours ago (4 children)

It's mostly a skill issue for services that go down when USE-1 has issues in AWS - if you actually know your shit, then you don't get these kinds of issues.

Case in point: Netflix runs on AWS and experienced no issues during this thing.

And yes, it's scary that so many high-profile companies are this bad at the thing they spend all day doing

[–] B0rax@feddit.org 4 points 8 hours ago* (last edited 7 hours ago)

Case in point: Netflix runs on AWS and experienced no issues during this thing.

But Netflix did encounter issues. For example the account cancel page did not work.

[–] village604@adultswim.fan 18 points 13 hours ago (2 children)

Yeah, if you're a major business and don't have geographic redundancy for your service, you need to rework your BCDR plan.

[–] Bob_Robertson_IX@discuss.tchncs.de 9 points 13 hours ago (1 children)
[–] village604@adultswim.fan 9 points 13 hours ago

So does an outage, but I get that the C-suite can only think one quarter at a time

[–] sugar_in_your_tea@sh.itjust.works 5 points 13 hours ago* (last edited 13 hours ago)

Absolutely this. We are based out of one region, but also have a second region as a quick disaster recovery option, and we have people 24/7 who can manage the DR process. We're not big enough to have live redundancy, but big enough that an hour of downtime would be a big deal.

[–] corsicanguppy@lemmy.ca 3 points 11 hours ago

I love the "git gud" response. Sacred cashcows?

[–] tourist@lemmy.world 4 points 12 hours ago (1 children)

What's the general plan of action when a company's base region shits the bed?

Keep dormant mirrored resources in other regions?

I presumed the draw of us-east-1 was its lower cost, so if any solutions involve spending slightly more money, I'm not surprised high profile companies put all their eggs in one basket.

[–] corsicanguppy@lemmy.ca 4 points 11 hours ago

I presumed the draw of us-east-1 was its lower cost

At no time is pub-cloud cheaper than priv-cloud.

The draw is versatility, as change didn't require spinning up hardware. No one knew how much the data costs would kill the budget, but now they do.

[–] tburkhol@lemmy.world 28 points 16 hours ago (1 children)

It is still a logical argument, especially for smaller shops. I mean, you can (as self-hosters know) set up automatic backups, failover systems, and all that, but it takes significant time & resources. Redundant internet connectivity? Redundant power delivery? Spare capacity to handle a 10x demand spike? Those are big expenses for small, even mid-sized business. No one really cares if your dentist's office is offline for a day, even if they have to cancel appointments because they can't process payments or records.

Meanwhile, theoretically, reliability is such a core function of cloud providers that they should pay for experts' experts and platinum standard infrastructure. It makes any problem they do have newsworthy.

I mean,it seems silly for orgs as big and internet-centric as Fortnite, Zoom, or forturne-500 bank to outsource their internet, and maybe this will be a lesson for them.

[–] village604@adultswim.fan 5 points 13 hours ago (1 children)

It's also silly for the orgs to not have geographic redundancy.

[–] killabeezio@lemmy.zip 2 points 8 hours ago (1 children)

No it's not. It's very expensive to run and there are a lot of edge cases. It's much easier to have regional redundancy for a fraction of the cost.

[–] village604@adultswim.fan 4 points 5 hours ago* (last edited 5 hours ago)

The organizations they were talking about and I was referring to have a global presence

Plus, it's not significantly more expensive to have a cold standby in a different geographic location in AWS.

[–] ms_lane@lemmy.world 10 points 16 hours ago

They zigged when we all zagged.

Decentralisation has always been the answer.

[–] mhzawadi@lemmy.horwood.cloud 8 points 16 hours ago

yeah, so many things now use AWS in some way. So when AWS has a cold, the internet shivers

[–] relativestranger@feddit.nl 1 points 15 hours ago

sidekicks in '09. had so many users here affected.

never again.

[–] wirebeads@lemmy.ca 1 points 16 hours ago

A single point of failure you pay them for.