A community to talk about the Fediverse and all it's related services using ActivityPub (Mastodon, Lemmy, Mbin, etc).
If you wanted to get help with moderating your own community then head over to !moderators@lemmy.world!
Learn more at these websites: Join The Fediverse Wiki, Fediverse.info, Wikipedia Page, The Federation Info (Stats), FediDB (Stats), Sub Rehab (Reddit Migration)
Thank you very much for sharing this, it makes all Lemmy instances safer. Good job!
Im curious how they could execute that postgres archive command to write a marker. Did they use the oauth token to be able to do that?
Just not sure how they can run postgres queries as a normal user. What made that possible?
ouch. i really feel for you.
Really appreciate the transparency. Time to change my blahaj password!
Then they transfered a file to /tmp/exp which was linux kernel CVE-2026-43500, nicknamed ‘Dirty Frag’, an RxRPC local privilege escalation. I had not patched these internal servers that nobody should have access to against this.
Lessons Learned #1:
Install your patches.
"But I have a firewall!"
That is not a sufficient control.
Install.
Your.
Fucking.
Patches!
"Just patch" is advice for a windows administrator, where updates break everything so you have to sit and baby them and apply them manually.
On Linux, there are ways to enable automatic security updates, including automatic reboots, so you can safely receive the mitigations your distro provides. That way, you don't have to worry about forgetting to patch (until the distro release becomes unmaintained, at least).
Now, dirty frag was a zero day, meaning that it was released and probably in the wild before a mitigation was pushed out to handle it. So you did need to apply an actual configuration patch... unless you had some form of kernel based isolation, which I mention as #2 of my other comment in this thread: https://programming.dev/post/52129409/24414213
"Should" is a four-letter word in fields like safety and security.
well, there goes 75% of shitposts on lemmy.
Also a vital support network for LGBT+ people, I guess?
Don't worry, we're back again :)
Also a vital support network for LGBT+ people, I guess?
Mhm and the instance where the only women's community is hosted :(
Our piefed instance (which hosts our remaining communities) should be back up soon!
i know this isn't my place as a cis white dude, but can anarchist.nexus be a resource for a women's space? i've been looking for gaps in the threadiverse we could nurture and foster. my #1 concern lately with the makeup of the hegemony of the threadiverse lately has been:
my commitment to you as a cis white dude admin would be to stay the fuck out of the way of moderators looking to host a women's issues community here, within reason. obviously it wouldn't be okay to break instance rules against bigotry and such, but i can offer my commitment to shutting my mouth and knowing my role
edit: also, i bet @poVoq@slrpnk.net and @ProdigalFrog@slrpnk.net from my second original home instance would be happy to have a women's issues community hosted on slrpnk.net, i'm still figuring out what my new relationship and responsibility to the threadiverse is now, but i think just like how there's more than one world news community, there could be more than one women's issues community so that people have a preferred and fallback space on days like today. the one thing i don't want to diminish in this comment is the incredible work people like @ada@lemmy.blahaj.zone have done in creating a welcoming space on the threadiverse for marginalized people. i just don't want that to be work that is only done alone. we should be collaborative towards a better, freer, world
For what it's worth, I absolutely want to see duplicate communities across multiple instances. Even if it's a backup community, folks shouldn't be left without a space if an instance goes offline, or if an admin goes rogue. I want to see more communities for vulnerable folk across more instances.
I wish there were more queer first instances too, and hopefully, this incident pushes someone in to spinning one up!
There used to be a women's space on slrpnk.net (although a bit unfortunately named after a controversial Reddit community), but we could never really find women willing to moderate it longer term and the amount of dudes showing up with trollish comments became a bit too much for us admins to handle.
the amount of dudes showing up with trollish comments became a bit too much for us admins to handle.
The mods in womensstuff work really, really hard.
Maybe! LadyButterfly is the mod of the womensstuff community and she does a great job, maybe we can get this info to her.
I really appreciate your acknowledgement of some of the issues with the diversity in the threadiverse. And appreciate your support <3
@LadyButterfly@piefed.blahaj.zone , when your instance is online again let's have a discussion about what resources i can make available to you that will center your community's needs without someone like me colonizing the discussion.
(unless i have the wrong ladybutterfly, then my apologies)
That’s her :)
Thanks for sharing the details 👍
Excellent writeup, and I appreciate the transparency. I have some suggestions on how to mitigate something like this from happening in the future.
This is simpler than trying to control postgres permissions granularity. Even if one application that connects to a database gets owned, it doesn't have access to other postgres databases, preventing data leaks/exfiltration.
Kata containers is a container runtime, that is virtual machine.
There is also Gvisor and Syd Box, which are application kernels. Application kernels are reimplimentations of the parts of the Linux kernel needed to run apps, and in this case both Gvisor (Go) and Syd Box (Rust) are in memory safe langauges.
Kata containers are faster, but you will need nested virtualization in order to use them. Application kernels are slower, but you can install them anywhere, including hosts where virtualization is disabled (like a VPS that doesn't let you enable nested virtualization.
Both take a tiny bit more resources intensive due to no longer being able to share the host kernel, but for most part, it is worth it. They don't bring an entire kernel along, just what is needed to run apps.
Both offer similar levels of isolation, and preventing applications running inside them from touching the host kernel directly. They effectively manage to prevent issues like copy fail, dirty frag, and so on, from owning your host.
They are fairly easy to install, docker has some docs here: https://docs.docker.com/engine/daemon/alternative-runtimes/ . But if you are using podman or kubernetes, you can also install them there.
A large part of the draw of stable Linux like Debian or Red Hat, is that they only do security updates. They don't do feature updates, or even bug fixes (except for critical ones). In doing so, there is essentially a guarantee of reliability, where it is impossible for updates to break anything.
This makes it possible to enable automatic security updates, and you can even configure it to automatically reboot in order to load a new kernel that includes mitigations against issues like dirty frag. Make sure your docker containers are configured to automatically restart and everything will be smooth.
"Just patch" is a good but it is never enough, and I am frustrated hearing it so frequently. The way I view it is, any time I have to patch, what I really need to do is to improve my security architecture so I never have to "patch" this specific issues again. Patches are the exact kind of security toil that I complain about in this comment.
Use a separate DBMS (that is, a separate postgres/mariasql/etc container) for each service. Give each one service unique passwords, which you can define in the docker compose.
unique passwords is good practice, but separate db server for each of the services is extreme. it brings much more resource consumption. the solution here is being subscribed to security releases and updating soon. those application kernels also sound like a good idea. and as I understand, postgres permissions were not at fault, the permission system had a bug.
Even if one application that connects to a database gets owned, it doesn't have access to other postgres databases, preventing data leaks/exfiltration.
except that because of the bug, anyone with query permission could have become postgres superuser.
except that because of the bug, anyone with query permission could have become postgres superuser.
If a user can't log in to a DBMS, they don't have query permission.
separate db server for each of the services is extreme. it brings much more resource consumption.
Yes. It consumes more resources. But it's not that much more, and you can make it fit easily. Many users using docker compose unwittingly do this since docker composes often bring their own database containers. When done consciously, you make a trade off for peace of mind.
the solution here is being subscribed to security releases and updating soon.
I addressed takes like these in the last part of my previous comment. The linked comment also elaborates on my opinions about manual updates, manually watching security releases, and other forms of security toil.
Fookin' hell. Nightmare scenario.
Oof, that sucks :(
Damn, i hope the femcelmemes crew is fine
I've been wondering why it's down, that's my home instance. That's unfortunate :/
Oof, hope it works out for them 😣
Bummer! I was wondering what had happened. Thanks for the update!
This is horrible. And I say this as someone who's had horrible experiences with users over there, particularly when it comes to bullying and harassment. They didn't deserve to get shut down like this. Hope they're able to fix whatever the problem is and get back up and running again.