DesertCreosote

joined 1 year ago
[–] DesertCreosote@lemm.ee 1 points 4 months ago

I’m one of the admins who manage CrowdStrike at my company.

We have all automatic updates disabled, because when they were enabled (according to the CrowdStrike best practices guide they gave us), they pushed out a version with a bug that overwhelmed our domain servers. Now we test everything through multiple environments before things make it to production, with at least two weeks of testing before we move a version to the next environment.

This was a channel file update, and per our TAM and account managers in our meeting after this happened, there’s no way to stop that file from being pushed, or to delay it. Supposedly they’ll be adding that functionality in now.

[–] DesertCreosote@lemm.ee 8 points 4 months ago (1 children)

Yes, CrowdStrike says they don’t need to do conventional AV definitions updates, but the channel file updates sure seem similar to me.

The file they pushed out consisted of all zeroes, which somehow corrupted their agent and caused the BSOD. I wasn’t on the meeting where they explained how this happened to my company; I was one of the people woken up to deal with the initial issue, and they explained this later to the rest of my team and our leadership while I was catching up on missed sleep.

I would have expected their agent to ignore invalid updates, which would have prevented this whole thing, but this isn’t the first time I’ve seen examples of bad QA and/or their engineering making assumptions about how things will work. For the amount of money they charge, their product is frustratingly incomplete. And asking them to fix things results in them asking you to submit your request to their Ideas Portal, so the entire world can vote on whether it’s a good idea, and if enough people vote for it they will “consider” doing it. My company spends a fortune on their tool every year, and we haven’t been able to even get them to allow non-case-sensitive searching, or searching for a list of hosts instead of individuals.

[–] DesertCreosote@lemm.ee 25 points 4 months ago (5 children)

Speaking as someone who manages CrowdStrike in my company, we do stagger updates and turn off all the automatic things we can.

This channel file update wasn’t something we can turn off or control. It’s handled by CrowdStrike themselves, and we confirmed that in discussions with our TAM and account manager at CrowdStrike while we were working on remediation.

[–] DesertCreosote@lemm.ee 7 points 11 months ago (1 children)

Can't speak for the person you're replying to, but I'm a security engineer and stuff still makes its way to me that you would think would get filtered out by others (and isn't my job to fix). It just takes the right person thinking "this is obviously a problem with $system, let's just send it straight over to them so they can fix it quickly!" And then we get the fun job of proving it's not us and has no relation to us.

We got a ticket today for packet loss between two systems, neither of which have any of our tools on them...