this post was submitted on 21 Jul 2024
191 points (76.5% liked)
Technology
59534 readers
3143 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Please, enlighten me how you'd remotely service a few thousand Bitlocker-locked machines, that won't boot far enough to get an internet connection, with non-tech-savvy users behind them. Pray tell what common "basic hygiene" practices would've helped, especially with Crowdstrike reportedly ignoring and bypassing the rollout policies set by their customers.
Not saying the rest of your post is wrong, but this stood out as easily glossed over.
A decade ago I worked for a regional chain of gyms with locations in 4 states.
I was in TN. When a system would go down in SC or NC, we originally had three options:
I got sick of this. So I researched options and found an open source software solution called FOG. I ran a server in our office and had little optiplex 160s running a software client that I shipped to each club. Then each machine at each club was configured to PXE boot from the fog client.
The server contained images of every machine we commonly used. I could tell FOG which locations used which models, and it would keep the images cached on the client machines.
If everything was okay, it would chain the boot to the os on the machine. But I could flag a machine for reimage and at next boot, the machine would check in with the local FOG client via PXE and get a complete reimage from premade images on the fog server.
The corporate office was physically connected to one of the clubs, so I trialed the software at our adjacent club, and when it worked great, I rolled it out company wide. It was a massive success.
So yes, I could completely reimage a computer from hundreds of miles away by clicking a few checkboxes on my computer. Since it ran in PXE, the condition of the os didn’t matter at all. It never loaded the os when it was flagged for reimage. It would even join the computer to the domain and set up that locations printers and everything. All I had to tell the low-tech gymbro sales guy on the phone to do was reboot it.
This was free software. It saved us thousands in shipping fees alone. And brought our time to fix down from days to minutes.
There ARE options out there.
This works great for stationary pcs and local servers, does nothing for public internet connected laptops in hands of users.
The only fix here is staggered and tested updates, and apparently this update bypassed even deffered update settings that crowdstrike themselves put into their software.
The only winning move here was to not use crowdstrike.
Absolutely. 100%
But don’t let perfect be the enemy of good. A fix that gets you 40% of the way there is still 40% less work you have to do by hand. Not everything has to be a fix for all situations. There’s no such thing as a panacea.
Sure. At the same time one needs to manage resources.
I was all in on laptop deployment automation. It cut down on a lot of human error issues and having inconsistent configuration popping up all the time.
But it needs constant supervision, even if not constant updates. More systems and solutions lead to neglect if not supplied well. So some "would be good to have" systems just never make the cut, because as overachieving I am, I'm also don't want to think everything is taken care of when it clearly isn't.
Yeah. I find a base image and post-install config with group policy or Ansible to be far more reliable.
Yea we're doing something similiar. Only update base images for bigger OS updates or if something breaks or can break.
The general idea is to have config that works for both new PCs and the ones that are already in use. Saves on maintaining two configuration methods.
You were all in, but was the company all in? How many employees? It sounds like you innovated. Let's say that the company you worked for was spending millions on vendors that promised solutions but rarely delivered. If instead they gave you $400k a year, a $1 million/year budget & 10 employees.. I'm guessing you could have managed the laptop deployment automation, along with some other significant projects as well.
Instead though, people with good ideas, even loyal to the company, are competing against sales and marketing reps from billion dollar companies, and upper management are easily swooned.
I'm the only one to swoon here, and I'm as sceptical as one can be.
I'm also a cost and my budget is on paper only. Non-IT management is complicit in crappy IT.
Completely fair, man.