Major IT outage affecting banks, airlines, media outlets across the world

Reading into the updates some more... I'm starting to think this might just destroy CloudStrike as a company altogether. Between the mountain of lawsuits almost certainly incoming and the total destruction of any public trust in the company, I don't see how they survive this. Just absolutely catastrophic on all fronts.

If all the computers stuck in boot loop can't be recovered... yeah, that's a lot of cost for a lot of businesses. Add to that all the immediate impact of missed flights and who knows what happening at the hospitals. Nightmare scenario if you're responsible for it.
This sort of thing is exactly why you push updates to groups in stages, not to everything all at once.
- Looks like the laptops are able to be recovered with a bit of finagling, so fortunately they haven't bricked everything.
  And yeah staged updates or even just... some testing? Not sure how this one slipped through.
Agreed, this will probably kill them over the next few years unless they can really magic up something.
They probably don't get sued - their contracts will have indemnity clauses against exactly this kind of thing, so unless they seriously misrepresented what their product does, this probably isn't a contract breach.
If you are running crowdstrike, it's probably because you have some regulatory obligations and an auditor to appease - you aren't going to be able to just turn it off overnight, but I'm sure there are going to be some pretty awkward meetings when it comes to contract renewals in the next year, and I can't imagine them seeing much growth
- Nah. This has happened with every major corporate antivirus product. Multiple times. And the top IT people advising on purchasing decisions know this.
- Don't most indemnity clauses have exceptions for gross negligence? Pushing out an update this destructive without it getting caught by any quality control checks sure seems grossly negligent.
- explain to the project manager with crayons why you shouldn't do this
  Can't; the project manager ate all the crayons
- Why is it bad to do on a Friday? Based on your last paragraph, I would have thought Friday is probably the best week day to do it.
- rolling out an update to production that there was clearly no testing
  Or someone selected "env2" instead of "env1" (#cattleNotPets names) and tested in prod by mistake.
  Look, it's a gaffe and someone's fired. But it doesn't mean fuck ups are endemic.
- Was it not possible for MS to design their safe mode to still “work” when Bitlocker was enabled? Seems strange.
I think you're on the nose, here. I laughed at the headline, but the more I read the more I see how fucked they are. Airlines. Industrial plants. Fucking governments. This one is big in a way that will likely get used as a case study.
- The London Stock Exchange went down. They're fukd.
Yeah saw that several steel mills have been bricked by this, that's months and millions to restart
- Got a link? I find it hard to believe that a process like that would stop because of a few windows machines not booting.
Testing in production will do that
- Not everyone is fortunate enough to have a seperate testing environment, you know? Manglement has to cut cost somewhere.
What lawsuits do you think are going to happen?
- They can have all the clauses they like but pulling something like this off requires a certain amount of gross negligence that they can almost certainly be held liable for.
- Forget lawsuits, they're going to be in front of congress for this one
Don't we blame MS at least as much? How does MS let an update like this push through their Windows Update system? How does an application update make the whole OS unable to boot? Blue screens on Windows have been around for decades, why don't we have a better recovery system?
- Crowdstrike runs at ring 0, effectively as part of the kernel. Like a device driver. There are no safeguards at that level. Extreme testing and diligence is required, because these are the consequences for getting it wrong. This is entirely on crowdstrike.
- This didn't go through Windows Update. It went through the ctowdstrike software directly.

The amount of servers running Windows out there is depressing to me

Make a kernel-level antivirus
Make it proprietary
Don't test updates... for some reason??

I mean I know it's easy to be critical but this was my exact thought, how the hell didn't they catch this in testing?
- I have had numerous managers tell me there was no time for QA in my storied career. Or documentation. Or backups. Or redundancy. And so on.
- Completely justified reaction. A lot of the time tech companies and IT staff get shit for stuff that, in practice, can be really hard to detect before it happens. There are all kinds of issues that can arise in production that you just can't test for.
  But this... This has no justification. A issue this immediate, this widespread, would have instantly been caught with even the most basic of testing. The fact that it wasn't raises massive questions about the safety and security of Crowdstrike's internal processes.
- From what I've heard and to play a devil's advocate, it coincidented with Microsoft pushing out a security update at basically the same time, that caused the issue. So it's possible that they didn't have a way how to test it properly, because they didn't have the update at hand before it rolled out. So, the fault wasn't only in a bug in the CS driver, but in the driver interaction with the new win update - which they didn't have.
You left out
Pushed a new release on a Friday
You left out Profit
Oh... Wait...Hang on a sec.
Lots of security systems are kernel level (at least partially) this includes SELinux and AppArmor by the way. It's a necessity for these things to actually be effective.
- You missed most important line:
  Make it proprietary

never do updates on a Friday.

Yeah my plans of going to sleep last night were thoroughly dashed as every single windows server across every datacenter I manage between two countries all cried out at the same time lmao

I always wondered who even used windows server given how marginal its marketshare is. Now i know from the news.
- Marginal? You must be joking. A vast amount of servers run on Windows Server. Where I work alone we have several hundred and many companies have a similar setup. Statista put the Windows Server OS market share over 70% in 2019. While I find it hard to believe it would be that high, it does clearly indicate it's most certainly not a marginal percentage.
- This is a crowdstrike issue specifically related to the falcon sensor. Happens to affect only windows hosts.
- Well, I've seen some, but they usually don't have automatic updates and generally do not have access to the Internet.
- It's only marginal for running custom code. Every large organization has at least a few of them running important out-of-the-box services.
- Not too long ago, a lot of Customer Relationship Management (CRM) software ran on MS SQL Server. Businesses made significant investments in software and training, and some of them don't have the technical, financial, or logistical resources to adapt - momentum keeps them using Windows Server.
  For example, small businesses that are physically located in rural areas can't use cloud based services because rural internet is too slow and unreliable. Its not quite the case that there's no amount of money you can pay for a good internet connection in rural America, but last time I looked into it, Verizon wanted to charge me $20,000 per mile to run a fiber optic cable from the nearest town to my client's farm.
- Almost everyone, because the Windows server market share isn’t marginal at all.
- My current company does and I hate it so much. Who even got that idea in the first place? Linux always dominated server-side stuff, no?
How many coffee cups have you drank in the last 12 hours?
- I work in a data center
  I lost count
- There was a point where words lost all meaning and I think my heart was one continuous beat for a good hour.
Did you feel a great disturbance in the force?
- Oh yeah I felt a great disturbance (900 alarms) in the force (Opsgenie)
How's it going, Obi-Wan?

Here's the fix: (or rather workaround, released by CrowdStrike) 1)Boot to safe mode/recovery 2)Go to C:\Windows\System32\drivers\CrowdStrike 3)Delete the file matching "C-00000291*.sys" 4)Boot the system normally

It's disappointing that the fix is so easy to perform and yet it'll almost certainly keep a lot of infrastructure down for hours because a majority of people seem too scared to try to fix anything on their own machine (or aren't trusted to so they can't even if they know how)
- They also gotta get the fix through a trusted channel and not randomly on the internet. (No offense to the person that gave the info, it’s maybe correct but you never know)
- This sort of fix might not be accessible to a lot of employees who don't have admin access on their company laptops, and if the laptop can't be accessed remotely by IT then the options are very limited. Trying to walk a lot of nontechnical users through this over the phone won't go very well.
- Might seem easy to someone with a technical background. But the last thing businesses want to be doing is telling average end users to boot into safe mode and start deleting system files.
  If that started happening en masse we would quickly end up with far more problems than we started with. Plenty of users would end up deleting system32 entirely or something else equally damaging.
- It might not even be that. A lot of places have many servers (and even more virtual servers) running crowdstrike. Some places also seem to have it on endpoints too.
  That's a lot of machines to manually fix.
- And people need to travel to remote machines to do this in person
- I wouldn’t fix it if it’s not my responsibly at work. What if I mess up and break things further?
  When things go wrong, best to just let people do the emergency process.
I'm on a bridge still while we wait for Bitlocker recovery keys, so we can actually boot into safemode, but the Bitkocker key server is down as well...
- Gonna be a nice test of proper backups and disaster recovery protocols for some organisations
Man, it sure would suck if you could still get to safe mode from pressing f8. Can you imagine how terrible that'd be?
- You hold down Shift while restarting or booting and you get a recovery menu. I don’t know why they changed this behaviour.
- That was the dumbest thing to learn this morning.
A driver failure, yeesh. It always sucks to deal with it.
Not that easy when it's a fleet of servers in multiple remote data centers. Lots of IT folks will be spending their weekend sitting in data center cages.

This is going to be a Big Deal for a whole lot of people. I don't know all the companies and industries that use Crowdstrike but I might guess it will result in airline delays, banking outages, and hospital computer systems failing. Hopefully nobody gets hurt because of it.

Big chunk of New Zealands banks apparently run it, cos 3 of the big ones can't do credit card transactions right now
- It was mayhem at PakNSave a bit ago.
- cos 3 of the big ones can’t do credit card transactions right now
  Bitcoin still up and running perhaps people can use that
Several 911 systems were affected or completely down too

CrowdStrike: It's Friday, let's throw it over the wall to production. See you all on Monday!

^^so ^^hard ^^picking ^^which ^^meme ^^to ^^use
- Good choice, tho. Is the image AI?
We did it guys! We moved fast AND broke things!
When your push to prod on Friday causes a small but measurable drop in global GDP.
- Actually, it may have helped slow climate change a little
- Definitely not small, our website is down so we can't do any business and we're a huge company. Multiply that by all the companies that are down, lost time on projects, time to get caught up once it's fixed, it'll be a huge number in the end.
They did it on Thursday. All of SFO was BSODed for me when I got off a plane at SFO Thursday night.
Was it actually pushed on Friday, or was it a Thursday night (US central / pacific time) push? The fact that this comment is from 9 hours ago suggests that the problem existed by the time work started on Friday, so I wouldn't count it as a Friday push. (Still, too many pushes happen at a time that's still technically Thursday on the US west coast, but is already mid-day Friday in Asia).
- I'm in Australia so def Friday. Fu crowdstrike.

Wow, I didn't realize CrowdStrike was widespread enough to be a single point of failure for so much infrastructure. Lot of airports and hospitals offline.

The Federal Aviation Administration (FAA) imposed the global ground stop for airlines including United, Delta, American, and Frontier.

Flights grounded in the US.

The System is Down

Ironic. They did what they are there to protect against. Fucking up everyone's shit

Maybe centralizing everything onto one company's shoulders wasn't such a great idea after all...
- Wait, monopolies are bad? This is the first I've ever heard of this concept. So much so that I actually coined the term "monopoly" just now to describe it.
- The too big to fail philosophy at its finest.
CrowdStrike has a new meaning... literally Crowd Strike.
- They virtually blew up airports
Since when has any antivirus ever had the intent of actually protecting against viruses? The entire antivirus market is a scam.

An offline server is a secure server!

Honestly my philosophy these days, when it comes to anything proprietary. They just can't keep their grubby little fingers off of working software.
At least this time it was an accident.
There is nothing unsafer than local networks.
AV/XDR is not optional even in offline networks. If you don't have visibility on your network, you are totally screwed.

The thought of a local computer being unable to boot because some remote server somewhere is unavailable makes me laugh and sad at the same time.

Clownstrike

Crowdshite haha gotem
- CrowdCollapse

https://www.theregister.com/ has a series of articles on what's going on technically.

Latest advice...

There is a faulty channel file, so not quite an update. There is a workaround...

Boot Windows into Safe Mode or WRE.
Go to C:\Windows\System32\drivers\CrowdStrike
Locate and delete file matching "C-00000291*.sys"
Boot normally.

For a second I thought this was going to say "go to C:\Windows\System32 and delete it."
I've been on the internet too long lol.
Yeah I had to do this with all my machines this morning. Worked.
Working on our units. But only works if we are able to launch command prompt from the recovery menu. Otherwise we are getting a F8 prompt and cannot start.
- Bootable USB stick?

Yep, stuck at the airport currently. All flights grounded. All major grocery store chains and banks also impacted. Bad day to be a crowdstrike employee!

My flight was canceled. Luckily that was a partner airline. My actual airline rebooked me on a direct flight. Leaves 3 hours later and arrives earlier. Lower carbon footprint. So, except that I'm standing in queue so someone can inspect my documents it's basically a win for me. 😆

Yep, this is the stupid timeline. Y2K happening to to the nuances of calendar systems might have sounded dumb at the time, but it doesn't now. Y2K happening because of some unknown contractor's YOLO Friday update definitely is.

I'm so exhausted... This is madness. As a Linux user I've busy all day telling people with bricked PCs that Linux is better but there are just so many. It never ends. I think this is outage is going to keep me busy all weekend.

What are you, an apostle? Lol. This issue affects Windows, but it's not a Windows issue. It's wholly on CrowdStrike for a malformed driver update. This could happen to Linux just as easily given how CS operates. I like Linux too, but this isn't the battle.
🙄 and then everyone clapped
- Yeah it's all fun and games until you actually convince someone and then you gotta explain how a bootloader works to someone who still calls their browser "Google"
A month or so ago a crowdstrike update was breaking some of our Linux vms with newer kernels. So it's not just the os.
- How? I'm really curious to learn.
This isn't really a Windows vs Linux issue as far as I'm aware. It was a bad driver update made by a third party. I don't see why Linux couldn't suffer from the same kind of issue.
We should dunk on Windows for Windows specific flaws. Like how Windows won't let me reinstall a corrupted Windows Store library file because admins can't be trusted to manage Microsoft components on their own machine.
- I am no expert. But afaik drivers normally are integrated into the kernel and intensively tested by several parties before getting onto your computer. Only for proprietary drivers this would be problematic under Linux.
You're comment I came looking for. You get a standing ovation or something.

A few years ago when my org got the ask to deploy the CS agent in linux production servers and I also saw it getting deployed in thousands of windows and mac desktops all across, the first thought that came to mind was "massive single point of failure and security threat", as we were putting all the trust in a single relatively small company that will (has?) become the favorite target of all the bad actors across the planet. How long before it gets into trouble, either because if it's own doing or due to others?

I guess that we now know

My work PC is affected. Nice!

Plot twist: you're head of IT
Same! Got to log off early 😎
Dammit, hit us at 5pm on Friday in NZ
- 4:00PM here in Aus. Absolutely perfect for an early Friday knockoff.
Noice!

My dad needed a CT scan this evening and the local ER's system for reading the images was down. So they sent him via ambulance to a different hospital 40 miles away. Now I'm reading tonight that CrowdStrike may be to blame.

crowdstrike sent a corrupt file with a software update for windows servers. this caused a blue screen of death on all the windows servers globally for crowdstrike clients causing that blue screen of death. even people in my company. luckily i shut off my computer at the end of the day and missed the update. It's not an OTA fix. they have to go into every data center and manually fix all the computer servers. some of these severs have encryption. I see a very big lawsuit coming...

I don't see how they can recover from that. They will get lawsuits from all around the world.
- I'm never financially recovering from this. - George Kurtz
they have to go into every data center and manually fix all the computer servers.
Jesus christ, you would think that (a) the company would have safeguards in place and (b) businesses using the product would do better due diligence. Goes to show thwre are no grown ups in the room inside these massive corporations that rule every aspect of our lives.
I'm calling it now. In the future there will be some software update for your electric car, and due to some jackass, millions of cars will end up getting bricked in the middle of the road where they have to manually be rebooted.
- Laid off one too many persons, finance bros taking over
- We’re already halfway there! https://www.reddit.com/r/Piracy/comments/18rnhdh/cant_drive_your_car_due_to_update/#lightbox
- I work for one of these behemoths, and there are a lot of adults in the room. When we began our transition off the prior, well known corporate AV, I never even heard of crowd strike.
  The adults were asking reasonable questions: why such an aggressive migration timeline? Why can't we have our vendor recommended exclusion lists applied? Why does this need to be installed here when previously agentless technologies was sufficient? Why is crowd strike spending monies on a Superbowl ad instead of investing back into the technology?
  Either something fucky is a foot, as in this was mandated to our higher ups to m make the switch (why?), or, as is typically the case, the decision was made already and this 'due diligence' is all window dressing to CYA.
  Who gives a shit about fines on SLAs if your vendor is going to foot the bill.
. they have to go into every data center and manually fix all the computer servers
Do they not have IPMI/BMC for the servers? Usually you can access KVM over IP and remotely power-off/power-on/reboot servers without having to physically be there. KVM over IP shows the video output of the system so you can use it to enter the UEFI, boot in safe/recovery mode, etc.
I've got IPMI on my home server and I'm just some random guy on the internet, so I'd be surprised if a data center didn't.
- I’d be surprised if a data center didn’t.
  Then you'd be surprised.
- Sometimes there are options that are reasonable for individual users that don't scale well to enterprise environments.
  Also, the effectively gives attackers a secondary attack surface in addition to the normal remote access technologies that require the machine to be up and running to work.

We had a bad CrowdStrike update years ago where their network scanning portion couldn’t handle a load of DNS queries on start up. When asked how we could switch to manual updates we were told that wasn’t possible. So we had to black hole the update endpoint via our firewall, which luckily was separate from their telemetry endpoint. When we were ready to update, we’d have FW rules allowing groups to update in batches. They since changed that but a lot of companies just hand control over to them. They have both a file system and network shim so it can basically intercept **everything **

Honestly kind of excited for the company blogs to start spitting out their ~~disaster recovery~~ crisis management stories.

I mean - this is just a giant test of ~~disaster recovery~~ crisis management plans. And while there are absolutely real-world consequences to this, the fix almost seems scriptable.

If a company uses IPMI (~~Called~~ Branded AMT and sometimes vPro by Intel), and their network is intact/the devices are on their network, they ought to be able to remotely address this.
But that’s obviously predicated on them having already deployed/configured the tools.

I mean - this is just a giant test of disaster recovery plans.
Anyone who starts DR operations due to this did 0 research into the issue. For those running into the news here...
CrowdStrike Blue Screen solution
CrowdStrike blue screen of death error occurred after an update. The CrowdStrike team recommends that you follow these methods to fix the error and restore your Windows computer to normal usage.

Rename the CrowdStrike folder Delete the “C-00000291*.sys” file in the CrowdStrike directory Disable CSAgent service using the Registry Editor
No need to roll full backups... As they'll likely try to update again anyway and bsod again. Caching servers are a bitch...
- I think we’re defining disaster differently. This is a disaster. It’s just not one that necessitates restoring from backup.
  Disaster recovery is about the plan(s), not necessarily specific actions. I would hope that companies recognize rerolling the server from backup isn’t the only option for every possible problem.
  I imagine CrowdStrike pulled the update, but that would be a nightmare of epic dumbness if organizations got trapped in a loop.
- Note this is easy enough to do if systems are booting or you dealing with a handful, but if you have hundreds of poorly managed systems, discard and do again.
- Nah... just boot into safemode > cmd prompt: CD C:\Windows\System32\drivers\CrowdStrike
  Then: del C-00000291*.sys
  Exit/reboot.
I'm here right now just to watch it unfold in real time. Unfortunately Reddit is looking juicer on that front.
https://libreddit.northboot.xyz
IPMI (Called AMT and sometimes vPro by Intel),
IPMI is not AMT. AMT/vPro is closed protocol, right? Also people are disabling AMT, because of listed risks, which is too bad; but it's easier than properly firewalling it.
Better to just say "it lets you bring up the console remotely without windows running, so machines can be fixed by people who don't have to come into the office".
- Ah, you’re right. A poor turn of phrase.
  I meant to say that intel brands their IPMI tools as AMT or vPro. (And completely sidestepped mentioning the numerous issues with AMT, because, well, that’s probably a novel at this point.)
On desktops, nobody does. Servers, yes, all the time.
- Depends on your management solutions. Intel vPro can allow remote access like that on desktops & laptops even if they’re on WiFi and in some cases cellular. It’s gotta be provisioned first though.
- VPro is massively a desktop feature. Servers have proper IPMI/iDrac with more features, and VPro fills (part of) that management gap for desktops. It's pretty cool it's not spoiled or disabled. Time to reburn all the desktops in east-07? VPro will be the way.

lol

too bad me posting this will bump the comment count though. maybe we should try to keep the vote count to 404

I can only see 368 comments rn, there must be some weird-ass puritan server blocking .ml users. It's not beehaw as I can see comments from there.
I can only conclude that it is probably some liberals trying to block "Tankies" and no comment of value was lost.

Been at work since 5AM... finally finished deleting the C-00000291*.sys file in CrowdStrike directory.

182 machines total. Thankfully the process in of itself takes about 2-3 minutes. For virtual machines, it's a bit of a pain, at least in this org.

lmao I feel kinda bad for those companies that have 10k+ endpoints to do this to. Eff... that. Lot's of immediate short term contract hires for that, I imagine.

How do you deal with places with thousands of remote endpoints??
- That's one of those situations where they need to immediately hire local contractors to those remote sites. This outage literally requires touching the equipment. lol
  I'd even say, fly out each individual team member to those sites.. but even the airports are down.
- Call the remote people in, deputize anyone who can work a command line, and prioritize the important stuff.
Can you program some keyboard-presenting device to automate this? Still requires plugging in something of course...what a mess.
- Yeah, there are USB sticks that identify as keyboards and run every keystroke saved in a text file on its memory in sequence. Neat stuff. The primary use case is of course corrupting systems or bruteforcing passwords without touching anything... But they work really well for executing scripts semi-automated.
- Yep I have one of these, I think it's called tiny. Very similar to an Arduino, and very easy to program.
Lot's of immediate short term contract hires for that, I imagine.
I think sysadmins union should be created today

My favourite thing has been watching sky news (UK) operate without graphics, trailers, adverts or autocue. Back to basics.

Linux and Mac just got free advertisment.

The words 'Mac' and 'free' aren't allowed in the same sentence.
- Also, enterprise infrastructure running on a Mac? It’s something I’ve never heard of in over a decade and a half of working in tech. And now I’m curious. Is it a thing?
Heard linux systems had a similar issue a few months ago
- Yeah the more I read it seems some crucial security software deployed bad patch.
  I'm sure whatever OS would dominate would face widespread issues just by the numbers.
  But to be fair it's fun to make fun of Windows/MSFT :p

AWS No!!!

Oh wait it's not them for once.

TFW the green check is actually correct.

Annoyingly, my laptop seems to be working perfectly.

That's the burden when you run Arch, right?
- lol he said it's working

One possible fix is to delete a particular file while booting in safe mode. But then they'll need to fix each system manually. My company encrypts the disks as well so it's going to be a even bigger pain (for them). I'm just happy my weekend started early.

- Yeah that would be case in most laptops. So if bitlocker is involved as well what could be the possible fix.
You have ta have access to boot in safe mode too, I guess I can't on my work pc for example.
What a shitty workaround & might crowd strike burn in hell lol
- Enjoy your weekend unless you are in IT

CrowdStrike Holdings, Inc. is an American cybersecurity technology company based in Austin, Texas.

Never trust a texan

To be fair Austin is the least Texan of Texas
- Is it still at all Austiney though? With all the companies moving to Austin, I wonder how much of the original Austin is left.
Totally Texas!

Huh. I guess this explains why the monitor outside of my flight gate tonight started BSoD looping. And may also explain why my flight was delayed by an additional hour and a half...

oh joy. can’t wait to have to fix this for all of our clients today…

Stop running production services on M$. There is a better backend OS.

The issue was caused by a third-party vendor, though. A similar issue could have happened on other OSes too. There's relatively intrusive endpoint security systems for MacOS and Linux too.
- That's the annoying thing here. Everyone, particularly Lemmy where everyone runs Linux and FOSS, thinks this is a Microsoft/Windows issue. It's not, it's a Crowdstrike issue.
Crowdstrike did the same to Linux servers previously.
There’s a better frontend OS
Doesn’t mean people want to go away from what they know
- There's a shit ton more reasons than that, but in short: I highly doubt anyone suggesting a company just up and leave the MS ecosystem has spent any considerable amount of time in a sysadmin position.

My company used to use something else but after getting hacked switched to crowdstrike and now this. Hilarious clownery going on. Fingers crossed I'll be working from home for a few days before anything is fixed.

This is a better article. It's a CrowdStrike issue with an update (security software)

I agree that's a better article, thanks for sharing
- Np man. Thanks for mentioning it.

I see a lot of hate ITT on kernel-level EDRs, which I wouldn't say they deserve. Sure, for your own use, an AV is sufficient and you don't need an EDR, but they make a world of difference. I work in cybersecurity doing Red Teamings, so my job is mostly about bypassing such solutions and making malware/actions within the network that avoids being detected by it as much as possible, and ever since EDRs started getting popular, my job got several leagues harder.

The advantage of EDRs in comparison to AVs is that they can catch 0-days. AV will just look for signatures, a known pieces or snippets of malware code. EDR, on the other hand, looks for sequences of actions a process does, by scanning memory, logs and hooking syscalls. So, if for example you would make an entirely custom program that allocates memory as Read-Write-Execute, then load a crypto dll, unencrypt something into such memory, and then call a thread spawn syscall to spawn a thread on another process that runs it, and EDR would correlate such actions and get suspicious, while for regular AV, the code would probably look ok. Some EDRs even watch network packets and can catch suspicious communication, such as port scanning, large data extraction, or C2 communication.

Sure, in an ideal world, you would have users that never run malware, and network that is impenetrable. But you still get at avarage few % of people running random binaries that came from phishing attempts, or around 50% people that fall for vishing attacks in your company. Having an EDR increases your chances to avoid such attack almost exponentionally, and I would say that the advantage it gives to EDRs that they are kernel-level is well worth it.

I'm not defending CrowdStrike, they did mess up to the point where I bet that the amount of damages they caused worldwide is nowhere near the amount damages all cyberattacks they prevented would cause in total. But hating on kernel-level EDRs in general isn't warranted here.

Kernel-level anti-cheat, on the other hand, can go burn in hell, and I hope that something similar will eventually happen with one of them. Fuck kernel level anti-cheats.

The problem here isn't if you should run EDR or not it's that people need to take the risk and responsibilities seriously and the cure needs to to be better than the disease.
If you need to hand remote kernel level access over to a company it's in you to make sure they have the security, QA and basic competency to shoulder that responsibility.
And when it comes out that they don't even run or verify their code before deploying it to millions of machines all at once, it's on you for buying not vetting them.
Just like users should check files and where they come from before running them IT professionals need to do the same.

No one bother to test before deploying to all machines? Nice move.

I was quite surprised when I heard the news. I had been working for hours on my PC without any issues. It pays off not to use Windows.

It's not a flaw with Windows causing this.
The issue is with a widely used third party security software that installs as a kernel level driver. It had an auto update that causes bluescreening moments after booting into the OS.
This same software is available for Linux and Mac, and had similar issues with specific Linux distros a month ago. It just didn't get reported on because it didn't have as wide of an impact.
- They skimp on QA?
- had similar issues with specific Linux distros a month ago. It just didn’t get reported on because it didn’t have as wide of an impact.
  Because most data center admins using linux are not so stupid to subscribe to remote updates from a third party. Linux issues happen when critical package vulnerabilities make it into the repo.
- Still a MS issue. Both testing and rollout procedures were inadequate

Irrelevant but I keep reading "crowd strike" as "counter strike" and it's really messing with me

Think of it as ClownStrike, they will be known as a bunch of clowns after this.
Bomb Carrier: "I'm going A"
Random Teammate: "Okay, everyone rush B!"

Huh, so that's why the office couldn't order pizza last night lmfao

This is proof you shouldn't invest everything in one technology. I won't say everyone should change to Linux because it isn't immune to this, but we need to push companies to support several OS

The issue here is kernel level applications that can brick a box. Anti viruses compete for resources, no one should run 2 at once
- That's why I run 3 at once...

I picked the right week to be on PTO hahaha

If these affected systems are boot looping, how will they be fixed? Reinstall?

There is a fix people have found which requires manual booting into safe mode and removal of a file causing the BSODs. No clue if/how they are going to implement a fix remotely when the affected machines can't even boot.
- Probably have to go old-skool and actually be at the machine.
- Do you have any source on this?
It is possible to edit a folder name in windows drivers. But for IT departments that could be more work than a reimage
- Having had to fix >100 machines today, I'm not sure how a reimage would be less work. Restoring from backups maybe, but reimage and reconfig is so painful
- It’s just one file to delete.

Everyone is assuming it’s some intern pushing a release out accidentally or a lack of QA but Microsoft also pushed out July security updates that have been causing bsods on the 9th(?). These aren’t optional either.

What’s the likelihood that the CS file was tested on devices that hadn’t got the latest windows security update and it was an unholy union of both those things that’s caused this meltdown. The timelines do potentially line up when you consider your average agile delivery cadence.

I don’t think so. I do updates every two months so I haven’t updated Windows at all in July and it still crashed my servers
- Microsoft installs security updates automatically.

Apparently at work "some servers are experiencing problems". Sadly, none of the ones I need to use :(

Meanwhile Kaspersky: thinks if so incompetent people can even make antivirus at all

don't rely on one desktop OS too much. diversity is the best.

Dont rely on corpo trash at al.

So that's why my work laptop is down for the count today. I'm even getting that same error as the thumbnail picture

Servers on Windows? Even domain controllers can be Linux-based.

I'm a long-time Samba fan, but even I wouldn't run them as DCs in a production environment.
- I meant sssd, that I've seen working as a DC, if my memory serves me right. But I've never been the person setting up a domain controller, so.
  EDIT: No, it doesn't serve me right, from quick googling sssd can't into DC's. Kurwa.
- Are there any 9P fans?
Old servers. Also Crowdstrike took down Linux servers a few years ago.

A lot of people I work with were affected, I wasn't one of them. I had assumed it was because I put my machine to sleep yesterday (and every other day this week) and just woke it up after booting it. I assumed it was an on startup thing and that's why I didn't have it.

Our IT provider already broke EVERYTHING earlier this month when they remote installed" Nexthink Collector" which forced a 30+ minute CHKDSK on every boot for EVERYONE, until they rolled out a fix (which they were at least able to do remotely), and I didn't want to have to deal with that the week before I go in leave.

But it sounds like it even happened to running systems so now I don't know why I wasn't affected, unless it's a windows 10 only thing?

Our IT have had some grief lately, but at least they specified Intel 12th gen on our latest CAD machines, rather than 13th or 14th, so they've got at least one win.

Your computer was likely not powered on during the time window between the fucked update pushing out and when they stopped pushing it out.
- That makes sense, although I must have just missed it, for people I work with to catch it.

Xfinity H&I network it down so I can't watch Star Trek. I get an error msg connection failure. Other channels work though.

Interesting day

This is the best summary I could come up with:

There are reports of IT outages affecting major institutions in Australia and internationally.

The ABC is experiencing a major network outage, along with several other media outlets.

Crowd-sourced website Downdetector is listing outages for Foxtel, National Australia Bank and Bendigo Bank.

Follow our live blog as we bring you the latest updates.

The original article contains 52 words, the summary contains 52 words. Saved 0%. I'm a bot and I'm open source!

The original article contains 52 words, the summary contains 52 words. Saved 0%.
Good bot!

I legit have never been more happy to be unemployed.

A bunch of shitty sysadmins/cybersec people just learned why you don't blindly deploy new updates to production without testing them first.

I've used Crowdstrike before. It has support for deploying version N to a pilot group, N-1 to the test environment and N-2 to production.

Apparently the slow rollout was skipped (on Crowdstrike's end) for this
That's true about the test group deployments, but it turned out this one was not an agent update under that control system. It's a Channel File update that goes out to all endpoints automatically.

The internet is just a way to communicate data. The only people affected by this bug are customers of a Windows based "cybersecurity" company.
- Just a little bit of a correction. Crowd strike publishes agents for Windows, Linux, and MacOS. Its just the particular broken agent is for Windows. They had a 1/3 chance of taking down any one of their three client bases
- The company is not Windows based, they offer clients and agents for Linux and Mac as well (and there are some scattered reports they fucked up some of their Linux customers like this last month).
  The Windows version of their software is what is broken.
Interesting how ARPA net (the internet) was build to with stand these issues, but companies like Microsoft and Amazon (and no regulation) have completely reversed it's original intent.

It's a fair point but I would rather diversify and also use something that is open / less opaque

ReactOS for the win!

Some intern is getting their ass beat right now, never release into prod without extensive test.

If an intern can release to prod without extensive testing there are bigger issues.
Given the scope of the potential impact, if anyone can release to prod and have that deploy to all customers without some form of a canary release strategy, then there are still issues.
It's likely not an intern's fault. Likely a C suite not authorizing the testing infrastructures requested by the developers and sysops people.

Buy the dip!

Webroot had something similar ish earlier this year. Such a pain.

Windows moment 🤣

This is why you create restore points if using windows.

Those things never worked for me... Problems always persisted or it failed to apply the restore point. This is from the XP and Windows 7 days, never bothered with those again. To Microsoft's credit, both W7 and W10 were a lot more stable negating the need for it.
- I can't say about XP or 7 but they've definitely saved my bacon on Win10 before on my home system. And the company I work for has them automatically created and it made dealing with the problem much easier as there was a restore point right before the crowdstrike update. No messing around with the file system drivers needed.
  I'd really recommend at least creating one at a state when your computer is working ok, it doesn't hurt anything even if it doesn't work for you for whatever reason. It's just important to understand that it's not a cure all, it's only designed to help with certain issues (primarily botched updates and file system trouble).

I'm used to IT doing a lot of their work on the weekends as to not impact operations.

play stupid games win stupid prizes

Bahaha 😂😂 continue using proprietary software, that's all you are going to get in addition to privacy issues... Switch to Linux.

Not easy to switch a secured 4,000+ workstation business. Plus, a lot of companies get their support, license, and managed email from one vendor. It's bundled in such a way that it would cost MORE to deploy Linux. (And that very much on purpose)
- It's entertaining to me that our brand of monopolistic / oligarchic capitalism itself disincentivizes one-time costs that are greatly outweighed by the risk of future occurrences. Even when those one-time costs would result in greater stability and lower prices...and not even on that big of a time horizon. There is an army of developers that would be so motivated to work on a migration project like this. But then I guess execs couldn't jet set around the world to hang out at the Crowdstrike F1 hospitality tent every weekend.
(Crowd strike also runs on Linux. This could have happened to Linux too)
- Crowd can run on linux, but it is running on all Linux machines? I've seen support for some RedHat machines, show me if there's more

Good ol microsloth

As much as it pains me to say it, it's not really Microsoft at fault here, it's CrowdStrike.
- MS should have done testing as well. They can't dodge this

Why do people run windows servers when Linux exists, it’s literally a no brainer.

Servers weren't much of a problem, they're mostly virtual and could be just restored from a backup. The several hundred workstations were a problem. They needed a physical touch. All are encrypted with BitLocker, requiring passkeys stored in AD. Over half are laptops. Most of those don't have wired ethernet ports, and an account with local admin rights hasn't logged in since the day they were imaged. Throw in a proper LAPS config, where randomly generated passwords of three dozen characters in length are also stored in AD...
... Yeah, today was a bad day.
They run Windows and all this third party software because they would rather pay subscriptions and give up control of their business than retain skilled staff. It has nothing todo with Linux vs Windows. Linux won't stop doors falling off Boeing planes. It is the myopia of modern business culture.
No brainer... Blames windows... For a 3rd party issue... The same 3rd party that's done the same thing here to Linux recently.... No brainer achieved.
My last companyoffered apps as either hosted or on-prem. Once they decided the on-prem version had to be Windows for our customer base, it made sense to have one build, one installer. I’m very happy not to be there today.
My current employer is all Linux servers and Engineers use Mac laptops. The only ones affected were Management and HR, LoL. Back of the line for you, we have customers to help!
Because all software runs from Linux right...
- It could if more people just used Linux
I built a home virtual server host on Proxmox. It destroyed itself 3 times in 6 months. I gave up and installed server 2019 with HyperV. Has been rock solid for years.
Are you a Linux user?
Hey everyone, shut up! This guy is a Linux user and he’s here to tell us about using Linux.
- Look! We found the anti anti-microsoft user in their natural habitat of talking shit on social media. They are from the same family bootlickers and related to the Tossaladers. Don't mention you are an alternative OS person around them or they may go into a blind rage.

This is a a ruse to make Work From Home end.

All IT people should go on general strike now.

????
How would this have been prevented by working on site?
A crowd strike, you could call it
"Never attribute to malice that which can be adequately explained by stupidity."
This is just blatantly incorrect - 99% of these outages are going to be fixed remotely.
- Not at my company. We're all stuck in BSOD boot loops thanks to BitLocker, and our BIOS is password protected by IT. This is going to take weeks for them to manually update, on site, all the computers one by one.
- Eh. This particular issue is making machines bluescreen.
  Virtualized assets, If there's a will there's a way. Physical assets with REALLY nice KVMs... you can probably mount up an ISO to boot into to remove the stupid definitions causing this shit. Everything else? Yeah... you probably need to be there physically to fix it.
  But I will note that many companies by policy don't allow USB insertion... virtually or not. Which will make this considerably harder across the board. I agree that I think the majority could be fixed remotely. I don't think the "other" categories are only 1%... I think there's many more systems that probably required physical intervention. And more importantly... it doesn't matter if it's 100% or 0.0001%... If that one system is the one that makes the company money... % population doesn't matter.

It's Russia, or Iran or China or even our "ally" Saudi Arabia. So really, it's time to reset the clock to pre 1989. Cut Russia and China off completely, no investment, no internet, no students no tourist nothing. These people mean and are continually doing us harm and we still plod along and some unscrupulous types become agents for personal profit. Enough.

It was a bad update...
Let's ignore that it was an American company taking down EVERYONE'S stuff, I guess
I mean, I’m not completely against that… but Crowdstrike is an American company. Did you comment on the wrong post?

best day ever. the digitards get a wakeup call. how often have been lectured by imbeciles how great whatever dumbo closed source is. "i need photoshop", "windows powershell and i get work done", "azure and onedrive and teams...best shit ever", " go use NT, nobody will use a GNU".

yeah well, i hope every windows user would be kept of the interwebs for a year and mac users just burn in hell right away. lazy scum that justifies shitting on society for their own comfort. while everyone needs a drivers license, dumb fucking parents give tiktok to their kids...idiocracy will have a great election this winter.

So when’s the last time you touched some grass? It’s a lovely day outside. Maybe go to a pet shelter and see some puppies? Are you getting enough fiber? Drinking enough water? Why not call a friend and hang out?
- Yeah, I agree with the sentiment, but this is pretty intense.
- Bold of you to assume this person has friends!
Literally every program you outlined here are not Windows exclusive. If you let people lecture you on this shit that's on you.
While I get your point on the over reliance on Microsoft, some of us are going to be stuck spending the whole day trying to fix this shit. You could show some compassion.
- no. after decades...not anymore. again and again. there is no good excel, apple users are not "educated". just assholes caring about themselves and avoiding the need to learn. i did not yet hear any valid argument.
  why not show compassion for driving bmw series 5 without a drivers license around kindergard?
  i'm seriously just done with every windows,android,osx user forever. digital trumps. thats all they are. me. me. me.
You guys remember that world of warcraft episode of south park?
This is him.
You kinda went off the rails, there.
It's fun to dunk on the closed source stuff, but this is a release engineering thing that isn't the fault of a software license.
Render unto Caesar, dude.
It's a nice rant, but it also combined government regulation and monocultures and indolence. You should cut that up into several different rants, bringing out the proper one in a more focused spiel for a timely and dramatic win. The lack of cohesion also means you're taking longer to get the rant out, and if it's too long to hold focus you'll reduce your points awarded for the dunk.
Otherwise, lots of street-preacher potential. Solid effort.