Global IT Outage

[gr8oldies]

I know as of the late 90s, WHIO/WHKO/WCLR/WZLR in Dayton still backed everything up to cart. I'm sure that's not happening now.

I have friends and family who couldn't get on flights, and every rental car in my area was spoken for yesterday.

 

[Mark Roberts]

The reliance on computers and corporations will be the downfall of Civilization. I get the cheapening of services for increased profit for the shareholders. "Gotta make the quarter's numbers". But if you can't deliver your service because the geniuses at Microsoft and their vendors can't properly test an update, how is that increasing profits? I realize a huge number of IT folks only do programs that run on an Microsoft operating system. If I was running things they would have to learn Linux or sharpen their fast food skills.

A really smart IT guy told me to switch over to a Linux system. He claims the hackers use Linux because they don't want to infect their own computers. Linux is a derivative of Unix which the old Lucent (now Nokia) switches ran on 99.9999% reliably. Also Linux doesn't have as many updates.

Several things here:

Linux updates are entirely dependent upon the distribution. Some are slower about it; others, such as Fedora, can update as often as daily.

In April, Crowdstrike had similar issues to those of yesterday with servers running Debian (one of the major Linux distributions) and RockyLinux. Crowdstrike was more easily removed on those servers, probably via a "live CD" or a bootable USB-drive equivalent; the cause was obvious when Crowdstrike was removed and the servers could then reboot normally.

Nokia ran its own heavily-customized version of Linux on its switches and firewalls. Nokia managed the patches and other updates.

A tip for 9 to 5 office environments only. Turn off your PC at when you leave the office. Turn off the automatic updates. Unless it is an national emergency wait a couple of days to do the updates. If it is a bad update let someone else experienced it.

That's not how it works. Corporate Windows environments are centrally managed through GPOs (Group Policy Objects) which limit the privileges that individual users have on a system. Updates are pushed via SCCM (Software Center Configuration Manager) or Intune (SCCM's replacement). Individual users may have some ability to control the timing of updates, but this is entirely dependent on what the SCCM/Intune administrators have configured. If you turn off a PC, wait a few days, then turn it on again, one of the first things that will happen (if the SCCM/Intune administrators have so configured it) is that an update will be pushed. Beyond that, some corporate environments have a policy requiring PCs to remain "on" during non-working hours.

More thoughts in the General thread.

(edit to add Intune, which is cloud-based compared to SCCM and which has been Microsoft's future direction for a year or two)

 

[Kelly A]

One does wonder if this will cause companies to rethink some of their policies and plans for "the cloud" in the future.

Of course it won't. The cost benefits long term of cloud computing outweighs a rare instance like this. And from what I can tell, this situation was unique to only users of CrowdStrike cyber security services. That included Microsoft Azure customers and independent Microsoft users that also had CrowdStrike services. It's a lot of users, but users of Amazon Web Services, Linux, and Apple products were unaffected.
Airlines have had major IT incidents included in their own private data centers. Just remember a couple years ago when Southworst was completely grounded when their systems got overwhelmed.

 

[Triple Fake Jerry]

Now just imagine a snafu affecting the national power grid. And how many hundreds of thousands will die who are dependent on steady electricity who are on medical devices.

 

[Mark Roberts]

Now just imagine a snafu affecting the national power grid.

Actually, there are two grids, essentially east and west, plus Texas. And part of one of my jobs was to develop and analyze just such scenarios.

 

[secondchoice]

Nokia ran its own heavily-customized version of Linux on its switches and firewalls. Nokia managed the patches and other updates.


That's not how it works. Corporate Windows environments are centrally managed through GPOs (Group Policy Objects)

A 9 to 5 office is usually small and will not have IT folks there everyday and shouldn't be considered a corporate environment. A modern form of the old Windows N / T or 2000 is technical overkill for small business.

Nokia got access to a version of Unix when they took over Lucent. The 5E and 4 (for cellular) switches ran Unix. The version we used ran years without updates. Part of Lucent's "old school" thinking (that allowed me to stay employed long enough with low seniority to get vested in the pension plan after the Tech Meltdown) was the operating system's software could only be accessed by a physical connection on the premise. You had to

A few companies had a "secret" dial up connection that had limited cabilities similar to what was available to the control console on the office: assignment of numbers to cable pairs, DSL services, long disturbance carrier, etc.